A presentation at OWASP DevSlop by Jacquie Grindrod
Orchestration from A to Z
Agenda 01 02 Orchestration A, B, Cs Lock It Down
Agenda 03 04 Runtime Q&A
Jacquie Grindrod Dev Advocate HashiCorp @devopsjacquie /jacquelyne-grindrod @gogococo ● Bit of background about me,
01 A, B, Cs of Orchestration
Workloads ● When I say workloads, I mean a task. Possibly a service.
Automation ● ● ● ● Automation helps make business more efficient by reducing/replacing human interaction with systems and instead uses software to perform tasks/reduce cost. With automation, we’re typically talking about automating one task or chain of tasks. They’re linear. I tend to think of simple delivery pipelines: Check out code, build code/artifacts, push to storage, deploy to environment. https://www.redhat.com/en/topics/automation/what-is-orchestration#:~:text=Orchestration%20is%20the%20automated%20configuration,isn’t%20a%20scalable%20strategy.
Schedulers ● ● ● ● Scheduling typically refers to the ability for an administrator to load a service file into a host system that establish how to run a specific container. We could think about the process of booking a meeting in a teammates calendar here. We can see their total load capacity (how many hours they have available in a work day) and and scheduling restraints they have (possibly their calendar is already full) and then make a decision on when/where to schedule based on that info. Perhaps another teammate can help and is more available, or perhaps another time is better. One of the key things schedulers are responsible for is host selection, based on scheduling restraints. https://www.digitalocean.com/community/tutorials/the-docker-ecosystem-scheduling-and-orchestration#:~:text=Orchestration%20is%20a%20broad%20term,to%20run%20a%20specific%20container.
Orchestrators ● ● ● ● ● ● ● ● Which brings us to orchestration. It’s generally the automation of a process or workflow that involves many steps across multiple systems. We typically expect an Orchestrator to handle things like cluster management, automated configuration, scheduling of workloads, keeping a desired state (ex I want to have 4 instances of this task, if it crashes, restart it), statefulsets or attached volumes to make sure when our workloads restart that they come back with the correct data and other details like rolling deployments. Generally, we want it to do management & co-ordination of computer systems, apps and services. ● There’s different options for how to handle orchestration but we’ll be focusing on Nomad today.
Nomad ● ● ● So let’s talk a bit about Nomad. It’s a HashiCorp product for Workload Orchestration. It’s available free and is OSS, and also has an enterprise version for those who need a bit more. You’re probably more familiar with hearing “container orchestration”. The reason I’m saying workload is that while we support containers, it doesn’t HAVE to be a container.
Do One Thing Well ● ● Both HashiCorp & Linux share this philosophy of do one thing well.
K8s vs Nomad ● I don’t want everyone watching to have this burning question
02 Lock It Down ● For this demo, I finally got to do something I’ve been wanting to do for years - Start with Security First. I have a background implementing asks from our security team in a healthcare compliant environment, and one of the things our team talked about was trying to build security in first. Given that this is OWASP DevSlop, it’s only fitting that we’re going to talk about security first and it will be a returning theme. While I’m not a security expert, I did talk to one to help prep for this talk. He shared a lot of awesome tips, information & examples with me, but he told me to make sure I say this, because security experts love it.
Nomad Security ● https://www.nomadproject.io/docs/internals/security Things to think about for basic nomad/k8s Probably provisioning it up into a cloud env that abstracts a lot away Hardening at HostOS layer? But since Nomad is a single binary , generally easy to plug in systemD security features Ex Nomad servers don’t need to run as root. Clients, probably do (for access to stuff like Namespaces and Consul Connect) Was creating a nomad user:group to the docker group Then Nomad can spin it all up etc but gets rough with consul-connect. Caveat - docker groups sorta has root permissions
Defence in Depth ● Defense in depth is a concept used in Information security in which multiple layers of security controls are placed throughout an information technology system.
Top 3 Threats External Attacks ● ● ● ● ● Don’t have mtls, but do have ACLs, but exposed to internet During the first couple seconds, whoever hits it first will get the ACL token Now they have free run until you triage (ex maybe to run a bitcoin miner for that hour or smth) During initial bootstrap process, at the least need a mtls cert to avoid that
Top 3 Threats Supply Chain Attacks ● Package management/docker images.
Top 3 Threats Internal ● ● ● ● ● Top threat’s are usually internal Uncomfortable to bring up but worth the discussion Whether or not they’re intentional And the best protection against that is good logging/visibility
Tips & Tricks Favourite Tool ● ● ● Favourite tools: Falco & FluentD Centralized Logging + Observability are the 2 strongest supports for security. If you’re package installing in a docker container, it’s a security smell. Pro tip - if using the docker runtime, most security tools for docker will run well (ex Snyk) Nomad exec command Great to play in when simulating a compromised instance
https://github.com/picatz/terraform-google-nomad ● ● ● ● ● ● ● ● ● ● ● Module Features Automated load balancer configuration to access the Nomad Server API. Automatically enables mTLS, and generates certifcates. Automatically enables gossip encryption, and generates the gossip key. Automatically generates SSH credentials for the bastion host. ACLs enabled by default. Only the Docker task driver is enabled by default. Runs the Docker daemon with no-new-privileges=true and icc=false set by default. Installs the gVisor container runtime by default (runsc). Installs HashiCorp’s Consul service mesh.
03 Run Here, There, Everywhere! You could enter a subtitle here in case you need it
Running Nomad ● https://www.nomadproject.io/docs https://www.nomadproject.io/intro https://learn.hashicorp.com/nomad Job Spec Task Drivers
04 Q&A!
Thanks! @devopsjacquie /jacquelyne-grindrod @gogococo CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik. Please keep this slide for attribution.
Whether orchestration is a buzzword you’ve heard and plan on looking into, or you’ve already taken a stab at it and found yourself looking for more - this talk is for you! Jacquie will be walking us through the A,B,Cs of scheduling workloads with an example application and Nomad as an orchestrator. We’ll discover how different types of workloads affect orchestration, such as stateful versus stateless applications and scheduling with security constraints. She’ll also be sharing lessons she learned from applying these foundational concepts to datacenter and public cloud environments in the digital healthcare space. You’ll walk away from this with a better understanding of how to apply these concepts with your orchestrator of choice.
The following resources were mentioned during the presentation or are useful additional information.
This was the SockShop demo code that I used at the time.
for free. You
can too.