Orchestration with Nomad from A(rtifacts) to Z(ones)

Orchestration from A to Z

Agenda 01 02 Orchestration A, B, Cs Lock It Down

Agenda 03 04 Runtime Q&A

Jacquie Grindrod Dev Advocate HashiCorp @devopsjacquie /jacquelyne-grindrod @gogococo ● Bit of background about me,

01 A, B, Cs of Orchestration

Workloads ● When I say workloads, I mean a task. Possibly a service.

Automation ● ● ● ● Automation helps make business more efficient by reducing/replacing human interaction with systems and instead uses software to perform tasks/reduce cost. With automation, we’re typically talking about automating one task or chain of tasks. They’re linear. I tend to think of simple delivery pipelines: Check out code, build code/artifacts, push to storage, deploy to environment. https://www.redhat.com/en/topics/automation/what-is-orchestration#:~:text=Orchestration%20is%20the%20automated%20configuration,isn’t%20a%20scalable%20strategy.

Schedulers ● ● ● ● Scheduling typically refers to the ability for an administrator to load a service file into a host system that establish how to run a specific container. We could think about the process of booking a meeting in a teammates calendar here. We can see their total load capacity (how many hours they have available in a work day) and and scheduling restraints they have (possibly their calendar is already full) and then make a decision on when/where to schedule based on that info. Perhaps another teammate can help and is more available, or perhaps another time is better. One of the key things schedulers are responsible for is host selection, based on scheduling restraints. https://www.digitalocean.com/community/tutorials/the-docker-ecosystem-scheduling-and-orchestration#:~:text=Orchestration%20is%20a%20broad%20term,to%20run%20a%20specific%20container.

Orchestrators ● ● ● ● ● ● ● ● Which brings us to orchestration. It’s generally the automation of a process or workflow that involves many steps across multiple systems. We typically expect an Orchestrator to handle things like cluster management, automated configuration, scheduling of workloads, keeping a desired state (ex I want to have 4 instances of this task, if it crashes, restart it), statefulsets or attached volumes to make sure when our workloads restart that they come back with the correct data and other details like rolling deployments. Generally, we want it to do management & co-ordination of computer systems, apps and services. ● There’s different options for how to handle orchestration but we’ll be focusing on Nomad today.

Nomad ● ● ● So let’s talk a bit about Nomad. It’s a HashiCorp product for Workload Orchestration. It’s available free and is OSS, and also has an enterprise version for those who need a bit more. You’re probably more familiar with hearing “container orchestration”. The reason I’m saying workload is that while we support containers, it doesn’t HAVE to be a container.

• help us to automate deployments, app scaling and upgrades; - empower developers to manage deployments directly; - automatically manage failures; - hide complex details such as node management, letting users choose only what is needed to have the application up and running. Some cool stuff: As mentioned earlier, Docker is supported, but any type of application can be used, on whatever type of operating system is needed (Linux, Windows, BSD and OSX are all supported). Clusters can be created, adding nodes from different data centers and/or different regions. I’d LOVE to redo this as we have way more features now but we did a 1 million container challenge https://www.hashicorp.com/c1m

Do One Thing Well ● ● Both HashiCorp & Linux share this philosophy of do one thing well.

K8s vs Nomad ● I don’t want everyone watching to have this burning question

02 Lock It Down ● For this demo, I finally got to do something I’ve been wanting to do for years - Start with Security First. I have a background implementing asks from our security team in a healthcare compliant environment, and one of the things our team talked about was trying to build security in first. Given that this is OWASP DevSlop, it’s only fitting that we’re going to talk about security first and it will be a returning theme. While I’m not a security expert, I did talk to one to help prep for this talk. He shared a lot of awesome tips, information & examples with me, but he told me to make sure I say this, because security experts love it.

Nomad Security ● https://www.nomadproject.io/docs/internals/security Things to think about for basic nomad/k8s Probably provisioning it up into a cloud env that abstracts a lot away Hardening at HostOS layer? But since Nomad is a single binary , generally easy to plug in systemD security features Ex Nomad servers don’t need to run as root. Clients, probably do (for access to stuff like Namespaces and Consul Connect) Was creating a nomad user:group to the docker group Then Nomad can spin it all up etc but gets rough with consul-connect. Caveat - docker groups sorta has root permissions

Defence in Depth ● Defense in depth is a concept used in Information security in which multiple layers of security controls are placed throughout an information technology system.

Top 3 Threats External Attacks ● ● ● ● ● Don’t have mtls, but do have ACLs, but exposed to internet During the first couple seconds, whoever hits it first will get the ACL token Now they have free run until you triage (ex maybe to run a bitcoin miner for that hour or smth) During initial bootstrap process, at the least need a mtls cert to avoid that

Top 3 Threats Supply Chain Attacks ● Package management/docker images.

Top 3 Threats Internal ● ● ● ● ● Top threat’s are usually internal Uncomfortable to bring up but worth the discussion Whether or not they’re intentional And the best protection against that is good logging/visibility

Tips & Tricks Favourite Tool ● ● ● Favourite tools: Falco & FluentD Centralized Logging + Observability are the 2 strongest supports for security. If you’re package installing in a docker container, it’s a security smell. Pro tip - if using the docker runtime, most security tools for docker will run well (ex Snyk) Nomad exec command Great to play in when simulating a compromised instance

https://github.com/picatz/terraform-google-nomad ● ● ● ● ● ● ● ● ● ● ● Module Features Automated load balancer configuration to access the Nomad Server API. Automatically enables mTLS, and generates certifcates. Automatically enables gossip encryption, and generates the gossip key. Automatically generates SSH credentials for the bastion host. ACLs enabled by default. Only the Docker task driver is enabled by default. Runs the Docker daemon with no-new-privileges=true and icc=false set by default. Installs the gVisor container runtime by default (runsc). Installs HashiCorp’s Consul service mesh.

03 Run Here, There, Everywhere! You could enter a subtitle here in case you need it

Running Nomad ● https://www.nomadproject.io/docs https://www.nomadproject.io/intro https://learn.hashicorp.com/nomad Job Spec Task Drivers

04 Q&A!

Thanks! @devopsjacquie /jacquelyne-grindrod @gogococo CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik. Please keep this slide for attribution.