A presentation at Cisco Devnet Create in in Santa Clara, CA, USA by Karthik Gaekwad
DevSecOps 10 Tips for in a Cloud Native Security Cloud Native World Karthik Gaekwad Austin Developer Week 2018 http://bit.ly/cloudnative-security Karthik Gaekwad @iteration1 Devnet Create 2019 1
Hello • I’m Karthik Gaekwad • NOT a DBA • Cloud Native Evangelist at Oracle Cloud • https://cloudnative.oracle.com/ • Past: Developer on the Oracle Managed Kubernetes Team @iteration1 @iteration1
Hello • Been in Industry 15 years. • In general, I like building stuff with friends. • Maintainer for Gauntlt- Open source security scanner. • Love Teaching and building community. • Run Devopsdays Austin, Container Days, Cloud Austin. • Chair All Day Devops Cloud Native track. • LinkedIn Learning Author for Learning Kubernetes (and more). @iteration1
The Cloud Native Journey Core to Edge Docker Speed Efficiency Agility Kubernetes Phase I Developer Focus Container Adoption Focus Applications Automation Community Developer adoption Dev/Test apps Simple orchestration Individual developers Phase II DevOps Focus Application Deployment DevOps deployment Production apps Advanced orchestration Teams & lines of business Phase III Business Focus (end-to-end) Intelligent Operations End-to-end integration Digital business apps Serverless, DevSecOps, & ML Cloud native enterprises 4 @iteration1
CNCF Survey: August 2018 How Does Your Company Use Containers and Where? Lots of adoption on dev/staging Continued production increase @iteration1
CNCF Survey: August 2018 How Does Your Company Use Containers and Where? Adoption over public and on-prem @iteration1
Kubernetes Dominates Container Management Your company/organization manages containers with: Winner! Kubernetes @iteration1
Top 5 challenges to cloud native adoption… Monitoring Security Lack of Training Cultural Challenges Complexity 0 5 10 15 20 25 30 35 40 45 Percentages @iteration1
Kubernetes & Cloud Native Challenges • Managing, maintaining, upgrading Kubernetes Control Plane • API Server, etcd, scheduler etc…. • Managing, maintaining, upgrading Kubernetes Data Plane • In place upgrades, deploy parallel cluster etc…. • Figuring out container networking & storage • Overlays, persistent storage etc… - it should just work • Managing Teams • How do I manage & control team access to my clusters? • Security, security, security Source: Oracle Customer Survey 2018 9 @iteration1
How Are Teams Addressing Complexity, Training Issues? Customer Managed Fully-Managed App Management App Management App Deployment Scaling App Deployment Scaling YOU High Availability High Availability Platform Backup & Recovery Platform Backup & Recovery Upgrades & Patching Upgrades & Patching Software Installation Server Provisioning Software Installation Server Provisioning Rack and Stack Rack and Stack Power, HVAC Power, HVAC Fully managed Benefits ü Faster Time to Deploy ü Lower Risk ü Accelerate Innovation @iteration1
Which brings us to security…
Where no news, is good news!
Unsecured K8s dashboards • Unsecured Kubernetes Dashboard with account creds. • Used this to mine cryptocurrency. • 2017: Aviva • 2018: Tesla, Weight Watchers • https://redlock.io/blog/cryptojackingtesla @iteration1
Kubelet credentials hack • Shopify: Server Side request Forgery • Get kubelet certs/private key • Root access to any container in part of infrastructure. • https://hackerone.com/reports/341876 @iteration1
CVE’s Happen… Even more relevant with increased production usage of containers… @iteration1
CVE’s Happen… Privilege Escalation Container Escaping @iteration1
@iteration1
@iteration1
WAT? @iteration1
How did we get here?
“Kubernetes is too complicated”
“Kubernetes is too complicated” “We hope it’ll get easier”
What is your stratetter! e b t e g o t t ? t n r a a t s Iw o t e r e h w t u B
Let’s look at: • Attack Surface • More importantly, how to limit damage • Security related features in K8s • The more you know, the better you build • Opensource Tooling to help • Because we all need help
Attack Surface
Attack Surface Goal: Reduce the attack surface •Analysis for: • Host(s) • Container (Images and running) • Kubernetes Cluster @iteration1
Attack Surface: Host • These are the machines you’re running Kubernetes on. • Age old principles of Linux still apply: • • • • Enable SELinux AppArmor Seccomp Hardened Images • Goal: Minimize privilege to applications running on the host • Good news: Already a wealth of information on this subject! • http://lmgtfy.com/?q=how+to+reduce+attack+surface+linux @iteration1
Attack Surface: Container Images GOAL: Know your base image when building containers @iteration1
Attack Surface: Container Images GOAL: Know your base image when building containers **BTW, this is just a ruby helloworld app @iteration1
Attack Surface: Container Images GOAL: Know your base image when building containers Full disclosure: I’m karthequian; I created this as a ruby 101 container for learning purposes only @iteration1
Attack Surface: Container Images GOAL: Know your base image when building containers • When in doubt, stick to an official images! • Or start from a sane base image (example: alpine linux) @iteration1
Attack Surface: Container Images GOAL: Smaller the image, the better • Less things for an attacker to exploit. • Quicker to push, quicker to pull. @iteration1
Attack Surface: Container Images GOAL: Don’t rely on :latest tag • :latest image yesterday might not be :latest image tomorrow • Instead, you’d want to know what specific version you’re operating with. • Side benefit: If there is a new vulnerability announced for OS version x.y.z, you know immediately whether you’re running that version! @iteration1
Attack Surface: Container Images GOAL: Check for vulnerabilities periodically • Plenty of ways to do this in registries. We’ll cover more in the tooling section @iteration1
Attack Surface: Running Containers GOAL: Don’t run as root • Containers running as root might be completely unnecessary for the actual application. • If compromised, attacker can do a lot more things.. • Pod security policies can help (we’ll see how later). @iteration1
Attack Surface: Running Containers GOAL: Limit host mounts • Be wary of images that require broad access to paths on the host. • Limit your host mount to a smaller subset of directories. • Reduces blast radius on compromise. @iteration1
Attack Surface: Kubernetes Cluster
Kubernetes Cluster- TLS TLS ALL THE THINGS @iteration1
Kubernetes Cluster- TLS @iteration1
Kubernetes Cluster- TLS • TLS Checklist: 1. User and Master 2. Nodes and Master 3. Everything etcd @iteration1
CVE’s GOAL: Have an upgrade strategy • Because…CVE’s are fixed in new minor versions. • Don’t treat K8s as “install once, run all the time”. • Make your K8s install repeatable for different versions. • ..Or use a Managed Provider. • Either automatically patch for you, or tell you what to do. 41 @iteration1
We’re a little better off now. But what else to do?
K8s Features How can the platform help me make secure choices? @iteration1
K8s Features • Kubernetes Secrets • Authentication • Authorization • Audit Logging • Network Policies • Pod security policies • Open Policy Agent @iteration1
Kubernetes Secrets • GOAL: Use Kubernetes secrets to store sensitive data instead of config maps. • Also look at: secrets encryption provider. • Controls how etcd encrypts API data • —experimental-encryption-provider-config • https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ @iteration1
Authentication and Authorization • Do you know how you are authenticating with Kubernetes? • Many ways to Authenticate • • • • • • Client Certs Static token file Service Account tokens OpenID Webhook Mode And more (https://kubernetes.io/docs/reference/access-authn-authz/authentication/) @iteration1
Goal: Pick a strategy that fits your use case Whatever you do, DO NOT YOLO!
If you DO NOT YOLO… You can pick an authz strategy..
Authentication and Authorization https://kubernetes.io/docs/reference/acc ess-authn-authz/authorization/ @iteration1
Authentication and Authorization • Pro tip: Nobody uses ABAC anymore. Don’t be that guy…. • RBAC is the defacto standard • Based on roles and role bindings • Good set of defaults: https://github.com/uruddarraju/kubernetes-rbac-policies • Can use multiple authorizers together, but can get confusing. • 1st authorizer to authorize passes authz @iteration1
Kubernetes Cluster- Audit Logs • Wat? • “Kubernetes auditing provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system.” • Answers: What/when/who/where information on security events. • Your job: Periodically watch Kubernetes Audit logs • https://kubernetes.io/docs/tasks/debug-application-cluster/audit/ @iteration1
Kubernetes Cluster- Network Policies • Consider adding a network policy to the cluster… • Default Policy: All pods can talk to all other pods. • Consider limiting this with a Network Policy • https://kubernetes.io/docs/concepts/services-networking/network-policies/ @iteration1
Kubernetes Cluster- Pod Security Policies • Consider adding Pod Security policies • PodSecurityPolicy: A Defined set of conditions a pod must run with. • Think of this as authorization for pods. @iteration1
Kubernetes Cluster: Pod Security Policies Capability for an admin to control specific actions https://kubernetes.io/docs/concepts/policy/pod-security-policy/#what-is-a-pod-security-policy @iteration1
Open Policy Agent • Policy based control for your whole environment. • Full featured Policy Engine to offload policy decisions from each application/service. • Deploy OPA alongside your service • Add policy data to OPA’s store • Query OPA on decisions. • Great idea, still early, watch this space… • Standardize policies for all clusters • https://www.openpolicyagent.org/ 56 @iteration1
Tooling Opensource Tooling
Keep tabs on the CNCF Security landscape https://landscape.cncf.io/landscape=security-complia
CNCF Projects • “The Update Framework” • Is a framework or a methodology. • Used for secure software updates. • Based on ideas surrounding trust and integrity. • Is a project. • Based on TUF. • A solution to secure software updates and distribution. • Used in Docker Trusted Registry. @iteration1
Clair • Open source project for the static analysis of vulnerabilities in containers. • Find vulnerable images in your repo. • Built into quay.io, but you can add to your own repo. • https://github.com/coreos/clair @iteration1
@iteration1
Harbor • Newer! CNCF Project • Registry product • Supports vulnerability scanning, image signing and identity control • Scope is larger than clair @iteration1
Harbor
Kube-bench • Checks whether a Kubernetes cluster is deployed according to security best practices. • Run this after creating your K8s cluster. • https://github.com/aquasecurity/kube-bench • Defined by the CIS Benchmarks Docs: https://www.cisecurity.org/cisbenchmarks/ • Run it against your Kubernetes Master, or Kubernetes node. @iteration1
Kube-bench example @iteration1
Kubesec • Helps you quantify risk for Kubernetes resources. • Run against your K8s applications (deployments/pods/daemonsets etc) • https://kubesec.io/ from controlplane • Can be used standalone, or as a kubectl plugin (https://github.com/stefanprodan/kubectl-kubesec) @iteration1
Kubesec example @iteration1
Kubeaudit • Opensourced from Shopify. • Auditing your applications in your K8s cluster. • https://github.com/Shopify/kubeaudit • Little more targeted than Kubesec. @iteration1
@iteration1
Kubeaudit example @iteration1
Put it all together… 71 @iteration1
Apply It! • Day 1: • Know what version of Docker and Kubernetes you use. • Understand if your control and data plane nodes are hardened. • Understand how your Docker containers are built. • Find out how you authenticate and authorize for your clusters. 72 @iteration1
Apply It! •Week 1: •Build an Automation Pipeline: • To build Docker images on code pushes • Versioning strategy for code • To build your Kubernetes clusters 73 @iteration1
Apply It! • 1st Month • Sanitize your code: • Know your base images • Implement versioning for your containers • Invest in a registry (or tooling) that does vulnerability scanning • Kubernetes: • Have an upgrade strategy in place • Analyze secrets/sensitive cluster data • Turn on audit logging 74 @iteration1
Apply It! • 3 Months: • Continuously Monitor • Tooling like Kubesec/Kube-audit • Plan how to address vulnerabilities/CVE’s • K8s: • Strategy for Pod Security Policies • Strategy for Network Policies • Run scans (like kube-bench) on cluster creation 75 @iteration1
Apply It! •6 Months: •Re-ask day 1 questions. •Review strategies- is it working? What needs tweaking? •Review tooling- are there new tools that help? Are existing tools working? •Review CVE’s 76 @iteration1
Couple more resources to look at: • 11 ways not to get hacked: https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked • K8s security (from Image Hygiene to Network Policy): https://speakerdeck.com/mhausenblas/kubernetes-security-fromimage-hygiene-to-network-policies @iteration1
KEEP CALM AND KUBE ON @iteration1
Presentation on different security strategies for K8s presented at Cisco Devnet Create 2019
for free. You
can too.