Practical Approaches to Cloud Native Security

A presentation at RSA Conference in March 2019 in San Francisco, CA, USA by Karthik Gaekwad

Slide 1

Slide 1

SESSION ID: ASD-W10 Practical Approaches to Cloud Native Security Karthik Gaekwad Principal Engineer Oracle Inc @iteration1 #RSAC

Slide 2

Slide 2

Slides are here: http://bitly.com/cloudnative-rsa

Slide 3

Slide 3

#RSAC Hello • I’m Karthik Gaekwad • NOT a DBA • https://cloudnative.oracle.com/ • Cloud Native Evangelist at Oracle Cloud • Past: Developer on the Oracle Managed Kubernetes Team @iteration1

Slide 4

Slide 4

#RSAC Hello Been in Industry for 15 years. In general, I like building stuff with friends. – Maintainer for Gauntlt- Open source security scanner. Love Teaching and building community. – Run Devopsdays Austin, Container Days, Cloud Austin. – Chair All Day Devops Cloud Native track. – LinkedIn Learning Author for Learning Kubernetes (and more).

Slide 5

Slide 5

#RSAC I mustache you a question…

Slide 6

Slide 6

The Cloud Native Journey #RSAC Core to Edge Docker Speed Efficiency Agility Kubernetes Phase I Developer Focus Container Adoption Focus Applications Automation Community Developer adoption Dev/Test apps Simple orchestration Individual developers Phase II DevOps Focus Application Deployment DevOps deployment Production apps Advanced orchestration Teams & lines of business 6 Phase III Business Focus (end-to-end) Intelligent Operations End-to-end integration Digital business apps Serverless, DevSecOps, & ML Cloud native enterprises

Slide 7

Slide 7

#RSAC CNCF Survey: August 2018 How Does Your Company Use Containers and Where? Lots of adoption on dev/staging Continued production increase

Slide 8

Slide 8

#RSAC CNCF Survey: August 2018 How Does Your Company Use Containers and Where? Adoption over public and on-prem

Slide 9

Slide 9

#RSAC Kubernetes Dominates Container Management Your company/organization manages containers with: Winner! Kubernetes

Slide 10

Slide 10

#RSAC Many Projects… Good News, Bad News… Good usage in dev/prod But…

Slide 11

Slide 11

#RSAC Top 5 challenges to cloud native adoption… Monitoring Security Lack of Training Cultural Challenges Complexity 0 5 10 15 Percentages 20 25 30 35 40 45

Slide 12

Slide 12

#RSAC Kubernetes & Cloud Native Challenges Managing, maintaining, upgrading Kubernetes Control Plane – API Server, etcd, scheduler etc…. Managing, maintaining, upgrading Kubernetes Data Plane – In place upgrades, deploy parallel cluster etc…. Figuring out container networking & storage – Overlays, persistent storage etc… - it should just work Managing Teams – How do I manage & control team access to my clusters? Security, security, security Source: Oracle Customer Survey 2018 12

Slide 13

Slide 13

How Are Teams Addressing Complexity, Training Issues? Customer Managed Fully-Managed App Management App Management App Deployment Scaling High Availability Platform Backup & Recovery Upgrades & Patching Software Installation Server Provisioning Rack and Stack Power, HVAC #RSAC App Deployment Scaling YOU High Availability Platform Backup & Recovery Upgrades & Patching Software Installation Server Provisioning Rack and Stack Power, HVAC Benefits ü Faster Time to Deploy ü Lower Risk ü Accelerate Innovation

Slide 14

Slide 14

#RSAC Which brings us to security…

Slide 15

Slide 15

#RSAC Where no news, is good news!

Slide 16

Slide 16

#RSAC Unsecured K8s dashboards Unsecured Kubernetes Dashboard with account creds. Used this to mine cryptocurrency. 2017: Aviva 2018: Tesla, Weight Watchers https://redlock.io/blog/cryptojacking-tesla

Slide 17

Slide 17

#RSAC Kubelet credentials hack Shopify: Server Side request Forgery Get kubelet certs/private key Root access to any container in part of infrastructure. https://hackerone.com/reports/341876

Slide 18

Slide 18

#RSAC CVE’s Happen… Even more relevant with increased production usage of containers…

Slide 19

Slide 19

#RSAC CVE’s Happen… Privilege Escalation Container Escaping

Slide 20

Slide 20

#RSAC

Slide 21

Slide 21

#RSAC

Slide 22

Slide 22

#RSAC

Slide 23

Slide 23

#RSAC

Slide 24

Slide 24

#RSAC WAT?

Slide 25

Slide 25

#RSAC How did we get here?

Slide 26

Slide 26

#RSAC “Kubernetes is too complicated”

Slide 27

Slide 27

#RSAC “Kubernetes is too complicated” “We hope it’ll get easier”

Slide 28

Slide 28

#RSAC ! What is your strate r e ett b t e g o t t ? t n r a a t s Iw o t e r e h w t u B

Slide 29

Slide 29

#RSAC Let’s look at: Attack Surface – More importantly, how to limit damage Security related features in K8s – The more you know, the better you build Opensource Tooling to help – Because we all need help

Slide 30

Slide 30

#RSAC Attack Surface

Slide 31

Slide 31

#RSAC Attack Surface Goal: Reduce the attack surface Analysis for: –Host(s) –Container (Images and running) –Kubernetes Cluster

Slide 32

Slide 32

#RSAC Attack Surface: Host These are the machines you’re running Kubernetes on. Age old principles of Linux still apply: – – – – Enable SELinux AppArmor Seccomp Hardened Images Goal: Minimize privilege to applications running on the host Good news: Already a wealth of information on this subject! – http://lmgtfy.com/?q=how+to+reduce+attack+surface+linux

Slide 33

Slide 33

#RSAC Attack Surface: Container Images GOAL: Know your base image when building containers

Slide 34

Slide 34

#RSAC Attack Surface: Container Images GOAL: Know your base image when building containers **BTW, this is just a ruby helloworld app

Slide 35

Slide 35

#RSAC Attack Surface: Container Images GOAL: Know your base image when building containers **BTW, this is just a ruby helloworld app

Slide 36

Slide 36

#RSAC Attack Surface: Container Images GOAL: Know your base image when building containers Full disclosure: I’m karthequian; I created this as a ruby 101 container for learning purposes only

Slide 37

Slide 37

#RSAC Attack Surface: Container Images GOAL: Know your base image when building containers When in doubt, stick to an official images! Or start from a sane base image (example: alpine linux)

Slide 38

Slide 38

#RSAC Attack Surface: Container Images GOAL: Smaller the image, the better Less things for an attacker to exploit. Quicker to push, quicker to pull.

Slide 39

Slide 39

#RSAC Attack Surface: Container Images GOAL: Don’t rely on :latest tag :latest image yesterday might not be :latest image tomorrow Instead, you’d want to know what specific version you’re operating with. Side benefit: If there is a new vulnerability announced for OS version x.y.z, you know immediately whether you’re running that version!

Slide 40

Slide 40

#RSAC Attack Surface: Container Images GOAL: Check for vulnerabilities periodically Plenty of ways to do this in registries. We’ll cover more in the tooling section

Slide 41

Slide 41

#RSAC Attack Surface: Running Containers GOAL: Don’t run as root Containers running as root might be completely unnecessary for the actual application. If compromised, attacker can do a lot more things.. Pod security policies can help (we’ll see how later).

Slide 42

Slide 42

#RSAC Attack Surface: Running Containers GOAL: Limit host mounts Be wary of images that require broad access to paths on the host Limit your host mount to a smaller subset of directories Reduces blast radius on compromise

Slide 43

Slide 43

#RSAC Attack Surface: Kubernetes Cluster

Slide 44

Slide 44

#RSAC Kubernetes Cluster- TLS TLS ALL THE THINGS

Slide 45

Slide 45

#RSAC Kubernetes Cluster- TLS TLS Checklist: 1. Nodes and Master 2. User and Master 3. Everything etcd 4. Kubelet to API Server

Slide 46

Slide 46

#RSAC Kubernetes Cluster- TLS

Slide 47

Slide 47

#RSAC Kubernetes Cluster- TLS TLS Checklist: 1. User and Master 2. Nodes and Master 3. Everything etcd 4. Kubelet to API Server

Slide 48

Slide 48

#RSAC CVE’s GOAL: Have an upgrade strategy Because…CVE’s are fixed in new minor versions. Don’t treat K8s as “install once, run all the time”. Make your K8s install repeatable for different versions. ..Or use a Managed Provider. – Either automatically patch for you, or tell you what to do. 48

Slide 49

Slide 49

We’re a little better off now. #RSAC But what else to do?

Slide 50

Slide 50

#RSAC K8s Features How can the platform help me make secure choices?

Slide 51

Slide 51

#RSAC K8s Features Kubernetes Secrets Authentication Authorization Audit Logging Network Policies Pod security policies

Slide 52

Slide 52

#RSAC Kubernetes Secrets GOAL: Use Kubernetes secrets to store sensitive data instead of config maps. Also look at: secrets encryption provider. – Controls how etcd encrypts API data – —experimental-encryption-provider-config https://kubernetes.io/docs/tasks/administer-cluster/encryptdata/

Slide 53

Slide 53

#RSAC Authentication and Authorization Do you know how you are authenticating with Kubernetes? Many ways to Authenticate – – – – – – Client Certs Static token file Service Account tokens OpenID Webhook Mode And more (https://kubernetes.io/docs/reference/access-authn-authz/authentication/)

Slide 54

Slide 54

#RSAC Goal: Pick a strategy that fits your use case Whatever you do, DO NOT YOLO!

Slide 55

Slide 55

#RSAC If you DO NOT YOLO… You can pick an authz strategy..

Slide 56

Slide 56

#RSAC Authentication and Authorization https://kubernetes.io/docs/reference/acc ess-authn-authz/authorization/

Slide 57

Slide 57

#RSAC Authentication and Authorization Pro tip: Nobody uses ABAC anymore. Don’t be that guy…. RBAC is the defacto standard – Based on roles and role bindings – Good set of defaults: https://github.com/uruddarraju/kubernetes-rbac-policies Can use multiple authorizers together, but can get confusing. – 1st authorizer to authorize passes authz

Slide 58

Slide 58

#RSAC Kubernetes Cluster- Audit Logs Wat? “Kubernetes auditing provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system.” Answers: What/when/who/where information on security events. Your job: Periodically watch Kubernetes Audit logs https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

Slide 59

Slide 59

#RSAC

Slide 60

Slide 60

#RSAC Kubernetes Cluster- Network Policies Consider adding a network policy to the cluster… Default Policy: All pods can talk to all other pods. Consider limiting this with a Network Policy https://kubernetes.io/docs/concepts/services-networking/network-policies/

Slide 61

Slide 61

#RSAC Kubernetes Cluster- Pod Security Policies Consider adding Pod Security policies PodSecurityPolicy: A Defined set of conditions a pod must run with. Think of this as authorization for pods.

Slide 62

Slide 62

#RSAC Kubernetes Cluster: Pod Security Policies Capability for an admin to control specific actions https://kubernetes.io/docs/concepts/policy/pod-security-policy/#what-is-a-pod-security-policy

Slide 63

Slide 63

#RSAC Open Policy Agent Policy based control for your whole environment. Full featured Policy Engine to offload policy decisions from each application/service. – Deploy OPA alongside your service – Add policy data to OPA’s store – Query OPA on decisions. Great idea, still early, watch this space… Standardize policies for all clusters https://www.openpolicyagent.org/ 63

Slide 64

Slide 64

#RSAC Tooling Opensource Tooling

Slide 65

Slide 65

#RSAC Keep tabs on the CNCF Security landscape https://landscape.cncf.io/landscape=security-complia

Slide 66

Slide 66

#RSAC CNCF Projects “The Update Framework” Is a project. Is a framework or a methodology. Based on TUF. Used for secure software updates. Based on ideas surrounding trust and integrity. A solution to secure software updates and distribution. Used in Docker Trusted Registry.

Slide 67

Slide 67

#RSAC Clair Open source project for the static analysis of vulnerabilities in containers. Find vulnerable images in your repo. Built into quay.io, but you can add to your own repo. https://github.com/coreos/clair

Slide 68

Slide 68

#RSAC

Slide 69

Slide 69

#RSAC Harbor Newer! CNCF Project Registry product Supports vulnerability scanning, image signing and identity control Scope is larger than clair

Slide 70

Slide 70

#RSAC Harbor

Slide 71

Slide 71

#RSAC Kube-bench Checks whether a Kubernetes cluster is deployed according to security best practices. Run this after creating your K8s cluster. https://github.com/aquasecurity/kube-bench Defined by the CIS Benchmarks Docs: https://www.cisecurity.org/cis-benchmarks/ Run it against your Kubernetes Master, or Kubernetes node.

Slide 72

Slide 72

#RSAC Kube-bench example

Slide 73

Slide 73

#RSAC Kube-hunter Penetration Testing tool for Kubernetes clusters. Looks for weaknesses in your cluster. 2 ways to run: – External to cluster (what an attacker sees) – Internal to cluster (install as a pod) Active and Passive mode. – Active mode changes stuff (probably don’t use in Prod) Read more: https://kube-hunter.aquasec.com/ ONLY RUN ON YOUR OWN CLUSTER. 73

Slide 74

Slide 74

#RSAC Kube-hunter 74

Slide 75

Slide 75

#RSAC Kubesec Helps you quantify risk for Kubernetes resources. Run against your K8s applications (deployments/pods/daemonsets etc) https://kubesec.io/ from controlplane Can be used standalone, or as a kubectl plugin (https://github.com/stefanprodan/kubectl-kubesec)

Slide 76

Slide 76

#RSAC Kubesec example

Slide 77

Slide 77

#RSAC Kubeaudit Opensourced from Shopify. Auditing your applications in your K8s cluster. https://github.com/Shopify/kubeaudit Little more targeted than Kubesec.

Slide 78

Slide 78

#RSAC

Slide 79

Slide 79

#RSAC Kubeaudit example

Slide 80

Slide 80

#RSAC Sonobuoy by Heptio Sonobuoy is a diagnostic tool. Tells you whether your cluster is built correctly per best practices. vendor-agnostic conformance tests to let you test the cluster. End to End tests test that APIs required for your deployment are supported by the cluster. Can take a long time to finish (1-3 hours). https://scanner.heptio.com/ 80

Slide 81

Slide 81

#RSAC Test Output summary Stored in a file called e2e.log 81

Slide 82

Slide 82

Put it all together… 82

Slide 83

Slide 83

#RSAC Apply It! Day 1: Know what version of Docker and Kubernetes you use. Understand if your control and data plane nodes are hardened. Understand how your Docker containers are built. Find out how you authenticate and authorize for your clusters. 83

Slide 84

Slide 84

#RSAC Apply It! Week 1: Build an Automation Pipeline: –To build Docker images on code pushes –Versioning strategy for code –To build your Kubernetes clusters 84

Slide 85

Slide 85

#RSAC Apply It! 1st Month Sanitize your code: – Know your base images – Implement versioning for your containers – Invest in a registry (or tooling) that does vulnerability scanning Kubernetes: – Have an upgrade strategy in place – Analyze secrets/sensitive cluster data – Turn on audit logging 85

Slide 86

Slide 86

#RSAC Apply It! 3 Months: Continuously Monitor – Tooling like sonobuoy/Kubesec/Kube-audit/kube-hunter// Plan how to address vulnerabilities/CVE’s K8s: – Strategy for Pod Security Policies – Strategy for Network Policies – Run scans (like kube-bench) on cluster creation 86

Slide 87

Slide 87

#RSAC Apply It! 6 Months: Re-ask day 1 questions. Review strategies- is it working? What needs tweaking? Review tooling- are there new tools that help? Are existing tools working? Review CVE’s 87

Slide 88

Slide 88

#RSAC Couple more resources to look at: 11 ways not to get hacked: https://kubernetes.io/blog/2018/07/18/11-ways-not-to-gethacked K8s security (from Image Hygiene to Network Policy): https://speakerdeck.com/mhausenblas/kubernetes-securityfrom-image-hygiene-to-network-policies

Slide 89

Slide 89

#RSAC KEEP CALM AND KUBE ON @iteration1