A presentation at Texas Cyber Summit in in San Antonio, TX, USA by Karthik Gaekwad
FaaS Track to Serverless Security Texas Cyber Summit @iteration1
Karthik Gaekwad Cloud Native Advocate, Oracle Cloud Infrastructure Live in Austin Run Devopsdays and Devsecopsdays Austin @iteration1
Shoutout @wickett Principal Security Engineer @Verica Follow James’ work @wickett @iteration1
What are we upto today? * * * * * Serverless changes the security landscape Where security fits into serverless The Secure WIP model for serverless A quick look at lambhack Serverless provider security tips @iteration1
What is Serverless? @iteration1
Serverless encourages functions as deploy units, coupled with third party services that allow running end-to-end applications without worrying about system operation. @iteration1
Isn’t that a PaaS? @iteration1
@iteration1
Serverless is IT Value @iteration1
…without worrying about system operation — About 2 minutes ago @iteration1
Yasss! Ops (and security) for free! @iteration1
Tech burden can only be transferred @iteration1
Ops burden to rationalize serverless model — @patrickdebois @iteration1
Applies to security too @iteration1
Security burden is not created or destroyed (in serverless), merely transferred @iteration1
Security is in crisis @iteration1
Inequitable Labor Distribution @iteration1
10:1 Dev:Ops @iteration1
100:10:1 Dev:Ops:Sec @iteration1
The new OSI model @iteration1
Security knows the crisis is real @iteration1
Companies are spending a great deal on security, but we read of massive computer-related attacks. Clearly something is wrong. The root of the problem is twofold: we’re protecting the wrong things, and we’re hurting productivity in the process. @iteration1
And the survey says @iteration1
While engineering teams are busy deploying leading-edge technologies, security teams are still focused on fighting yesterday’s battles. SANS 2018 DevSecOps Survey @iteration1
95% of security professionals spend their time protecting legacy applications @iteration1
“many security teams work with a worldview where their goal is to inhibit change as much as possible” @iteration1
Serverless model doesn’t fit into security team’s worldview @iteration1
How do we change this? @iteration1
WIP @iteration1
Secure WIP for Serverless → The code you Write → The code you Inherit → The container you were Provided @iteration1
Secure WIP means collaboration DevSecOps @iteration1
WIP @iteration1
How to WIP? @iteration1
Security seperation of concerns
OWASP Serverless Top 10 (2017) OWASP Serverless Top 10 @iteration1
VERY relevant in serverless * * * * * A1 Injection A5 Broken Access Control A6 Security Misconfiguration A9 Components with known vulnerabilities A10 Insufficient Logging & Monitoring ..talk about these as we go along.. @iteration1
Secure WIP @iteration1
WIP Write @iteration1
OWASP A1-Injection Issue: Hostile Incoming Data * Same issues as in traditional apps, but more prevalent. * Frontend frameworks made this transparent before. * Need to pay more attention now. @iteration1
Injection What should I do? → Input Validation FTW. → Seperate data from commands/queries. → Sanitize data being stored. → Use Whitelist validation strategy (if possible). @iteration1
Injection- Whitelist & Blacklisting Whitelisting only passes expected data. In contrast, blacklisting relies on programmers predicting all unexpected data. As a result, easier to make mistakes with blacklisting. @iteration1
OWASP A5-Broken Access Control Issue: Users acting outside their intended permissions. * URL Modificiation Example: lambhack demo with uname * Metadata, Header manipulation * Token Expiration (or lack thereof) @iteration1
Broken Access Control What do I do? → Deny by default strategy → Use an access control mechanism → Rate limit against automated tooling → Log the failures (but NOT sensitive data) @iteration1
Serverless Myth @iteration1
You can’t do command execution through the API gateway — Anonymous Developer @iteration1
@iteration1
Vulnerable Lambda + API Gateway stack → Wanted to see make the point that appsec is relevant in serverless → Born from the heritage of WebGoat, Rails Goat … @iteration1
Lambhack → A Vulnerable Lambda + API Gateway stack → Open Source, MIT licensed → Includes arbitrary code execution in a query string @iteration1
Basically a reverse shell in http query string for lambda @iteration1
// Handler is our lambda handler invoked by the lambda.Start function call func Handler(ctx context.Context, request events.APIGatewayProxyRequest) (Response, error) { output := “Your function executed successfully!” if len(request.QueryStringParameters[“q”]) > 0 { // Source of our hacky code… output = runner.Run(request.QueryStringParameters[“q”]) log.Print(“Request %v, q=%v, %v”, string(request.QueryStringParameters[“q”]), string(output)) log.Print(output) } resp := Response{ StatusCode: 200, Body: output, Headers: map[string]string{ “Content-Type”: “application/text”, }, } }
return resp, nil
$ make deploy MacbookHome:lambhack karthik$ make deploy rm -rf ./bin ./vendor Gopkg.lock dep ensure -v Root project is “github.com/karthequian/lambhack” 2 transitively valid internal packages 2 external packages imported from 1 projects (0) ✓ select (root) (1) ? attempt github.com/aws/aws-lambda-go with 2 pkgs; 24 versions to try (1) try github.com/aws/aws-lambda-go@v1.13.2 (1) ✓ select github.com/aws/aws-lambda-go@v1.13.2 w/5 pkgs ✓ found solution with 5 packages from 1 projects (1/1) Wrote github.com/aws/aws-lambda-go@v1.13.2 env GOOS=linux go build -ldflags=”-s -w” -o bin/hello hello/main.go sls deploy Serverless: Packaging service… Serverless: Excluding development dependencies… Serverless: Uploading CloudFormation file to S3… Serverless: Uploading artifacts… Serverless: Uploading service myservice.zip file to S3 (3.11 MB)… Serverless: Validating template… Serverless: Updating Stack… Serverless: Checking Stack update progress… Serverless: Stack update finished… Service Information service: myservice stage: dev region: us-east-1 stack: myservice-dev resources: 10 api keys: None endpoints: GET - https://13grnm4qgi.execute-api.us-east-1.amazonaws.com/dev/hello functions: hello: myservice-dev-hello layers: None Serverless: Removing old service artifacts from S3… Serverless: Run the “serverless” command to setup monitoring, troubleshooting and testing. @iteration1
Description=”API Gateway URL” Key=APIGatewayURL Value=”https://XXXX.execute-api.us-east-1.amazonaws.com/prod” @iteration1
Run uname -a curl “<URL>/lambhack/c?args=uname+-a” returns Linux 169.254.54.149 4.14.133-97.112.amzn2.x86_64 \ 1 SMP Wed Aug 7 22:41:25 UTC 2019 x86_64 x86_64 \ x86_64 GNU/Linux @iteration1
/proc/version curl “<URL>/lambhack/c?args=cat+/proc/version” returns “Linux version 4.14.94-73.73.amzn1.x86_64 \ (mockbuild@gobi-build-64001) \ (gcc version 7.2.1 20170915 \ (Red Hat 7.2.1-2) (GCC)) \ #1 SMP Tue Jan 22 20:25:24 UTC 2019\n”
Look in /tmp curl “<URL>/lambhack/c?args=ls+-la+/tmp;+sleep+1” returns total 8 drwx——— 2 sbx_user1064 482 4096 Feb 21 22:35 . drwxr-xr-x 21 root root 4096 Feb 21 17:51 .. @iteration1
I can haz web proxy curl “<URL>/lambhack/c?args=curl+https://www.example.com;+sleep+1” returns <!doctype html> <html> <head> <title>Example Domain</title> <meta charset=”utf-8” /> …
github.com/wickett/lambhack @iteration1
AppSec Thoughts from Lambhack → Lambda has limited Blast Radius, but not zero → Monitoring/Logging plays a key role here → Detect longer run times → Higher error rate occurrences → Log actions of lambdas @iteration1
WIP Inherit @iteration1
It all seems so simple… 222 Lines of Code 5 direct dependencies 54 total deps (incl. indirect) (example thanks to snyk.io) @iteration1
460,046 Lines of Code @iteration1
Most defect density studies range from .5 to 10 defects per KLOC @iteration1
More importantly, defect density is not zero @iteration1
Vulnerabilities are just exploitable defects @iteration1
OWASP-A9 Components with known vulnerabilities What should I do? * Monitor dependencies continuously. * If you use a Docker based system, use the registry scanning tools. * Watch for CVE’s (they will happen). @iteration1
OWASP-A6 Security Misconfiguration Issue: Configuration or misconfiguration * Function permissiveness and roles (too much privilege) * Configuration for services (supporting cloud based services) * Security configuration left in logging @iteration1
OWASP-A6 Security Misconfiguration What should I do? * * * * * Limit your blast radius Harden security provider config (IAM/storage) Scan for global bucket read/write access Principle of least privilege Enterprise setting: MFA to access cloud console @iteration1
OWASP-A6 Principle of least privilege The practice of limiting access rights for users to the bare minimum permissions they need to perform their work. @iteration1
Most common attacks → Crypto Mining (via remote code execution) → Hijacking business flow → Denial of wallet → Data misconfiguration Via puresec whitepaper @iteration1
WIP Provided @iteration1
Platform Help @iteration1
Vendor Best Practices → Oracle Cloud Infrastructure → AWS → Google Cloud → Azure @iteration1
General Hygiene Recommendations * * * * * Disable root access keys Manage users with profiles Secure your keys in your deploy system Secure keys in dev system Use provider IAM and MFA @iteration1
Focus on IAM Roles and Policies @iteration1
@iteration1
Oracle Cloud Infrastructure → Oracle Functions based on Open Source Code → Fn Project: https://fnproject.io/ @iteration1
Oracle Cloud Infrastructure → IAM, MFA, Policy → Limit your blast radius with Compartments → Limit specific user/group access to specific compartments → Key Management Service → Security guidance @iteration1
AWS @iteration1
AWS lets you roll your own @iteration1
Choose your own adventure → Your very own Honeypot → Defend scanners and attack tooling → Parsing reputation lists → Deal with whitelisting/blacklisting → Tuning WAF Regex rules @iteration1
Cool, but figure out the importance! @iteration1
Azure → Lots of great resources in the docs! → Check out Security Center and Sentinel → Security Center → Security Policy → Key Vault Service @iteration1
@iteration1
Google Cloud → Follow IAM and data best practices → Security command → Storage best practices @iteration1
What about roll your own? → Knative → OpenFaaS → Fn → and others… @iteration1
Kubernetes Security → Many Faas providers can use K8s to deploy/scale → Understand how to K8s → Use K8s best practices → Starting point- Devsecops in a Cloudnative world @iteration1
The New Security Playbook * * * * Speed up delivery instead of blocking Empathy towards devs and ops Normal - provide value by making security normal Automate - security testing in every phase @iteration1
Security’s Path to Influence 1. Identify Resource Misutilization 2. Add Telemetry and Feedback Loops 3. Automate and Monitor Across the Software Pipeline 4. Influence Organizational Culture @iteration1
Conclusions * * * * Use the Secure WIP model Involve security team in serverless New Security Playbook Foster discussion on where to apply controls @iteration1
Moar Reccomendations * Learn from infosec * LASCON X in Austin in October * And…. @iteration1
Moar++ NEW! → 1st time in Austin! → Goal: “Talk about effective collaboration between dev, ops and security in our cloud (native) world.” → DevSecOpsDays Austin 2019 → December 16th, 2019 @iteration1
Keep In Touch @iteration1 theagileadmin.com cloudnative.oracle.com @iteration1
Bonus slides: Thought provoking talk: Gone in 60 Milliseconds Intrusion and Exfiltration in Server-less Architecture https://media.ccc.de/v/33c3-7865gonein60_milliseconds @iteration1
Security in FaaS isn’t what you’re used to. With enterprises quickly moving to serverless, there’s a need to address the topic of security.
Karthik Gaekwad shares what he’s learned in the application security sphere that still applies in the modern world of FaaS. Using lambhack, a sample vulnerable serverless application written in Go, Karthik explains how to think about security in the serverless world and details security strategies and pitfalls viewed through a serverless lens. You’ll leave with a solid understanding of how to approach security conversations about serverless applications in the enterprise.
for free. You
can too.