A presentation at Philly DevOps February Meetup in in Philadelphia, PA, USA by Chris Munns
What happened in Serverless in 2019 Chris Munns Sr Manager/Principal Developer Advocate AWS Serverless © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
About me Chris Munns - munns@amazon.com, @chrismunns • Sr Manager/Principal Developer Advocate – Serverless • New Yorker (ehhh…ish.. kids/burbs/ya know?) • Previously: • AWS Business Development Manager – DevOps, July ‘15 - Feb ‘17 • AWS Solutions Architect Nov ‘11- Dec ‘14 • Formerly on operations teams @Etsy and @Meetup • Little time at a hedge fund, Xerox and a few other startups • Rochester Institute of Technology: Applied Networking and Systems Administration ’05 • Internet infrastructure geek © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why are we here today? © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://secure.flickr.com/photos/mgifford/4525333972
AWS shipped a ton of improvements to serverless services in 2019! © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Photo by Chris Munns
Serverless applications AWS Lambda © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless applications – starts with a function Function Node.js Python Java C# Go Ruby Runtime API © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless applications – add events Event source Changes in data state Requests to endpoints Changes in Resource state © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Function Node.js Python Java C# Go Ruby Runtime API
Serverless applications – add any other services Event source Changes in data state Requests to endpoints Changes in Resource state © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Function Node.js Python Java C# Go Ruby Runtime API Services
Today we’ll focus on these 3 main areas + shared Event source Function Shared capabilities © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services
“Tell me your biggest pain points right now” © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Photo by Zhen Hu on Unsplash
Starting with Lambda Event source Function Shared capabilities © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services
3 main ways that Lambda got better in 2019 Behind the curtain © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Visible but no-op Customer enabled
Behind the curtain The Lambda service team is dedicated to making things you don’t even see or know about better: “a reduction of 20 ms jitter in the invoke duration for Lambda Functions across all runtimes..”– May 2019
Lambda + VPC, no longer a cold-start pain point! ß Before: 14.8 sec duration After: 933ms duration à © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda + VPC, no longer a cold-start pain point! ß Before: 14.8 sec duration After: 933ms duration à © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda states Serverless applications with Lambda often require other non-Lambda resources Creation/modification of those resources can impact function execution time The states lifecycle allows us to move that resource modification before execution reducing the pain involved Now: VPC resources are created/modified before your function is “Active” for executions © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda states + VPC creation/modification NEW!!! Read more! © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cold starts and you © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Photo by Chris Munns
Cold Starts and you A cold start is what occurs when a system needs to create a new resource in response to an event/request. For Lambda: • • Happens when new execution environments are needed to handle requests Typically <1% of all invokes for “production workloads” • • As measured by functions with consistent invokes over a period of time, aka not dev, excludes rarely invoked Can vary from <100ms to a > 1 second Why this variance? © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available Yes Complete invocation Invoke handler warm start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available Yes Complete invocation Invoke handler warm start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. This is the path that most invocations end up taking in applications with somewhat consistent traffic patterns. A small <1% of requests would cause a cold-start and then ~99% would end on warmed environments
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available No Find available compute resource “full” cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available No Find available compute resource “full” cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Download customer code
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available No Find available compute resource Download customer code Start execution environment “full” cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available No Find available compute resource Download customer code Execute INIT Start execution environment “full” cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available No Find available compute resource Download customer code Execute INIT Start execution environment “full” cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
INIT - Pre-handler code, dependencies, variables Executes at the initial start of an execution environment. • Import only what you need • Where possible trim down SDKs and other libraries to the specific bits required • • Pre-handler code is great for establishing connections, but be prepared to then handle reconnections in further executions REMEMBER – execution environments are reused • • • Lazily load variables in the global scope Don’t load it if you don’t need it – cold starts are affected Clear out used variables so you don’t run into left-over state © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Import sdk Import http-lib Import ham-sandwich Pre-handler-secret-getter() Pre-handler-db-connect() Function myhandler(event, context) { ….
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available Complete invocation Invoke handler No Find available compute resource Download customer code Execute INIT Start execution environment “full” cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available Yes Complete invocation Invoke handler warm start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Subsequent invokes could then follow the warm path
Concurrency and you © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Photo by Kolleen Gladden on Unsplash
Concurrency and you A single Lambda execution environment can only process a single event at a time. • • Regardless of event source or invoke type Batches pulled from Amazon Kinesis Data Streams, Amazon SQS, or Amazon DynamoDB Streams count as a single event Concurrent requests will require new execution environments to be created • Limited in concurrency by burst rate per account per region © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concurrency and you Time Request 1 Cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Execution
Concurrency and you Time Request 1 Cold start Execution This execution environment is blocked for this entire time © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concurrency and you Time Request 1 Cold start Execution Request 2 Cold start Execution Request 3 Cold start Execution © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concurrency and you Time Request 1 Cold start Execution Request 2 Cold start Execution Request 3 Cold start Execution Request 4 Request 5 Cold start Cold start Execution Execution The first environment is now free and warm © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concurrency and you Time Request 1 Cold start Execution Request 2 Cold start Execution Request 3 Cold start Execution Request 4 Request 5 Cold start Cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Request 6 Execution Execution Execution
Concurrency and you Time Request 1 Cold start Execution Request 2 Cold start Execution Request 3 Cold start Execution Request 4 Request 5 Cold start Cold start Request 6 Execution Request 7 Execution Request 8 Execution Execution Request 10 Execution Request 9 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cold start Ex
Concurrency and you Time Request 1 Cold start Request 2 Cold start Execution Request 3 Cold start Execution Request 4 Request 5 Request 6 Execution Cold start Cold start Execution Request 7 Execution Request 8 Execution Execution Request 10 Execution Request 9 Concurrency 3 5 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 4 6 5 Cold start Ex
Lambda per function concurrency controls Concurrency a shared pool by default = 1000 per account per region Separate using per function concurrency settings • Acts as reservation Also acts as max concurrency per function • Especially critical for downstream resources like databases “Kill switch” – set per function concurrency to zero © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introducing AWS Lambda Provisioned Concurrency © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Photo by Erwan Hesry on Unsplash
Introducing AWS Lambda Provisioned Concurrency Pre-creates execution environments all the way up through the INIT phase. • • • • • Mostly for interactive workloads that are heavily latency sensitive Greatly improved consistency across the full long tail of performance Little to no changes to your code or way you use Lambda Integrated with AWS Auto Scaling Adds a cost factor for per concurrency provisioned but a lower duration cost for execution • This could end up saving you money when heavily utilized © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. NEW!!!
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available Complete invocation Invoke handler No Find available compute resource Download customer code Execute INIT Start execution environment “full” cold start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Function configured with Provisioned Concurrency Find available compute resource Download customer code Execute INIT Start execution environment Provisioned concurrency start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Function lifecycle Request made to Lambda’s API Service identifies if warm execution environments is available Yes Complete invocation Invoke handler warm start © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. This becomes the default for all provisioned concurrency execution environments
Provisioned Concurrency – things to know • • Reduces the start time to your function handler to <100ms Can’t configure for $LATEST • • • • • Use versions/aliases Soft limit of 500 provisioned execution environment creation per minute No changes to function handler code performance Requests above provisioned concurrency follow on-demand Lambda limits and behaviors for cold-starts, bursting, pricing Still limited by overall account concurrency per limit region © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provisioned Concurrency – things to know In order to maintain availability and help with scaling we provision more resources than you request • • You don’t pay for these extra execution environments But because they execute INIT, they could consume other resources from your account (DB connections, calls to other services) In order to keep execution environments fresh, we still reap them regularly but will pre-create execution environments behind the scenes • • You’ll see INITs in your logs every few hours without configuration changes or other events This won’t impact performance We give less CPU burst to Provisioned Concurrency than On-Demand during INIT and so code could take longer to execute © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provisioned Concurrency – pricing https://aws.amazon.com/lambda/pricing/ : • Provisioned Concurrency is calculated from the time you enable it on your function until it is disabled, rounded up to the nearest 5 minutes. The price depends on the amount of memory you allocate to your function and the amount of concurrency that you configure on it. No free tier. • Duration is calculated from the time your code begins executing until it returns or otherwise terminates, rounded up to the nearest 100ms. The price depends on the amount of memory you allocate to your function. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provisioned Concurrency - configuration © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provisioned Concurrency - configuration © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provisioned Concurrency - configuration © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provisioned Concurrency - AutoScaling © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provisioned Concurrency - configuration © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
On events Event source Function Shared capabilities © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services
Lambda API API provided by the Lambda service SDK clients 1. Lambda directly invoked via invoke API Used by all other services that invoke Lambda across all models Supports sync and async Can pass any event payload structure you want Lambda function © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Client included in every SDK
Lambda execution model Synchronous (push) Amazon API Gateway Asynchronous (event) Amazon SNS Amazon DynamoDB Amazon S3 Amazon Kinesis /order Lambda function © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Stream (Poll-based) reqs Lambda function changes AWS Lambda service function
HTTP APIs for Amazon API Gateway Achieve up to 67% cost reduction and 50% latency reduction compared to REST APIs. HTTP APIs are also easier to configure than REST APIs, allowing customers to focus more time on building applications. Reduce application costs by up to 67% Reduce application latency by up to 50% © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Configure HTTP APIs easier and faster than before
© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. w ie ev Pr Amazon API Gateway – HTTP APIs
Synchronous APIs POST /order Client © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 201 Order service
Synchronous APIs POST /order Order service Client POST /order retry after failure Client © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 201 Order service
Synchronous APIs POST /order Client 201 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. POST /invoice Order service 201 Invoice service
Synchronous APIs POST /order Client 201 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. POST /invoice Order service 201 Invoice service
Synchronous APIs POST /order POST /invoice Who owns the retry? Does OrderFor how long?Invoice 201client service the ever know?201Etc.. service Client This effectively creates a “tight coupling” where failures become harder to recover from © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thinking asynchronously Client Service A Service B Client Service A Service B Synchronous Asynchronous commands events © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Asynchronous APIs POST /order Client 201 POST /invoice Order service Get /invoice Client 200 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Invoice service Invoice service
If you don’t need a response, execute async Use the Lambda APIs to start an asynchronous execution • Built-in queue (Amazon SQS behind the scenes) • Automatic retries • Dead letter queue for failed events client = boto3.client(“lambda”) client.invoke_async( FunctionName=”test” InvokeArgs=json_payload ) © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Topics, streams, queues, and buses Amazon SNS Amazon SQS © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon EventBridge Amazon Kinesis Data Streams
Ways to compare Scale/concurrency controls Durability Persistence Consumption models Retries Pricing © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concurrency across models SNS/API No event store Queue based Stream based © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recent announcements for async event sources Amazon SQS FIFO as invoke source for Lambda Amazon SNS Dead Letter Queues (DLQs) For streamed events: • • • • • • NEW!!! MaximumRetryAttempts, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError, On-failure destination Batch Window Parallelization Factor For async events: • • • MaximumRetryAttempts MaximumEventAgeInSeconds ICYMI: Serverless pre:Invent 2019 © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Directed vs. Observable events Joe, please create an invoice. Directed Commands © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ok, here it is
Directed vs. Observable events I don’t care about this Joe, please create an invoice. Customer X just ordered a widget I’ll send an invoice Ok, here it is I’ll add that to the sales report Directed Observable Commands Events © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Event passing with Amazon EventBridge Amazon Simple Notification Service HTTPS server on premises Amazon Kinesis Data Firehose Amazon S3 Amazon Athena Amazon Elastic Container Service Task Lambda based service Amazon EventBridge AWS Step Functions Workflow Amazon Comprehend AWS Lambda © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon SageMaker
Events with Amazon EventBridge • Order service Amazon EventBridge © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Your services can both produce messages onto the bus and consume just the messages they need from the bus Services don’t need to know about each other, just about the bus.
Amazon EventBridge © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EventBridge – schemas! NEW!!! © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Backend services + data stores Event source Function Shared capabilities © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services
Introducing: RDS Proxy (Preview) Fully managed, highly available database proxy feature for Amazon RDS. Pools and shares DB connections to make applications more scalable, resilient to database failures, and secure. Pool and share app scaling availability DB failover times © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. data security access controls Fully managed compatible
Scaling Serverless apps with Amazon RDS Proxy Lambda functions RDS Proxy SQL / TLS Connection Pool Amazon RDS AWS Secrets Manager © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS IAM
Two options for security: Control is in your hands Developers can use either IAM or native DB credentials to connect 1 n o ti p O 1: t tion p e es a St equ ntic R uthe n A ke To Amazon RDS Proxy SQL / TLS SQL / TLS Connect with native DB creds © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 2: Connect with IAM token Amazon RDS 2 n o ti p O AWS IAM
Introducing Lambda Event Destinations For asynchronous invocations, capture Success or Failure record contains details about the request and response in JSON format • Contains more information than data sent to a DLQ • Can send both outcomes to same destination or • Can send success to one destination, failure to another • © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. NEW!!! AWS Lambda AWS Lambda Amazon SNS Amazon SQS Amazon EventBridge
Shared capabilities and services Event source Function Shared capabilities © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services
How can we measure this? © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Photo by Chris Munns
AWS X-Ray Profile and troubleshoot serverless applications: • Lambda instruments incoming requests for all supported languages and can capture calls made in code • API Gateway inserts a tracing header into HTTP calls as well as reports data back to X-Ray itself © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. var AWSXRay = require(‘aws-xray-sdk-core‘); var AWS = AWSXRay.captureAWS(require(‘aws-sdk’)); S3Client = AWS.S3();
X-Ray trace example © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recent announcements for X-Ray & Amazon CloudWatch CloudWatch ServiceLens • a “single pane of glass” monitoring tool CloudWatch Synthetics • Generate “canary” tests for your APIs and other services Embedded Metric Format for CloudWatch Logs CloudWatch Contributor Insights • Identify top talkers, common patterns in logs X-Ray Trace Maps • Map a single event though your distributed application X-Ray integration with CloudWatch Synthetics © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ICYMI: Serverless pre:Invent 2019
What about tools? © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Photo by Carlos Irineu da Costa on Unsplash
AWS Serverless Application Model (AWS SAM) AWS CloudFormation extension optimized for serverless Special serverless resource types: functions, APIs, tables, layers, and applications Supports anything AWS CloudFormation supports Open specification (Apache 2.0) https://aws.amazon.com/serverless/sam © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SAM template AWSTemplateFormatVersion: ‘2010-09-09’ Transform: AWS::Serverless-2016-10-31 Resources: GetProductsFunction: Type: AWS::Serverless::Function Properties: Handler: index.getProducts Runtime: nodejs8.10 CodeUri: src/ Policies: - DynamoDBReadPolicy: TableName: !Ref ProductTable Events: GetResource: Type: Api Properties: Path: /products/{productId} Method: get ProductTable: Type: AWS::Serverless::SimpleTable © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Just 20 lines to create: • Lambda function • IAM role • API Gateway • DynamoDB table
AWS SAM Command Line Interface (AWS SAM CLI) CLI tool for local development, debugging, testing, deploying, and monitoring of serverless applications Supports API Gateway “proxy-style” and Lambda service API testing Response object and function logs available on your local machine Uses open source docker-lambda images to mimic Lambda’s execution environment such as timeout, memory limits, runtimes Can tail production logs from CloudWatch Logs Can help you build in native dependencies https://aws.amazon.com/serverless/sam © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
New with SAM CLI Completely improved deploy experience! • Interactive mode • Config file for deployed stacks • ChangeSet visibility • Stack events streamed © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s never been better to build with serverless @ AWS © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Photo by Markus Spiske on Unsplash
FIN/ACK • We are constantly working to improve our services for you • • Tell us your pains! Lot’s of exciting customer stories happening • • Please give me feedback! Don’t take just my word for it: https://www.datadoghq.com/state-ofserverless/ There’s lots of places where we’re working to make serverless better for you. Keep your eyes open! • Check the ICYMI blog posts on the AWS Compute blog! © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://rebrand.ly/PhillyDevOps
aws.amazon.com/serverless © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chris Munns munns@amazon.com @chrismunns © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://www.flickr.com/photos/theredproject/3302110152/
for free. You
can too.