How Netlify migrated to multicloud while no one noticed

A presentation at Velocity Conf by Ryan Neal

How Netlify Migrated to a Multicloud Architecture And no one noticed

Engineer of things
Tech Passions
Distributed Systems
Streaming Data System
Infrastructure Automation
System Design
Worked
Raytheon
Palantir Middle East
Ye l p
Netlify

full CI/CD
prerendering
content delivery
lambda deployment
routing layer
split testing
identity provider
dns provider
... What is Netlify? Netlify is the simplest way to build, deploy, and manage web projects on the JAMstack. We're changing the way the web is built by collapsing the modern front-end development process into a single, simplified workflow.

full CI/CD
prerendering
content delivery
lambda deployment
routing layer
split testing
identity provider
dns provider
... What is Netlify? Over
5 million sites
4,000 requests/sec
1,200 deploys/hour

What is Netlify? Over
5 million sites
4,000 requests/sec
1,200 deploys/hour
full CI/CD
prerendering
content delivery
lambda deployment
routing layer
split testing
identity provider
dns provider
...

What am I going to talk about? 1. Intro to the system 2. Why we did all this work 3. How we accomplished it 4. The actual migration 5. Next steps

DB Cluster Origin Cluster CDN Content Delivery origin origin origin origin origin origin cdn node cdn node cdn node cdn node cdn node cdn node Cloud
Files DNS

Plan for failure
Redundancy is a priority
Everything is horizontally scalable
Everything runs in cluster
Health checking for everything

DB Cluster Origin Cluster CDN But where does it live? origin origin origin origin origin origin

DB Cluster Origin Cluster CDN And when it fails? origin origin origin origin origin origin DNS

CDN stays up
Keep serving cached content
Higher traffic sites are going to be happier What happens when things go wrong?

Providers fail with no notice
Degraded perf > outage
Same cloud is fastest Multicloud Setup

AWS
Elastic Compute Cloud
S3
GCP
Compute Engine
Cloud Storage
Providers fail with no notice
Degraded perf > outage
Same cloud is fastest Multicloud Setup
RAX
Cloud Servers
Cloud Files

Steps to Multicloud 1. Double check assumptions 2. Replicate all the objects 3. Prepare the database 4. Make the origin services cloud agnostic 5. Test everything 6. Do the actual cutover

Assumption Checking https://github.com/rybit/cloud-bench

Replicate it all origin origin origin origin primary CF 1 2 3 {
"_id" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7",
"size" : 9935,
"sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", "created_at" : ISODate("2018-06-07T21:02:29.240Z"),
}

BlobSync
Done out of band from the request cycle
Constantly queries for unreplicated blobs
Pulls object down, pushes to the other clouds
Records progress and errors BlobSync

BlobSync {
"_id" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7",
"size" : 9935,
"sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", "created_at" : ISODate("2018-06-07T21:02:29.240Z"), “m”: 1 } BlobSync
Done out of band from the request cycle
Constantly queries for unreplicated blobs
Pulls object down, pushes to the other clouds
Records progress and errors

Replicate it all {
"_id" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7",
"size" : 9935,
"sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", "created_at" : ISODate("2018-06-07T21:02:29.240Z"), “m”: 1, “r”: true } Replication Flag Spares index in mongo origin origin origin origin primary CF 1 2 3

State of the world origin origin origin origin primary CF CDN

State of the world origin origin origin origin primary CF GCS S3 BlobSync CDN

State of the world origin origin origin origin CF GCS S3 BlobSync CDN primary

Cloud Agnostic Origin Services
Generic cloud storage interface
Automatic failover
Prefer staying in cloud
Forceable overrides

Smart Resolution origin CDN primary {
"sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", “m”: 7 }

Smart Resolution primary {
"sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", “m”: 7 } origin CDN

State of the world origin origin origin origin CF origin origin origin origin GCS origin origin origin origin S3 BlobSync CDN primary

Pulling the trigger 1. Spin up enough origin services 2. Fail over the DB
3. Update the consul entry 4. Aggressively stare at monitors

primary State of the world origin origin origin origin CF origin origin origin origin GCS origin origin origin origin S3 BlobSync CDN

Redundant everything
Cloud agnostic origin and CDN
Programmable infrastructure
Out of band replication
Smart routing
Automated failover Summary

So now what?
Setup trickle of traffic to live standby
Automate the traffic switch
Speedup network scale up
More monitoring

ryan@ rybit
Find me to talk! @ry_boflavin

Ryan Neal
@rybit

1 / 103

Netlify’s CDN runs on six different cloud providers, but up until recently, its origin servers relied on only one. That meant its origin servers were subject to the performance and uptime characteristics of a single provider. The company needed make sure that in the face of a major outage from an underlying provider anywhere in its network, Netlify’s service would continue with minimal interruption.

Ryan Neal explains how Netlify planned, tested, and executed its first multicloud migration that could direct traffic to Google Cloud (GCP), AWS, and Rackspace Cloud on demand, without any service interruptions. Along the way, Ryan shares lessons learned and key takeaways you can apply to your own infrastructure decisions.

How Netlify migrated to multicloud while no one noticed

Who am I?

Who am I?

Dog Dad

Who am I?

Dog Dad

Engineer

Engineer of things

Tech Passions

Distributed Systems

Streaming Data System

Infrastructure Automation

System Design

Worked

Raytheon

Palantir Middle East

Ye l p

full CI/CD

prerendering

content delivery

lambda deployment

routing layer

split testing

identity provider

dns provider

full CI/CD

prerendering

content delivery

lambda deployment

routing layer

split testing

identity provider

dns provider

full CI/CD

prerendering

content delivery

lambda deployment

routing layer

split testing

identity provider

dns provider

Plan for failure

Redundancy is a priority

Everything is horizontally scalable

Everything runs in cluster

CDN stays up

Keep serving cached content

CDN stays up

Keep serving cached content

CDN stays up

Keep serving cached content

Providers fail with no notice

Degraded perf > outage

AWS

Elastic Compute Cloud

S3

GCP

Compute Engine

Cloud Storage

Providers fail with no notice

Degraded perf > outage

Same cloud is fastest Multicloud Setup

RAX

Cloud Servers

AWS

Elastic Compute Cloud

S3

GCP

Compute Engine

Cloud Storage

Providers fail with no notice

Degraded perf > outage

Same cloud is fastest Multicloud Setup

RAX

Cloud Servers

BlobSync

Done out of band from the request cycle

Constantly queries for unreplicated blobs

Pulls object down, pushes to the other clouds

BlobSync { "_id" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", "size" : 9935, "sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", "created_at" : ISODate("2018-06-07T21:02:29.240Z"), “m”: 1 } BlobSync

BlobSync {
"_id" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7",
"size" : 9935,
"sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", "created_at" : ISODate("2018-06-07T21:02:29.240Z"), “m”: 1 } BlobSync

BlobSync {
"_id" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7",
"size" : 9935,
"sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", "created_at" : ISODate("2018-06-07T21:02:29.240Z"), “m”: 1 } CF BlobSync

BlobSync {
"_id" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7",
"size" : 9935,
"sha" : "9c74b7c31a3c04634ddf1d54d2339e0163dcf4a7", "created_at" : ISODate("2018-06-07T21:02:29.240Z"), “m”: 1 } CF S3 GCS BlobSync