The Jump to Hyperspace
๐ Light speed, antientropy, & moving past the cloud ๐ค
Slide 2
The Jump to Hyperspace
๐ Light speed, antientropy, & moving past the cloud ๐ค
Slide 3
Slide 4
I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail ~Abraham Maslow
Slide 5
I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail ~Abraham Maslow
Slide 6
Slide 7
[โฆ] by 2025, 75% of data will be processed outside the traditional data centre or cloud ~ Gartner (also Dell & IBM) https://www.gartner.com/smarterwithgartner/what-edge-computing-means-for-infrastructure-and-operations-leaders
Meta ๐ฎ โข R&D at Fission and others โข Future looking, emerging space โข Edge, web3, dweb โข Go โup a layerโ from BEAM to the internet itself
Slide 11
Meta ๐ฎ โข R&D at Fission and others โข Future looking, emerging space โข Edge, web3, dweb โข Go โup a layerโ from BEAM to the internet itself
Slide 12
โHistorical Reasonsโ
How We Got Here ๐ ๐ฒ
Slide 13
Motivation ๐ญ
1994 Set the Tone
Slide 14
Motivation ๐ญ
1994 Set the Tone
Slide 15
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
Slide 16
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ
Slide 17
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
Slide 18
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
๐
Slide 19
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
โ
๐
Slide 20
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
โ
๐ช
๐
Slide 21
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
โ
๐ช
๐
Slide 22
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
โ
๐ช
๐
Slide 23
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
โ
๐ช
๐
Slide 24
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
โ
๐ช
๐
Slide 25
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐
๐ฅ ๐ข
โ
๐ช
๐
Slide 26
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐ ๐ ๐
๐ฅ ๐ข ๐ฅ ๐ข ๐ฅ ๐ข
โ
๐ช
๐
Slide 27
Motivation ๐ญ
The Dream of the 90s is Alive on the Web ๐ถ
๐ ๐ ๐
๐ฅ ๐ข ๐ฅ ๐ข ๐ฅ ๐ข
โ
๐ช
๐
๐
Motivation ๐ญ
Less is More
๐ ๐ ๐
๐ฅ ๐ฅ ๐ฅ
โ ๐ ๐ ๐ โ ๐ โServerlessโ ๐ ๐ (AKA more โservers)
๐ฉ
๐
Slide 32
โฆand so it was for many yearsโฆ
Slide 33
โฆand so it was for many yearsโฆ ๐ฆโ๐๐พ๐ฐ๐ข๐
Slide 34
Motivation ๐ญ
Natural Consequences ๐
Slide 35
Motivation ๐ญ
Natural Consequences ๐
โข Single source of truth (โtheโ database)
Slide 36
Motivation ๐ญ
Natural Consequences ๐
โข Single source of truth (โtheโ database) โข Server-centric โข โFull stack developmentโ โข DevOps, Docker, k8s โข How to train enough engineers?
Slide 37
Motivation ๐ญ
Natural Consequences ๐
โข Single source of truth (โtheโ database) โข Server-centric โข โFull stack developmentโ โข DevOps, Docker, k8s โข How to train enough engineers?
Slide 38
Motivation ๐ญ
Natural Consequences ๐
โข Single source of truth (โtheโ database) โข Server-centric โข โFull stack developmentโ โข DevOps, Docker, k8s โข How to train enough engineers? โข Infrastructure Hegemony โข AWS (60%), GCP, Azure
Slide 39
Motivation ๐ญ
Sending a โDirectโ Message
Slide 40
Motivation ๐ญ
Sending a โDirectโ Message
Slide 41
Motivation ๐ญ
Sending a โDirectโ Message
Slide 42
Motivation ๐ญ
Sending a โDirectโ Message
โณ๐๐ข๐ช
Slide 43
New Environment ๐ฐ
Users vs Cloud Infra
Source: AWS
Slide 44
New Environment ๐ฐ
Users vs Cloud Infra
Source: AWS
Slide 45
New Environment ๐ฐ
Users vs Cloud Infra 6
7
6
2 1 1
1
1 Source: AWS
Slide 46
New Environment ๐ฐ
Users vs Cloud Infra 6
7
6
2 1 1
1
1 Source: AWS
Slide 47
New Environment ๐ฐ
Users vs Cloud Infra 6
7
6
2 1 1
1
1 Source: AWS
Slide 48
New Environment ๐ฐ
Users vs Cloud Infra 6
7
6
2 1 1
1
1 Source: AWS
Slide 49
New Environment ๐ฐ
Users vs Cloud Infra 6
7
6
2 1 1
1
1 Source: AWS
Slide 50
New Environment ๐ฐ
Users vs Cloud Infra 6
7
2
6
50M
1 1
1
1 Source: AWS
Slide 51
New Environment ๐ฐ
Users vs Cloud Infra
371 million 56M/centre
6
7
2
6
50M
1 1
1
1 Source: AWS
Slide 52
New Environment ๐ฐ
Users vs Cloud Infra
371 million 56M/centre
6
7
2
6
50M
1 1 ~435 million 435M/centre
1
1 Source: AWS
Slide 53
New Environment ๐ฐ
Users vs Cloud Infra
371 million 56M/centre
6
7
2
6
50M
1 1 ~435 million 435M/centre
1
~1.4 billion 1400M/centre
1 Source: AWS
Slide 54
New Environment ๐ฐ
Users vs Cloud Infra
371 million 56M/centre
6
7
2
6
50M
1 1 ~435 million 435M/centre
1
~1.4 billion 1400M/centre
1 Source: AWS
Slide 55
Video Killed the Radio Star
A New Environment ๐
Slide 56
New Environment ๐ฐ
New Environment
Slide 57
New Environment ๐ฐ
New Environment Then ๐
Now ๐
Slide 58
New Environment ๐ฐ
New Environment Need
Then ๐
Now ๐
Convenient ๐
Critical ๐จ
Slide 59
New Environment ๐ฐ
New Environment Then ๐
Now ๐
Need
Convenient ๐
Critical ๐จ
Location
Data Centre ๐ข
Powerful Clients (M1, IoT) โ๐๐
Slide 60
New Environment ๐ฐ
New Environment Then ๐
Now ๐
Need
Convenient ๐
Critical ๐จ
Location
Data Centre ๐ข
Powerful Clients (M1, IoT) โ๐๐
๐ฅ
๐ฑ
Access
Slide 61
New Environment ๐ฐ
New Environment Then ๐
Now ๐
Need
Convenient ๐
Critical ๐จ
Location
Data Centre ๐ข
Powerful Clients (M1, IoT) โ๐๐
๐ฅ
๐ฑ
Bandwidth ๐
Latency โฒ
Access Bottleneck
Slide 62
New Environment ๐ฐ
New Environment Then ๐
Now ๐
Need
Convenient ๐
Critical ๐จ
Location
Data Centre ๐ข
Powerful Clients (M1, IoT) โ๐๐
๐ฅ
๐ฑ
Bandwidth ๐
Latency โฒ
๐บ๐ธ๐ช๐บ
๐๐๐ โฆ ๐
Access Bottleneck Market
Slide 63
High Volume ๐ข
Science Fiction โ Reality
โข Remote surgery โข Extended reality โข Autonomous vehicles โข Location transparency โข Competitive cloud gaming โข Realtime manufacturing โข Continuous ML
Literal brain surgery over 5G 3,000km ~ 1,900mi
Slide 64
Slide 65
Sensor data explosion will kill the cloud. [โฆ] existing infrastructure will not be able to handle the volumes or the rates We are absolutely going to return to a peer-to-peer computing [โฆ] not unlike distributed computing We are going to move to a world of data-centric programming. ~ a16z, โThe End of Cloud Computingโ
Slide 66
Sensor data explosion will kill the cloud. [โฆ] existing infrastructure will not be able to handle the volumes or the rates We are absolutely going to return to a peer-to-peer computing [โฆ] not unlike distributed computing We are going to move to a world of data-centric programming. ~ a16z, โThe End of Cloud Computingโ
โ
Slide 67
Sensor data explosion will kill the cloud. [โฆ] existing infrastructure will not be able to handle the volumes or the rates We are absolutely going to return to a peer-to-peer computing [โฆ] not unlike distributed computing We are going to move to a world of data-centric programming. โ
~ a16z, โThe End of Cloud Computingโ
โ
Slide 68
Sensor data explosion will kill the cloud. [โฆ] existing infrastructure will not be able to handle the volumes or the rates We are absolutely going to return to a peer-to-peer computing [โฆ] not unlike distributed computing We are going to move to a world of data-centric programming. โ
~ a16z, โThe End of Cloud Computingโ
๐คพ
โ
๐
Slide 69
Sensor data explosion will kill the cloud. [โฆ] existing infrastructure will not be able to handle the volumes or the rates We are absolutely going to return to a peer-to-peer computing [โฆ] not unlike distributed computing We are going to move to a world of data-centric programming. โ
~ a16z, โThe End of Cloud Computingโ
๐คพ
โ
๐
Slide 70
The World is Changing
Relativistic Computing ๐ธ
Slide 71
Slide 72
The ship that made the Kessel Run in less than 12 parsecs ~ Han Solo
Slide 73
The ship that made the Kessel Run in less than 12 parsecs ~ Han Solo
Slide 74
New Environment ๐ฐ
Mooreโs Law. But make it networked.
Slide 75
Low Latency ๐
Latency is a Physical Limit ๐ง
โข Bandwidth max not even close โข Speed of light causality โข Edge dominates < 40ms โข Best at ~8ms โข 1ms applications exist โข โUltra Reliable Low Latencyโ
Slide 76
Low Latency ๐
Latency is a Physical Limit ๐ง
โข Bandwidth max not even close โข Speed of light causality โข Edge dominates < 40ms โข Best at ~8ms โข 1ms applications exist โข โUltra Reliable Low Latencyโ
f
Source: Ericsson http://cscn2017.ieee-cscn.org/ iles/2017/08/Janne_Peisa_Ericsson_CSCN2017.pdf
Slide 77
Low Latency ๐
Latency is a Physical Limit ๐ง
โข Bandwidth max not even close โข Speed of light causality โข Edge dominates < 40ms โข Best at ~8ms โข 1ms applications exist โข โUltra Reliable Low Latencyโ
f
Source: Ericsson http://cscn2017.ieee-cscn.org/ iles/2017/08/Janne_Peisa_Ericsson_CSCN2017.pdf
Slide 78
Low Latency ๐
What 8ms Looks Like
Slide 79
Low Latency ๐
What 8ms Looks Like Austin โก San Francisco Ideal Vacuum ๐ซ
Slide 80
Low Latency ๐
What 8ms Looks Like Austin โก San Francisco Ideal Vacuum ๐ซ
Austin ๐ (almost) Atlanta Ideal Vacuum ๐ซ
Slide 81
Low Latency ๐
What 8ms Looks Like Austin โก San Francisco Ideal Vacuum ๐ซ
Austin ๐ (almost) Atlanta Ideal Vacuum ๐ซ
Austin ๐ New Orleans Ideal Fibre ๐งถ
Slide 82
Low Latency ๐
Causal Islands ๐๐
Slide 83
Low Latency ๐
Causal Islands ๐๐
Slide 84
Low Latency ๐
Causal Islands ๐๐
Slide 85
Low Latency ๐
Causal Islands ๐๐
Slide 86
New Environment ๐ฐ
Friendly Neighbourhood Compute
Slide 87
New Environment ๐ฐ
Friendly Neighbourhood Compute
โข 5G networks & Starlink ๐ฐ โข Straight line point-to-point โข PoP directly on the tower ๐ผ๐
Slide 88
New Environment ๐ฐ
Friendly Neighbourhood Compute
โข 5G networks & Starlink ๐ฐ โข Straight line point-to-point โข PoP directly on the tower ๐ผ๐ โข Walmartโs Edge โข 90% Americans < 10mi [16km]
Slide 89
High Volume ๐ข
A New Topology
Slide 90
High Volume ๐ข
A New Topology
๐คณ
Slide 91
High Volume ๐ข
A New Topology
๐ฐ
๐คณ
๐ผ
๐พโ
Slide 92
High Volume ๐ข
A New Topology
๐ฐ
๐คณ
๐ผ
๐พโ
๐ข ๐พโ
Slide 93
High Volume ๐ข
A New Topology
๐ฐ
๐คณ
๐ผ
๐พโ
๐ข ๐พโ
โ ๐พโ โ ๐พ โ ๐พ โ ๐พ ๐พโ โ
Slide 94
High Volume ๐ข
A New Topology
๐ฐ
๐คณ
๐ผ
๐พโ
๐ฐ
๐ฐ
๐ข
โ ๐พโ
๐พโ
โ ๐พ โ ๐พ โ ๐พ ๐พโ โ
Slide 95
High Volume ๐ข
A New Topology
๐ฐ
Local ๐คณ First
๐ผ
๐พโ
๐ฐ
๐ฐ
๐ข
โ ๐พโ
๐พโ
โ ๐พ โ ๐พ โ ๐พ ๐พโ โ
Slide 96
High Volume ๐ข
A New Topology
๐ฐ
Local ๐คณ First
๐ผ
Realtime, Storage, Caching, OLTP ๐พโ
๐ฐ
๐ฐ
๐ข
โ ๐พโ
๐พโ
โ ๐พ โ ๐พ โ ๐พ ๐พโ โ
Slide 97
High Volume ๐ข
A New Topology
๐ฐ
Local ๐คณ First
๐ผ
Realtime, Storage, Caching, OLTP ๐พโ
๐ฐ Relay, Replication, Consistency, Tasks ๐พโ
๐ข
๐ฐ
โ ๐พโ โ ๐พ โ ๐พ โ ๐พ ๐พโ โ
Slide 98
High Volume ๐ข
A New Topology
๐ฐ
Local ๐คณ First
๐ผ
Realtime, Storage, Caching, OLTP ๐พโ
๐ฐ Relay, Replication, Consistency, Tasks ๐พโ
๐ข
๐ฐ
โ
Aggregation, Batching, Training, โ ๐พ โ ๐พ โ ๐พ OLAP โ ๐พ
๐พโ โ
Slide 99
High Volume ๐ข
A New Topology
๐ฐ
Local ๐คณ First ๐
๐ผ
Realtime, Storage, Caching, OLTP ๐พโ
๐ฐ Relay, Replication, Consistency, Tasks ๐พโ
๐ข
๐ฐ
โ
Aggregation, Batching, Training, โ ๐พ โ ๐พ โ ๐พ OLAP โ ๐พ
๐พโ โ ๐
Slide 100
Functional Programming
On the Edge ๐ง
Slide 101
Slide 102
Instead of [โฆ] โwhich database would be best to hold presences?โ, we could ask โhow can we best replicate data in a distributed system without the user having to worry about it?โ With Elixir, you are empowered to tackle problems that in other platforms would feel impossible to solve ~ Chris McCord, What Makes Phoenix Presence Special
Slide 103
On the Edge ๐ง
Phoenix LiveView
Slide 104
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
Slide 105
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
Slide 106
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ
๐พ๐พ๐พ๐พ ๐ โ
WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
Slide 107
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ
๐พ๐พ๐พ๐พ ๐ โ
WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
Slide 108
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ
๐พ๐พ๐พ๐พ ๐ โ
WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
Slide 109
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ
๐พ๐พ๐พ๐พ ๐ โ
WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
Slide 110
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ
๐พ๐พ๐พ๐พ ๐ โ โ
WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
Slide 111
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ
๐พ๐พ๐พ๐พ ๐ โ โ
WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
๐ฅ ๐พ
Slide 112
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ
๐พ๐พ๐พ๐พ ๐ โ โ
WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
๐ฅ ๐พ
Slide 113
On the Edge ๐ง
Phoenix LiveView Users ๐จ๐ซ๐ฉ๐ญ๐งโ๐ท Client ๐ฅ
๐พ๐พ๐พ๐พ ๐ โ โ
WSS / REST / GraphQL โ Controller Logic โ Data Store ๐ DevOps ๐ค Developer ๐ฉ๐ป
๐ฅ ๐พ
๐ฅ ๐พ
Slide 114
On the Edge ๐ง
Phoenix Inside Out ๐ฃ
๐ฅ
๐พ๐พ
๐ฅ
๐พ๐พ๐พ
โ
๐พ ๐
Slide 115
On the Edge ๐ง
Mo Distributed Mo Problems Property
Consequence
Casual islands
No single source of truth!
Local first
How to do coordination?
Replicate/run on any machine
ACLs fail (for reads, many writes)
Slide 116
On the Edge ๐ง
CAP โ PACELC ๐ฆ๐ฆ
Slide 117
On the Edge ๐ง
CAP โ PACELC ๐ฆ๐ฆ
โข If network partition, either: โข Availability (A) โ Uptime! โข Consistency (C)
Slide 118
On the Edge ๐ง
CAP โ PACELC ๐ฆ๐ฆ
โข If network partition, either: โข Availability (A) โ Uptime!
C
โข Consistency (C) A
Slide 119
On the Edge ๐ง
CAP โ PACELC ๐ฆ๐ฆ
โข If network partition, either: โข Availability (A) โ Uptime!
C
โข Consistency (C)
P
A
Slide 120
On the Edge ๐ง
CAP โ PACELC ๐ฆ๐ฆ
โข If network partition, either: โข Availability (A) โ Uptime!
C
โข Consistency (C)
P
โข Else (E) running normally, either: โข Latency (L) โ Speed! โข Consistency (C)
A
Slide 121
On the Edge ๐ง
CAP โ PACELC ๐ฆ๐ฆ
โข If network partition, either: โข Availability (A) โ Uptime!
C
โข Consistency (C)
P
E
โข Else (E) running normally, either: โข Latency (L) โ Speed! โข Consistency (C)
A
L
Slide 122
On the Edge ๐ง
CAP โ PACELC ๐ฆ๐ฆ
โข If network partition, either: โข Availability (A) โ Uptime!
C
โข Consistency (C)
P
E
โข Else (E) running normally, either: โข Latency (L) โ Speed! โข Consistency (C)
A
L
Slide 123
On the Edge ๐ง
CAP โ PACELC ๐ฆ๐ฆ
โข If network partition, either: โข Availability (A) โ Uptime!
C
โข Consistency (C)
P
E
โข Else (E) running normally, either: โข Latency (L) โ Speed! โข Consistency (C)
A
L PA/EL
Slide 124
On the Edge ๐ง
New Assumptions, New Approach
Slide 125
On the Edge ๐ง
New Assumptions, New Approach
โข New assumptions โ new architecture โ new features โข Local-first โ network e cient
ffi
โข Data can run anywhere โ commons networks
Slide 126
On the Edge ๐ง
New Tools
Slide 127
On the Edge ๐ง
ff
New Tools โข Conflict-free Replicated Data Type (CRDT) โข Software Transactional Memory (STM) โข Di erential Datalog (DDlog) โข Merkleization (hash chaining) โข Cryptographic compute (MPC/FHE/NIZK) โข REST < bidirectional sync < relative data views
Slide 128
On the Edge ๐ง
ff
New Tools โข Conflict-free Replicated Data Type (CRDT) โข Software Transactional Memory (STM) โข Di erential Datalog (DDlog) โข Merkleization (hash chaining) โข Cryptographic compute (MPC/FHE/NIZK) โข REST < bidirectional sync < relative data views
Slide 129
On the Edge ๐ง
ff
New Tools โข Conflict-free Replicated Data Type (CRDT) โข Software Transactional Memory (STM) โข Di erential Datalog (DDlog) โข Merkleization (hash chaining) โข Cryptographic compute (MPC/FHE/NIZK) โข REST < bidirectional sync < relative data views
Slide 130
On the Edge ๐ง
ff
New Tools โข Conflict-free Replicated Data Type (CRDT) โข Software Transactional Memory (STM) โข Di erential Datalog (DDlog) โข Merkleization (hash chaining) โข Cryptographic compute (MPC/FHE/NIZK) โข REST < bidirectional sync < relative data views
Slide 131
Fixing The Leaky Pipes
Identity & Access ๐ฟ
Slide 132
Fixing the Leaky Pipes ๐ฟ
ACL Read & Write
Slide 133
Fixing the Leaky Pipes ๐ฟ
ACL Read & Write
๐ง๐พ
Data dominates. If youโve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming. Rob Pike, 5 Rules of Programming
Slide 183
Itโs All About that Data ๐
Gossiping Out of Order ๐
Slide 184
Itโs All About that Data ๐
Gossiping Out of Order ๐
Slide 185
Itโs All About that Data ๐
Gossiping Out of Order ๐
๐จ
Slide 186
Itโs All About that Data ๐
Gossiping Out of Order ๐
๐จ
Slide 187
Itโs All About that Data ๐
Gossiping Out of Order ๐
tโ
๐จ
Slide 188
Itโs All About that Data ๐
Gossiping Out of Order ๐ ๐ข
๐จ
๐ด
๐ก
๐ด
tโ
๐ข
๐ข
๐ก
Slide 189
Itโs All About that Data ๐
Gossiping Out of Order ๐ ๐ข
๐จ
๐ด
๐ก
๐ด
tโ
๐ข
๐ข
๐ก
Slide 190
Itโs All About that Data ๐
Gossiping Out of Order ๐ ๐ข
๐จ
๐ด
๐ก
๐ด
tโ
๐ข
๐ข
๐ก
Slide 191
Itโs All About that Data ๐
Gossiping Out of Order ๐ ๐ข
๐จ
๐ด
๐ก
๐ด
tโ
๐ข
๐ข
๐ก
Slide 192
Itโs All About that Data ๐
Gossiping Out of Order ๐ ๐ข
๐จ
๐ด
๐ก
๐ด
tโ
๐ข
๐ข
๐ก
Slide 193
Itโs All About that Data ๐
Gossiping Out of Order ๐ ๐ข
๐จ
๐ด
๐ก
๐ด
tโ
๐ข
๐ข
๐ก
โ
Slide 194
Itโs All About that Data ๐
Entropy Isnโt What It Used To Be
Slide 195
Itโs All About that Data ๐
Entropy Isnโt What It Used To Be
Slide 196
Itโs All About that Data ๐
Abstract
โข Network agnostic โข Any number of replicas
Slide 197
Itโs All About that Data ๐
Abstract
Commutative
โข Network agnostic โข Any number of replicas
Slide 198
Itโs All About that Data ๐
Abstract
Commutative
merge
merge
โข Network agnostic โข Any number of replicas
Slide 199
Itโs All About that Data ๐
Abstract
Commutative
merge
merge
โข Network agnostic โข Any number of replicas
Slide 200
Itโs All About that Data ๐
Abstract
Commutative
merge
merge
โข Network agnostic โข Any number of replicas
Slide 201
Itโs All About that Data ๐
Abstract
Commutative
merge
merge
โข Network agnostic โข Any number of replicas
Slide 202
Itโs All About that Data ๐
Abstract
Commutative
merge
merge
โข Network agnostic โข Any number of replicas
Associative
Slide 203
Itโs All About that Data ๐
Abstract
Associative
Commutative
merge
merge
โข Network agnostic โข Any number of replicas
merge
merge
Slide 204
Itโs All About that Data ๐
Abstract
Associative
Commutative
merge
merge
โข Network agnostic โข Any number of replicas
merge
merge
Slide 205
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Slide 206
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
(
) merge
Slide 207
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
(
) merge
Slide 208
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
(
) merge
Slide 209
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
(
) merge
merge
Slide 210
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
(
) merge
merge
Slide 211
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
(
) merge
merge
Slide 212
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
Slide 213
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
fill
Slide 214
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
fill
Slide 215
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
fill
Slide 216
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
fill
Slide 217
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
fill
fill
Slide 218
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
fill
fill
Slide 219
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
fill
fill
Slide 220
Itโs All About that Data ๐
Abstract
Associative
Commutative
( merge
merge
โข Network agnostic โข Any number of replicas
) merge
merge
Idempotent
(
) merge
merge
fill
fill
Slide 221
Itโs All About that Data ๐
PNCounter
Slide 222
Itโs All About that Data ๐
PNCounter
Slide 223
Itโs All About that Data ๐
PNCounter
Slide 224
Itโs All About that Data ๐
PNCounter
Slide 225
Itโs All About that Data ๐
PNCounter
Slide 226
Itโs All About that Data ๐
PNCounter
Slide 227
Itโs All About that Data ๐
PNCounter
Slide 228
Itโs All About that Data ๐
PNCounter
Slide 229
Itโs All About that Data ๐
PNCounter
Slide 230
Itโs All About that Data ๐
PNCounter
Slide 231
Itโs All About that Data ๐
PNCounter
Slide 232
Itโs All About that Data ๐
PNCounter
Slide 233
Itโs All About that Data ๐
PNCounter
Slide 234
Itโs All About that Data ๐
PNCounter
Slide 235
Itโs All About that Data ๐
Syncing Tables Seems Inefficient
Slide 236
Itโs All About that Data ๐
Syncing Tables Seems Inefficient user_id
username
company
start_date
inserted_at
1
expede
Fission
AUG-2019
FEB-2020
2
bmann
โ
OCT-2020
Slide 237
Itโs All About that Data ๐
Syncing Tables Seems Inefficient user_id
username
company
start_date
inserted_at
1
expede
Fission
AUG-2019
FEB-2020
2
bmann
Itโs All About that Data ๐
Syncing Tables Seems Inefficient
Whoโs clock? Meaningful or coincidence?
user_id
username
company
start_date
inserted_at
1
expede
Fission
AUG-2019
FEB-2020
2
bmann
Itโs All About that Data ๐
Syncing Tables Seems Inefficient
Whoโs clock? Meaningful or coincidence?
๐? user_id
username
company
start_date
inserted_at
1
expede
Fission
AUG-2019
FEB-2020
2
bmann
Itโs All About that Data ๐
Syncing Tables Seems Inefficient
Whoโs clock? Meaningful or coincidence?
๐? ๐ชต? user_id
username
company
start_date
inserted_at
1
expede
Fission
AUG-2019
FEB-2020
2
bmann
Itโs All About that Data ๐
Syncing Tables Seems Inefficient
Whoโs clock? Meaningful or coincidence?
๐? ๐ชต? ๐ค? user_id
username
company
start_date
inserted_at
1
expede
Fission
AUG-2019
FEB-2020
2
bmann
Itโs All About that Data ๐
Relationships XYZ
Name: @expede
From JAN-2000
๐ Work: Fission
From AUG-2019
KEB
๐ Red Switches
Switches: Blue
From JAN-2020
From FEB-2020
Wireless
Slide 243
Itโs All About that Data ๐
Relationships XYZ
Name: @expede
From JAN-2000
๐ Work: Fission
From AUG-2019
KEB
๐ Red Switches
Switches: Blue
From JAN-2020
From FEB-2020
Wireless
Slide 244
Itโs All About that Data ๐
A Sequel to SQL: Nonlinear DBs XYZ
Name: @expede
From JAN-2000
ABC
Name: @bmann
From DEC-1999
KEB
Type: Wireless
Always
๐
XYZ
Work: Fission
From AUG-2019
๐
KEB
Switches: Red
From JAN-2020
KEB
Owner:XYZ
From JAN-2020
KEB
Switches: Blue
From FEB-2020
Slide 245
Itโs All About that Data ๐
A Sequel to SQL: Nonlinear DBs XYZ
Name: @expede
From JAN-2000
ABC
Name: @bmann
From DEC-1999
KEB
Type: Wireless
Always
๐
XYZ
Work: Fission
From AUG-2019
๐
KEB
Switches: Red
From JAN-2020
KEB
Owner:XYZ
From JAN-2020
KEB
Switches: Blue
From FEB-2020
Slide 246
Itโs All About that Data ๐
A Sequel to SQL: Nonlinear DBs XYZ
Name: @expede
From JAN-2000
ABC
Name: @bmann
From DEC-1999
KEB
Type: Wireless
Always
๐
XYZ
Work: Fission
From AUG-2019
๐
KEB
Switches: Red
From JAN-2020
KEB
Owner:XYZ
From JAN-2020
KEB
Switches: Blue
From FEB-2020
Slide 247
Itโs All About that Data ๐
A Sequel to SQL: Nonlinear DBs XYZ
Name: @expede
From JAN-2000
ABC
Name: @bmann
From DEC-1999
KEB
Type: Wireless
Always
๐
XYZ
Work: Fission
From AUG-2019
๐
KEB
Switches: Red
From JAN-2020
KEB
Owner:XYZ
From JAN-2020
KEB
Switches: Blue
From FEB-2020
Slide 248
Itโs All About that Data ๐
A Sequel to SQL: Nonlinear DBs XYZ
Name: @expede
From JAN-2000
ABC
Name: @bmann
From DEC-1999
KEB
Type: Wireless
Always
๐
XYZ
Work: Fission
From AUG-2019
๐
KEB
Switches: Red
From JAN-2020
KEB
Owner:XYZ
From JAN-2020
KEB
Switches: Blue
From FEB-2020
๐ข
Slide 249
Itโs All About that Data ๐
A Sequel to SQL: Nonlinear DBs XYZ
Name: @expede
From JAN-2000
ABC
Name: @bmann
From DEC-1999
KEB
Type: Wireless
Always
๐
XYZ
Work: Fission
From AUG-2019
๐
KEB
Switches: Red
From JAN-2020
KEB
Owner:XYZ
From JAN-2020
KEB
Switches: Blue
From FEB-2020
๐ฉ๐๐ฑ
๐ข
Slide 250
Itโs All About that Data ๐
A Sequel to SQL: Nonlinear DBs XYZ
Name: @expede
From JAN-2000
ABC
Name: @bmann
From DEC-1999
KEB
Type: Wireless
Always
๐
XYZ
Work: Fission
From AUG-2019
๐
KEB
Switches: Red
From JAN-2020
๐จ๐ณ๐ป
KEB
Owner:XYZ
From JAN-2020
๐ข
KEB
Switches: Blue
From FEB-2020
๐ฉ๐๐ฑ
Slide 251
Itโs All About that Data ๐
A Sequel to SQL: Nonlinear DBs โกโฌ
XYZ
Name: @expede
From JAN-2000
ABC
Name: @bmann
From DEC-1999
KEB
Type: Wireless
Always
๐
XYZ
Work: Fission
From AUG-2019
๐
KEB
Switches: Red
From JAN-2020
๐จ๐ณ๐ป
KEB
Owner:XYZ
From JAN-2020
๐ข
KEB
Switches: Blue
From FEB-2020
๐ฉ๐๐ฑ
Slide 252
Collaboration & Memoization & E ects at Scale (oh my!)
ff
Universal Compute ๐
Slide 253
Universal Compute ๐
โWith a Little Scale From My Friendsโ
Slide 254
Universal Compute ๐
Throughput
โWith a Little Scale From My Friendsโ
Parallelization
Slide 255
Universal Compute ๐
โWith a Little Scale From My Friendsโ
Throughput
Ideal (Linear)
Parallelization
Slide 256
Universal Compute ๐
โWith a Little Scale From My Friendsโ Ideal (Linear)
Throughput
Amdahlโs Law
Parallelization
Slide 257
Universal Compute ๐
โWith a Little Scale From My Friendsโ Ideal (Linear)
Throughput
Amdahlโs Law
Universal Scaling Law
Parallelization
Slide 258
Universal Compute ๐
โWith a Little Scale From My Friendsโ Ideal (Linear)
Throughput
Amdahlโs Law
Incoherence, Data Contention
Universal Scaling Law
Parallelization
Slide 259
Universal Compute ๐
โWith a Little Scale From My Friendsโ
Throughput
Global Adaptive Memoization
๐คฏ
Ideal (Linear)
Amdahlโs Law
Incoherence, Data Contention
Universal Scaling Law
Parallelization
Slide 260
Universal Compute ๐
The Compute Commons ๐๐ฌ
Slide 261
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
Slide 262
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
Slide 263
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
Slide 264
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
Slide 265
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
Slide 266
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
Slide 267
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
Slide 268
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
Slide 269
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
Slide 270
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
๐
Slide 271
Universal Compute ๐
The Compute Commons ๐๐ฌ
Side E ect Stream
Pure E ect Stream
Pure Function Stream
ff
ff
Base Event Stream
tโ
๐
Slide 272
Universal Compute ๐
GenEffect ๐
Slide 273
Universal Compute ๐
GenEffect ๐
Slide 274
Universal Compute ๐
GenEffect ๐
Slide 275
Universal Compute ๐
GenEffect ๐
Slide 276
Universal Compute ๐
GenEffect ๐
Slide 277
Where Do We Go From Here?
๐ญ
Slide 278
Where Do We Go From Here?
New Environment Then ๐
Now ๐
Need
Convenient ๐
Critical ๐จ
Location
Data Centre ๐ข
Powerful Clients (M1, IoT) โ๐๐
๐ฅ
๐ฑ
Bandwidth ๐
Latency โฒ
๐บ๐ธ๐ช๐บ
๐๐๐ โฆ ๐
Access Bottleneck Market
Slide 279
Where Do We Go From Here?
Relativistic Programming ๐ซ
Slide 280
Where Do We Go From Here?
A Better Future on the Edge
Slide 281
Where Do We Go From Here?
A Better Future on the Edge Embrace the distributed nature of the internet:
Slide 282
Where Do We Go From Here?
A Better Future on the Edge Embrace the distributed nature of the internet: 1. Only replicate what you need to
Slide 283
Where Do We Go From Here?
A Better Future on the Edge Embrace the distributed nature of the internet: 1. Only replicate what you need to 2. Remember that data propagates relativistically
Slide 284
Where Do We Go From Here?
A Better Future on the Edge Embrace the distributed nature of the internet: 1. Only replicate what you need to 2. Remember that data propagates relativistically 3. Free yourself from intrinsic time
Itโs All About that Data ๐
Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}}
โข Single-source server/client โข DNS: hostname โ IP address โข PIDs: number โ address โข Focused: physical network
ff
โข Referential opacity (same PID, di erent data)
Slide 290
Itโs All About that Data ๐
Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}}
โข Single-source server/client โข DNS: hostname โ IP address โข PIDs: number โ address
P H Y S I C A L L O C AT I O N ๐บ
โข Focused: physical network
ff
โข Referential opacity (same PID, di erent data)
Slide 291
Itโs All About that Data ๐
Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}}
โข Single-source server/client โข DNS: hostname โ IP address โข PIDs: number โ address
V I R T UA L A D D R E S S ๐ฌ P H Y S I C A L L O C AT I O N ๐บ
โข Focused: physical network
ff
โข Referential opacity (same PID, di erent data)
Slide 292
Itโs All About that Data ๐
Consistent Keys
%{hash(content) => content}
โข Above virtual address โข Focused: data itself โข Same for everyone & everywhere โข Perfect for caching โข Immutable data++ โข Consistent pointers โ consistent data
Slide 293
Itโs All About that Data ๐
Consistent Keys
%{hash(content) => content}
โข Above virtual address โข Focused: data itself
CONTENT ID โ
โข Same for everyone & everywhere โข Perfect for caching โข Immutable data++ โข Consistent pointers โ consistent data
P H Y S I C A L L O C AT I O N ๐บ
Slide 294
Itโs All About that Data ๐
Consistent Keys
%{hash(content) => content}
โข Above virtual address โข Focused: data itself โข Same for everyone & everywhere โข Perfect for caching โข Immutable data++ โข Consistent pointers โ consistent data
CONTENT ID โ V I R T UA L A D D R E S S ๐ฌ P H Y S I C A L L O C AT I O N ๐บ
Slide 295
Itโs All About that Data ๐
Hash-Based Relationships
Slide 296
Itโs All About that Data ๐
Hash-Based Relationships
{
}
CID ~ Data PID Qm123456โฆ: { data: nil, links: [ {name: โcompanyโ, hash: Qmabcโฆ} {name: โindustryโ, hash: Qmzyxโฆ} ] }
Slide 297
Itโs All About that Data ๐
Hash-Based Relationships
{
}
{ Qm123456โฆ: { data: nil, links: [ {name: โcompanyโ, hash: Qmabcโฆ} {name: โindustryโ, hash: Qmzyxโฆ} ] }
CID ~ Data PID
Qmabcdefโฆ: { data: โFissionโ, links: [ {name: โcityโ, hash: Qm1gbโฆ}, {name: โaboutโ, hash: Qm0eNโฆ} ] } }
Itโs All About that Data ๐
Content IDs Are Easy
Slide 300
Decentralized Systems ๐
Different Clients ~ Schema Drift ๐
Source: Project Cambria, Ink & Switch https://www.inkandswitch.com/cambria.html
Slide 301
High Volume ๐ข
Evolving Toolbox Cloud Geographic Sync Serverless Commons Networks P2P
Local First
ff
O line
Slide 302
On the Edge ๐ง
Why Elixir?
โข Data-oriented, immutable โข Code-as-data ๐ data-as-code โข Shared-nothing architectures โข Easy concurrency, distributed systems โข Actor model โ OCAP โข Build up complexity from simple parts
Slide 303
Slide 304
We have a system that applies cutting edge CS research to tackle day-to-day problems in the applications we all write. Phoenix Presence - has no single point of failure - has no single source of truth -[โฆ] - self heals ~ Chris McCord, โWhat Makes Phoenix Presence Specialโ
Slide 305
Low Latency ๐
Earth is Just Too Big! ๐
โข Ideal 40ms one-way โข SF โก Austin, Vancouver, London, Tokyo โข SF โ Cape Town, Sidney, Rio de Janeiro
Credit: Keenan Crane http://www.cs.cmu.edu/~kmcrane/Projects/ModelRepository/
Slide 306
Low Latency ๐
Earth is Just Too Big! ๐
โข Ideal 40ms one-way โข SF โก Austin, Vancouver, London, Tokyo โข SF โ Cape Town, Sidney, Rio de Janeiro
Credit: Keenan Crane http://www.cs.cmu.edu/~kmcrane/Projects/ModelRepository/
Slide 307
What does this all mean?
Consequence ๐
Slide 308
Consequence ๐ธ
Primary Progression
Slide 309
Consequence ๐ธ
Primary Progression 1. Embrace the distributed nature of the internet ๐ค
Slide 310
Consequence ๐ธ
Primary Progression 1. Embrace the distributed nature of the internet ๐ค 2. Universal IDs (without coordination)
Slide 311
Consequence ๐ธ
Primary Progression 1. Embrace the distributed nature of the internet ๐ค 2. Universal IDs (without coordination) 3. Direct access control
Slide 312
Consequence ๐ธ
Primary Progression 1. Embrace the distributed nature of the internet ๐ค 2. Universal IDs (without coordination) 3. Direct access control 4. Cache friendliness
Slide 313
Consequence ๐ธ
Primary Progression 1. Embrace the distributed nature of the internet ๐ค 2. Universal IDs (without coordination) 3. Direct access control 4. Cache friendliness 5. Relativistic databases
Slide 314
Consequence ๐ธ
Primary Progression 1. Embrace the distributed nature of the internet ๐ค 2. Universal IDs (without coordination) 3. Direct access control 4. Cache friendliness 5. Relativistic databases 6. Interoperable formats (across time & team)
Slide 315
Consequence ๐ธ
Primary Progression 1. Embrace the distributed nature of the internet ๐ค 2. Universal IDs (without coordination) 3. Direct access control 4. Cache friendliness 5. Relativistic databases 6. Interoperable formats (across time & team)
ffi
7. E cient replication
Slide 316
Itโs All About that Data ๐
Minimal CRDT From Smaller Blocks
Slide 317
Itโs All About that Data ๐
Minimal CRDT From Smaller Blocks
Slide 318
Itโs All About that Data ๐
Minimal CRDT From Smaller Blocks
Slide 319
Itโs All About that Data ๐
Minimal CRDT From Smaller Blocks
Slide 320
Itโs All About that Data ๐
Minimal CRDT From Smaller Blocks
Sibling / Concurrent
Slide 321
Getting Ready ๐ฑ
Data > Compute
โข Pure data-focus frees you! โข Clarify โrealโ dependencies on data โข Direct auth, OCAP