Living On the Edge

A presentation at CodeBEAM BR by Brooklyn Zelenka

[…] by 2025, 75% of data will be processed outside the traditional data centre or cloud ~ IBM (paraphrasing a Gartner study)

Brooklyn Zelenka @expede • CTO at Fission • https://fission.codes • Infra & browser SDK for edge apps • PLT, distributed systems • Specs: DIF, ETH Core • Meetups: Vancouver FP, Code & Co ee YVR ff • Libs: Witchcraft, Exceptional, Rescue, &c

WebNative 🚀 Meta 🔮 • R&D from Fission & others • Future looking / an emerging area • Interesting tech, very exciting • …but not all problems solved today • Some advantages to flexible tech even before the network changes • Universal Hostless Substrate (2019)

WebNative 🚀 Fission R&D • Local first • Edge only • No servers • Fully distributed • Encrypted at Rest, E2EE • User owned data @FISSIONCodes

WebNative 🚀 Overview Part I: Motivation Part II: On the Edge How we got here Why BEAM What changed? Primer All About Data A Few Techniques

Motivation 🎭 Scaling Up 💁 💁 💁 🖥 🖥 🖥 🐙 ⚙ ⚙ ⚙ 🔐 🔐 🔐 🗃 🗃 🗃

Motivation 🎭 Scaling Up 💁 💁 💁 🖥 🖥 🖥 ⚙ 🗃 “The Cloud” 🔐 🐙☁☁☁☁☁☁ ⚙ 🗃 🔐 ⚙ 🗃 🔐

Motivation 🎭 Abstracting 💁 💁 💁 🖥 🖥 🖥 ⚙ 🗃 “Serverless” 🔐 🐙 ⚙ 🗃 λλλλλλλλλλλλλ 🔐 ⚙ 🗃 🔐

Motivation 🎭 Natural Consequences 🍃 • Server-focus • More stack to learn • DevOps, Docker, k8s

Motivation 🎭 Natural Consequences 🍃 • Server-focus • More stack to learn • DevOps, Docker, k8s • Single source of truth • i.e. “the database”

Motivation 🎭 Natural Consequences 🍃 • Server-focus • More stack to learn • DevOps, Docker, k8s • Single source of truth • i.e. “the database” • Client concerned with data sync

Motivation 🎭 What Even is a “Server”? 🧐
Auth gatekeeper (because multi-tenant data) 2. Resource availability 3. Out-of-band compute (e.g. batch tasks, cron, OLAP)

Motivation 🎭 Network Topology 🧠 ⚙ 👨🎤 👩🌾 Centralized 🧑🎨

Motivation 🎭 Network Topology 🧠 💾 ⚙ ⚙ 👨🎤 👩🌾 Centralized 🤖 🛠 🐙 🧑🎨 🧑🎨 👩🌾 👨🎤 Hub (e.g. gateway or load balanced)

Motivation 🎭 Network Topology 🧠 💾 ⚙ ⚙ 👨🎤 👩🌾 Centralized ⚙ 🤖 🛠 🐙 🧑🎨 🧑🎨 👩🌾 🧑🎨 👨🎤 Hub (e.g. gateway or load balanced) 🧑🎨 🤖 💾 ⚙ 👨🎤 👩🌾 Hierarchical or pipelined

A Challenger Emerges A New Environment 🛰

New Environment 🛰 New Assumptions • Powerful client devices (e.g. M1 chips, smartphones, IoT) • Latency is the bottleneck • Mobile (i.e. smartphone) use only growing • Lose connection, drop when switching towers • Do more with the existing physical network • Not unlike how Moore’s Law lead to more parallelism

New Environment 🛰 New Biz Who Dis? • Paradigm shift means new opportunities

New Environment 🛰 New Biz Who Dis? • Paradigm shift means new opportunities • 5G networks & Starlink • Put an edge PoP right on the base station • Low-latency compute across the street

Low Latency 🐇 Latency is a Physical Barrier 🚧 • Speed of light / speed of causality • <40ms = edge dominates • 8ms is ideal • Ultra Reliable Low Latency (URLLC)

Low Latency 🐇 Latency is a Physical Barrier 🚧 • Speed of light / speed of causality • <40ms = edge dominates • 8ms is ideal • Ultra Reliable Low Latency (URLLC) f Source: Ericsson http://cscn2017.ieee-cscn.org/ iles/2017/08/Janne_Peisa_Ericsson_CSCN2017.pdf

Low Latency 🐇 Spherical Cow Assumption 🐮 • No compute, straight line, in a vacuum, guaranteed delivery, etc • 40ms • São Paulo ➡ NYC, Vancouver, Stockholm • São Paulo ❌ Sidney, Tokyo, Seoul Credit: Keenan Crane http://www.cs.cmu.edu/~kmcrane/Projects/ModelRepository/

Low Latency 🐇 What 8ms Looks Like Montevideo ➡ Rio de Janeiro Ideal Vacuum

Low Latency 🐇 What 8ms Looks Like Montevideo ➡ Rio de Janeiro Ideal Vacuum Brasilia 🔁 Salvador Ideal Vacuum

Low Latency 🐇 What 8ms Looks Like Montevideo ➡ Rio de Janeiro Ideal Vacuum Brasilia 🔁 Salvador Ideal Vacuum Brasilia 🔁 Barreiras Ideal Fiber

Low Latency 🐇 Light Cone & Relativistic Ordering

Low Latency 🐇 Light Cone & Relativistic Ordering Source: Duesentrieb via Wikimedia Commons

High Volume 🌊 Unprecedented Volume 🦖 • We have high scale NOW? Only more devices & usage in the future! • Sensors everywhere: IoT devices, continuous health data • Geospatial data (e.g. autonomous vehicles, XR)

High Volume 🌊 Feedback Cycle Source: Microsoft • Remote surgery • Extended reality • Location transparency • Competitive cloud gaming Source: YouTube, South China Morning Post • Realtime manufacturing • Continuous ML training Source: Google & Bungie

Sensor data explosion will kill the cloud. Sensors will produce massive amounts of data, but the existing infrastructure will not be able to handle the volumes or the rates […] We are absolutely going to return to a peer-to-peer computing model […] not unlike the distributed computing model We are going to move to a world of data-centric programming. ~ a16z, “The End of Cloud Computing”

High Volume 🌊 Edge Absorbs Cloud (and MEC)

High Volume 🌊 Edge Absorbs Cloud (and MEC) 🛰 Local 🤳 First 🗼 💾⚙ 🛰 🛰 🏢 ☁ 💾⚙ 💾⚙ ⚙ 💾 ⚙ 💾 ⚙ 💾 💾⚙ ⚙

High Volume 🌊 Edge Absorbs Cloud (and MEC) 🛰 Local 🤳 First Realtime, Storage, Caching, OLTP 💾⚙ 🗼 🛰 🛰 🏢 ☁ 💾⚙ 💾⚙ ⚙ 💾 ⚙ 💾 ⚙ 💾 💾⚙ ⚙

High Volume 🌊 Edge Absorbs Cloud (and MEC) 🛰 Local 🤳 First Realtime, Storage, Caching, OLTP 💾⚙ 🗼 🛰 Relay, Replication, Consistency, Tasks 🏢 💾⚙ 🛰 ☁ 💾⚙ ⚙ 💾 ⚙ 💾 ⚙ 💾 💾⚙ ⚙

High Volume 🌊 Edge Absorbs Cloud (and MEC) 🛰 Local 🤳 First Realtime, Storage, Caching, OLTP 💾⚙ 🗼 🛰 Relay, Replication, Consistency, Tasks 🏢 💾⚙ 🛰 ☁ Aggregation, Batching, Training, ⚙ 💾 ⚙ 💾 OLAP 💾⚙ ⚙ 💾 💾⚙ ⚙

Consequence 🛸 New Assumptions, New Approach

Consequence 🛸 New Assumptions, New Approach • New features naturally fall out of the architecture • Recognize that we’re increasingly connected/networked • Local-first means network e cient (in the normal case) ffi • Data can run anywhere = commons networks

Consequence 🛸 Tackling the Fallacies Latency is zero Bandwidth is infinite Transport cost is zero The network is secure There is one administrator The network is reliable The network is homogeneous Topology doesn’t change

Consequence 🛸 Tackling the Fallacies Latency is zero Treat latency directly (speed of causality) Treat (order of causality / relativistic) Bandwidth is infinite Apps continue to work with zero bandwidth Only push when & what needed Transport cost is zero Minimize network use The network is secure Assume that the pipes are broken Direct access control There is one administrator Fine grained, delegate capabilities (OCAP) The network is reliable Time, delivery, & order independence The network is homogeneous Device agnostic Topology doesn’t change atomic unit is the edge device (same like the atomic unit is the actor)

Consequence 🛸 Giving Up Topological Control

Consequence 🛸 Data, Data, Data 💾 • Only UI & data are essential

Consequence 🛸 Data, Data, Data 💾 • Only UI & data are essential • New primitives • Consistency (CRDTs, STM, Distributed Datalog) • State transfer ➡ state synchronization ➡ state views

On the Edge 🧗 Why Functional Programming • Data-oriented • Pure functions on data is just data • Shared nothing architectures • Immutability, easy concurrency • Manage complexity by being declarative • What > how • Data > process

On the Edge 🧗 Why the BEAM Specifically • Low conceptual distance from actor model to OCAP • Community experience with distributed systems • Used to building up complexity from simple parts • We’re already using a bunch of this! • e.g. Phoenix Presence 👉 👉 👉

What’s special about Phoenix’s implementation is we have a system that applies cutting edge CS research to tackle day-to-day problems in the applications we all write. Phoenix Presence - has no single point of failure - has no single source of truth - relies entirely on the standard library with no operational dependencies - self heals ~ Chris McCord, “What Makes Phoenix Presence Special”

What if we turn Phoenix Live View Upside Down? 🔁

On the Edge 🧗 Phoenix LiveView Users 👨🏫👩🏭🧑⚕👷 Client 🖥 WSS / REST / GraphQL ↕ Controller Logic ⚙ Data Store 🗃 DevOps 📤 Developer 👩💻

On the Edge 🧗 Upside Down ⚙ 🖥 🗃💾 💾💾💾 🖥 💾💾

Data dominates. If you’ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming. Rob Pike, 5 Rules of Programming

It’s All About the Data 📊 f Problems! Property Consequence Run anywhere No process in charge of access control Casual islands Inconsistent views of data (or downtime) Unstable topology No consistent connections Local irst In accessible, no replicas

It’s All About the Data 📊 CAP ➡ PACELC 📦🦌

It’s All About the Data 📊 CAP ➡ PACELC 📦🦌 • If network partition (P) • Choose between: • Availability (A) ✅ Local-first & uptime • Consistency (C)

It’s All About the Data 📊 CAP ➡ PACELC 📦🦌 C • If network partition (P) • Choose between: • Availability (A) ✅ Local-first & uptime • Consistency (C) A

It’s All About the Data 📊 CAP ➡ PACELC 📦🦌 C • If network partition (P) • Choose between: P • Availability (A) ✅ Local-first & uptime • Consistency (C) A

It’s All About the Data 📊 CAP ➡ PACELC 📦🦌 C • If network partition (P) • Choose between: P • Availability (A) ✅ Local-first & uptime • Consistency (C) • Else (E) when running normally: • Choose between: • Latency (L) ✅ • Consistency (C) A

It’s All About the Data 📊 CAP ➡ PACELC 📦🦌 C • If network partition (P) • Choose between: P • Availability (A) ✅ Local-first & uptime E • Consistency (C) • Else (E) when running normally: • Choose between: • Latency (L) ✅ • Consistency (C) A L

It’s All About the Data 📊 Mutable Content • Predominantly single-source (per file) server/client • %{node_id => %{path => content}} • DNS maps names to IP addresses • PIDs associate processes with numbers • e.g. send(:example@42.123.45.6, :ping) • Focused on the physical network • Referential opacity ff • Calling same PID often will return di erent data

It’s All About the Data 📊 Consistent Keys • A layer of abstraction above location • %{hash(content) => content} • Hash AKA “content identifier” or CID • Special “universal” relationship to content • Focused on the data • Stored anywhere, same ID • E cient caching • Immutable data++ ffi • Not just consistent pointers; consistent data V I R T UA L A D D R E S S P H Y S I C A L L O C AT I O N

It’s All About the Data 📊 Consistent Keys • A layer of abstraction above location • %{hash(content) => content} CONTENT ID • Hash AKA “content identifier” or CID • Special “universal” relationship to content • Focused on the data • Stored anywhere, same ID • E cient caching • Immutable data++ ffi • Not just consistent pointers; consistent data V I R T UA L A D D R E S S P H Y S I C A L L O C AT I O N

It’s All About the Data 📊 Hash-Based Relationships

It’s All About the Data 📊 Hash-Based Relationships (CID ~ Data PID) { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabcdef…} {name: “license”, hash: Qmzyxwvu…} ] } }

It’s All About the Data 📊 Hash-Based Relationships (CID ~ Data PID) { { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabcdef…} {name: “license”, hash: Qmzyxwvu…} ] } } Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb5sn…}, {name: “about”, hash: Qmzyxwvu…} ] } }

It’s All About the Data 📊 Content IDs Are Easy [no network version]

It’s All About the Data 📊 Partial Dependencies

It’s All About the Data 📊 Partial Dependencies t

It’s All About the Data 📊 This all works…

It’s All About the Data 📊 Out of Order Delivery 💌 ✉ 📧 ⚙ 📧 💌 💌

It’s All About the Data 📊 Commutative Monoid (AKA Minimal CRDT)

It’s All About the Data 📊 Commutative Monoid (AKA Minimal CRDT) Sibling / Concurrent

Decentralized Systems 🌈 Scale Curve Adapted from http://www.perfdynamics.com/Manifesto/USLscalability.html

Decentralized Systems 🌈 Scale Curve Linear Ideal Adapted from http://www.perfdynamics.com/Manifesto/USLscalability.html

Decentralized Systems 🌈 Scale Curve Linear Ideal Amdahl’s Law Adapted from http://www.perfdynamics.com/Manifesto/USLscalability.html

Decentralized Systems 🌈 Scale Curve Linear Ideal Amdahl’s Law Data Contention Adapted from http://www.perfdynamics.com/Manifesto/USLscalability.html Universal Scaling Law

Decentralized Systems 🌈 Scale Curve 🤯 Linear Ideal Shared Adaptive Memoization (“Theoretical) Amdahl’s Law Data Contention Adapted from http://www.perfdynamics.com/Manifesto/USLscalability.html Universal Scaling Law

Decentralized Systems 🌈 Conflict Free Effects 🕊🧱 Side Effect Stream Pure Effect Stream Pure Function Stream Base Event Stream

Decentralized Systems 🌈 Conflict Free Effects 🕊🧱 Side Effect Stream Pure Effect Stream Pure Function Stream Base Event Stream t

Decentralized Systems 🌈 Different Clients ~ Schema Drift Source: Project Cambria, Ink & Switch https://www.inkandswitch.com/cambria.html

Secure Decentralized Data Access Fixing the Leaky Pipes 🚿

Fixing the Leaky Pipes 🚿 Object Capability Model (OCAP)

Fixing the Leaky Pipes 🚿 Object Capability Model (OCAP) • ACL is “reactive auth” / OCAP is “proactive auth”

Fixing the Leaky Pipes 🚿 Object Capability Model (OCAP) • ACL is “reactive auth” / OCAP is “proactive auth” • OCAP contains all the info about access

Fixing the Leaky Pipes 🚿 Object Capability Model (OCAP) • ACL is “reactive auth” / OCAP is “proactive auth” • OCAP contains all the info about access • Generally some reference, proof, or key • …not unlike having a PID • Rights to anything directly created (parenthood) • The right to delegate subset of access to another (introduction)

Fixing the Leaky Pipes 🚿 3rd-Party Subdelegation & Attenuation

Fixing the Leaky Pipes 🚿 3rd-Party Subdelegation & Attenuation 🖥 🔟 ⚙ 2⃣ 🛠

Fixing the Leaky Pipes 🚿 Direct Access Control •Advantages •Proactive •Proactive •Works o ine •Revocation •Attenuation •Give up (more) access stats •Easy to understand rules •User control (GDPR, CCPA) •Interoperable ffl •Challenges

Fixing the Leaky Pipes 🚿 Hierarchal Read Access

Fixing the Leaky Pipes 🚿 Cryptree 🎄 JSON Binary Encrypted Node 🔒 AES256 + 🔑 Virtual Node = Index 🔑 🔑 Metadata 🔑

Fixing the Leaky Pipes 🚿 Cryptree Sketch ✍

Fixing the Leaky Pipes 🚿 Cryptree Sketch ✍ Local stateful, remote stateless

How to Do O ine & Distributed Auth ffl Universal Auth & ID 🗝

Universal Auth & ID 🗝 Universal IDs • W3C, DIF, Microsoft

Universal Auth & ID 🗝 Universal IDs • W3C, DIF, Microsoft • Based on public-key cryptography

Universal Auth & ID 🗝 Universal IDs • W3C, DIF, Microsoft • Based on public-key cryptography • Truly “universal” user IDs

Universal Auth & ID 🗝 Universal IDs • W3C, DIF, Microsoft • Based on public-key cryptography • Truly “universal” user IDs • Agnostic about backing

Universal Auth & ID 🗝 Universal IDs • W3C, DIF, Microsoft • Based on public-key cryptography • Truly “universal” user IDs • Agnostic about backing • For users, devices, and more

Universal Auth & ID 🗝 OAuth vs UCAN Sequence

Universal Auth & ID 🗝 OAuth vs UCAN Sequence (Verifiable & user originated)

Universal Auth & ID 🗝 🛂 External OIDC Server 🤖 Service A 👽 Service B 💁 User

Universal Auth & ID 🗝 🛂 External OIDC Server 🤖 Service A 👽 💁 Service B User ff UCAN with 💁 ID / email Describes o er for 🤖

Universal Auth & ID 🗝 🛂 External OIDC Server 🤖 Service A 👽 💁 Service B User UCAN with 💁 ID / email Describes o er for 🤖 ff OIDC Login

Universal Auth & ID 🗝 🛂 External OIDC Server 🤖 Service A 👽 💁 Service B User UCAN with 💁 ID / email Describes o er for 🤖 OIDC Login ff OIDC Token

Universal Auth & ID 🗝 🛂 External OIDC Server 🤖 Service A 👽 💁 Service B User UCAN with 💁 ID / email Describes o er for 🤖 OIDC Login OIDC Token ff ff O er for 🤖+💁 Secured with signature 👽 and HMAC 💁🛂

Universal Auth & ID 🗝 🛂 External OIDC Server 🤖 Service A 👽 💁 Service B User UCAN with 💁 ID / email Describes o er for 🤖 OIDC Login OIDC Token O er for 🤖+💁 Secured with signature 👽 and HMAC 💁🛂 ff ff 💁’s OIDC token?

Instead of immediately asking “which database would be best to hold presences?”, we could ask “how can we best replicate data in a distributed system without the user having to worry about it?”. The platforms you build on top of drive the design decisions you make in your products. With Elixir, you are empowered to tackle problems that in other platforms would feel impossible to solve without tradeoffs with heavy dependencies. ~ Chris McCord, What Makes Phoenix Presence Special

Getting Ready 🍱 Data > Compute • Focus on data & structure • Clarify “real” dependencies on data • Start thinking about the properties in your code • Adopt OCAP • Use abstraction for declarative interfaces

🇧🇷 Thank You, CodeBEAM BR 🎉 brooklyn@fission.codes https://fission.codes github.com/expede @expede

Brooklyn Zelenka
@expede

1 / 211

The BEAM is exceptional for distributed systems, concurrency, and soft-realtime workloads. This makes it a wonderful fit for the cloud computing era of the past decade, but what’s next? Network assumptions are changing for the first time in 20+ years: new technologies like StarLink and 5G are leading to a shift in edge and low-latency computing. To handle the new causal limits and bumping into the speed of light, we will need to be able to grapple with multiple sources of truth, leverage commons networks & dynamic swarms, and allow for the movement of compute to data. In this keynote, Brooklyn will share some R&D being done at Fission and elsewhere for edge- and local-first computing, a few techniques that you can apply today for distributed computing, and vision for how Elixir can position itself as the premier ecosystem for this developing era.

Living On the Edge

Link for this presentation:

HTML code for embedding:

Share on social media: