The Mess We’re (Still) In 🌈🧘 Unbounded Parallelism, True Names, & Keeping CALM 🎱✨

The Mess We’re (Still) In

The Mess We’re (Still) In A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable — Leslie Lamport

The Mess We’re (Still) In

The Mess We’re (Still) In Meanwhile…

The Mess We’re (Still) In Meanwhile… …who? 🤨

The Mess We’re (Still) In

The Mess We’re (Still) In Keep Calm and Dig In 🕳⛏

The Mess We’re (Still) In

@FissionCodes Brooklyn Zelenka Quinn Wilton @expede @wilton_quinn

@FissionCodes Brooklyn Zelenka Quinn Wilton @expede @wilton_quinn • CTO at Fission • Applied Researcher at Fission • Distributed auth, data, compute, and discovery • Building a planetary scale database for local-first apps • Author of Witchcraft, Algae, Exceptional, &c • Contributed to Lumen, Gleam, Burrito, Witchcraft, &c

So Many Problems 🪨🪐 🌌

Problems 🪨🪐🌌 High Level

Massive State Space

Problems 🪨🪐🌌 High Level

Massive State Space 2. Place Oriented Programming

Problems 🪨🪐🌌 High Level

Massive State Space 2. Place Oriented Programming 3. Dependencies, Limited APIs, Inconsistency

Problems 🪨🪐🌌 State Space is Big

Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 (2 ) ≈ 10 48

Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 🌍

atoms on Earth

Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 🌍

atoms on Earth

Six 32-bit Numbers 32 6 (2 ) ≈ 10 57

Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 🌍 🌞 🪐 🌍 🌒

atoms on Earth

atoms in solar system

Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 🌍 🌞 🪐 🌍 🌒

atoms on Earth

atoms in solar system

Single Receiver (2 32+1 6 ) × 6! ≈ 10 62

Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 Single Receiver 32+1 6 (2 ) × 6! ≈ 10 🌍 🌞 🌌 🪐 🌍 🌒

atoms on Earth

atoms in solar system

62

atoms in Milky Way

Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 32+1 6 (2 🌍 🌞 🌌 🌍 🌒

atoms in solar system

62 ) × 6! ≈ 10 🪐

atoms on Earth

Two Receivers Single Receiver

atoms in Milky Way

((2 32+1 6 2 ) × 6!) ≈ 10 124

Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 Two Receivers Single Receiver 32+1 6 (2 ) × 6! ≈ 10 62 32+1 6 ((2 2 124 ) × 6!) ≈ 10 🌍 🌞 🌌 🤯 🪐 🌍 🌒

atoms on Earth

atoms in solar system

atoms in Milky Way

More than observable universe

Problems 🪨🪐🌌

Problems 🪨🪐🌌 Distributed systems introduce significant nondeterminism to our programs. Sources of non-determinism include unsynchronized parallelism, unreliable components, and networks with unpredictable delays. As a result, a distributed program can exhibit a large space of possible behaviors on a given input. — Hellerstein & Alvaro, Keeping CALM: When Distributed Consistency is Easy

Problems 🪨🪐🌌 The Great 73-Hour Roblox Outage of 2021 https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://www.theverge.com/2021/10/30/22754107/roblox-down-outage-chipotle-server-issues-status

Problems 🪨🪐🌌 The Great 73-Hour Roblox Outage of 2021 👀 https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://www.theverge.com/2021/10/30/22754107/roblox-down-outage-chipotle-server-issues-status

Problems 🪨🪐🌌 The Great 73-Hour Roblox Outage of 2021 🧨 👀 https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://www.theverge.com/2021/10/30/22754107/roblox-down-outage-chipotle-server-issues-status

Problems 🪨🪐🌌 And Yet…

Problems 🪨🪐🌌 And Yet… These metastable failures have caused widespread outages at large internet companies, lasting from minutes to hours. Paradoxically, the root cause of these failures is often features that improve the efficiency or reliability of the system. – Bronson et al, Metastable Failures in Distributed Systems

Problems 🪨🪐🌌 Metastable Mechanism

Problems 🪨🪐🌌 Metastable Mechanism ⚡ ⚖

Problems 🪨🪐🌌 Metastable Mechanism ⚡ 🛑 ⚖

Problems 🪨🪐🌌 Metastable Mechanism ⚡ 🛑 • Retries / let it crash • Work amplification ⚖ • General thrash 🫡

Problems 🪨🪐🌌 Places Fight Light 💫🤺

Problems 🪨🪐🌌 Places Fight Light 💫🤺 The limitation of local knowledge is the fundamental fact about the setting in which we work, and it is a very powerful limitation – Nancy Lynch, A Hundred Impossibility Proofs for Distributed Computing

Problems 🪨🪐🌌 Places Fight Light 💫🤺 Sending a “Direct” Message

Problems 🪨🪐🌌 Places Fight Light 💫🤺 Sending a “Direct” Message 7.2x ⏳🔋🛢🪠

Problems 🪨🪐🌌 Data Behind Walls 🏰

Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration Bob’s Photo Gallery 🖼 ⛔ 🔒 👩🎤 Alice’s Music Player 🎶 🔒 🧑🎤 Carol’s Videogame 👾 🔒

Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration

Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration 💿 💾 📀

Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration Bob’s Photo Gallery 🖼 💿 💾 📀

Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration Bob’s Photo Gallery 🖼 💿 💾 📀 Carol’s Videogame 👾

Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration Bob’s Photo Gallery 🖼 💿 💾 Alice’s Music Player 🎶 📀 Carol’s Videogame 👾

Problems 🪨🪐🌌 Data Behind Walls 🏰 Inconsistency

Problems 🪨🪐🌌 Data Behind Walls 🏰 Inconsistency • Even with FOSS!

Problems 🪨🪐🌌 Data Behind Walls 🏰 Inconsistency • Even with FOSS! • Annual migration to the latest hipster HTTP client • HTTPotion → HTTPoison → Hackney → Tesla → Finch → Req

Problems 🪨🪐🌌 Data Behind Walls 🏰

Problems 🪨🪐🌌 Data Behind Walls 🏰 If people in a few hundred years from now want to see what their ancestors wrote, what will they find, a mess of badly formatted crap?! — Joe Armstrong, Why Markdown Sucks

Cause & E ect ff Mental Framework 🏗🧠

Mental Framework 🏗🧠 Causal Islands 🏖🏝

Mental Framework 🏗🧠 Causal Islands 🏖🏝 “Causal Subjectivity”

Mental Framework 🏗🧠

Mental Framework 🏗🧠 What is the family of problems that can be consistently computed in a distributed fashion without coordination, and what problems lie outside that family? —Hellerstein & Alvaro, Keeping CALM: When Distributed Consistency is Easy

Mental Framework 🏗🧠 Gossiping Out of Order 🙊

Mental Framework 🏗🧠 Gossiping Out of Order 🙊 t→ 🎨

Mental Framework 🏗🧠 Gossiping Out of Order 🙊 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡

Mental Framework 🏗🧠 Gossiping Out of Order 🙊 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡 ❎

Mental Framework 🏗🧠 Monotone Functions

Mental Framework 🏗🧠 Monotone Functions 🍎 🍋 🍊

Mental Framework 🏗🧠 Monotone Functions 🍎 🍋 🍊 🍎 🍑 🍋 🍒 🍊 🍓 🍌 🍍

Mental Framework 🏗🧠 Monotone Functions 🍎 🍑 🍎 🍋 🍊 ⊆ 🍋 🍒 🍊 🍓 🍌 🍍

Mental Framework 🏗🧠 Monotone Functions get(:red) 🍎 🍑 🍎 🍋 🍊 🍎 ⊆ 🍋 🍒 🍊 🍓 🍌 🍍

Mental Framework 🏗🧠 Monotone Functions get(:red) 🍎 🍑 🍎 🍋 🍊 get(:red) ⊆ 🍋 🍒 🍊 🍓 🍌 🍍 🍎 🍎 🍒 🍓

It’s All About that Data 📊 PNCounter

Grappling With Reality Towards a Solution ⚗✨

Towards a Solution ⚗✨ Evolving Toolbox Serverless Networked Data Cloud Commons Networks Local-First Blockchain O ffl P2P ine

Towards a Solution ⚗✨ Evolving Toolbox Serverless Networked Data Radical shifts how we think about auth, locality of reference, ownership, and reliability Cloud Commons Networks Local-First Blockchain O ffl P2P ine

Towards a Solution ⚗✨ Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}} • Single-source server/client • DNS: hostname → IP address • PIDs: number → address • Focused: physical network ff • Referential opacity (same PID, di erent data)

Towards a Solution ⚗✨ Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}} • Single-source server/client • DNS: hostname → IP address • PIDs: number → address P H Y S I C A L L O C AT I O N 🗺 • Focused: physical network ff • Referential opacity (same PID, di erent data)

Towards a Solution ⚗✨ Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}} • Single-source server/client • DNS: hostname → IP address • PIDs: number → address V I R T UA L A D D R E S S 📬 P H Y S I C A L L O C AT I O N 🗺 • Focused: physical network ff • Referential opacity (same PID, di erent data)

Towards a Solution ⚗✨ Consistent Keys %{hash(content) => content} • Above virtual address • Focused: data itself • Same for everyone & everywhere • Perfect for caching • Immutable data++ • Consistent pointers → consistent data V I R T UA L A D D R E S S 📬 P H Y S I C A L L O C AT I O N 🗺

Towards a Solution ⚗✨ Consistent Keys %{hash(content) => content} • Above virtual address • Focused: data itself • Same for everyone & everywhere • Perfect for caching • Immutable data++ • Consistent pointers → consistent data CONTENT ID ☃ V I R T UA L A D D R E S S 📬 P H Y S I C A L L O C AT I O N 🗺

Towards a Solution ⚗✨ Hash-Based Relationships

Towards a Solution ⚗✨ Hash-Based Relationships { } CID ~ Data PID Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] }

Towards a Solution ⚗✨ Hash-Based Relationships { } { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] } CID ~ Data PID Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb…}, {name: “about”, hash: Qm0eN…} ] } }

Towards a Solution ⚗✨ Hash-Based Relationships { } { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] } CID ~ Data PID Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb…}, {name: “about”, hash: Qm0eN…} ] } } Qm123456…/company/about/ceo ⇒ “Boris Mann”

Towards a Solution ⚗✨ Content IDs Are Easy

Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy

Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999%

Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999% 🖼 99.0% 🖼 99.99% 🖼 99.999%

Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999% 👩🎤 🖼 99.0% 🖼 99.99% 🖼 99.999%

Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999% 👩🎤 🖼 99.0% 🖼 99.99% 🖼 99.999% 👩🎤 🧑🎤

Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999% 👩🎤 🖼 99.0% 🖼 99.99% 🖼 99.999% 11-nines 👩🎤 🧑🎤

Towards a Solution ⚗✨ Reliability from Unreliable Components

Towards a Solution ⚗✨ Reliability from Unreliable Components 1 2 3 4 5 6 7 8 9 10

Towards a Solution ⚗✨ Downtime per Year (linear scale) Reliability from Unreliable Components 1 2 3 4 5 6 #Independent Machines 7 8 9 10

Towards a Solution ⚗✨ Downtime per Year (linear scale) Reliability from Unreliable Components 87 hours 1 2 3 4 5 6 #Independent Machines 7 8 9 10

Towards a Solution ⚗✨ Downtime per Year (linear scale) Reliability from Unreliable Components 87 hours 52 mins 1 2 3 4 5 6 #Independent Machines 7 8 9 10

Towards a Solution ⚗✨ Downtime per Year (linear scale) Reliability from Unreliable Components 87 hours 52 mins 32 sec 1 2 3 4 5 6 #Independent Machines 7 8 9 10

Towards a Solution ⚗✨ Downtime per Year (log scale) Reliability from Unreliable Components 87 hours 103s 1s 10-6s 316 fs 10-12s 1 2 3 4 5 6 #Independent Machines 7 8 9 10

Towards a Solution ⚗✨ Downtime per Year (log scale) Reliability from Unreliable Components 87 hours 2O 103s rde rs M agn itud e/ 1s Ma chi n e 10-6s 316 fs 10-12s 1 2 3 4 5 6 #Independent Machines 7 8 9 10

Towards a Solution ⚗✨ Why Sync Whole Tables?