A presentation at CodeBEAM A Coruña in June 2022 in A Coruña, Spain by Brooklyn Zelenka
The Mess We’re (Still) In 🌈🧘 Unbounded Parallelism, True Names, & Keeping CALM 🎱✨
The Mess We’re (Still) In
The Mess We’re (Still) In A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable — Leslie Lamport
The Mess We’re (Still) In Meanwhile…
The Mess We’re (Still) In Meanwhile… …who? 🤨
The Mess We’re (Still) In Keep Calm and Dig In 🕳⛏
@FissionCodes Brooklyn Zelenka Quinn Wilton @expede @wilton_quinn
@FissionCodes Brooklyn Zelenka Quinn Wilton @expede @wilton_quinn • CTO at Fission • Applied Researcher at Fission • Distributed auth, data, compute, and discovery • Building a planetary scale database for local-first apps • Author of Witchcraft, Algae, Exceptional, &c • Contributed to Lumen, Gleam, Burrito, Witchcraft, &c
So Many Problems 🪨🪐 🌌
Problems 🪨🪐🌌 High Level
Problems 🪨🪐🌌 State Space is Big
Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 (2 ) ≈ 10 48
Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 🌍
Six 32-bit Numbers 32 6 (2 ) ≈ 10 57
Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 🌍 🌞 🪐 🌍 🌒
Single Receiver (2 32+1 6 ) × 6! ≈ 10 62
Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 Single Receiver 32+1 6 (2 ) × 6! ≈ 10 🌍 🌞 🌌 🪐 🌍 🌒
62
Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 32+1 6 (2 🌍 🌞 🌌 🌍 🌒
62 ) × 6! ≈ 10 🪐
Two Receivers Single Receiver
((2 32+1 6 2 ) × 6!) ≈ 10 124
Problems 🪨🪐🌌 State Space is Big Five 32-bit Numbers 32 5 48 (2 ) ≈ 10 Six 32-bit Numbers 32 6 57 (2 ) ≈ 10 Two Receivers Single Receiver 32+1 6 (2 ) × 6! ≈ 10 62 32+1 6 ((2 2 124 ) × 6!) ≈ 10 🌍 🌞 🌌 🤯 🪐 🌍 🌒
More than observable universe
Problems 🪨🪐🌌
Problems 🪨🪐🌌 Distributed systems introduce significant nondeterminism to our programs. Sources of non-determinism include unsynchronized parallelism, unreliable components, and networks with unpredictable delays. As a result, a distributed program can exhibit a large space of possible behaviors on a given input. — Hellerstein & Alvaro, Keeping CALM: When Distributed Consistency is Easy
Problems 🪨🪐🌌 The Great 73-Hour Roblox Outage of 2021 https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://www.theverge.com/2021/10/30/22754107/roblox-down-outage-chipotle-server-issues-status
Problems 🪨🪐🌌 The Great 73-Hour Roblox Outage of 2021 👀 https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://www.theverge.com/2021/10/30/22754107/roblox-down-outage-chipotle-server-issues-status
Problems 🪨🪐🌌 The Great 73-Hour Roblox Outage of 2021 🧨 👀 https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://www.theverge.com/2021/10/30/22754107/roblox-down-outage-chipotle-server-issues-status
Problems 🪨🪐🌌 And Yet…
Problems 🪨🪐🌌 And Yet… These metastable failures have caused widespread outages at large internet companies, lasting from minutes to hours. Paradoxically, the root cause of these failures is often features that improve the efficiency or reliability of the system. – Bronson et al, Metastable Failures in Distributed Systems
Problems 🪨🪐🌌 Metastable Mechanism
Problems 🪨🪐🌌 Metastable Mechanism ⚡ ⚖
Problems 🪨🪐🌌 Metastable Mechanism ⚡ 🛑 ⚖
Problems 🪨🪐🌌 Metastable Mechanism ⚡ 🛑 • Retries / let it crash • Work amplification ⚖ • General thrash 🫡
Problems 🪨🪐🌌 Places Fight Light 💫🤺
Problems 🪨🪐🌌 Places Fight Light 💫🤺 The limitation of local knowledge is the fundamental fact about the setting in which we work, and it is a very powerful limitation – Nancy Lynch, A Hundred Impossibility Proofs for Distributed Computing
Problems 🪨🪐🌌 Places Fight Light 💫🤺 Sending a “Direct” Message
Problems 🪨🪐🌌 Places Fight Light 💫🤺 Sending a “Direct” Message 7.2x ⏳🔋🛢🪠
Problems 🪨🪐🌌 Data Behind Walls 🏰
Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration Bob’s Photo Gallery 🖼 ⛔ 🔒 👩🎤 Alice’s Music Player 🎶 🔒 🧑🎤 Carol’s Videogame 👾 🔒
Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration
Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration 💿 💾 📀
Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration Bob’s Photo Gallery 🖼 💿 💾 📀
Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration Bob’s Photo Gallery 🖼 💿 💾 📀 Carol’s Videogame 👾
Problems 🪨🪐🌌 Data Behind Walls 🏰 Dependencies & Integration Bob’s Photo Gallery 🖼 💿 💾 Alice’s Music Player 🎶 📀 Carol’s Videogame 👾
Problems 🪨🪐🌌 Data Behind Walls 🏰 Inconsistency
Problems 🪨🪐🌌 Data Behind Walls 🏰 Inconsistency • Even with FOSS!
Problems 🪨🪐🌌 Data Behind Walls 🏰 Inconsistency • Even with FOSS! • Annual migration to the latest hipster HTTP client • HTTPotion → HTTPoison → Hackney → Tesla → Finch → Req
Problems 🪨🪐🌌 Data Behind Walls 🏰 If people in a few hundred years from now want to see what their ancestors wrote, what will they find, a mess of badly formatted crap?! — Joe Armstrong, Why Markdown Sucks
Cause & E ect ff Mental Framework 🏗🧠
Mental Framework 🏗🧠 Causal Islands 🏖🏝
Mental Framework 🏗🧠 Causal Islands 🏖🏝 “Causal Subjectivity”
Mental Framework 🏗🧠
Mental Framework 🏗🧠 What is the family of problems that can be consistently computed in a distributed fashion without coordination, and what problems lie outside that family? —Hellerstein & Alvaro, Keeping CALM: When Distributed Consistency is Easy
Mental Framework 🏗🧠 Gossiping Out of Order 🙊
Mental Framework 🏗🧠 Gossiping Out of Order 🙊 t→ 🎨
Mental Framework 🏗🧠 Gossiping Out of Order 🙊 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡
Mental Framework 🏗🧠 Gossiping Out of Order 🙊 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡 ❎
Mental Framework 🏗🧠 Monotone Functions
Mental Framework 🏗🧠 Monotone Functions 🍎 🍋 🍊
Mental Framework 🏗🧠 Monotone Functions 🍎 🍋 🍊 🍎 🍑 🍋 🍒 🍊 🍓 🍌 🍍
Mental Framework 🏗🧠 Monotone Functions 🍎 🍑 🍎 🍋 🍊 ⊆ 🍋 🍒 🍊 🍓 🍌 🍍
Mental Framework 🏗🧠 Monotone Functions get(:red) 🍎 🍑 🍎 🍋 🍊 🍎 ⊆ 🍋 🍒 🍊 🍓 🍌 🍍
Mental Framework 🏗🧠 Monotone Functions get(:red) 🍎 🍑 🍎 🍋 🍊 get(:red) ⊆ 🍋 🍒 🍊 🍓 🍌 🍍 🍎 🍎 🍒 🍓
It’s All About that Data 📊 PNCounter
Grappling With Reality Towards a Solution ⚗✨
Towards a Solution ⚗✨ Evolving Toolbox Serverless Networked Data Cloud Commons Networks Local-First Blockchain O ffl P2P ine
Towards a Solution ⚗✨ Evolving Toolbox Serverless Networked Data Radical shifts how we think about auth, locality of reference, ownership, and reliability Cloud Commons Networks Local-First Blockchain O ffl P2P ine
Towards a Solution ⚗✨ Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}} • Single-source server/client • DNS: hostname → IP address • PIDs: number → address • Focused: physical network ff • Referential opacity (same PID, di erent data)
Towards a Solution ⚗✨ Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}} • Single-source server/client • DNS: hostname → IP address • PIDs: number → address P H Y S I C A L L O C AT I O N 🗺 • Focused: physical network ff • Referential opacity (same PID, di erent data)
Towards a Solution ⚗✨ Mutable Pointers send(:example@42.123.45.6, :ping) %{node_id => %{path => content}} • Single-source server/client • DNS: hostname → IP address • PIDs: number → address V I R T UA L A D D R E S S 📬 P H Y S I C A L L O C AT I O N 🗺 • Focused: physical network ff • Referential opacity (same PID, di erent data)
Towards a Solution ⚗✨ Consistent Keys %{hash(content) => content} • Above virtual address • Focused: data itself • Same for everyone & everywhere • Perfect for caching • Immutable data++ • Consistent pointers → consistent data V I R T UA L A D D R E S S 📬 P H Y S I C A L L O C AT I O N 🗺
Towards a Solution ⚗✨ Consistent Keys %{hash(content) => content} • Above virtual address • Focused: data itself • Same for everyone & everywhere • Perfect for caching • Immutable data++ • Consistent pointers → consistent data CONTENT ID ☃ V I R T UA L A D D R E S S 📬 P H Y S I C A L L O C AT I O N 🗺
Towards a Solution ⚗✨ Hash-Based Relationships
Towards a Solution ⚗✨ Hash-Based Relationships { } CID ~ Data PID Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] }
Towards a Solution ⚗✨ Hash-Based Relationships { } { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] } CID ~ Data PID Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb…}, {name: “about”, hash: Qm0eN…} ] } }
Towards a Solution ⚗✨ Hash-Based Relationships { } { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] } CID ~ Data PID Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb…}, {name: “about”, hash: Qm0eN…} ] } } Qm123456…/company/about/ceo ⇒ “Boris Mann”
Towards a Solution ⚗✨ Content IDs Are Easy
Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy
Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999%
Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999% 🖼 99.0% 🖼 99.99% 🖼 99.999%
Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999% 👩🎤 🖼 99.0% 🖼 99.99% 🖼 99.999%
Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999% 👩🎤 🖼 99.0% 🖼 99.99% 🖼 99.999% 👩🎤 🧑🎤
Towards a Solution ⚗✨ Decoupling, Abundance, Redundancy 🖼 99.99999% 👩🎤 🖼 99.0% 🖼 99.99% 🖼 99.999% 11-nines 👩🎤 🧑🎤
Towards a Solution ⚗✨ Reliability from Unreliable Components
Towards a Solution ⚗✨ Reliability from Unreliable Components 1 2 3 4 5 6 7 8 9 10
Towards a Solution ⚗✨ Downtime per Year (linear scale) Reliability from Unreliable Components 1 2 3 4 5 6 #Independent Machines 7 8 9 10
Towards a Solution ⚗✨ Downtime per Year (linear scale) Reliability from Unreliable Components 87 hours 1 2 3 4 5 6 #Independent Machines 7 8 9 10
Towards a Solution ⚗✨ Downtime per Year (linear scale) Reliability from Unreliable Components 87 hours 52 mins 1 2 3 4 5 6 #Independent Machines 7 8 9 10
Towards a Solution ⚗✨ Downtime per Year (linear scale) Reliability from Unreliable Components 87 hours 52 mins 32 sec 1 2 3 4 5 6 #Independent Machines 7 8 9 10
Towards a Solution ⚗✨ Downtime per Year (log scale) Reliability from Unreliable Components 87 hours 103s 1s 10-6s 316 fs 10-12s 1 2 3 4 5 6 #Independent Machines 7 8 9 10
Towards a Solution ⚗✨ Downtime per Year (log scale) Reliability from Unreliable Components 87 hours 2O 103s rde rs M agn itud e/ 1s Ma chi n e 10-6s 316 fs 10-12s 1 2 3 4 5 6 #Independent Machines 7 8 9 10
Towards a Solution ⚗✨ Why Sync Whole Tables?
— OCT-2020
— OCT-2020 kb_id owner_id mode switches inserted_at 42 1 Wireless Blue JAN-2020
Towards a Solution ⚗✨ Relationships XYZ Name: @expede From JAN-2000 🔒 Work: Fission From AUG-2019 KEB 🔒 Red Switches Switches: Blue From JAN-2020 From FEB-2020 Wireless
Towards a Solution ⚗✨ A Sequel to SQL: Nonlinear DBs XYZ Name: @expede From JAN-2000 ABC Name: @bmann From DEC-1999 KEB Type: Wireless Always 🔒 XYZ Work: Fission From AUG-2019 🔒 KEB Switches: Red From JAN-2020 KEB Owner:XYZ From JAN-2020 KEB Switches: Blue From FEB-2020
Towards a Solution ⚗✨ A Sequel to SQL: Nonlinear DBs XYZ Name: @expede From JAN-2000 ABC Name: @bmann From DEC-1999 KEB Type: Wireless Always 🔒 XYZ Work: Fission From AUG-2019 🔒 KEB Switches: Red From JAN-2020 KEB Owner:XYZ From JAN-2020 KEB Switches: Blue From FEB-2020 🏢
Towards a Solution ⚗✨ A Sequel to SQL: Nonlinear DBs XYZ Name: @expede From JAN-2000 ABC Name: @bmann From DEC-1999 KEB Type: Wireless Always 🔒 XYZ Work: Fission From AUG-2019 🔒 KEB Switches: Red From JAN-2020 KEB Owner:XYZ From JAN-2020 KEB Switches: Blue From FEB-2020 👩🚀📱 🏢
Towards a Solution ⚗✨ A Sequel to SQL: Nonlinear DBs XYZ Name: @expede From JAN-2000 ABC Name: @bmann From DEC-1999 KEB Type: Wireless Always 🔒 XYZ Work: Fission From AUG-2019 🔒 KEB Switches: Red From JAN-2020 👨🍳💻 KEB Owner:XYZ From JAN-2020 🏢 KEB Switches: Blue From FEB-2020 👩🚀📱
Towards a Solution ⚗✨ A Sequel to SQL: Nonlinear DBs ➡⬅ XYZ Name: @expede From JAN-2000 ABC Name: @bmann From DEC-1999 KEB Type: Wireless Always 🔒 XYZ Work: Fission From AUG-2019 🔒 KEB Switches: Red From JAN-2020 👨🍳💻 KEB Owner:XYZ From JAN-2020 🏢 KEB Switches: Blue From FEB-2020 👩🚀📱
Towards a Solution ⚗✨
Towards a Solution ⚗✨ With over 1.5 million publications per year and more than 50 million total peer-reviewed articles, the rate and volume of novel discoveries has surpassed our ability to fully utilize and understand what is known – William E. Byrd et al, mediKanren: a System for Biomedical Reasoning
Towards a Solution ⚗✨ Standardized Knowledge Graphs
Towards a Solution ⚗✨ Data Integration
Towards a Solution ⚗✨ Data Integration A ‘high-speed Dr. House’ for medical breakthroughs – University of Alabama News, on mediKanren
Your Turn Call to Action 🦸
Call to Action 🦸
Call to Action 🦸 We have a system that applies cutting edge CS research to tackle day-to-day problems in the applications we all write. Phoenix Presence - has no single point of failure - has no single source of truth -[…] - self heals ~ Chris McCord, “What Makes Phoenix Presence Special”
Call to Action 🦸 Next Steps
Call to Action 🦸 Next Steps 1. Embrace the distributed nature of the network 🫂
Call to Action 🦸 Next Steps 1. Embrace the distributed nature of the network 🫂 2. Put data into interoperable forms
Call to Action 🦸 Next Steps 1. Embrace the distributed nature of the network 🫂 2. Put data into interoperable forms 3. Better living through replication
🎉 Thank You, A Coruña 🇪🇸 https://fission.codes {brooklyn,quinn}@fission.codes github.com/expede | github.com/quinnwilton @expede | @wilton_quinn
View The Mess We’re (Still) In on Notist.
Dismiss
Cowritten and presented with Quinn Wilton (Twitter: @wilton_quinn)