A presentation at CascadiaJS in in Sunriver, OR 97707, USA by Brooklyn Zelenka
🪐 The Dark Forest 🚀 Private CRDTs as Global Database or: “How to Power a New Internet” github.com/wnfs-wg github.com/fission-codes
BROOKLYN ZELENKA @expede Cofounder & CTO Fission Codes
BROOKLYN ZELENKA @expede Cofounder & CTO Fission Codes
BROOKLYN ZELENKA ✨ WNFS ✨ Dialog CAR Pool IPVM @expede Cofounder & CTO Fission Codes
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨
The Situation 🤨 2. Too Much In The Way
The Situation 🤨 2. Too Much In The Way Users 👨🏫👩🏭🧑⚕👷 Developer 👩💻
The Situation 🤨 2. Too Much In The Way Users 👨🏫👩🏭🧑⚕👷 Browser 🖥 REST / JSON-RPC / GraphQL ↕ Server ⚙ Data Store 🗃 DevOps 📤 Developer 👩💻
The Situation 🤨 2. Too Much In The Way Users 👨🏫👩🏭🧑⚕👷 Browser 🖥 REST / JSON-RPC / GraphQL ↕ Server ⚙ Data Store 🗃 DevOps 📤 Developer 👩💻
The Situation 🤨 3. Toxic Data • 2005: Credit card info in DB • 2015: Personal info in DB • 2025: Nothing in DB
What If… 🤔
What If… 🤔 Networked Data, Not Apps
What If… 🤔 Networked Data, Not Apps f f f Alice’s Stu Bob’s Stu Carol’s Stu
What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 f f f Alice’s Stu Bob’s Stu Carol’s Stu
What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 Alice’s Stu Bob’s Stu Carol’s Stu Social Media f f f 📜
What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 Alice’s Stu Bob’s Stu Music Player f f f 🎶 Carol’s Stu Social Media 📜
What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 Auth Service Alice’s Stu Bob’s Stu Music Player f f f 🎶 Carol’s Stu Social Media 📜 💂
What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 Alice’s Stu Bob’s Stu Music Player f f f 🎶 Carol’s Stu Social Media 📜
What If… 🤔 Open Protocols Everywhere
What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation Networking 🔌
What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… Networking 🔌
What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… • Auth: interop without pre-negotiation Auth 🎟 Networking 🔌
What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… • Auth: interop without pre-negotiation • Data: ubiquitous “dumb” storage Data 💾 Auth 🎟 Networking 🔌
What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… • Auth: interop without pre-negotiation • Data: ubiquitous “dumb” storage • Compute: local & remote lambdas Compute ⚙ Data 💾 Auth 🎟 Networking 🔌
What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… • Auth: interop without pre-negotiation • Data: ubiquitous “dumb” storage • Compute: local & remote lambdas Compute ⚙ Data 💾 Auth 🎟 Networking 🔌
What If… 🤔 Three Techniques
What If… 🤔 Three Techniques Global Primary Keys 🗺 Portable Private Data 💾 BFT Concurrency 🔀
What If… 🤔 Three Techniques Global Primary Keys 🗺 Content Addressing Portable Private Data 💾 BFT Concurrency 🔀
What If… 🤔 Three Techniques Global Primary Keys 🗺 Portable Private Data 💾 Content Addressing Cryptrees / Dark Forest BFT Concurrency 🔀
What If… 🤔 Three Techniques Global Primary Keys 🗺 Portable Private Data 💾 BFT Concurrency 🔀 Content Addressing Cryptrees / Dark Forest Hash-Linked CRDTs
Content Addressing Truly Global Links 🗺
Content Addressing 🗺
Content Addressing 🗺 The limitation of local knowledge is the fundamental fact about the setting in which we work, and it is a very powerful limitation – Nancy Lynch, A Hundred Impossibility Proofs for Distributed Computing
Content Addressing 🗺 Addressing Stack
Content Addressing 🗺 Addressing Stack Physical Location 🗺 send(42.123.45.6, path) = content1 send(42.123.45.6, path) = content2
Content Addressing 🗺 Addressing Stack Virtual Address 📬 Physical Location 🗺 {DNS: IP} send(42.123.45.6, path) = content1 send(42.123.45.6, path) = content2
Content Addressing 🗺 Addressing Stack Content ID ☃ Virtual Address 📬 Physical Location 🗺 {hash(content): content} {DNS: IP} send(42.123.45.6, path) = content1 send(42.123.45.6, path) = content2
Content Addressing 🗺 Hash-Based Relationships
Content Addressing 🗺 Hash-Based Relationships { } Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] }
Content Addressing 🗺 Hash-Based Relationships { } { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] } } Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb…}, {name: “about”, hash: Qm0eN…} ] }
Content Addressing 🗺 Hash-Based Relationships { } { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] } } Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb…}, {name: “about”, hash: Qm0eN…} ] } Qm123456…/company/about/ceo => “Boris Mann”
Content Addressing 🗺 Hard & Soft Links
Content Addressing 🗺 Hard & Soft Links alice.fission.name
Content Addressing 🗺 Hard & Soft Links alice.fission.name bob.fission.name
Content Addressing 🗺 Hard & Soft Links alice.fission.name Hard Links New for the web! Direct reference 2 pointers ~ deduplicate bob.fission.name
Content Addressing 🗺 Hard & Soft Links alice.fission.name Hard Links New for the web! Direct reference 2 pointers ~ deduplicate bob.fission.name
Content Addressing 🗺 Hard & Soft Links alice.fission.name Hard Links New for the web! Direct reference 2 pointers ~ deduplicate bob.fission.name Soft Links Symlink or web link (URL) Pointer ~ latest Latest may break
Content Addressing 🗺 Content Addressed Storage Is Simple
Content Addressing 🗺 Content Addressed Storage Is Simple
Content Addressing 🗺 Content Addressed Storage Is Simple
Content Addressing 🗺 Content Addressed Storage Is Simple
Content Addressing 🗺 Content Addressed Storage Is Simple
Content Addressing 🗺 Content Addressed Storage Is Simple
Content Addressing 🗺 Content Addressed Storage Is Simple
Content Addressing 🗺 Content Addressed Storage Is Simple
Content Addressing 🗺 Upshot • Names that leak no additional info • A consistent way to reference data • Self-verifying data for low/no-trust environment • Hard links on the web
Private Data Substrate Distributed Storage When The Pipes Are Broken 💾
Private Data Substrate 💾 Virtual Files & Directories
Private Data Substrate 💾 Virtual Files & Directories File Node Raw Data Metadata
Private Data Substrate 💾 Virtual Files & Directories File Node Virtual Node Raw Data Consistent interface Arbitrary metadata Tags, creators, MIME, sources, &c Metadata
Private Data Substrate 💾 Virtual Files & Directories File Node Virtual Node Raw Data Consistent interface Arbitrary metadata Tags, creators, MIME, sources, &c Metadata Directory Node Index Metadata
Private Data Substrate 💾 Persistent Versioning
Private Data Substrate 💾 Persistent Versioning Photos@r0 Vacation Avatars@r0 beach.png caricature.jpg Revision 0
Private Data Substrate 💾 Persistent Versioning Photos@r0 Vacation Avatars@r0 beach.png caricature.jpg Revision 0 🕙
Private Data Substrate 💾 Persistent Versioning Photos@r0 Photos@r1 Vacation Avatars@r0 Avatars@r1 beach.png caricature.jpg headshot.png Revision 0 Revision 1 🕙
Private Data Substrate 💾 Persistent Versioning Photos@r0 ChildHasEvent Vacation Avatars@r0 beach.png caricature.jpg Photos@r1 InsertNew Revision 0 headshot.png Revision 1 🕙 Avatars@r1
Private Data Substrate 💾 Persistent Versioning Photos@r0 ChildHasEvent Vacation Avatars@r0 beach.png caricature.jpg Photos@r1 InsertNew Revision 0 headshot.png Revision 1 🕙 Avatars@r1
Private Data Substrate 💾 It’s All Just Maps
Private Data Substrate 💾 It’s All Just Maps
Private Data Substrate 💾 It’s All Just Maps
Private Data Substrate 💾 It’s All Just Maps
Private Data Substrate 💾 It’s All Just Maps
Private Data Substrate 💾 It’s All Just Maps
Private Data Substrate 💾
Private Data Substrate 💾 Cryptography is a tool for turning lots of different problems into key management problems Dr. Lea Kissner, Google’s Global Lead of Privacy Technologies
Private Data Substrate 💾 SubtleCrypto (WebCrypto) Source: caniuse.com
Private Data Substrate 💾 Non-Extractable Keys
Private Data Substrate 💾 Non-Extractable Keys
Private Data Substrate 💾 Non-Extractable Keys 🗝
Private Data Substrate 💾 Non-Extractable Keys 🦹 🗝
Private Data Substrate 💾 Non-Extractable Keys 🦹 🗝
Private Data Substrate 💾 Non-Extractable Keys 🦹 🏰🏰🏰🏰🏰🏰🏰 🗝
Private Data Substrate 💾 Non-Extractable Keys 🦹 📝 🏰🏰🏰🏰🏰🏰🏰 🗝
Private Data Substrate 💾 Non-Extractable Keys 🦹 📝 🏰🏰🏰🏰🏰🏰🏰 🗝
Private Data Substrate 💾 Non-Extractable Keys 🦹 🏰🏰🏰🏰🏰🏰🏰 📝 🗝
Private Data Substrate 💾 Non-Extractable Keys 🦹 📝 🏰🏰🏰🏰🏰🏰🏰 🗝
Private Data Substrate 💾 Self-Authenticating & Self-Authorizing
Private Data Substrate 💾 Self-Authenticating & Self-Authorizing 123 ACF CF4 C4A 0FC 1F3 A83 ED2 D55 823 247 81D F0A B92
Private Data Substrate 💾 Self-Authenticating & Self-Authorizing A7B ACF CF4 👩💻 C4A 0FC 1F3 A83 ED2 D55 823 247 81D F0A B92
Private Data Substrate 💾 Offline Access Control / Photos/ Sunriver/ food.jpg Iceland/ stage.png Docs/ notes.md harpa.svg
Private Data Substrate 💾 Offline Access Control / 🗝 Photos/ Sunriver/ food.jpg Iceland/ stage.png Docs/ notes.md harpa.svg
Private Data Substrate 💾 Offline Access Control / 🗝 Photos/ Sunriver/ Iceland/ Docs/ notes.md 🗝 food.jpg stage.png harpa.svg
Private Data Substrate 💾 Offline Access Control / 🗝 Photos/ Docs/ 🗝 Sunriver/ Iceland/ notes.md 🗝 food.jpg stage.png harpa.svg
Private Data Substrate 💾 Offline Access Control / / 🗝 Photos/ Docs/ Photos/ 🗝 🗝 Sunriver/ food.jpg Iceland/ stage.png notes.md harpa.svg pdx.jpg plane.png Docs/
Private Data Substrate 💾 Offline Access Control Photos/ Sunriver/ food.jpg stage.png notes.md pdx.jpg plane.png
Private Data Substrate 💾 Offline Access Control Shared Sunriver/ food.jpg stage.png Photos/ notes.md pdx.jpg plane.png
Private Data Substrate 💾 Private Nodes
Private Data Substrate 💾 Private Nodes Binary Encrypted Node 🔒 Encrypted Node 🔒 Encrypted Node 🔒
Private Data Substrate 💾 Private Nodes Binary CBOR Encrypted Node 🔒 Virtual Node Encrypted Node 🔒 Index Encrypted Node 🔒 🔑 🔑 Metadata
Private Data Substrate 💾 Private Nodes Binary CBOR Encrypted Node 🔒 AES256 Encrypted Node 🔒 Virtual Node 🔑 Index Encrypted Node 🔒 🔑 🔑 Metadata
Private Data Substrate 💾 Private Nodes Binary CBOR Encrypted Node 🔒 AES256 Encrypted Node 🔒 Virtual Node 🔑 Index Encrypted Node 🔒 🔑 🔑 Metadata
Private Data Substrate 💾 Reconstruction Virtual Node Virtual Node 🔑 Index 🔑 🔑 Metadata 🔑 🔑 🔑 Index Metadata
Private Data Substrate 💾 Dark Forest
Private Data Substrate 💾 Dark Forest
Fault-Safe Concurrency Safely Merge Unreliable Updates 🔀
Fault-Safe Concurrency 🔀 Causal Islands 🏖🏝
Fault-Safe Concurrency 🔀 Causal Islands 🏖🏝
Fault-Safe Concurrency 🔀 Causal Islands 🏖🏝
Fault-Safe Concurrency 🔀 Causal Islands 🏖🏝
Fault-Safe Concurrency 🔀 Gossiping Out of Order
Fault-Safe Concurrency 🔀 Gossiping Out of Order
Fault-Safe Concurrency 🔀 Gossiping Out of Order 🎨
Fault-Safe Concurrency 🔀 Gossiping Out of Order 🎨
Fault-Safe Concurrency 🔀 Gossiping Out of Order t→ 🎨
Fault-Safe Concurrency 🔀 Gossiping Out of Order 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡
Fault-Safe Concurrency 🔀 Gossiping Out of Order 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡
Fault-Safe Concurrency 🔀 Gossiping Out of Order 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡
Fault-Safe Concurrency 🔀 Gossiping Out of Order 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡
Fault-Safe Concurrency 🔀 Gossiping Out of Order 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡
Fault-Safe Concurrency 🔀 Gossiping Out of Order 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡 ❎
Fault-Safe Concurrency 🔀 Entropy Isn’t What It Used to Be
Fault-Safe Concurrency 🔀 Entropy Isn’t What It Used to Be
Fault-Safe Concurrency 🔀 Properties Save the Day
Fault-Safe Concurrency 🔀 Properties Save the Day Commutative
Fault-Safe Concurrency 🔀 Properties Save the Day Commutative merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Commutative merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Commutative merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Commutative merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Commutative merge merge Associative
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative merge merge merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative merge merge merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge ( ) merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge ( ) merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge ( ) merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge ( ) merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge ( ) merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge ( ) merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill fill
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill fill
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill fill
Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill fill
Fault-Safe Concurrency 🔀 Temporal Confluence
Fault-Safe Concurrency 🔀 Temporal Confluence BFT-CRDTs Persistent data structure Automatic file-level reconciliation Pluggable sub-file reconciliation (forthcoming) Basis of upcoming BFT Datalog “at scale” work 😉
Fault-Safe Concurrency 🔀 Temporal Confluence BFT-CRDTs Persistent data structure Automatic file-level reconciliation Pluggable sub-file reconciliation (forthcoming) Basis of upcoming BFT Datalog “at scale” work 😉 Single File History / “Causal Shadow”
Fault-Safe Concurrency 🔀 Temporal Confluence BFT-CRDTs F B ⭐E⭐ Persistent data structure Automatic file-level reconciliation D Pluggable sub-file reconciliation (forthcoming) Basis of upcoming BFT Datalog “at scale” work 😉 A ⭐C⭐ Single File History / “Causal Shadow”
Fault-Safe Concurrency 🔀 Multiplayer Docs (Incl. Encryption) F B ⭐E⭐ D A ⭐C⭐
Fault-Safe Concurrency 🔀 Multiplayer Docs (Incl. Encryption) F B ⭐E⭐ D A ⭐C⭐
Fault-Safe Concurrency 🔀 Multiplayer Docs (Incl. Encryption) F B ⭐E⭐ D A ⭐C⭐
Fault-Safe Concurrency 🔀 Multiplayer Docs (Incl. Encryption) F B ⭐E⭐ D A ⭐C⭐
Fault-Safe Concurrency 🔀 Alternate History, Same State
Fault-Safe Concurrency 🔀 Alternate History, Same State null
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶 😺 null 🐙
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐶 😺 null 🐙
Fault-Safe Concurrency 🔀 Alternate History, Same State 😺🐙 🐶😺 🐶 😺 null 🐙
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙 😺🐙 🐶😺 🐶 😺 null 🐙
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙 😺🐙 🐶😺 🐶 😺 null 🐙 🐙🦋
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙 😺🐙 🐶😺 🐶 😺🐙🦋 😺 null 🐙 🐙🦋
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙🦋 🐶😺 🐙 😺🐙 🐶😺 🐶 😺🐙🦋 😺 null 🐙 🐙🦋
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙🦋 🐶😺 🐙 😺🐙 🐶😺 🐶 😺🐙🦋 😺 null 🐙 🐙🦋
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙🦋 🐶😺 🐙 😺🐙 🐶😺 🐶 😺🐙🦋 😺 null 🐙 🐙🦋
Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙🦋 🐶😺 🐙 😺🐙 🐶😺 🐶 😺🐙🦋 😺 null 🐙 🐙🦋
Wrap Up Thoughts & Field Notes 🎁
Wrap Up 🎁 Requirements Diagram Next Gen Distributed Apps Web 2.0 Features Collaborative Portable Data & Runtime Hostile Hosts e.g. Browsers Multiuser Nonextractable Signing Keys Concurrent Local First Access Control Authenticated Data Content Addressed Authorized Data Encryption at Rest Capability Auth Extensible Arbitrary Metadata Kernel vs Userspace
Wrap Up 🎁 Requirements Diagram Next Gen Distributed Apps Web 2.0 Features Collaborative Portable Data & Runtime Hostile Hosts e.g. Browsers Multiuser Nonextractable Signing Keys Concurrent Local First Access Control Authenticated Data Content Addressed Authorized Data Encryption at Rest Capability Auth Extensible Arbitrary Metadata Kernel vs Userspace
Wrap Up 🎁 Better Together
🏔🎉 Thank You, CascadiaJS 🇨🇦🇺🇸 github.com/ucan-wg github.com/wnfs-wg More in-depth at Strange Loop 2022 (or talk to us today)
P2P social networks, local-first apps, and resilient client-side apps need to be private, concurrent, BFT, and run everywhere. Let’s look at some recent tools & techniques that make this even possible.
for free. You
can too.