The Dark Forest: Private CRDTs as Global Database 

🪐 The Dark Forest 🚀 Private CRDTs as Global Database or: “How to Power a New Internet” github.com/wnfs-wg github.com/fission-codes

BROOKLYN ZELENKA @expede Cofounder & CTO Fission Codes

BROOKLYN ZELENKA ✨ WNFS ✨ Dialog CAR Pool IPVM @expede Cofounder & CTO Fission Codes

The Situation 🤨

Services In 2022

The Situation 🤨

Services In 2022 Alice Bob Carol

The Situation 🤨

Services In 2022 Alice Bob Music Service 🎶 Carol

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Music Service 🎶 Carol Carol’s Music

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Music Service 🎶 Carol Carol’s Music Auth Service 💂

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Music Service 🎶 Carol Carol’s Music Auth Service 💂

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Music Service 🎶 Carol Carol’s Music Auth Service 💂

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Bob’s Photos Music Service Photo Service 🎶 Carol 🖼 Carol’s Music Auth Service 💂

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Bob’s Photos Music Service Photo Service 🎶 Carol 🖼 Carol’s Music Auth Service 💂 Social Media 📜

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Bob’s Photos Bob’s Posts Music Service Photo Service Social Media 🎶 Carol 🖼 Carol’s Music 📜 Carol’s Posts Auth Service 💂

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Bob’s Photos Bob’s Posts Music Service Photo Service Social Media 🎶 Carol 🖼 Carol’s Music 📜 Carol’s Posts Auth Service 💂

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Bob’s Photos Music Service Photo Service 🎶 Carol Bob’s Posts 🖼 Carol’s Music Carol’s Posts Auth Service 💂

The Situation 🤨

Services In 2022 Alice Alice’s Music Bob Bob’s Music Bob’s Photos Music Service Photo Service 🎶 Carol 🖼 Carol’s Music Auth Service 💂

The Situation 🤨 2. Too Much In The Way

The Situation 🤨 2. Too Much In The Way Users 👨🏫👩🏭🧑⚕👷 Developer 👩💻

The Situation 🤨 2. Too Much In The Way Users 👨🏫👩🏭🧑⚕👷 Browser 🖥 REST / JSON-RPC / GraphQL ↕ Server ⚙ Data Store 🗃 DevOps 📤 Developer 👩💻

The Situation 🤨 3. Toxic Data • 2005: Credit card info in DB • 2015: Personal info in DB • 2025: Nothing in DB

What If… 🤔

What If… 🤔 Networked Data, Not Apps

What If… 🤔 Networked Data, Not Apps f f f Alice’s Stu Bob’s Stu Carol’s Stu

What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 f f f Alice’s Stu Bob’s Stu Carol’s Stu

What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 Alice’s Stu Bob’s Stu Carol’s Stu Social Media f f f 📜

What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 Alice’s Stu Bob’s Stu Music Player f f f 🎶 Carol’s Stu Social Media 📜

What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 Auth Service Alice’s Stu Bob’s Stu Music Player f f f 🎶 Carol’s Stu Social Media 📜 💂

What If… 🤔 Networked Data, Not Apps Photo Gallery 🖼 Alice’s Stu Bob’s Stu Music Player f f f 🎶 Carol’s Stu Social Media 📜

What If… 🤔 Open Protocols Everywhere

What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation Networking 🔌

What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… Networking 🔌

What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… • Auth: interop without pre-negotiation Auth 🎟 Networking 🔌

What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… • Auth: interop without pre-negotiation • Data: ubiquitous “dumb” storage Data 💾 Auth 🎟 Networking 🔌

What If… 🤔 Open Protocols Everywhere • HTTP being open lead to tons of innovation • Open protocols for… • Auth: interop without pre-negotiation • Data: ubiquitous “dumb” storage • Compute: local & remote lambdas Compute ⚙ Data 💾 Auth 🎟 Networking 🔌

What If… 🤔 Three Techniques

What If… 🤔 Three Techniques Global Primary Keys 🗺 Portable Private Data 💾 BFT Concurrency 🔀

What If… 🤔 Three Techniques Global Primary Keys 🗺 Content Addressing Portable Private Data 💾 BFT Concurrency 🔀

What If… 🤔 Three Techniques Global Primary Keys 🗺 Portable Private Data 💾 Content Addressing Cryptrees / Dark Forest BFT Concurrency 🔀

What If… 🤔 Three Techniques Global Primary Keys 🗺 Portable Private Data 💾 BFT Concurrency 🔀 Content Addressing Cryptrees / Dark Forest Hash-Linked CRDTs

Content Addressing Truly Global Links 🗺

Content Addressing 🗺

Content Addressing 🗺 The limitation of local knowledge is the fundamental fact about the setting in which we work, and it is a very powerful limitation – Nancy Lynch, A Hundred Impossibility Proofs for Distributed Computing

Content Addressing 🗺 Addressing Stack

Content Addressing 🗺 Addressing Stack Physical Location 🗺 send(42.123.45.6, path) = content1 send(42.123.45.6, path) = content2

Content Addressing 🗺 Addressing Stack Virtual Address 📬 Physical Location 🗺 {DNS: IP} send(42.123.45.6, path) = content1 send(42.123.45.6, path) = content2

Content Addressing 🗺 Addressing Stack Content ID ☃ Virtual Address 📬 Physical Location 🗺 {hash(content): content} {DNS: IP} send(42.123.45.6, path) = content1 send(42.123.45.6, path) = content2

Content Addressing 🗺 Hash-Based Relationships

Content Addressing 🗺 Hash-Based Relationships { } Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] }

Content Addressing 🗺 Hash-Based Relationships { } { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] } } Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb…}, {name: “about”, hash: Qm0eN…} ] }

Content Addressing 🗺 Hash-Based Relationships { } { Qm123456…: { data: nil, links: [ {name: “company”, hash: Qmabc…} {name: “industry”, hash: Qmzyx…} ] } } Qmabcdef…: { data: “Fission”, links: [ {name: “city”, hash: Qm1gb…}, {name: “about”, hash: Qm0eN…} ] } Qm123456…/company/about/ceo => “Boris Mann”

Content Addressing 🗺 Hard & Soft Links

Content Addressing 🗺 Hard & Soft Links alice.fission.name

Content Addressing 🗺 Hard & Soft Links alice.fission.name bob.fission.name

Content Addressing 🗺 Hard & Soft Links alice.fission.name Hard Links New for the web! Direct reference 2 pointers ~ deduplicate bob.fission.name

Content Addressing 🗺 Hard & Soft Links alice.fission.name Hard Links New for the web! Direct reference 2 pointers ~ deduplicate bob.fission.name Soft Links Symlink or web link (URL) Pointer ~ latest Latest may break

Content Addressing 🗺 Content Addressed Storage Is Simple

Content Addressing 🗺 Upshot • Names that leak no additional info • A consistent way to reference data • Self-verifying data for low/no-trust environment • Hard links on the web

Private Data Substrate Distributed Storage When The Pipes Are Broken 💾

Private Data Substrate 💾 Virtual Files & Directories

Private Data Substrate 💾 Virtual Files & Directories File Node Raw Data Metadata

Private Data Substrate 💾 Virtual Files & Directories File Node Virtual Node Raw Data Consistent interface Arbitrary metadata Tags, creators, MIME, sources, &c Metadata

Private Data Substrate 💾 Virtual Files & Directories File Node Virtual Node Raw Data Consistent interface Arbitrary metadata Tags, creators, MIME, sources, &c Metadata Directory Node Index Metadata

Private Data Substrate 💾 Persistent Versioning

Private Data Substrate 💾 Persistent Versioning Photos@r0 Vacation Avatars@r0 beach.png caricature.jpg Revision 0

Private Data Substrate 💾 Persistent Versioning Photos@r0 Vacation Avatars@r0 beach.png caricature.jpg Revision 0 🕙

Private Data Substrate 💾 Persistent Versioning Photos@r0 Photos@r1 Vacation Avatars@r0 Avatars@r1 beach.png caricature.jpg headshot.png Revision 0 Revision 1 🕙

Private Data Substrate 💾 Persistent Versioning Photos@r0 ChildHasEvent Vacation Avatars@r0 beach.png caricature.jpg Photos@r1 InsertNew Revision 0 headshot.png Revision 1 🕙 Avatars@r1

Private Data Substrate 💾 It’s All Just Maps

Private Data Substrate 💾

Private Data Substrate 💾 Cryptography is a tool for turning lots of different problems into key management problems Dr. Lea Kissner, Google’s Global Lead of Privacy Technologies

Private Data Substrate 💾 SubtleCrypto (WebCrypto) Source: caniuse.com

Private Data Substrate 💾 Non-Extractable Keys

Private Data Substrate 💾 Non-Extractable Keys 🗝

Private Data Substrate 💾 Non-Extractable Keys 🦹 🗝

Private Data Substrate 💾 Non-Extractable Keys 🦹 🏰🏰🏰🏰🏰🏰🏰 🗝

Private Data Substrate 💾 Non-Extractable Keys 🦹 📝 🏰🏰🏰🏰🏰🏰🏰 🗝

Private Data Substrate 💾 Non-Extractable Keys 🦹 🏰🏰🏰🏰🏰🏰🏰 📝 🗝

Private Data Substrate 💾 Non-Extractable Keys 🦹 📝 🏰🏰🏰🏰🏰🏰🏰 🗝

Private Data Substrate 💾 Self-Authenticating & Self-Authorizing

Private Data Substrate 💾 Self-Authenticating & Self-Authorizing 123 ACF CF4 C4A 0FC 1F3 A83 ED2 D55 823 247 81D F0A B92

Private Data Substrate 💾 Self-Authenticating & Self-Authorizing A7B ACF CF4 👩💻 C4A 0FC 1F3 A83 ED2 D55 823 247 81D F0A B92

Private Data Substrate 💾 Offline Access Control / Photos/ Sunriver/ food.jpg Iceland/ stage.png Docs/ notes.md harpa.svg

Private Data Substrate 💾 Offline Access Control / 🗝 Photos/ Sunriver/ food.jpg Iceland/ stage.png Docs/ notes.md harpa.svg

Private Data Substrate 💾 Offline Access Control / 🗝 Photos/ Sunriver/ Iceland/ Docs/ notes.md 🗝 food.jpg stage.png harpa.svg

Private Data Substrate 💾 Offline Access Control / 🗝 Photos/ Docs/ 🗝 Sunriver/ Iceland/ notes.md 🗝 food.jpg stage.png harpa.svg

Private Data Substrate 💾 Offline Access Control / / 🗝 Photos/ Docs/ Photos/ 🗝 🗝 Sunriver/ food.jpg Iceland/ stage.png notes.md harpa.svg pdx.jpg plane.png Docs/

Private Data Substrate 💾 Offline Access Control Photos/ Sunriver/ food.jpg stage.png notes.md pdx.jpg plane.png

Private Data Substrate 💾 Offline Access Control Shared Sunriver/ food.jpg stage.png Photos/ notes.md pdx.jpg plane.png

Private Data Substrate 💾 Private Nodes

Private Data Substrate 💾 Private Nodes Binary Encrypted Node 🔒 Encrypted Node 🔒 Encrypted Node 🔒

Private Data Substrate 💾 Private Nodes Binary CBOR Encrypted Node 🔒 Virtual Node Encrypted Node 🔒 Index Encrypted Node 🔒 🔑 🔑 Metadata

Private Data Substrate 💾 Private Nodes Binary CBOR Encrypted Node 🔒 AES256 Encrypted Node 🔒 Virtual Node 🔑 Index Encrypted Node 🔒 🔑 🔑 Metadata

Private Data Substrate 💾 Reconstruction Virtual Node Virtual Node 🔑 Index 🔑 🔑 Metadata 🔑 🔑 🔑 Index Metadata

Private Data Substrate 💾 Dark Forest

Fault-Safe Concurrency Safely Merge Unreliable Updates 🔀

Fault-Safe Concurrency 🔀 Causal Islands 🏖🏝

Fault-Safe Concurrency 🔀 Gossiping Out of Order

Fault-Safe Concurrency 🔀 Gossiping Out of Order 🎨

Fault-Safe Concurrency 🔀 Gossiping Out of Order t→ 🎨

Fault-Safe Concurrency 🔀 Gossiping Out of Order 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡

Fault-Safe Concurrency 🔀 Gossiping Out of Order 🟢 🎨 🔴 🟡 🔴 t→ 🟢 🟢 🟡 ❎

Fault-Safe Concurrency 🔀 Entropy Isn’t What It Used to Be

Fault-Safe Concurrency 🔀 Properties Save the Day

Fault-Safe Concurrency 🔀 Properties Save the Day Commutative

Fault-Safe Concurrency 🔀 Properties Save the Day Commutative merge merge

Fault-Safe Concurrency 🔀 Properties Save the Day Commutative merge merge Associative

Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative merge merge merge merge

Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge

Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge ( ) merge

Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge ( ) merge merge

Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge

Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill

Fault-Safe Concurrency 🔀 Properties Save the Day Associative Commutative ( merge merge ) merge merge Idempotent ( ) merge merge fill fill

Fault-Safe Concurrency 🔀 Temporal Confluence

Fault-Safe Concurrency 🔀 Temporal Confluence BFT-CRDTs Persistent data structure Automatic file-level reconciliation Pluggable sub-file reconciliation (forthcoming) Basis of upcoming BFT Datalog “at scale” work 😉

Fault-Safe Concurrency 🔀 Temporal Confluence BFT-CRDTs Persistent data structure Automatic file-level reconciliation Pluggable sub-file reconciliation (forthcoming) Basis of upcoming BFT Datalog “at scale” work 😉 Single File History / “Causal Shadow”

Fault-Safe Concurrency 🔀 Temporal Confluence BFT-CRDTs F B ⭐E⭐ Persistent data structure Automatic file-level reconciliation D Pluggable sub-file reconciliation (forthcoming) Basis of upcoming BFT Datalog “at scale” work 😉 A ⭐C⭐ Single File History / “Causal Shadow”

Fault-Safe Concurrency 🔀 Multiplayer Docs (Incl. Encryption) F B ⭐E⭐ D A ⭐C⭐

Fault-Safe Concurrency 🔀 Alternate History, Same State

Fault-Safe Concurrency 🔀 Alternate History, Same State null

Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶 😺 null 🐙

Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐶 😺 null 🐙

Fault-Safe Concurrency 🔀 Alternate History, Same State 😺🐙 🐶😺 🐶 😺 null 🐙

Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙 😺🐙 🐶😺 🐶 😺 null 🐙

Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙 😺🐙 🐶😺 🐶 😺 null 🐙 🐙🦋

Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙 😺🐙 🐶😺 🐶 😺🐙🦋 😺 null 🐙 🐙🦋

Fault-Safe Concurrency 🔀 Alternate History, Same State 🐶😺 🐙🦋 🐶😺 🐙 😺🐙 🐶😺 🐶 😺🐙🦋 😺 null 🐙 🐙🦋

Wrap Up Thoughts & Field Notes 🎁

Wrap Up 🎁 Requirements Diagram Next Gen Distributed Apps Web 2.0 Features Collaborative Portable Data & Runtime Hostile Hosts e.g. Browsers Multiuser Nonextractable Signing Keys Concurrent Local First Access Control Authenticated Data Content Addressed Authorized Data Encryption at Rest Capability Auth Extensible Arbitrary Metadata Kernel vs Userspace

Wrap Up 🎁 Better Together

🏔🎉 Thank You, CascadiaJS 🇨🇦🇺🇸 github.com/ucan-wg github.com/wnfs-wg More in-depth at Strange Loop 2022 (or talk to us today)