Keynote — Illuminating the Darkest of Magics: Toward a Disipline of Library Design

I L L U M I N AT I N G T H E DARKEST OF MAGICS ✨ TOWARDS A DISCIPLINE OF LIBRARY DESIGN “

Like punning, programming is a play on words ” AL AN J. PERLIS

Experience is the name everyone gives to their mistakes # O S C A R W I L D E , “L A DY W I N D E R M E R E ’ S FA N “

M E TA T A L K A B O U T TA L K

M E TA B R O O K LY N Z E L E N K A @EXPEDE CEO & Chief Scientist at FISSION $ https://tools.fission.codes Ethereum Core Dev % Authored a bunch of standards Previously Elixir consultant & trainer Witchcraft, Algae, Quark, Exceptional & Open Sourceress ’ PLT Enthusiast λ Founded VanFP

M E TA B R O O K LY N Z E L E N K A @EXPEDE CEO & Chief Scientist at FISSION $ https://tools.fission.codes Ethereum Core Dev % Authored a bunch of standards Previously Elixir consultant & trainer Witchcraft, Algae, Quark, Exceptional & Open Sourceress ’ PLT Enthusiast λ Founded VanFP T E G E M CO KERS! C I T S

M E TA STRUCTURE ( • Top-down: general to concrete 1. Modes of Thought 2. Library Principles 3. Library Pragmatics • Goals • Turn intuitive design into explicit choices • Learn from my mistakes experiences!

M E TA WHY FOCUS ON LIBRARIES? ” Language Theory Library Design Application Design

M E TA H I G H L E V E L C O D E I L L U S T R AT I O N S O N LY

PA R T I MODES OF THOUGHT S E T T I N G T H E S TA G E )

The limits of my language mean the limits of my world

• LUDWIG WIT TGENSTEIN

MODES OF THOUGHT TOOLS > BIOLOGY +,- • Possibly use our minds less than our ancestors • Better mental tools = lower cognitive load ., • Metric > Imperial • Arabic numerals > Roman numerals • 24-hours and 360-degrees have nice divisors Because it’s easier now Flavours /0 1. Algorithms (“how”) 1 Much of undergrad CS ⚙ Mechanistic ➗ e.g. Long division 2. Abstraction (“what”) 4 A lot of language & UX 5 e.g. Space & geometry 6 e.g. Analogy & metaphor ” e.g. Narrative

MODES OF THOUGHT MODES OF THOUGHT ⚙6 •→• ↓ ↓ •→• • Operational / Mechanistic • Denotational / Equational • Humanist / Linguistic / HCI 7 8

MODES OF THOUGHT W H AT ’ S S O G R E A T A B O U T F P A N Y W AY ? 9 • Modularity • Composition • Can describe general solutions via abstraction! • Make the large feel like the small & fewer patterns to memorize : • Maleable code

MODES OF THOUGHT O P E R AT I O N A L V S D E N O TAT I O N A L T H I N K I N G • Enum.map/2 can be seen through the lens of… • Collections • A cleaner for loop (operational) • Run a function on every item in a structure • Programs • Transforms one program into a new one ;

MODES OF THOUGHT FUNCTIONAL ALCHEMY ⚗ @spec long_form(integer()) :: String.t() Enum.map/2 @spec long_forms([integer()]) :: [String.t()]

MODES OF THOUGHT FUNCTIONAL ALCHEMY ⚗ integer() String.t() Enum.map/2 [integer()] [String.t()]

MODES OF THOUGHT W H AT ’ S S O T E R R I B L E A B O U T F P A N Y W AY ? = • The Church Chasm • If you have a hammer, everything looks like a nail • “Use anonymous functions for everything! Aren’t closures amazing?!” • Can focus on mechanics rather than concepts • Explicit state = lots of explicit handling, “pass the world”, &c • A tendency towards cleverness

MODES OF THOUGHT ACTOR ABYSS > • Each step is very simple • Reasoning about dynamic organisms is hard • Remember to store your data for crash recovery • Called collaborator may not be there • Exactly why frameworks & GenStage are great! • Abstract this all away for many use cases • More concrete than arrows & better matches the Elixir ethos • Complexity grows faster than linear • Takeaway: provide structured abstractions

MODES OF THOUGHT {:tarpit, “tradeoffs”} • Surface simplicity has a cost • Complex in the large • Restricted domain in the small

PA R T I I LIBRARY PRINCIPLES ” LIB LONG AND PROSPER ?

The utility of a language as a tool of thought increases with the range of topics it can treat, but decreases with the amount of vocabulary and the complexity of grammatical rules which the user must keep in mind @ KENNETH IVERSON

LIBRARY PRINCIPLES ” C L A S S I F I C AT I O N A Category Data Representation / Interface Domain Specific Language (DSL) Example Poison, ExAws Ecto, Algae, Absinthe Reusable Patterns Enum, Quark, Exceptional, Witchcraft Fe a t u re E mu l a t i o n TypeClass, Credo, Dialyxir Framework / System Phoenix, Nerves

LIBRARY PRINCIPLES ” SPECTRUM B m o i d I Wide n La a u g e g L S D n k r o i e o t c a n w a r c e f i e r l t m e t p t a a p r n I P A F Narrow

LIBRARY PRINCIPLES ” R E L AT I O N S H I P S C Reusable Patterns Framework & System Representation & Interface Feature Emulation DSL

LIBRARY PRINCIPLES ” R E L AT I O N S H I P S C Reusable Patterns Framework & System Representation & Interface Feature Emulation DSL

LIBRARY PRINCIPLES ” I, LIBRARY D •Ethos •Generality •Orthogonality •Independence •Completeness •Extensibility

LIBRARY PRINCIPLES ” ETHOS T H E W I S D O M TO B A L A N C E C O N S I ST E N CY, C H A N G E , A N D R E ST R A I N T “

LIBRARY PRINCIPLES ” ETHOS A library must remain consistent with the character of the host language and broadest community standards.

LIBRARY PRINCIPLES ” ETHOS C H A R A C T E R M O T I V AT I O N E • Two styles of Scala 1. Java without the braces 2. Haskell fan fiction • “I’d need to convert my entire project for this lib to make sense” F • Seen in production with Ruby, Swift, JS, and more • It doesn’t have to be this way -

LIBRARY PRINCIPLES ” ETHOS A M AT T E R O F C H A R A C T E R • Ethos — noun. (/ˈiːθɒs/ or US: /ˈiːθoʊs/) The fundamental character or spirit of a culture; the underlying sentiment that informs the beliefs, customs, or practices of a group or society; dominant assumptions of a people or period • Like all interesting things, it fuzzy • Feel as close to the host language as possible • But also blending in features from a target

LIBRARY PRINCIPLES ” ETHOS A “STRONG PERSONALITY” ; • Strong character: C, Haskell, Rust, Elm, Golang • Erlang designed for practicality (telecom switches) • Elixir has no clear language-level raison d’être • It’s like tofu: very easy to mix flavours in!

LIBRARY PRINCIPLES ” ETHOS T H E P I P E S S T R AT E G Y • A break in ethos from Erlang • Fits Elixir’s strategy • Tradeoffs • Operational/mechanistic mode of thought • Clean • Cleans up a lot of code • Easy to follow • Focus or “zoom” effect G • Similar to OO’s fluent interfaces • Zooming doesn’t really apply to actors • Helps to tell a story • Takeaway: consider you metaphors

LIBRARY PRINCIPLES ” ETHOS CASE STUDY: Exceptional’S PIPES • Concept: Flow-ability is very core to Elixir’s ethos • Kernel.|>/2 • Consistent flow metaphor / punning on existing metaphor • Exceptional: ~>/2 and >>>/2

LIBRARY PRINCIPLES ” ETHOS E D G E C A S E — G U E S T L A N G U A G E S Y N TA X E S • Which language is this? • Does it matter?

LIBRARY PRINCIPLES ” GENERALITY THE FLEXIBILITY TO WORK ANYWHERE

LIBRARY PRINCIPLES ” GENERALITY Each entity must be useful in as many scenarios as possible, but no more.

LIBRARY PRINCIPLES ” GENERALITY SWEE T SPOTS H Generality Power • Low information • High information • Few assumptions • Can make many assumptions • Many use cases • Tailored to few use cases GenServer Enum Libraries GENERALITY I Ecto.Schema ⚖ POWER - Applications

LIBRARY PRINCIPLES ” GENERALITY … T O O FA R ? K

LIBRARY PRINCIPLES ” INDEPENDENCE THE COURAGE TO VENTURE ALONE ⚔

LIBRARY PRINCIPLES ” INDEPENDENCE Each unit of code must be able to be taken on its own. It must not break nor alter the runtime semantics of either enclosed or exterior code.

LIBRARY PRINCIPLES ” INDEPENDENCE I N D E P E N D E N C E D AY M • Similar to referential transparency • Especially important with FP assumptions • Each chunk of code (horizontal or vertical) should be comprehensible on its own • Most common issues: • Behaviour depending on implicit or hidden state • Macro rewriting magic

LIBRARY PRINCIPLES ” INDEPENDENCE CASE STUDY: Algae All the features!

LIBRARY PRINCIPLES ” INDEPENDENCE CASE STUDY: Algae All the features! 42 77 1234 98 32

LIBRARY PRINCIPLES ” INDEPENDENCE CASE STUDY: Algae • Concept: Bootstrap structs into ADTs • Removes • Takeaways: • Composition • Nothing • Adds • Orthogonality • Structure DSL • Structured abstraction • Types with automatic default values • Turns into vanilla Elixir • Manual default values • Constructor helpers (e.g. Foo.new)

LIBRARY PRINCIPLES ” ORTHOGONALITY T H E Z E N O F U N D E R S TAT E M E N T

LIBRARY PRINCIPLES ” ORTHOGONALITY Each entity must be the best at a specific task. It must not conflict with other functions.

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R N

LIBRARY PRINCIPLES ” ORTHOGONALITY G E O M E T R I C M E TA P H O R N Components: 4 Results: effectively limitless

LIBRARY PRINCIPLES ” ORTHOGONALITY M O R E T H A N T H E S U M O F I T S PA R T S O • Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values! • More focused (does one thing) • More general (works everywhere) • Ad hoc function extension • Can we take this further? P

LIBRARY PRINCIPLES ” ORTHOGONALITY AN Exceptional EXAMPLE Q Works everywhere Any data Any error struct Abstracted out Q foo!/* from foo/* S ⏩ Q Any flow (esp. pipes) Super easy to test BONUS Disambiguate between nil value and actual errors

LIBRARY PRINCIPLES ” COMPLETENESS THE POWER OF HARMONIOUS TOOLS T

LIBRARY PRINCIPLES ” COMPLETENESS A library must introduce or integrate its concept(s) as completely as possible.

LIBRARY PRINCIPLES ” COMPLETENESS CASE STUDY: Witchcraft • Can go pretty far with completeness • Update, clean, and power up Kernel functions • Curried, async, flipped • What about orthogonality? V variants

LIBRARY PRINCIPLES ” COMPLETENESS KEEPIN’ IT WITCHY W • Ported things with immediate synergy • Renamed many functions for clarity • Sometimes very difficult • Operator >>> aliases • Changed lots of argument orders • :pipable |> everything_is() ✅

LIBRARY PRINCIPLES ” COMPLETENESS KEEPIN’ IT WITCHY W • Ported things with immediate synergy • Renamed many functions for clarity • Sometimes very difficult • Operator >>> aliases • Changed lots of argument orders • :pipable |> everything_is()

LIBRARY PRINCIPLES ” EXTENSIBILITY THE FREEDOM TO DO MORE

LIBRARY PRINCIPLES ” COMPLETENESS A library should provide the ability to adapt its functionality to new contexts without altering the library itself.

LIBRARY PRINCIPLES ” EXTENSIBILITY LIBRARY-LE VEL EXTENSION • Witchcraft lib hierarchy was a very good idea • Wish I had gone further • e.g. Algae’s DSL vs instances • Balance how much each lib should do • Extension as usage example • Market as more reusable Y

PA R T I I I L I B R A R Y P R A G M AT I C S FA I L U R E M O D E S & L O W H A N G I N G F R U I T Z

The art of programming is the art of organizing complexity, of mastering multitude, and avoiding its bastard chaos as effectively as possible [ EDSGER DIJKSTRA

L I B R A R Y P R A G M AT I C S YOUR INNER MAD SCIENTIST \ If you feel like this you’ve probably created a monster

L I B R A R Y P R A G M AT I C S MIND TRAPS >] • More dangerous in libraries than application code • Great definitional power comes with great responsibility to your users! • Identify the way you tend to fail • e.g. My failure mode is to enforce correctness on others • KISS principle ^ • Common trap: getting too cute with function names and metaphors • Names should be descriptive of what they do, not just be clever • Other traps include: no tests, incomplete scope, not general enough, &c

L I B R A R Y P R A G M AT I C S DO NOT FORCE PRACTICES ON USERS _ • Overall very happy with TypeClass • But contains my biggest regret ` • Enforced prop testing at compile time • Slow • Detracts people from using the lib V • Wanted to encourage good practice • Please give user choice • Generate test helpers ✅

L I B R A R Y P R A G M AT I C S A L L T H E O P E R AT O R S a _ ✅ • Super duper handy to have b • Limited number of operators in Elixir • Operator-space collisions c • Handy if you’re user has one lib per application, but you don’t know • Always provide a named variant • Operator Y

L I B R A R Y P R A G M AT I C S T H E O P T I O N T O D I S A M B I G U AT E ✅ What does this do?

L I B R A R Y P R A G M AT I C S FA K E A S M U C H A S P O S S I B L E • Started out by wanting to do everything by hand _ • Save yourself and others the trouble: • Excellent idea to just fake things by bootstrapping • Vanilla Elixir! • Algae: “just” a vanilla struct DSL • TypeClass: “just” vanilla protocols (plus some enforced checking) • I repeat: it’s all vanilla Elixir underneath the hood

L I B R A R Y P R A G M AT I C S SMOKE & MIRRORS & • Leverage what’s already there! • Stays compatible with entire ecosystem • Less work for you Y

L I B R A R Y P R A G M AT I C S G R E AT A R T I S T S S T E A L d • Don’t reinvent the wheel: steal! g • Cat Theory → Haskell & PureScript & Elm &c → Witchcraft a • Clean up the type class hierarchy h • Haskell 6 • Purescript • Fantasy Land Specification • and others • Some pipings and naming • Elm Elixir e f ⚙ )

L I B R A R Y P R A G M AT I C S G R E AT A R T I S T S S T E A L d • Don’t reinvent the wheel: steal! g • Cat Theory → Haskell & PureScript & Elm &c → Witchcraft a • Clean up the type class hierarchy h • Haskell 6 • Purescript • Fantasy Land Specification • and others • Some pipings and naming • Elm Elixir e e 6 e e e f f f f f ⚙ )

L I B R A R Y P R A G M AT I C S BENCH ALL THE THINGS i • Sometimes truly surprising results (both good and bad) • Bench different data types and sizes • Run analyses against application-specific functions or Kernel • TL;DR Kernel functions are really fast! • Optimized & battle tested • Use in defimpls whenever possible

L I B R A R Y P R A G M AT I C S T E X T D I A G R A M S > I M A G E S > O N LY W O R D S > N O T H I N G

L I B R A R Y P R A G M AT I C S PRODUCT THINKING: ROI • People use libs because of they make their lives better • Tradeoffs suck: have as few as possible • Low effort / high return (for the user) • Make it as easy as possible to set up • e.g. use Exceptional • Doesn’t conflict with other libs, Kernel, &c

FINAL THOUGHTS j WRAPPING UP a

FINAL THOUGHTS BIG THEMES • Standard libs are not sacred • But all lib code should feel like it belongs there! • Just because you can doesn’t mean you should • Great artists steal • Library-as-language-extension (think “version x.1”) • Focus on cohesion • Love your code ❤

b THANK YOU, PRAGUE l H E L L O @ B R O O K LY N Z E L E N K A . C O M GITHUB.COM/EXPEDE @EXPEDE