A presentation at Velocity Conference in in Berlin, Germany by Kelly Shortridge
C ONTROLLED C HAOS The Inevitable Marriage of DevOps & Security Kelly Shortridge (@swagitda_) Velocity Berlin 2019
Hi, I’m Kelly 2 @swagitda_
“Chaos isn’t a pit. Chaos is a ladder.” ― Petyr Baelish, Game of Thrones 3 @swagitda_
Infosec has a choice: marry DevOps or be rendered impotent & irrelevant 4 @swagitda_
Infosec won’t survive in a silo. It must be embedded in software delivery. 5 @swagitda_
DevOps can learn to carve its own path to secure software delivery @swagitda_
How can controlling chaos create a marriage of infosec and DevOps? 7 @swagitda_
Chaos Theory
Chaos engineering = continual experimentation to test resilience @swagitda_
“Things will fail” naturally extends into “things will be pwned” 11 @swagitda_
Security failure is when security controls don’t operate as intended 12 @swagitda_
What are the principles of chaotic security engineering? 13 @swagitda_
Game days: like planned firedrills 16 @swagitda_
Prioritize security game days based on potential business impacts 17 @swagitda_
Decision trees: start at target asset, work back to easiest attacker paths 18 @swagitda_
Determine the attacker’s least-cost path (hint: it doesn’t involve 0day) 19 @swagitda_
Your goal is to raise the cost of attack, ideally beginning at design 20 @swagitda_
Time to D.I.E.
We need a model promoting qualities that make systems more secure 22 @swagitda_
Enter the D.I.E. model by Sounil Yu: Distributed, Immutable, Ephemeral 23 @swagitda_
Distributed: multiple systems supporting the same overarching goal 24 @swagitda_
Distributed infrastructure reduces risk of DoS attacks by design 25 @swagitda_
A service mesh is like an on-demand VPN at the application level 26 @swagitda_
Attackers are forced to escalate privileges to access the iptables layer 27 @swagitda_
Immutable: infrastructure that doesn’t change after it’s deployed 28 @swagitda_
Immutable infra is more secure by design – ban shell access entirely 29 @swagitda_
Patching is no longer a nightmare with version-controlled images @swagitda_
Ephemeral: infrastructure with a very short lifespan (dies after a task) 31 @swagitda_
Ephemerality creates uncertainty for attackers (persistence = nightmare) 32 @swagitda_
Installing a rootkit on a resource that dies in minutes is a waste of effort 33 @swagitda_
Optimizing for D.I.E. reduces risk by design & supports resilience 34 @swagitda_
A Phoenix Rises
Begin with “dumb” testing before moving to “fancy” testing 36 @swagitda_
D.I.E.ing is an art, like everything else @swagitda_
Controlling Chaos: Distributed 38 @swagitda_
Distributed is mostly covered by the existing repertoire of chaos eng tools 39 @swagitda_
Repurpose these tools, but make attackers the source of failure 40 @swagitda_
Multi-region services present a fun opportunity to mess with attackers 41 @swagitda_
Shuffle IP blocks regularly to change attackers’ lateral movement game 42 @swagitda_
Test: inject failure into your service mesh to test authentication controls 43 @swagitda_
Controlling Chaos: Immutable 44 @swagitda_
Immutable infra is like a phoenix – it disappears & comes back a lot 45 @swagitda_
Volatile environments with continually moving parts raise the cost of attack 46 @swagitda_
Create rules like, “If there’s ever a write to disk, crash the node” 47 @swagitda_
Attackers must stay in-memory, which hopefully makes them cry 48 @swagitda_
Bonus: disallowing all local IO improves service reliability 49 @swagitda_
Metasploit Meterpreter + webshell: Touch passwords.txt & kaboom 50 @swagitda_
Build your Docker images with a garbage-filled “bamboozle layer” 51 @swagitda_
Mark garbage files as “unreadable” to craft enticing bait for attackers 52 @swagitda_
A potential goal: architect immutability turtles all the way down 53 @swagitda_
Test: inject attempts at writing to disk to ensure detection & reversion 54 @swagitda_
Treat changes to disk by adversaries similarly to failing disks: mercy kill 55 @swagitda_
Controlling Chaos: Ephemeral 56 @swagitda_
Most infosec bugs are stated-related – get rid of state, get rid of bugs 57 @swagitda_
Reverse uptime: longer host uptime adds greater security risk 58 @swagitda_
Test: change API tokens & test if services still accept old tokens 59 @swagitda_
Test: retrograde libraries, containers, other resources in CI/CD pipelines 60 @swagitda_
Test: inject hashes of old pieces of data to ensure no data persistence 61 @swagitda_
Leverage lessons from toll fraud – cloud billing becomes security signal 62 @swagitda_
Test: exfil TBs or run a cryptominer to inform billing spike detection 63 @swagitda_
Conclusion
Chaos/resilience are natural homes for infosec & represent its future. 65 @swagitda_
The future of infosec involves unified responsibility & accountability. 66 @swagitda_
Security can be innovative and fuel the engine of business as well. 67 @swagitda_
“You must have chaos within you to give birth to a dancing star.” ― Friedrich Nietzsche 68 @swagitda_
@swagitda_ /in/kellyshortridge kelly@greywire.net 69 @swagitda_
PDF: https://swagitda.com/speaking/Controlled-Chaos-Inevitable-Marriage-DevOps-Security-Shortridge-Velocity-2019.pdf
Software is eating the world, and security will be eaten as well if it doesn’t evolve. Kelly Shortridge exposes why chaos and resilience engineering represents the future of security programs—and why it catalyzes the dawn of defensive innovation. You’ll examine how adopting distributed, immutable, and ephemeral infrastructure (the “DIE” triad) can create powerful security benefits.
Finally, focusing on practical implementation, you’ll delve into the art of harnessing chaos for security and how to conduct security tests within the DIE model. Through this marriage of security and DevOps, security can transform from a frustrating cost center to a lean, mean, innovation machine that supports software delivery performance.
The following resources were mentioned during the presentation or are useful additional information.
Check out the “Decision Trees” section
Yu, Sounil (2019). Distributed Immutable Ephemeral - New Paradigms for the Next Era of Security.
Here’s what was said about this presentation on social media.
for free. You
can too.