Security - more than an operations topic!

A presentation at Open Rhein Main 2019 in September 2019 in Darmstadt, Germany by Alexander Reelsen

Slide 1

Slide 1

Security … more than an operations topic Alexander Reelsen @spinscale alex@elastic.co

Slide 2

Slide 2

Who owns application security? 🔧 Operations? 🧨 Developers? 🙈 Intruder? 👮 CISO? 💯 All of the above - and more…

Slide 3

Slide 3

Internal Stakeholders Application Developers Platform Administrators 3rd Party Integrators Quality Management Compliance & Legal

Slide 4

Slide 4

Internal Stakeholders Application Developers Platform Administrators 3rd Party Integrators Quality Management Compliance & Legal today’s topic

Slide 5

Slide 5

Slide 6

Slide 6

Slide 7

Slide 7

today’s topic

Slide 8

Slide 8

Elasticsearch in 10 seconds Search Engine (FTS, Analytics, Geo), real-time Distributed, scalable, highly available, resilient Interface: HTTP & JSON Centrepiece of the Elastic Stack Uneducated conservative guess: Tens of thousands of clusters worldwide, hundreds of thousands of instances

Slide 9

Slide 9

Today’s goal Improve security in your own apps!

Slide 10

Slide 10

Naming is hard Security vs. Safety vs. Resiliency Do not run as root Integrity checks OutOfMemoryException System.exit Stop writing data before running out of disk space

Slide 11

Slide 11

Agenda Sandboxing your code Sandboxing others people’s code Prevent system call executions Ensure a smooth ride into production

Slide 12

Slide 12

Sandbox Sandboxing all the code!

Slide 13

Slide 13

What is a sandbox? connect 192.168.1.1:9300 Your code write /var/log/elasticsearch.log unlink /var/lib/elasticsearch/… ✅ ✅ ✅

Slide 14

Slide 14

What is a sandbox? open /etc/passwd Your code connect bitcoin-miner.foo.bar unlink /var/lib/elasticsearch ⛔ ⛔ ⛔

Slide 15

Slide 15

What is a sandbox? sandbox ✅ Your code ⛔

Slide 16

Slide 16

Sandbox my own code?! Expect your code to be exploited Prevent unknown attack vectors Is it really your code being executed Simple security model Blocklist vs. allowlist

Slide 17

Slide 17

Java Security Manager Java Security Manager Java Program Policy

Slide 18

Slide 18

Java Security Manager Java Security Manager Policy FilePermission read /etc/elasticsearch Java Program FilePermission write /var/log/elasticsearch SocketPermission connect *

Slide 19

Slide 19

Java Security Manager Java Security Manager Policy Java Program

Slide 20

Slide 20

Java Security Manager Java Security Manager Policy Java Program

Slide 21

Slide 21

Java Security Manager Java Security Manager Policy Java Program

Slide 22

Slide 22

Java Security Manager Java Security Manager Policy Java Program

Slide 23

Slide 23

Java Security Manager Java Security Manager Policy Java Program

Slide 24

Slide 24

java.io.File

Slide 25

Slide 25

java.lang.SecurityManager

Slide 26

Slide 26

Security Manager Summary Extensible Requires knowledge of code execution within your dependencies! Many dependencies are not tested with the security manager, resulting in unknown code paths executions No OOM protection! No stack overflow protection! No protection against java agents

Slide 27

Slide 27

Elasticsearch & the security manager Elasticsearch plugins core modules

Slide 28

Slide 28

Elasticsearch & the security manager Elasticsearch plugins core modules modules & plugins lang-mustache lang-painless transport-netty4 repository-azure analysis-icu

Slide 29

Slide 29

Plugins & modules plugins are just zip files each can have its own jars/dependencies each is loaded with its own classloader each can have its own security permissions

Slide 30

Slide 30

Plugins & modules Java Security Manager Policy Elasticsearch Plugin

Slide 31

Slide 31

Plugins & modules Java Security Manager Elasticsearch Plugin Policy

Slide 32

Slide 32

Plugins & modules Elasticsearch Module Elasticsearch Plugin Policy Policy Elasticsearch Module Policy

Slide 33

Slide 33

Elasticsearch startup JVM Startup time

Slide 34

Slide 34

JVM Startup Elasticsearch startup time Read configuration file

Slide 35

Slide 35

time Read configuration file JVM Startup Elasticsearch startup Native system calls

Slide 36

Slide 36

time Native system calls Read configuration file JVM Startup Elasticsearch startup Set security manager

Slide 37

Slide 37

time Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Load plugins

Slide 38

Slide 38

time Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Bootstrap checks

Slide 39

Slide 39

time Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Network enabled

Slide 40

Slide 40

time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup

Slide 41

Slide 41

#noroot there is no reason to run code as root!

Slide 42

Slide 42

time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Do not run as root

Slide 43

Slide 43

Do not run as root

Slide 44

Slide 44

seccomp … or how I loved to abort system calls

Slide 45

Slide 45

time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Seccomp - prevent process forks

Slide 46

Slide 46

Seccomp - prevent process forks Security manager could fail Elasticsearch should still not be able to fork processes One way transition to tell the operating system to deny execve, fork, vfork, execveat system calls Works on Linux, Windows, Solaris, BSD, osx

Slide 47

Slide 47

Seccomp - prevent process forks

Slide 48

Slide 48

Seccomp - prevent process forks

Slide 49

Slide 49

seccomp sandbox seccomp ✅ Your code ⛔

Slide 50

Slide 50

bootstrap checks Annoying you now instead of devastating you later

Slide 51

Slide 51

time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Bootstrap checks

Slide 52

Slide 52

Bootstrap checks

Slide 53

Slide 53

Bootstrap checks

Slide 54

Slide 54

Bootstrap checks

Slide 55

Slide 55

bonus: ease-of-use don’t forget your users…

Slide 56

Slide 56

Mark sensitive settings

Slide 57

Slide 57

Register all your settings

Slide 58

Slide 58

Summary Security is hard - let’s go shopping!

Slide 59

Slide 59

Summary Development has big impact on security Operations is happy to help what is there out of the box Developers know their application best! Don’t reinvent, check out existing features! Developers are responsible for writing secure code! Before something happens!

Slide 60

Slide 60

Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co

Slide 61

Slide 61

Resources https://github.com/elastic/elasticsearch/ https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating https://www.elastic.co/blog/scripting https://www.elastic.co/blog/scripting-security https://docs.oracle.com/javase/9/security/toc.htm https://docs.oracle.com/javase/9/security/permissions-java-development-kit.htm https://www.elastic.co/blog/seccomp-in-the-elastic-stack

Slide 62

Slide 62

Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co