A presentation at Open Rhein Main 2019 in in Darmstadt, Germany by Alexander Reelsen
Security … more than an operations topic Alexander Reelsen @spinscale alex@elastic.co
Who owns application security? 🔧 Operations? 🧨 Developers? 🙈 Intruder? 👮 CISO? 💯 All of the above - and more…
Internal Stakeholders Application Developers Platform Administrators 3rd Party Integrators Quality Management Compliance & Legal
Internal Stakeholders Application Developers Platform Administrators 3rd Party Integrators Quality Management Compliance & Legal today’s topic
today’s topic
Elasticsearch in 10 seconds Search Engine (FTS, Analytics, Geo), real-time Distributed, scalable, highly available, resilient Interface: HTTP & JSON Centrepiece of the Elastic Stack Uneducated conservative guess: Tens of thousands of clusters worldwide, hundreds of thousands of instances
Today’s goal Improve security in your own apps!
Naming is hard Security vs. Safety vs. Resiliency Do not run as root Integrity checks OutOfMemoryException System.exit Stop writing data before running out of disk space
Agenda Sandboxing your code Sandboxing others people’s code Prevent system call executions Ensure a smooth ride into production
Sandbox Sandboxing all the code!
What is a sandbox? connect 192.168.1.1:9300 Your code write /var/log/elasticsearch.log unlink /var/lib/elasticsearch/… ✅ ✅ ✅
What is a sandbox? open /etc/passwd Your code connect bitcoin-miner.foo.bar unlink /var/lib/elasticsearch ⛔ ⛔ ⛔
What is a sandbox? sandbox ✅ Your code ⛔
Sandbox my own code?! Expect your code to be exploited Prevent unknown attack vectors Is it really your code being executed Simple security model Blocklist vs. allowlist
Java Security Manager Java Security Manager Java Program Policy
Java Security Manager Java Security Manager Policy FilePermission read /etc/elasticsearch Java Program FilePermission write /var/log/elasticsearch SocketPermission connect *
Java Security Manager Java Security Manager Policy Java Program
Java Security Manager Java Security Manager Policy Java Program
Java Security Manager Java Security Manager Policy Java Program
Java Security Manager Java Security Manager Policy Java Program
Java Security Manager Java Security Manager Policy Java Program
java.io.File
java.lang.SecurityManager
Security Manager Summary Extensible Requires knowledge of code execution within your dependencies! Many dependencies are not tested with the security manager, resulting in unknown code paths executions No OOM protection! No stack overflow protection! No protection against java agents
Elasticsearch & the security manager Elasticsearch plugins core modules
Elasticsearch & the security manager Elasticsearch plugins core modules modules & plugins lang-mustache lang-painless transport-netty4 repository-azure analysis-icu
Plugins & modules plugins are just zip files each can have its own jars/dependencies each is loaded with its own classloader each can have its own security permissions
Plugins & modules Java Security Manager Policy Elasticsearch Plugin
Plugins & modules Java Security Manager Elasticsearch Plugin Policy
Plugins & modules Elasticsearch Module Elasticsearch Plugin Policy Policy Elasticsearch Module Policy
Elasticsearch startup JVM Startup time
JVM Startup Elasticsearch startup time Read configuration file
time Read configuration file JVM Startup Elasticsearch startup Native system calls
time Native system calls Read configuration file JVM Startup Elasticsearch startup Set security manager
time Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Load plugins
time Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Bootstrap checks
time Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Network enabled
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup
#noroot there is no reason to run code as root!
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Do not run as root
Do not run as root
seccomp … or how I loved to abort system calls
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Seccomp - prevent process forks
Seccomp - prevent process forks Security manager could fail Elasticsearch should still not be able to fork processes One way transition to tell the operating system to deny execve, fork, vfork, execveat system calls Works on Linux, Windows, Solaris, BSD, osx
Seccomp - prevent process forks
Seccomp - prevent process forks
seccomp sandbox seccomp ✅ Your code ⛔
bootstrap checks Annoying you now instead of devastating you later
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Bootstrap checks
Bootstrap checks
Bootstrap checks
Bootstrap checks
bonus: ease-of-use don’t forget your users…
Mark sensitive settings
Register all your settings
Summary Security is hard - let’s go shopping!
Summary Development has big impact on security Operations is happy to help what is there out of the box Developers know their application best! Don’t reinvent, check out existing features! Developers are responsible for writing secure code! Before something happens!
Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co
Resources https://github.com/elastic/elasticsearch/ https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating https://www.elastic.co/blog/scripting https://www.elastic.co/blog/scripting-security https://docs.oracle.com/javase/9/security/toc.htm https://docs.oracle.com/javase/9/security/permissions-java-development-kit.htm https://www.elastic.co/blog/seccomp-in-the-elastic-stack
Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co
Have you ever had a huge cloud provider bill? A sudden spike in CPU usage across many services? Did your infrastructure turn into bitcoin miners because someone found a way to execute arbitrary code in the application you wrote? Security is neither an operations nor a developer responsibility, but both!
What did you do after the incident? Did you close this particular security whole or did you fix this whole type of vulnerability? In order achieve the latter this talk will talk about using seccomp as an operating system feature as well as the Java Security Manager and how to leverage those as application developer in order to improve security and reduce the risk of getting hacked.
for free. You
can too.