Security - more than an operations topic!

Security … more than an operations topic Alexander Reelsen @spinscale alex@elastic.co

Who owns application security? 🔧 Operations? 🧨 Developers? 🙈 Intruder? 👮 CISO? 💯 All of the above - and more…

Internal Stakeholders Application Developers Platform Administrators 3rd Party Integrators Quality Management Compliance & Legal

Internal Stakeholders Application Developers Platform Administrators 3rd Party Integrators Quality Management Compliance & Legal today’s topic

today’s topic

Elasticsearch in 10 seconds Search Engine (FTS, Analytics, Geo), real-time Distributed, scalable, highly available, resilient Interface: HTTP & JSON Centrepiece of the Elastic Stack Uneducated conservative guess: Tens of thousands of clusters worldwide, hundreds of thousands of instances

Today’s goal Improve security in your own apps!

Naming is hard Security vs. Safety vs. Resiliency Do not run as root Integrity checks OutOfMemoryException System.exit Stop writing data before running out of disk space

Agenda Sandboxing your code Sandboxing others people’s code Prevent system call executions Ensure a smooth ride into production

Sandbox Sandboxing all the code!

What is a sandbox? connect 192.168.1.1:9300 Your code write /var/log/elasticsearch.log unlink /var/lib/elasticsearch/… ✅ ✅ ✅

What is a sandbox? open /etc/passwd Your code connect bitcoin-miner.foo.bar unlink /var/lib/elasticsearch ⛔ ⛔ ⛔

What is a sandbox? sandbox ✅ Your code ⛔

Sandbox my own code?! Expect your code to be exploited Prevent unknown attack vectors Is it really your code being executed Simple security model Blocklist vs. allowlist

Java Security Manager Java Security Manager Java Program Policy

Java Security Manager Java Security Manager Policy FilePermission read /etc/elasticsearch Java Program FilePermission write /var/log/elasticsearch SocketPermission connect *

Java Security Manager Java Security Manager Policy Java Program

Java Security Manager Java Security Manager Policy Java Program

Java Security Manager Java Security Manager Policy Java Program

Java Security Manager Java Security Manager Policy Java Program

Java Security Manager Java Security Manager Policy Java Program

java.io.File

java.lang.SecurityManager

Security Manager Summary Extensible Requires knowledge of code execution within your dependencies! Many dependencies are not tested with the security manager, resulting in unknown code paths executions No OOM protection! No stack overflow protection! No protection against java agents

Elasticsearch & the security manager Elasticsearch plugins core modules

Elasticsearch & the security manager Elasticsearch plugins core modules modules & plugins lang-mustache lang-painless transport-netty4 repository-azure analysis-icu

Plugins & modules plugins are just zip files each can have its own jars/dependencies each is loaded with its own classloader each can have its own security permissions

Plugins & modules Java Security Manager Policy Elasticsearch Plugin

Plugins & modules Java Security Manager Elasticsearch Plugin Policy

Plugins & modules Elasticsearch Module Elasticsearch Plugin Policy Policy Elasticsearch Module Policy

Elasticsearch startup JVM Startup time

JVM Startup Elasticsearch startup time Read configuration file

time Read configuration file JVM Startup Elasticsearch startup Native system calls

time Native system calls Read configuration file JVM Startup Elasticsearch startup Set security manager

time Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Load plugins

time Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Bootstrap checks

time Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Network enabled

time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup

#noroot there is no reason to run code as root!

time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Do not run as root

Do not run as root

seccomp … or how I loved to abort system calls

time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Seccomp - prevent process forks

Seccomp - prevent process forks Security manager could fail Elasticsearch should still not be able to fork processes One way transition to tell the operating system to deny execve, fork, vfork, execveat system calls Works on Linux, Windows, Solaris, BSD, osx

Seccomp - prevent process forks

Seccomp - prevent process forks

seccomp sandbox seccomp ✅ Your code ⛔

bootstrap checks Annoying you now instead of devastating you later

time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Bootstrap checks

Bootstrap checks

Bootstrap checks

Bootstrap checks

bonus: ease-of-use don’t forget your users…

Mark sensitive settings

Register all your settings

Summary Security is hard - let’s go shopping!

Summary Development has big impact on security Operations is happy to help what is there out of the box Developers know their application best! Don’t reinvent, check out existing features! Developers are responsible for writing secure code! Before something happens!

Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co

Resources https://github.com/elastic/elasticsearch/ https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating https://www.elastic.co/blog/scripting https://www.elastic.co/blog/scripting-security https://docs.oracle.com/javase/9/security/toc.htm https://docs.oracle.com/javase/9/security/permissions-java-development-kit.htm https://www.elastic.co/blog/seccomp-in-the-elastic-stack

Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co