A presentation at JavaDay Odesa in in Odesa, Odessa Oblast, Ukraine, 65000 by Alexander Reelsen
Elasticsearch Securing a search engine while maintaining usability Alexander Reelsen @spinscale alex@elastic.co
Elastic Stack
Elasticsearch in 10 seconds Search Engine (FTS, Analytics, Geo), near real-time Distributed, scalable, highly available, resilient Interface: HTTP & JSON Centrepiece of the Elastic Stack (Kibana, Logstash, Beats, APM, ML, App Search, Enterprise Search) Uneducated conservative guess: Tens of thousands of clusters worldwide, hundreds of thousands of instances
Agenda Security: Feature or non-functional requirement? Security Manager Production Mode vs. Development Mode Plugins Scripting language: Painless
Security Feature or non-functional requirement?
Software has to be secure! O RLY? Defensive programming Do not persist specific data (PCI DSS) Not exploitable (pro tip: not gonna happen) No unintended resource access (directory traversal) Least privilege principle Reduced impact surface (DoS) https://www.theregister.co.uk/2017/03/26/miele_joins_internetofst_hall_of_shame/ Security as a non-functional requirement
Security as a feature Authentication Authorization (LDAP, users, PKI) TLS transport encryption Audit logging SSO/SAML/Kerberos
Security or safety or resiliency? Integrity checks Preventing OOMEs Prevent deep pagination Do not expose credentials in cluster state/REST APISs Stop writing data before running out of disk space Unable to call System.exit
„[T]HERE ARE KNOWN KNOWNS; THERE ARE THINGS WE KNOW WE KNOW. WE ALSO KNOW THERE ARE KNOWN UNKNOWNS; THAT IS TO SAY WE KNOW THERE ARE SOME THINGS WE DO NOT KNOW. BUT THERE ARE ALSO UNKNOWN UNKNOWNS – THERE ARE THINGS WE DO NOT KNOW WE DON’T KNOW.“ Donald Rumsfeld, former secretary of defense, IT Security Expert
„[T]HERE ARE KNOWN KNOWNS; THERE ARE THINGS WE KNOW WE KNOW. WE ALSO KNOW THERE ARE KNOWN UNKNOWNS; THAT IS TO SAY WE KNOW THERE ARE SOME THINGS WE DO NOT KNOW. BUT THERE ARE ALSO UNKNOWN UNKNOWNS – THERE ARE THINGS WE DO NOT KNOW WE DON’T KNOW.“ Donald Rumsfeld, former secretary of defense, IT Security Expert
„[T]HERE ARE KNOWN KNOWNS; THERE ARE THINGS WE KNOW WE KNOW. WE ALSO KNOW THERE ARE KNOWN UNKNOWNS; THAT IS TO SAY WE KNOW THERE ARE SOME THINGS WE DO NOT KNOW. BUT THERE ARE ALSO UNKNOWN UNKNOWNS – THERE ARE THINGS WE DO NOT KNOW WE DON’T KNOW.“ Donald Rumsfeld, former secretary of defense, IT Security Expert
„[T]HERE ARE KNOWN KNOWNS; THERE ARE THINGS WE KNOW WE KNOW. WE ALSO KNOW THERE ARE KNOWN UNKNOWNS; THAT IS TO SAY WE KNOW THERE ARE SOME THINGS WE DO NOT KNOW. BUT THERE ARE ALSO UNKNOWN UNKNOWNS – THERE ARE THINGS WE DO NOT KNOW WE DON’T KNOW.“ Donald Rumsfeld, former secretary of defense, IT Security Expert
Security Manager Have you ever called System.setSecurityManager()?
What is a sandbox? connect 192.168.1.1:9300 Your code write /var/log/elasticsearch.log unlink /var/lib/elasticsearch/… ✅ ✅ ✅
What is a sandbox? open /etc/passwd Your code connect bitcoin-miner.foo.bar unlink /var/lib/elasticsearch ⛔ ⛔ ⛔
What is a sandbox? sandbox ✅ Your code ⛔
Introduction Sandbox your java application Prevent certain calls by your application Policy file grants permissions FilePermission (read, write) SocketPermission (connect, listen, accept) URLPermission, PropertyPermission, …
Java Security Manager Java Security Manager Java Program Policy
Java Security Manager Java Security Manager Policy FilePermission read /etc/elasticsearch Java Program FilePermission write /var/log/elasticsearch SocketPermission connect *
Java Security Manager Java Security Manager Policy Java Program
Introduction
Introduction
DEMO
OHAI JLS https://docs.oracle.com/javase/specs/jls/se11/html/jls-17.html#jls-17.5.3
Drawbacks Hardcoded policies before startup DNS lookups are cached forever by default Forces you to think about dependencies! Many libraries are not even tested with the security manager, unknown code paths may be executed No OOM protection! No stack overflow protection! Granularity No protection against java agents
Reducing impact Bad things have less bad results
Reducing impact Elasticsearch integration of the Java Security Manager Least privilege principle Do not run as root No chance of forking a process Do not expose sensitive settings
Security Manager in Elasticsearch Initialization required before starting security manager Elasticsearch needs to read its configuration file first to find out about the file paths Native code needs to be executed first Solution: Start with empty security manager, bootstrap, apply secure security manager
Elasticsearch startup JVM Startup time
JVM Startup Elasticsearch startup time Read configuration file
time Read configuration file JVM Startup Elasticsearch startup Native system calls
time Native system calls Read configuration file JVM Startup Elasticsearch startup Set security manager
time Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Load plugins
time Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Bootstrap checks
time Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup Network enabled
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Elasticsearch startup
Security Manager in Elasticsearch Special security manager is used Does not set exitVM permissions, only a few special classes are allowed to call Thread & ThreadGroup security is enforced Also SpecialPermission was added, a special marker permission to prevent elevation by scripts
Security Manager in Elasticsearch ESPolicy allows for loading from files plus dynamic configuration (from the ES configuration file) Bootstrap check for java.security.AllPermission
#noroot there is no reason to run code as root!
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Do not run as root
Do not run as root
seccomp … or how I loved to abort system calls
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Seccomp - prevent process forks
Seccomp - prevent process forks Security manager could fail Elasticsearch should still not be able to fork processes One way transition to tell the operating system to deny execve, fork, vfork, execveat system calls Works on Linux, Windows, Solaris, BSD, osx
Seccomp - prevent process forks
Seccomp - prevent process forks
seccomp sandbox seccomp ✅ Your code ⛔
DEMO
Production mode vs Development mode Annoying you now instead of devastating you later
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Bootstrap checks
Is your dev setup equivalent to production? Development environments are rarely setup like production ones How to ensure certain preconditions in production but not for development? What is a good indicator?
Mode check
Bootstrap checks
Bootstrap checks
Bootstrap checks
Plugins … remaining secure
time Network enabled Bootstrap checks Load plugins Set security manager Native system calls Read configuration file JVM Startup Bootstrap checks
Plugins in 60 seconds plugins are just zip files each plugin can have its own jars/dependencies each plugin is loaded with its own classloader each plugin can have its own security permissions ES core loads a bunch of code as modules (plugins that ship with Elasticsearch)
Plugins & modules Java Security Manager Policy Elasticsearch Plugin
Plugins & modules Java Security Manager Elasticsearch Plugin Policy
Plugins & modules Elasticsearch Module Elasticsearch Plugin Policy Policy Elasticsearch Module Policy
Sample permissions
Sample permissions
Sample permissions
Introducing Painless A scripting language for Elasticsearch
Scripting: Why and how? Expression evaluation without needing to write java extensions for Elasticsearch Node ingest script processor Search queries (dynamic requests & fields) Aggregations (dynamic buckets) Templating (Mustache)
Scripting in Elasticsearch MVEL Groovy Expressions Painless
Painless - a secure scripting language Hard to take an existing programming language and make it secure, but remain fast Sandboxing Whitelisting over blacklisting, per method Opt-in to regular expressions Prevent endless loops Detect self references to prevent stack overflows
DEMO
Summary Security is hard - let’s go shopping!
Summary Not using the Security Manager - what’s your excuse? Scripting is important, is your implementation secure? Use operating system features! If you allow for plugins, remain secure! If you remove features, have alternatives!
Summary Development has big impact on security Operations is happy to help what is there out of the box Developers know their application best! Don’t reinvent, check out existing features! Developers are responsible for writing secure code! Before something happens!
Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co
Resources
Resources
Resources https://github.com/elastic/elasticsearch/ https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating https://www.elastic.co/blog/scripting https://www.elastic.co/blog/scripting-security https://docs.oracle.com/javase/9/security/toc.htm https://docs.oracle.com/javase/9/security/permissions-java-development-kit.htm https://www.elastic.co/blog/seccomp-in-the-elastic-stack https://github.com/spinscale/talk-elasticsearch-security-manager-and-seccomp
Bonus register all your settings
Mark sensitive settings
Register all your settings
Bonus deep pagination vs search_after
Pagination: Request N C Find the first 10 results for Elasticsearch
Pagination: Request N C Find the first 10 results for Elasticsearch
Pagination: Request N N N C N N Find the first 10 results for Elasticsearch
Pagination: Query Phase N N SortedPriorityQueue size = 50 N C N N Each node returns 10 results, create real top 10 out of 50
Pagination: Fetch phase N N N C N N ask for the real top 10
Pagination: Query Phase N N N C N N return real top 10
Pagination: Query N N N C N N Find the 10 results starting at position 90
Pagination: Query Phase N N SortedPriorityQueue size = 500 N C N N Each node returns 100 results, create real top 90-100 out of 500
Pagination: Query N N N C N N Find the 10 results starting at position 99990
Pagination: Query Phase N N SortedPriorityQueue size = 500000 N C N N Each node returns 100k results
Pagination: Query 1 N N C N 100 Find the 10 results starting at position 99990 over 100 nodes
Pagination: Query 1 N SortedPriorityQueue size = 10_000_000 N C N 100 Each node returns 100k results
Solution: search_after Do not use numerical positions Use keys where you stopped in the inverted index Let the client tell you what the last key was Just specify the last sort value from the last document returned as a starting point
Pagination: search_after 1 N N C N 100 Find the 10 results starting at sort key name foo over 100 nodes
Pagination: search_after 1 N SortedPriorityQueue size = 1000 N C N 100 Each node returns 10 results
Bonus replacing delete by query
delete_by_query removal/replace delete_by_query API was not safe API endpoint was removed extensive documentation was added what to do instead infrastructure for long running background tasks was added delete_by_query was reintroduced using above infra and doing the exact same thing as in the documentation data > convenience!
Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co
Security is often an afterthought when writing applications. Time pressure to finish features or developers not being aware of issues can be two out of many reasons. This talk will use the Elasticsearch codebase as an example of how to write a broadly used software, but keep security in mind. Not only pure Java features like the Java Security Manager will be covered or how to write a secure scripting engine, but also operating system features that can be leveraged.
The goal of this talk is most importantly to make you think about your own codebase and where you can invest time to improve security of it - with maybe less efforts than you would think.
for free. You
can too.