A presentation at JUG Essen in in Essen, Germany by Alexander Reelsen
Extending Elasticsearch Writing plugins, fast and easy! Alexander Reelsen @spinscale alex@elastic.co
Elasticsearch in 10 seconds Search Engine (FTS, Analytics, Geo), real-time Distributed, scalable, highly available, resilient Interface: HTTP & JSON Centrepiece of the Elastic Stack (Kibana, Logstash, Beats, APM, ML, App/Site/Enterprise Search) Uneducated guess: Tens of thousands of clusters worldwide, hundreds of thousands of instances
Agenda Overview Building Testing Releasing Extension points
Overview Requirements, gotchas and some details
Overview Plugins are zip files own jars/dependencies loaded with its own classloader own security permissions
Security Manager? Permissions? But why?! Sandbox your java application Prevent certain calls by your application Policy file grants permissions FilePermission (read, write) SocketPermission (connect, listen, accept) URLPermission, PropertyPermission, …
Security Manager? Permissions? But why?! Each plugin needs to be packaged with a policy that grants certain operations
Building gradle clean assemble
Overview A plugin needs to be released for an Elasticsearch patch release Use the gradle plugin! Batteries included, lots of them Integration tests, license checks, checkstyle, notice files
Testing gradle clean check
Overview Unit tests: ESTestCase Integration tests: YAML Download ES, install plugin, start ES, run tests against HTTP
ESTestCase Randomized testing Logger Thread leak detector Deprecation warnings detection Test method names have to begin with void test…() assertBusy/awaitBusy
Extension points class MyPlugin extends Plugin
Overview ActionPlugin: Implement own actions, REST endpoints AnalysisPlugin: Add custom parts to analysis chain ClusterPlugin: Implement own shard allocation logic DiscoveryPlugin: Add (often API based) host discovery IngestPlugin: Custom ingest processors MapperPlugin: Custom mapping data types NetworkPlugin: Network implementations (i.e. netty replacement) RepositoryPlugin: Snapshot/restore repository implementations ScriptPlugin: Script Engines, Script Contexts SearchPlugin: Queries, Highlighter, Suggester, Aggregations, Rescoring, Search Extensions ReloadablePlugin: Ensure a plugin can reload its state
Overview main class: org.elasticsearch.plugins.Plugin settings registration & filter custom cluster state metadata cluster state listener index modules (search listener, index event listener)
Plugin Loading [INFO ][o.e.p.PluginsService] [node] loaded module [lang-mustache]
Overview PluginService check for modules & plugins directory check jar hell check version create classloader reload lucene SPI load Plugin class instantiate Plugin class
Show me the code!
Ingest lang-detect processor PUT _ingest/pipeline/langdetect-pipeline { “processors”: [ { “langdetect” : { “field” : “my_field”, “target_field” : “language” } } ] }
Ingest lang-detect processor PUT /my-index/my-type/1?pipeline=langdetect-pipeline { “my_field” : “This is hopefully an english text, that will be detected.” } GET /my-index/my-type/1 { “my_field” : “This is hopefully an english text, that will be detected.”, “language”: “en” }
Custom query parser Ecommerce use-case clean data non expert searches
Custom query parser nike running hoodie xl
Custom query parser nike running hoodie xl
Custom query parser brand } } nike running hoodie xl size
Custom query parser PUT my_products { “settings”: { “index.queryparser.values” : { “brands” : [ “nike”, “adidas”, “puma”, “salomon” ], “size” : [ “m”, “xl”, “l”, “s”, “xs”, “xxs”, “xxl” ] } } }
Custom query parser GET my_products/_search { “query”: { “custom” : { “title”: “nike running hoodie XL” } } }
DEMO
Summary When is your plugin ready again?
Summary Writing own plugins is not that hard Testing plugins is incredibly easy with gradle Release via github or sonatype
Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co
Resources https://github.com/elastic/elasticsearch/ https://github.com/elastic/elasticsearch/tree/7.1/plugins/examples https://github.com/spinscale/elasticsearch-ingest-opennlp https://github.com/spinscale/elasticsearch-ingest-langdetect https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor
Thanks for listening! Questions? Alexander Reelsen @spinscale alex@elastic.co
A quick overview about how to write Elasticsearch plugins, what extension points exist, how plugins and modules are loaded when Elasticsearch starts up, how plugins are tested most efficiently with gradle. Also one ingest plugin and one query plugin were demoed as part of the presentation.
for free. You
can too.