What’s new in the Elastic Stack - 7.x Edition

What’s new in the Elastic Stack? Alexander Reelsen alex@elastic.co @spinscale

Agenda ‣ What’s new in 6.x? ‣ What’s new in 7.x? ‣ Q&A

What’s new in 6.x?

Elasticsearch 6.x 6.0 Zero downtime upgrades Cross cluster search Sequence id based recoveries Index sorting range based datatypes 6.1 Index splitting 6.2 Rank evaluation API 6.3 Rollup Java 10 support 6.4 Reloadable secure settings Field Aliases Korean analyzer 6.5 G1GC support, Java 11 Minimal snapshots (50% less) 6.6 Frozen indices BKD backed geoshapes 6.7 CCR SQL ILM Upgrade Assistant 6.8 ECK (Elastic for Kubernetes) Move security features into basic

What’s new in 7.x?

Kibana 7.x 7.0 7.2 Elastic UI Library Feature controls KQL by default (+ autocomplete) Maps in dashboards Responsive dashboards Export/Import saved objects Dark mode Metrics Explorer 7.1 ECK (Elastic for Kubernetes) Move security features into basic 7.3 Maps is now GA! Kerberos support

Beats 7.x 7.0 7.2 ECS script processor Filebeat: zeek, santa, netflow support, security analytics: palo alto networks, cisco encodings ASA, netflow Auditbeat: system module CoreDNS modules Metricbeat: Elasticsearch, Logstash & Kibana modules windows: sysmon & security module filebeat: container input Metricbeat: NATS, MSSQL, EC2, CouchDB 7.1 ECK (Elastic for Kubernetes) Move security features into basic 7.3 google cloud module, google pub/sub input database support: oracle, RDS, cockroachdb k8s monitoring configuration only metricbeat modules

Logstash 7.x 7.0 7.2 Default java pipeline execution Gradle based plugin workflow ILM support JMS input app search plugin 7.1 ECK (Elastic for Kubernetes) Move security features into basic 7.3 Pipeline to pipeline communication

Stack 7.x 7.0 7.2 ECS SIEM app Hadoop: Kerberos, Java8, Cascading APM: improved Java agent metrics/ removed framework support Clients: rewritten JS client, new Go client, High Level REST client 7.3 7.1 ECK (Elastic for Kubernetes) Move security features into basic SIEM anomaly detection APM: .NET agent is GA, SPA support, configure with APM UI

Elasticsearch 7.x 7.0 7.2 faster top-k retrieval search_as_you_type datatype adaptive replica selection enabled by replicated closed indices default distance_feature query No refresh on idle shards (faster indexing) date_nanos datatype 7.3 cluster coordination data frames script_score query rare_terms aggregation High Level REST client vector datatypes (dense & sparse) Single shard index by default voting only nodes ships with OpenJDK updateable synonyms 7.1 ECK (Elastic for Kubernetes) Move security features into basic 7.4 pinned queries SLM

Elasticsearch 7.0 - Rewritten cluster coordination Gone: discovery.zen.minimum_master_nodes Sub-second master election Simplifying growing/shrinking of cluster Cluster bootstrapping/Voting configuration Rolling upgrades from 6 to 7 work Formal verification via TLA+

Elasticsearch 7.0 - Faster top-k retrieval While querying, exclude documents that cannot make it into the top hits Search: Elasticsearch OR Kibana Term 1: Elasticsearch (max score 5.0) Term 2: Kibana (max score 3.0) If first k results all have a score > 3.0, then documents only containing Kibana can be ignored Number of potential candidates is reduced while running

Elasticsearch 7.0 - Faster top-k retrieval Scores may no longer be negative Total hits are not counted by default

Elasticsearch - Adaptive Replica Selection Problem: Coordinating node round robins requests between data nodes Underperforming node harms the whole cluster Adaptive replica selection Response time of previous requests Search execution time of the data node Queue size of the search threadpool on the data node

Elasticsearch - Nanosecond support new datatype: date_nanos stores nanoseconds since the epoch (reduced range!) internally: moved from Joda-Time to java time Aggregations: millisecond resolution! Beware: Upgrade path from 6.x!

Discussion … ask all the things!

Links Elasticsearch https://www.elastic.co/blog/easier-relevance-tuning-elasticsearch-7-0 https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand https://www.elastic.co/blog/creating-frozen-indices-with-the-elasticsearch-freeze-index-api https://www.elastic.co/blog/follow-the-leader-an-introduction-to-cross-cluster-replication-in-elasticsearch https://www.elastic.co/blog/moving-from-types-to-typeless-apis-in-elasticsearch-7-0 https://www.elastic.co/blog/improving-node-resiliency-with-the-real-memory-circuit-breaker https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch https://www.elastic.co/elasticon/conf/2018/sf/reliable-by-design-applying-formal-methods-to-distributedsystems https://github.com/elastic/elasticsearch-formal-models C3: https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-suresh.pdf Beats https://www.elastic.co/blog/introducing-auditbeat-system-module

Links https://www.elastic.co/blog/security-forelasticsearch-is-now-free https://www.elastic.co/blog/introducing-elasticcloud-on-kubernetes-the-elasticsearch-operator-andbeyond