Workshop: Logging with the Elastic Stack Alexander Reelsen @spinscale alex@elastic.co

Agenda • Why use a search engine for logging? • Log centralization • Logging challenges • Deployment • Demo & workshop • Logging patterns •Q&A

Prerequisite • docker • docker-compose • git • java

Prerequisites • git clone https://github.com/xeraa/java-logging • cd java-logging • ./gradlew assemble • docker-compose up —build

Logging? Why use Elastic Stack for logging?

But why? • Fundamental for debugging production issues • Logs are decentralized • Containers containing logs are ephemeral • Logs are not standardized • Correlations are hard

No standards… 1.2.3.4 - - [06/Nov/2014:19:10:38 +0600] “GET /news/foo.html HTTP/1.1” 404 177 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

No standards… Sep 12 10:15:08 rhincodon logd[64]: #DECODE failed to resolve UUID: [pc:0x7fff65ec1ac7 ns:0x06 type:0x82 flags:0x8208 main:A5 2374C3-0F9D-3062-A636-131B737C4589 pid:945]

No standards… [2019-09-12T10:23:45,900][INFO ][o.e.c.s.ClusterApplierService] [rhincodon] master node changed {previous [], current [{rhincodon}{q3RjloGxRdm176yLo9d9UA}{Vq4FpFklRbCyVFAVKU7ukQ} {127.0.0.1}{127.0.0.1:9300}{dim}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20}]}, term: 6, version: 57, reason: Publication{term=6, version=57}

Preprocessing to the rescue • Date normalization • Information extraction • Field normalization

Time series have a lifecycle • Recent data is more important • Recent data is queried more often • Older data less searched • Old data may require archival due to compliance

Time series is a search • Max response time per 10 minute window since yesterday • Documents: All documents from yesterday till now • Aggregate in 10 minute buckets (6*24) • For each bucket, extract max value

Dashboards & Time Series

Dashboards & Time Series

Dashboards & Time Series

Dashboards & Time Series

Dashboards & Time Series

Standardizing data

Elasticsearch overview

Elasticsearch in 10 seconds • Search Engine (FTS, Analytics, Geo), real-time • Distributed, scalable, highly available, resilient • Interface: HTTP & JSON • Centrepiece of the Elastic Stack

Elasticsearch - a distributed system node 1 p0

Elasticsearch - a distributed system node 1 p0 p1

Elasticsearch - a distributed system node 1 node 2 p0 p1

Elasticsearch - a distributed system node 1 node 2 p0 p1 node 3 node 4

Elasticsearch - a distributed system node 1 node 2 node 3 node 4 p0 p1 r0 r1

Ingest overview

Ingestion • Logstash: extensible dynamic data collection • Beats: specialized single purpose data shipper • your own rolled integration, it’s all HTTP!

Logstash

Logstash

Logstash

Beats • Filebeat • Metricbeat • Packetbeat • Winlogbeat • Auditbeat • Heartbeat • Functionbeat • Journalbeat

Filebeat s • Apache • Google Cloud • Logstash • Palo Alto Networks • Auditd • haproxy • MongoDB • PostgreSQL • AWS • IBM MQ • MSSQL • RabbitMQ • CEF • Icinga • MySQL • Redis • Cisco • IIS • nats • Santa • Coredns • Iptables • NetFlow • Suricata • Elasticsearch • Kafka • Nginx • Traefik • Envoyproxy • Kibana • Osquery • Zeek (Bro)

Metricbeat modules • Aerospike • Elasticsearch • Logstash • Redis • Apache • envoyproxy • Memcached • Statsd • aws • Etcd • MongoDB • System • Golang • MSSQL • traefik • Munin • uwsgi • MySQL • vSphere • Nats • Windows • Nginx • ZooKeeper • Beat • Ceph • Graphite • CockroachDB • HAProxy • consul • HTTP • coredns • Jolokia • Couchbase • Kafka • couchdb • Kibana • PostgreSQL • Docker • Kubernetes • Prometheus • Dropwizard • kvm • RabbitMQ • Oracle • PHP_FPM

Solutions

Elastic APM • Distributed tracing • APM server • Kibana application • Agents: Java, .NET, Node, Python, Ruby, RUM, Go • Alerting & ML integration

Elastic Logs

Elastic SIEM

Elastic Metrics

Elastic Uptime

Elastic Uptime

Elastic Infrastructure

Elastic Infrastructure

Deployment options

Distributions • zip, tar.gz, RPM, DEB • debian/rpm repositories, homebrew tap • Docker, Helm chart • K8s Operator (ECK)

Elastic Cloud

Elastic Cloud Enterprise

meetup.com RSVP stream demo Time series data…

logging workshop demo start your engines…

Logging patterns

Time based data • time based data has properties • current data gets indexed • more recent data gets searched more • old data is still required ‘just in case’

Homogeneous architecture

Hot warm architecture

Hot warm architecture Index

Hot warm architecture Index

Hot warm architecture Index

Index Lifecycle Management • Hot: read & write • Warm: frequently read • Cold: seldom read • Delete: no longer needed

Index Lifecycle Management: Hot • rollover • set priority • unfollow

Index Lifecycle Management: Warm • set priority • unfollow • read-only • allocate • shrink • forge merge

Index Lifecycle Management: cold • set priority • unfollow • allocate • freeze

More lifecycle topics • SLM: create snapshots based on cron • Rollup: Summarize and store historical data • Transform: Pivot data to entity centric indices

Architecture patterns

Start small

Grow big

https://ela.st/cfcamp-workshop-munich

Q&A