A presentation at Elastic Logging Workshop by Alexander Reelsen
Workshop: Logging with the Elastic Stack Alexander Reelsen @spinscale alex@elastic.co
Agenda • Why use a search engine for logging? • Log centralization • Logging challenges • Deployment • Demo & workshop • Logging patterns •Q&A
Prerequisite • docker • docker-compose • git • java
Prerequisites • git clone https://github.com/xeraa/java-logging • cd java-logging • ./gradlew assemble • docker-compose up —build
Logging? Why use Elastic Stack for logging?
But why? • Fundamental for debugging production issues • Logs are decentralized • Containers containing logs are ephemeral • Logs are not standardized • Correlations are hard
No standards… 1.2.3.4 - - [06/Nov/2014:19:10:38 +0600] “GET /news/foo.html HTTP/1.1” 404 177 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
No standards… Sep 12 10:15:08 rhincodon logd[64]: #DECODE failed to resolve UUID: [pc:0x7fff65ec1ac7 ns:0x06 type:0x82 flags:0x8208 main:A5 2374C3-0F9D-3062-A636-131B737C4589 pid:945]
No standards… [2019-09-12T10:23:45,900][INFO ][o.e.c.s.ClusterApplierService] [rhincodon] master node changed {previous [], current [{rhincodon}{q3RjloGxRdm176yLo9d9UA}{Vq4FpFklRbCyVFAVKU7ukQ} {127.0.0.1}{127.0.0.1:9300}{dim}{ml.machine_memory=17179869184, xpack.installed=true, ml.max_open_jobs=20}]}, term: 6, version: 57, reason: Publication{term=6, version=57}
Preprocessing to the rescue • Date normalization • Information extraction • Field normalization
Time series have a lifecycle • Recent data is more important • Recent data is queried more often • Older data less searched • Old data may require archival due to compliance
Time series is a search • Max response time per 10 minute window since yesterday • Documents: All documents from yesterday till now • Aggregate in 10 minute buckets (6*24) • For each bucket, extract max value
Dashboards & Time Series
Dashboards & Time Series
Dashboards & Time Series
Dashboards & Time Series
Dashboards & Time Series
Standardizing data
Elasticsearch overview
Elasticsearch in 10 seconds • Search Engine (FTS, Analytics, Geo), real-time • Distributed, scalable, highly available, resilient • Interface: HTTP & JSON • Centrepiece of the Elastic Stack
Elasticsearch - a distributed system node 1 p0
Elasticsearch - a distributed system node 1 p0 p1
Elasticsearch - a distributed system node 1 node 2 p0 p1
Elasticsearch - a distributed system node 1 node 2 p0 p1 node 3 node 4
Elasticsearch - a distributed system node 1 node 2 node 3 node 4 p0 p1 r0 r1
Ingest overview
Ingestion • Logstash: extensible dynamic data collection • Beats: specialized single purpose data shipper • your own rolled integration, it’s all HTTP!
Logstash
Logstash
Logstash
Beats • Filebeat • Metricbeat • Packetbeat • Winlogbeat • Auditbeat • Heartbeat • Functionbeat • Journalbeat
Filebeat s • Apache • Google Cloud • Logstash • Palo Alto Networks • Auditd • haproxy • MongoDB • PostgreSQL • AWS • IBM MQ • MSSQL • RabbitMQ • CEF • Icinga • MySQL • Redis • Cisco • IIS • nats • Santa • Coredns • Iptables • NetFlow • Suricata • Elasticsearch • Kafka • Nginx • Traefik • Envoyproxy • Kibana • Osquery • Zeek (Bro)
Metricbeat modules • Aerospike • Elasticsearch • Logstash • Redis • Apache • envoyproxy • Memcached • Statsd • aws • Etcd • MongoDB • System • Golang • MSSQL • traefik • Munin • uwsgi • MySQL • vSphere • Nats • Windows • Nginx • ZooKeeper • Beat • Ceph • Graphite • CockroachDB • HAProxy • consul • HTTP • coredns • Jolokia • Couchbase • Kafka • couchdb • Kibana • PostgreSQL • Docker • Kubernetes • Prometheus • Dropwizard • kvm • RabbitMQ • Oracle • PHP_FPM
Solutions
Elastic APM • Distributed tracing • APM server • Kibana application • Agents: Java, .NET, Node, Python, Ruby, RUM, Go • Alerting & ML integration
Elastic Logs
Elastic SIEM
Elastic Metrics
Elastic Uptime
Elastic Uptime
Elastic Infrastructure
Elastic Infrastructure
Deployment options
Distributions • zip, tar.gz, RPM, DEB • debian/rpm repositories, homebrew tap • Docker, Helm chart • K8s Operator (ECK)
Elastic Cloud
Elastic Cloud Enterprise
meetup.com RSVP stream demo Time series data…
logging workshop demo start your engines…
Logging patterns
Time based data • time based data has properties • current data gets indexed • more recent data gets searched more • old data is still required ‘just in case’
Homogeneous architecture
Hot warm architecture
Hot warm architecture Index
Hot warm architecture Index
Hot warm architecture Index
Index Lifecycle Management • Hot: read & write • Warm: frequently read • Cold: seldom read • Delete: no longer needed
Index Lifecycle Management: Hot • rollover • set priority • unfollow
Index Lifecycle Management: Warm • set priority • unfollow • read-only • allocate • shrink • forge merge
Index Lifecycle Management: cold • set priority • unfollow • allocate • freeze
More lifecycle topics • SLM: create snapshots based on cron • Rollup: Summarize and store historical data • Transform: Pivot data to entity centric indices
Architecture patterns
Start small
Grow big
https://ela.st/cfcamp-workshop-munich
Q&A
A logging workshop based on Philipp Krenn’s logging workshop github repository at https://github.com/xeraa/java-logging - some more explanations about logging challenges and proper architectures.
for free. You
can too.