A presentation at Webinar - Der Elastic Stack für Logs und Metriken by Alexander Reelsen
Der Elastic Stack für Logs und Metriken Alexander Reelsen | Community Advocate @spinscale alex@elastic.co
Logistics • Chat: Ensure you are writing messages to everyone and not just the panelists • Video: Ensure you select ‘Fit to Window’ at the top to see the whole screen • Chat: Write all your questions. We will answer them during the session or at the end • Recording will made available! 2
Agenda
Agenda • Logs & Metrics • Elastic Stack Introduction • Ingestion • DEMO • Q&A
Logs & Metrics?
What is a log? Nov 19 16:31:58 rhincodon syslogd[41]: ASL Sender Statistics timestamp message
What is a log? Nov 19 16:31:58 rhincodon syslogd[41]: ASL Sender Statistics timestamp host message
What is a structured log? Nov 19 16:31:58 rhincodon syslogd[41]: ASL Sender Statistics timestamp host process message
What is a log? Nov 19 16:31:58 timestamp • Not unique! • Granularity! • Timezone! • Year! • Defaults required!
Date normalization Nov 19 16:31:58
Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000
Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z
Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10
Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10 1420070400
Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10 1420070400 2019-11-19T17:05:38,752
Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10 1420070400 2019-11-19T17:05:38,752 16:06:02.858
Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10 1420070400 2019-11-19T17:05:38,752 16:06:02.858 2019-11-19T17:06:23.582+0100
Multi line events [2019-07-25T00:10:02,240][WARN ][o.e.i.IndexService ] [1563552203477145411] [migrate-bird-filebeat-7.0.0-alpha1-2019.07.2 5] failed to run task refresh - suppressing re-occurring exceptions unless the exception changes org.elasticsearch.index.engine.RefreshFailedEngineException: Refresh failed at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:919) ~[elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:632) ~[elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:690) ~[elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.index.IndexService.access$400(IndexService.java:92) ~[elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:832) ~[elasticsearch-5.6.4.jar:5 .6.4] at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:743) [elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasti csearch-5.6.4.jar:5.6.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_181] at java.lang.Thread.run(Unknown Source) [?:1.8.0_181] Caused by: org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path=”/data3/containers/1563552203477145411/es/data/nodes/0/indices/itne5EqpRE-vNw1wLMj2EA/1/index/ _8u.dim”))) at org.apache.lucene.codecs.CodecUtil.verifyAndCopyIndexHeader(CodecUtil.java:282) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33] at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:96) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk 2018-06-28 00:21:33] at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4945) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33] at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:529) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33] at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:481) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk 2018-06-28 00:21:33] at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33] at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33]
What is a metric? 1574179943 load_avg_1 1.70 timestamp id value • measurement at a point in time
Logs vs. Metrics • log: event based • metric: constant measurement
Log centralization • Access rights • Short lived containers • Search across services • Correlation • Retention • Alerting • Cost of storage/density
Data normalization • Timestamps • Field name convention (lowercase, tense) • Same field names across services • Elastic Common Schema https://www.elastic.co/guide/en/ecs/current/ecs-reference.html
Elastic Stack
Elastic Stack Visualize Store Ingest Ingest
Solutions APM Search Logs Uptime Metrics Analytics Maps SIEM
Deployment • Elastic Cloud • Elastic Cloud Enterprise • Elastic Cloud on K8s • Self hosted
Elastic Cloud
Elastic Cloud
Ingestion
Ingestion • Read data • Ship data • Modify data • Acknowledging • Fail safety
Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” https://www.elastic.co/products/logstash
Beats is the platform for singlepurpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch. https://www.elastic.co/products/beats
Ingestion - Beats • Filebeat • Metricbeat • Packetbeat • Winlogbeat • Auditbeat • Heartbeat • Functionbeat
Today’s setup • Elasticsearch/Kibana on Elastic Cloud • nginx running locally • Filebeat: Ingest HTTP logs • Metricbeat: Ingest metrics
DEMO
Next steps
APM & Distributed Tracing
APM & Distributed Tracing
Uptime
Uptime
SIEM
SIEM
Machine Learning
Machine Learning
Cloud: AWS
Cloud: Azure
Kibana Lens
Further information
https://discuss.elastic.co
Join a local meetup! https://community.elastic.co
https://www.elastic.co/elasticon/tour/frankfurt
Q&A
for free. You
can too.