Der Elastic Stack für Logs und Metriken

A presentation at Webinar - Der Elastic Stack für Logs und Metriken in November 2019 in by Alexander Reelsen

Slide 1

Slide 1

Der Elastic Stack für Logs und Metriken Alexander Reelsen | Community Advocate @spinscale alex@elastic.co

Slide 2

Slide 2

Logistics • Chat: Ensure you are writing messages to everyone and not just the panelists • Video: Ensure you select ‘Fit to Window’ at the top to see the whole screen • Chat: Write all your questions. We will answer them during the session or at the end • Recording will made available! 2

Slide 3

Slide 3

Agenda

Slide 4

Slide 4

Agenda • Logs & Metrics • Elastic Stack Introduction • Ingestion • DEMO • Q&A

Slide 5

Slide 5

Logs & Metrics?

Slide 6

Slide 6

What is a log? Nov 19 16:31:58 rhincodon syslogd[41]: ASL Sender Statistics timestamp message

Slide 7

Slide 7

What is a log? Nov 19 16:31:58 rhincodon syslogd[41]: ASL Sender Statistics timestamp host message

Slide 8

Slide 8

What is a structured log? Nov 19 16:31:58 rhincodon syslogd[41]: ASL Sender Statistics timestamp host process message

Slide 9

Slide 9

What is a log? Nov 19 16:31:58 timestamp • Not unique! • Granularity! • Timezone! • Year! • Defaults required!

Slide 10

Slide 10

Date normalization Nov 19 16:31:58

Slide 11

Slide 11

Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000

Slide 12

Slide 12

Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z

Slide 13

Slide 13

Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10

Slide 14

Slide 14

Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10 1420070400

Slide 15

Slide 15

Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10 1420070400 2019-11-19T17:05:38,752

Slide 16

Slide 16

Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10 1420070400 2019-11-19T17:05:38,752 16:06:02.858

Slide 17

Slide 17

Date normalization Nov 19 16:31:58 19/Jul/2015:08:13:42 +0000 2015-01-01T12:10:30.123456789Z 2019-10-10 1420070400 2019-11-19T17:05:38,752 16:06:02.858 2019-11-19T17:06:23.582+0100

Slide 18

Slide 18

Multi line events [2019-07-25T00:10:02,240][WARN ][o.e.i.IndexService ] [1563552203477145411] [migrate-bird-filebeat-7.0.0-alpha1-2019.07.2 5] failed to run task refresh - suppressing re-occurring exceptions unless the exception changes org.elasticsearch.index.engine.RefreshFailedEngineException: Refresh failed at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:919) ~[elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:632) ~[elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:690) ~[elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.index.IndexService.access$400(IndexService.java:92) ~[elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:832) ~[elasticsearch-5.6.4.jar:5 .6.4] at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:743) [elasticsearch-5.6.4.jar:5.6.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasti csearch-5.6.4.jar:5.6.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_181] at java.lang.Thread.run(Unknown Source) [?:1.8.0_181] Caused by: org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path=”/data3/containers/1563552203477145411/es/data/nodes/0/indices/itne5EqpRE-vNw1wLMj2EA/1/index/ _8u.dim”))) at org.apache.lucene.codecs.CodecUtil.verifyAndCopyIndexHeader(CodecUtil.java:282) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33] at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:96) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk 2018-06-28 00:21:33] at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4945) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33] at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:529) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33] at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:481) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk 2018-06-28 00:21:33] at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33] at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653) ~[lucene-core-6.6.1.jar:6.6.1 unknown - elk - 2018-06-28 00:21:33]

Slide 19

Slide 19

What is a metric? 1574179943 load_avg_1 1.70 timestamp id value • measurement at a point in time

Slide 20

Slide 20

Logs vs. Metrics • log: event based • metric: constant measurement

Slide 21

Slide 21

Log centralization • Access rights • Short lived containers • Search across services • Correlation • Retention • Alerting • Cost of storage/density

Slide 22

Slide 22

Data normalization • Timestamps • Field name convention (lowercase, tense) • Same field names across services • Elastic Common Schema https://www.elastic.co/guide/en/ecs/current/ecs-reference.html

Slide 23

Slide 23

Elastic Stack

Slide 24

Slide 24

Elastic Stack Visualize Store Ingest Ingest

Slide 25

Slide 25

Solutions APM Search Logs Uptime Metrics Analytics Maps SIEM

Slide 26

Slide 26

Deployment • Elastic Cloud • Elastic Cloud Enterprise • Elastic Cloud on K8s • Self hosted

Slide 27

Slide 27

Elastic Cloud

Slide 28

Slide 28

Elastic Cloud

Slide 29

Slide 29

Slide 30

Slide 30

Ingestion

Slide 31

Slide 31

Ingestion • Read data • Ship data • Modify data • Acknowledging • Fail safety

Slide 32

Slide 32

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” https://www.elastic.co/products/logstash

Slide 33

Slide 33

Beats is the platform for singlepurpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch. https://www.elastic.co/products/beats

Slide 34

Slide 34

Ingestion - Beats • Filebeat • Metricbeat • Packetbeat • Winlogbeat • Auditbeat • Heartbeat • Functionbeat

Slide 35

Slide 35

Today’s setup • Elasticsearch/Kibana on Elastic Cloud • nginx running locally • Filebeat: Ingest HTTP logs • Metricbeat: Ingest metrics

Slide 36

Slide 36

DEMO

Slide 37

Slide 37

Next steps

Slide 38

Slide 38

APM & Distributed Tracing

Slide 39

Slide 39

APM & Distributed Tracing

Slide 40

Slide 40

Uptime

Slide 41

Slide 41

Uptime

Slide 42

Slide 42

SIEM

Slide 43

Slide 43

SIEM

Slide 44

Slide 44

Machine Learning

Slide 45

Slide 45

Machine Learning

Slide 46

Slide 46

Cloud: AWS

Slide 47

Slide 47

Cloud: Azure

Slide 48

Slide 48

Kibana Lens

Slide 49

Slide 49

Further information

Slide 50

Slide 50

https://discuss.elastic.co

Slide 51

Slide 51

Join a local meetup! https://community.elastic.co

Slide 52

Slide 52

Slide 53

Slide 53

Slide 54

Slide 54

Slide 55

Slide 55

https://www.elastic.co/elasticon/tour/frankfurt

Slide 56

Slide 56

Q&A