ELASTICSEARCH: MINING METRICS FROM LOGS

18th October 2018 Jonathan Relf

About Commify

www.commify.com

Title Slide

MINING METRICS FROM LOGS

Seeing Inside

We need to be able to see inside software to know its working ok. Any inspection has a cost. We should be as concise & low impact as possible.

The Power of Logs

The Trouble with Logs

The Power of Metrics

The Trouble with Metrics

Initial reaction to metrics

The start of metrics at Esendex

'Pulse'

https://graphiteapp.org

'Pulse' - zoomed

Shifting balance

Elastic Stack

Centralised Logging

Metrics with Metadata

One solution that I’ve seen used with great success is a log document that’s like a “metric with metadata”. If there’s one thing to take away from today it’s this phrase.

Json Log documents

Correlations: 15 minute snapshot

Sample Metrics

Sample Metics - zoomed

Drilling down

Drilling down - zoomed

Kibana in action

Visualisation drill-downs

Dashboards

Using Elasticsearch with Grafana

Challenges

Next Steps

Timelion

Timelion screenshot

Machine Learning

Machine Learning screenshot

Elastic Stack benefits

Metrics with metadata proposition

Elastic Stack power

Additional X-Pack benefits

Summary

Questions?

About Me