A presentation at DigitalOcean Webinar Series by Aravind Putrevu
Elastic Stack Monitor your Services Aravind Putrevu Developer | Evangelist @aravindputrevu | aravindputrevu.in 1
Agenda 2 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo
Agenda 3 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo
Agenda 4 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo
Agenda 5 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo
Agenda 6 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo
Security Alerting Monitoring Elastic Stack No enterprise edition All new versions with 6.3 X-Pack Reporting Machine Learning Graph 7
Why Monitoring? Pet vs Cattle 8
Why Monitoring? Find out what’s happening? 9
Why Monitoring? Resolving Errors and bottlenecks 10
Why Elastic? SECURITY ANALYTICS
LOG ANALYTICS
METRICS ANALYTICS
BUSINESS ANALYTICS
SEARCH
APM
Protect your data
Be alerted on Detect anomalies Monitor your Elastic Stack changes
Find links in your data
Share your insights
1 1
Beats Elasticsearch Master Nodes (3) Log Files Metrics Custom UI Logstash Ingest Nodes (X) Wire Data Kibana your{beat} Data Nodes – Hot (X) Kafka Instances (X) Datastore Web APIs Redis Social Sensors Messaging Queue Data Notes – Warm (X) Nodes (X) X-Pack LDAP Hadoop Ecosystem ES-Hadoop AD X-Pack SSO Authentication Notification
Beats Elasticsearch Master Nodes (3) Log Files Metrics Custom UI Logstash Ingest Nodes (X) Wire Data Kibana your{beat} Data Nodes – Hot (X) Kafka Instances (X) Datastore Web APIs Redis Social Sensors Messaging Queue Data Notes – Warm (X) Nodes (X) X-Pack LDAP Hadoop Ecosystem ES-Hadoop AD X-Pack SSO Authentication Notification
Beats Lightweight data shippers Ship data from the source Ship and centralize in Elasticsearch Ship to Logstash for transformation and parsing Ship to Elastic Cloud Libbeat: API framework to build custom beats 30+ community Beats 14
The Beats family +40 community Beats Packetbeat Metricbeat Winlogbeat Network data Metrics Windows Event Logs Apachebeat, dockbeat, httpbeat, mysqlbeat, nginxbeat, redis beats, twitterbeat, and more 15 Auditbeat Filebeat Audit data Log files Heartbeat Uptime monitoring {your}beat
Logstash vs Beats ● Beats are lightweight data shippers that you install as agents on your servers ● Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources. 16
How beats work? • Small application Beats Framework • Install as agent on your servers • Written in Golang Log Files Metrics libbeat • No runtime dependencies • Single purpose Wire Data your{beat} 17
How beats work? 18
Classic Deployments VM 1 19 VM 2 VM n Filebeat Filebeat Filebeat Metricbeat Metricbeat Metricbeat
Kubernetes deployment Node 1 Node 2 ? 20 Node n
Elastic evolving ingest story DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform data node Store servers, containers CENTRALIZED COLLECTION Logstash network devices 21
Immediate insights with modules Logging • Turnkey experience for specific data types • Data to dashboard in just one step • Automated parsing and enrichment • Default dashboards, alerts, ML jobs Available with 22 Metrics Security
AUDITBEAT Logging modules WINLOGBEAT Applications Infrastructure 23 FILEBEAT System Databases Web servers • Linux / MacOS • MySQL • Apache • Windows Events • PostgreSQL • Nginx Containers Queues • Docker • Kafka • Kubernetes • Redis Audit data • Filesystem • System calls
METRICBEAT PACKETBEAT LOGSTASH Metrics modules Infrastructure System Containers Cloud Network • Linux • Docker • AWS • Netflow • MacOS • Kubernetes • Azure • Packets • DigitalOcean • TLS Envelope • GCP Storage • Windows • Perfmon Virtualization • vSphere 24 • Ceph
HEARTBEAT Metrics modules METRICBEAT PACKETBEAT LOGSTASH Applications Datastores Queues Uptime Web servers • MySQL • Kafka • Heartbeat • Apache • PostgreSQL • Redis • MongoDB • RabbitMQ • Couchbase 25 • Nginx Custom apps • JMX/Jolokia Other • Aerospike Caches • PHP-FPM • HAProxy • Graphite • Memcached • Golang • Zookeeper
With containers architecture, everything is a moving target We need specific tools to track things down 26
Docker deployment volume mounts volume mounts ... Web Apps Services Kibana Metricbeat Filebeat docker host Elasticsearch Networking 27 Docker API /proc filesystem Log files (/var/lib/docker/containers)
Kubernetes deployment Node 1 Node 2 Node n Filebeat Filebeat Filebeat Metricbeat Metricbeat Metricbeat Filebeat DaemonSet Metricbeat DaemonSet 28
Docker logs input Retrieve logs from Docker containers filebeat.prospectors: - type: docker containers.ids: - ‘’ Parse and ship /var/lib/docker/containers//*.log: {"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch:9200\r\n","stream":"stdout","t ime":"2018-02-11T23:29:19.236692181Z"} 29
Metadata processors Enrich events with useful metadata to correlate logs, metrics & traces add_cloud_metadata • cloud.region • cloud.instance_id • cloud.machine_type • cloud.provider add_docker_metadata add_kubernetes_metadata • docker.container.id • kubernetes.pod.name • docker.container.image • kubernetes.namespace • docker.container.name • kubernetes.labels • docker.container.labels • kubernetes.annotations • kubernetes.container.name • kubernetes.container.image 30
Metadata processors Example { "@timestamp": "2017-11-17T00:53:33.759Z", "message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "1234567", "provider": "digitalocean", "region": "blr1" } }, } 31
Metadata processors add_kubernetes_metadata internals API Server Docker Logs Pod start/stop events add_kubernetes_metadata Cont. ID pod watcher update 32 418a913c7076 c626cfdf38614 e5563a7cb80e 73de79be045c Metadata ……………… ……………… ……………… ……………... Parse Enrich Elasticsearch
Autodiscover Watch Docker events and react to changes metricbeat.autodiscover: providers: - type: docker templates: - condition: contains.docker.container.image: etcd config: - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379" 33
Autodiscover Watch Docker events and react to changes config template Events API Container start/stop events Beats
DEMO 35
What Next? 36
How Elastic Stack can help you? 37
● 100% Open Source ● Readymade UI in Kibana ● Language Agents alpha 38 beta
Resources • https://www.elastic.co/learn • https://www.elastic.co/blog/category/engineering • https://discuss.elastic.co/ • https://fb.com/groups/ElasticIndiaUserGroup • https://elastic.co/community 39
Fin! discuss.elastic.co | aravind@elastic.co | @aravindputrevu 40
for free. You
can too.