Monitor your {X} with Elastic Stack

Elastic Stack Monitor your Services Aravind Putrevu Developer | Evangelist @aravindputrevu | aravindputrevu.in 1

Agenda 2 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo

Agenda 3 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo

Agenda 4 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo

Agenda 5 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo

Agenda 6 1 Why Monitoring? 2 Why Elastic Stack? 3 Beats : Lightweight data shipper framework 4 Monitor All things with Beats 5 Demo

Security Alerting Monitoring Elastic Stack No enterprise edition All new versions with 6.3 X-Pack Reporting Machine Learning Graph 7

Why Monitoring? Pet vs Cattle 8

Why Monitoring? Find out what’s happening? 9

Why Monitoring? Resolving Errors and bottlenecks 10

Why Elastic? SECURITY ANALYTICS LOG ANALYTICS METRICS ANALYTICS BUSINESS ANALYTICS SEARCH APM
Protect your data Be alerted on Detect anomalies Monitor your Elastic Stack changes Find links in your data Share your insights 1 1

Beats Elasticsearch Master Nodes (3) Log Files Metrics Custom UI Logstash Ingest Nodes (X) Wire Data Kibana your{beat} Data Nodes – Hot (X) Kafka Instances (X) Datastore Web APIs Redis Social Sensors Messaging Queue Data Notes – Warm (X) Nodes (X) X-Pack LDAP Hadoop Ecosystem ES-Hadoop AD X-Pack SSO Authentication Notification

Beats Elasticsearch Master Nodes (3) Log Files Metrics Custom UI Logstash Ingest Nodes (X) Wire Data Kibana your{beat} Data Nodes – Hot (X) Kafka Instances (X) Datastore Web APIs Redis Social Sensors Messaging Queue Data Notes – Warm (X) Nodes (X) X-Pack LDAP Hadoop Ecosystem ES-Hadoop AD X-Pack SSO Authentication Notification

Beats Lightweight data shippers Ship data from the source Ship and centralize in Elasticsearch Ship to Logstash for transformation and parsing Ship to Elastic Cloud Libbeat: API framework to build custom beats 30+ community Beats 14

The Beats family +40 community Beats Packetbeat Metricbeat Winlogbeat Network data Metrics Windows Event Logs Apachebeat, dockbeat, httpbeat, mysqlbeat, nginxbeat, redis beats, twitterbeat, and more 15 Auditbeat Filebeat Audit data Log files Heartbeat Uptime monitoring {your}beat

Logstash vs Beats ● Beats are lightweight data shippers that you install as agents on your servers ● Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources. 16

How beats work? • Small application Beats Framework • Install as agent on your servers • Written in Golang Log Files Metrics libbeat • No runtime dependencies • Single purpose Wire Data your{beat} 17

How beats work? 18

Classic Deployments VM 1 19 VM 2 VM n Filebeat Filebeat Filebeat Metricbeat Metricbeat Metricbeat

Kubernetes deployment Node 1 Node 2 ? 20 Node n

Elastic evolving ingest story DISTRIBUTED COLLECTION Elasticsearch Beats ingest node Transform data node Store servers, containers CENTRALIZED COLLECTION Logstash network devices 21

Immediate insights with modules Logging • Turnkey experience for specific data types • Data to dashboard in just one step • Automated parsing and enrichment • Default dashboards, alerts, ML jobs Available with 22 Metrics Security

AUDITBEAT Logging modules WINLOGBEAT Applications Infrastructure 23 FILEBEAT System Databases Web servers • Linux / MacOS • MySQL • Apache • Windows Events • PostgreSQL • Nginx Containers Queues • Docker • Kafka • Kubernetes • Redis Audit data • Filesystem • System calls

METRICBEAT PACKETBEAT LOGSTASH Metrics modules Infrastructure System Containers Cloud Network • Linux • Docker • AWS • Netflow • MacOS • Kubernetes • Azure • Packets • DigitalOcean • TLS Envelope • GCP Storage • Windows • Perfmon Virtualization • vSphere 24 • Ceph

HEARTBEAT Metrics modules METRICBEAT PACKETBEAT LOGSTASH Applications Datastores Queues Uptime Web servers • MySQL • Kafka • Heartbeat • Apache • PostgreSQL • Redis • MongoDB • RabbitMQ • Couchbase 25 • Nginx Custom apps • JMX/Jolokia Other • Aerospike Caches • PHP-FPM • HAProxy • Graphite • Memcached • Golang • Zookeeper

With containers architecture, everything is a moving target We need specific tools to track things down 26

Docker deployment volume mounts volume mounts ... Web Apps Services Kibana Metricbeat Filebeat docker host Elasticsearch Networking 27 Docker API /proc filesystem Log files (/var/lib/docker/containers)

Kubernetes deployment Node 1 Node 2 Node n Filebeat Filebeat Filebeat Metricbeat Metricbeat Metricbeat Filebeat DaemonSet Metricbeat DaemonSet 28

Docker logs input Retrieve logs from Docker containers filebeat.prospectors: - type: docker containers.ids: - ‘’ Parse and ship /var/lib/docker/containers//*.log: {"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch:9200\r\n","stream":"stdout","t ime":"2018-02-11T23:29:19.236692181Z"} 29

Metadata processors Enrich events with useful metadata to correlate logs, metrics & traces add_cloud_metadata • cloud.region • cloud.instance_id • cloud.machine_type • cloud.provider add_docker_metadata add_kubernetes_metadata • docker.container.id • kubernetes.pod.name • docker.container.image • kubernetes.namespace • docker.container.name • kubernetes.labels • docker.container.labels • kubernetes.annotations • kubernetes.container.name • kubernetes.container.image 30

Metadata processors Example { "@timestamp": "2017-11-17T00:53:33.759Z", "message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "1234567", "provider": "digitalocean", "region": "blr1" } }, } 31

Metadata processors add_kubernetes_metadata internals API Server Docker Logs Pod start/stop events add_kubernetes_metadata Cont. ID pod watcher update 32 418a913c7076 c626cfdf38614 e5563a7cb80e 73de79be045c Metadata ……………… ……………… ……………… ……………... Parse Enrich Elasticsearch

Autodiscover Watch Docker events and react to changes metricbeat.autodiscover: providers: - type: docker templates: - condition: contains.docker.container.image: etcd config: - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379" 33

Autodiscover Watch Docker events and react to changes config template Events API Container start/stop events Beats

• module: etcd metricsets: ["leader", "self", "store"] hosts: " ${data.host}:2379"

1. autodiscover event 3. var expansion { "host": "10.4.15.9", "port": 2379, "docker": { 2. match "container": { condition "id": "13a2...d716" "name": "etcd", "image": "quay.io/coreos/etcd:v3.0.0", "labels": { "io.kubernetes.pod.name": "etcd-4dk4c", "io.kubernetes.pod.namespace": "kube-system" ... } } } } 34
2. launch module

• module: etcd hosts: " 10.4.15.9:237 metricsets: ["leader", "self", "store"] 9"

DEMO 35

What Next? 36

How Elastic Stack can help you? 37

● 100% Open Source ● Readymade UI in Kibana ● Language Agents alpha 38 beta

Resources • https://www.elastic.co/learn • https://www.elastic.co/blog/category/engineering • https://discuss.elastic.co/ • https://fb.com/groups/ElasticIndiaUserGroup • https://elastic.co/community 39

Fin! discuss.elastic.co | aravind@elastic.co | @aravindputrevu 40