A presentation at InfuxDays San Francisco 2019 in in San Francisco, CA, USA by David McKay
The Telegraf Toolbelt It can do that, really?
David McKay Developer Advocate 🏴 Scottish 🐭 Has 9 Pets at InfluxData 💙 Esoteric Programming Languages @rawkode ☸ Kubernetes Release Team
Establishing a Base Line The Plugin Ecosystem
Over 200 Plugins ➔ 169 ➔ 35 ➔ 15 ➔ 14 ➔ 9 ➔ 8 Inputs Outputs Processors Parsers Serializers Aggregators © 2019 InfluxData. All rights reserved. 4
V1.10 (May) ➔ Inputs ◆ ◆ ◆ ◆ ◆ ◆ ◆ Google Cloud PubSub Kinesis Consumer Kube Inventory Neptune Apex Nginx Upstream Checks Multifile Stack Driver ➔ Outputs ◆ Google Cloud PubSub ➔ Serializers © 2019 InfluxData. All rights reserved. ◆ ◆ Nowmetric Carbon2 5
V1.11 (June) ➔ Inputs ◆ ◆ ◆ ◆ ◆ ◆ ◆ bind9 Cisco GNMI Cisco MDT ECS & Fargate GitHub OpenWeatherMap PowerDNS ➔ Outputs ◆ ◆ Health Syslog ➔ Serializers ◆ ➔ © 2019 InfluxData. All rights reserved. Wavefront Aggregators ◆ Final 6
V1.12 (September) ➔ Inputs ◆ ◆ ◆ ◆ ◆ ◆ ◆ apcupsd Docker Logs Fireboard Logstash MarkLogic OpenNTPD uWSGI ➔ Outputs ◆ Exec ➔ Parsers ◆ ➔ © 2019 InfluxData. All rights reserved. Form Processors ◆ Date ◆ Pivot ◆ Unpivot ◆ Tag Limit 7
© 2019 InfluxData. All rights reserved. 8
There are 3249 telegraf.conf files on GitHub
© 2019 InfluxData. All rights reserved. 10
I could only get 1000 telegraf.conf files from GitHub
I used a sample of 1000 telegraf.conf files from GitHub
73% Use 10s (Default) 5.6% use 1s Interval 4% use 5s 2% use 1m 1% use 30s
Round Interval 90% True
Jitter 90% None
Omit Hostname 90% False
Output Plugins ➔ ➔ ➔ ➔ 71% 5% 2% .6% 1 2 0 3 Output Outputs Outputs Outputs © 2019 InfluxData. All rights reserved. 17
Output Plugins ➔ ➔ ➔ ➔ ➔ 72% 5% 2% .9% .6% InfluxDB File Prometheus Client Graphite Kafka © 2019 InfluxData. All rights reserved. 18
Input Plugins ➔ ➔ ➔ ➔ ➔ ➔ ➔ ➔ 17% 12% 10% 5% 5% 5% 5% 1 1 9 8 10 11 6 7 56 Input Inputs Inputs Inputs Inputs Inputs Inputs Inputs ?!?! © 2019 InfluxData. All rights reserved. 19
Input Plugins ➔ ➔ ➔ ➔ ➔ 58% 53% 52% 51% 47% CPU Mem Disk System DiskIO ➔ ➔ ➔ ➔ ➔ © 2019 InfluxData. All rights reserved. 47% 40% 31% 28% 23% Swap Process Kernel Docker Net 20
12% Invalid
Multiple Outputs Dual Write to InfluxDB 1 and 2 5.6%
Multiple Outputs [[outputs.influxdb]] urls = [“http://influxdb:8086”] [[outputs.influxdb_v2]] urls = [“http://influxdb2:9999”] © 2019 InfluxData. All rights reserved. 23
Remote Configuration Fetching Configuration at Runtime v1.10
Remote Configuration telegraf —config <http_uri> © 2019 InfluxData. All rights reserved. 25
Remote Configuration telegraf —config https://raw.githubusercontent.com /influxdata/telegraf/master/etc/telegraf.conf © 2019 InfluxData. All rights reserved. 26
Remote Configuration SOME_VAR=abc123 telegraf —config <http_uri> © 2019 InfluxData. All rights reserved. 27
Remote Configuration [agent] interval = “${INTERVAL}” [[outputs.influxdb_v2]] token = “${TOKEN}” © 2019 InfluxData. All rights reserved. 28
Output Resiliency Handling failures without losing metrics v0.10 (2015!)
Output Resiliency Use metric_buffer_limit to handle downtime of your outputs © 2019 InfluxData. All rights reserved. 30
Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf © 2019 InfluxData. All rights reserved. 31
Application Software Dependencies Bare Metal / VMs Buffer Batch Network Telegraf © 2019 InfluxData. All rights reserved. 32
Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf © 2019 InfluxData. All rights reserved. 33
Output Resiliency influxdb_listener http_listener v1.9 / 0.01% / It was me! © 2019 InfluxData. All rights reserved. 34
Output Resiliency influxdb_listener Allows Telegraf to serve as a proxy for the /write endpoint of the InfluxDB HTTP API. © 2019 InfluxData. All rights reserved. 35
[[inputs.influxdb_listener]] influxdb_listener service_address = “:8086”
© 2019 InfluxData. All rights reserved. 37
v1 Client Libraries
Output Resiliency http_listener_v2 Allows Telegraf to accept metrics over HTTP in any supported format © 2019 InfluxData. All rights reserved. 39
[[inputs.http_listener_v2]] http_listener_v2 service_address = “:8080” data_format = “json”
[[inputs.http_listener_v2]] data_format = “json” JSON Docs json_name_key = “name” tag_keys = [“go_version”]
Bring Your Own Telegraf Cloud Native Telegraf
Plugins come at a cost
➜ docker image ls byot-sample telegraf 19.3MB 254MB
Bring Your Own Telegraf FROM rawkode/telegraf:byo AS build FROM alpine:3.7 AS telegraf COPY —from=build /etc/telegraf /etc/telegraf COPY —from=build /binary /bin/telegraf ENTRYPOINT [ “/bin/telegraf” ] © 2019 InfluxData. All rights reserved. 45
Bring Your Own Telegraf BYOT BYOT Example Click Me! Click Me! © 2019 InfluxData. All rights reserved. 46
Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf © 2019 InfluxData. All rights reserved. 47
InfluxDB Application Telegraf © 2019 InfluxData. All rights reserved. Telegraf Software Dependencies Telegraf Bare Metal / VMs Telegraf Network 48
Internal Telegraf Exposes Metrics Too! v1.2 / 0.1%
Internal ➔ Keep an eye on buffer_size ➔ Make sure you alert on metrics_dropped
Health Output Sophisticated Health Checks v1.11 / 0%
@app.route(“/health”) def healthcheck(): return “OK”
@app.route(“/health”) def healthcheck(): return “OK”
Health Output [[outputs.health]] service_address = “http://:5559” namepass = [“web-metrics”] [[outputs.health.compares]] field = “response_time” lt = 0.300 © 2019 InfluxData. All rights reserved. 54
Kafka In, Kafka Out Controlling Ingress
InfluxDB Telegraf Telegraf Software Dependencies Telegraf Bare Metal / VMs Kafka Telegraf Application Telegraf © 2019 InfluxData. All rights reserved. Telegraf Network 56
Demo
We’ve covered less than 10 of the 200 plugins
Thank you @rawkode
Telegraf is an agent for collecting, processing, aggregating, and writing metrics. With over 200 plugins, Telegraf can fetch metrics from a variety of sources, allowing you to build aggregations and write those metrics to InfluxDB, Prometheus, Kafka, and more.
In this talk, we will take a look at some of the lesser known, but awesome, plugins that are often overlooked; as well as how to use Telegraf for monitoring of Cloud Native systems.
Here’s what was said about this presentation on social media.
for free. You
can too.