A presentation at Time Series Belgium in in Ghent, Belgium by David McKay
The Telegraf Toolbelt It can do that, really?
David McKay 🏴 Scottish 🐭 Has 9 Pets Developer Advocate 💙 Esoteric Programming Languages at InfluxData ☸ Kubernetes Release Team @rawkode 2 © 2019 InfluxData. All rights reserved. 🤔 Stoic
3 © 2019 InfluxData. All rights reserved.
Introduction Demo 4 © 2019 InfluxData. All rights reserved.
Telegraf is plugin based 5 © 2019 InfluxData. All rights reserved.
Over 200 Plugins ➔ 169 ➔ 35 ➔ 15 ➔ 14 ➔ 9 ➔ 8 6 Inputs Outputs Processors Parsers Serializers Aggregators © 2019 InfluxData. All rights reserved.
V1.10 (May) ➔ 7 Inputs ◆ Google Cloud PubSub ◆ Kinesis Consumer ◆ Kube Inventory ◆ Neptune Apex ◆ Nginx Upstream Checks ◆ Multifile ◆ Stack Driver © 2019 InfluxData. All rights reserved. ➔ Outputs ◆ ➔ Google Cloud PubSub Serializers ◆ ◆ Nowmetric Carbon2
V1.11 (June) ➔ 8 Inputs ◆ bind9 ◆ Cisco GNMI ◆ Cisco MDT ◆ ECS & Fargate ◆ GitHub ◆ OpenWeatherMap ◆ PowerDNS © 2019 InfluxData. All rights reserved. ➔ Outputs ◆ ◆ ➔ Serializers ◆ ➔ Health Syslog Wavefront Aggregators ◆ Final
V1.12 (September) ➔ 9 Inputs ◆ apcupsd ◆ Docker Logs ◆ Fireboard ◆ Logstash ◆ MarkLogic ◆ OpenNTPD ◆ uWSGI © 2019 InfluxData. All rights reserved. ➔ Outputs ◆ Exec ➔ Parsers ◆ ➔ Form Processors ◆ Date ◆ Pivot ◆ Unpivot ◆ Tag Limit
10 © 2019 InfluxData. All rights reserved.
There are 3249 telegraf.conf files on GitHub 11 © 2019 InfluxData. All rights reserved.
12 © 2019 InfluxData. All rights reserved.
I used a sample of 1000 telegraf.conf files from GitHub 13 © 2019 InfluxData. All rights reserved.
Output Plugins ➔ ➔ ➔ ➔ ➔ 14 72% 5% 2% .9% .6% InfluxDB File Prometheus Client Graphite Kafka © 2019 InfluxData. All rights reserved.
73% Use 10s (Default) 5.6% use 1s Interval 4% use 5s 2% use 1m 1% use 30s 15 © 2019 InfluxData. All rights reserved.
Round Interval 16 © 2019 InfluxData. All rights reserved. 90% True
Jitter 17 © 2019 InfluxData. All rights reserved. 90% None
Omit Hostname 18 © 2019 InfluxData. All rights reserved. 90% False
Output Plugins ➔ ➔ ➔ ➔ 19 71% 5% 2% .6% © 2019 InfluxData. All rights reserved. 1 2 0 3 Output Outputs Outputs Outputs
Input Plugins ➔ ➔ ➔ ➔ ➔ ➔ ➔ ➔ 20 17% 12% 10% 5% 5% 5% 5% 1 © 2019 InfluxData. All rights reserved. 1 9 8 10 11 6 7 56 Input Inputs Inputs Inputs Inputs Inputs Inputs Inputs ?!?!
Input Plugins ➔ ➔ ➔ ➔ ➔ 21 58% 53% 52% 51% 47% © 2019 InfluxData. All rights reserved. CPU Mem Disk System DiskIO ➔ ➔ ➔ ➔ ➔ 47% 40% 31% 28% 23% Swap Process Kernel Docker Net
Multiple Outputs 22 © 2019 InfluxData. All rights reserved.
Multiple Outputs [[outputs.influxdb]] urls = [“http://influxdb:8086”] [[outputs.influxdb_v2]] urls = [“http://influxdb2:9999”] 23 © 2019 InfluxData. All rights reserved.
Remote Configuration 24 © 2019 InfluxData. All rights reserved.
Remote Configuration telegraf —config <http_uri> 25 © 2019 InfluxData. All rights reserved.
Remote Configuration telegraf —config https://raw.githubusercontent.com /influxdata/telegraf/master/etc/telegraf.conf 26 © 2019 InfluxData. All rights reserved.
Remote Configuration SOME_VAR=abc123 telegraf —config <http_uri> 27 © 2019 InfluxData. All rights reserved.
Remote Configuration [agent] interval = “${INTERVAL}” [[outputs.influxdb_v2]] token = “${TOKEN}” 28 © 2019 InfluxData. All rights reserved.
Output Resiliency 29 © 2019 InfluxData. All rights reserved.
Output Resiliency Use metric_buffer_limit to handle downtime of your outputs 30 © 2019 InfluxData. All rights reserved.
Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf 31 © 2019 InfluxData. All rights reserved.
Application Software Dependencies Bare Metal / VMs Buffer Batch Network Telegraf 32 © 2019 InfluxData. All rights reserved.
Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf 33 © 2019 InfluxData. All rights reserved.
Output Resiliency influxdb_listener http_listener 34 © 2019 InfluxData. All rights reserved.
Output Resiliency influxdb_listener Allows Telegraf to serve as a proxy for the /write endpoint of the InfluxDB HTTP API. 35 © 2019 InfluxData. All rights reserved.
[[inputs.influxdb_listener]] influxdb_listener 36 © 2019 InfluxData. All rights reserved. service_address = “:8086”
37 © 2019 InfluxData. All rights reserved.
v1 Client Libraries 38 © 2019 InfluxData. All rights reserved.
Output Resiliency http_listener_v2 Allows Telegraf to accept metrics over HTTP in any supported format 39 © 2019 InfluxData. All rights reserved.
[[inputs.http_listener_v2]] http_listener_v2 40 © 2019 InfluxData. All rights reserved. service_address = “:8080” data_format = “json”
[[inputs.http_listener_v2]] data_format = “json” JSON Docs 41 © 2019 InfluxData. All rights reserved. json_name_key = “name” tag_keys = [“go_version”]
Bring Your Own Telegraf 42 © 2019 InfluxData. All rights reserved.
Plugins come at a cost 43 © 2019 InfluxData. All rights reserved.
➜ docker image ls byot-sample telegraf 44 © 2019 InfluxData. All rights reserved. 15.2MB 254MB
Bring Your Own Telegraf FROM rawkode/telegraf:byo AS build FROM alpine:3.7 AS telegraf COPY —from=build /etc/telegraf /etc/telegraf COPY —from=build /binary /bin/telegraf ENTRYPOINT [ “/bin/telegraf” ] 45 © 2019 InfluxData. All rights reserved.
Bring Your Own Telegraf BYOT BYOT Example Click Me! Click Me! 46 © 2019 InfluxData. All rights reserved.
Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf 47 © 2019 InfluxData. All rights reserved.
InfluxDB Application 48 © 2019 InfluxData. All rights reserved. Telegraf Telegraf Software Dependencies Telegraf Bare Metal / VMs Telegraf Network
Internal 49 © 2019 InfluxData. All rights reserved.
Internal 50 © 2019 InfluxData. All rights reserved. ➔ Keep an eye on buffer_size ➔ Make sure you alert on metrics_dropped
Health Output 51 © 2019 InfluxData. All rights reserved.
@app.route(“/health”) def healthcheck(): return “OK” 52 © 2019 InfluxData. All rights reserved.
@app.route(“/health”) def healthcheck(): return “OK” 53 © 2019 InfluxData. All rights reserved.
Health Output [[outputs.health]] service_address = “http://:5559” namepass = [“web-metrics”] [[outputs.health.compares]] field = “response_time” lt = 0.300 54 © 2019 InfluxData. All rights reserved.
Kafka In, Kafka Out 55 © 2019 InfluxData. All rights reserved.
InfluxDB Telegraf Telegraf Software Dependencies Telegraf Bare Metal / VMs Kafka Telegraf Application 56 © 2019 InfluxData. All rights reserved. Telegraf Telegraf Network
Serverless? 57 © 2019 InfluxData. All rights reserved.
Demo 58 © 2019 InfluxData. All rights reserved.
We’ve covered less than 10 of the 200 plugins 59 © 2019 InfluxData. All rights reserved.
https://speakerdeck.com/rawkode https://github.com/rawkode/influxdb-examples @rawkode
Telegraf is an agent for collecting, processing, aggregating, and writing metrics. With over 200 plugins, Telegraf can fetch metrics from a variety of sources, allowing you to build aggregations and write those metrics to InfluxDB, Prometheus, Kafka, and more.
In this talk, we will take a look at some of the lesser known, but awesome, plugins that are often overlooked; as well as how to use Telegraf for monitoring of Cloud Native systems.
for free. You
can too.