The Telegraf Toolbelt It can do that, really?

David McKay Developer Advocate 🏴󠁧󠁢󠁳󠁣󠁴󠁿 Scottish 🐭 Has 9 Pets at InfluxData 💙 Esoteric Programming Languages @rawkode ☸ Kubernetes Release Team

Establishing a Base Line The Plugin Ecosystem

Over 200 Plugins ➔ 169 ➔ 35 ➔ 15 ➔ 14 ➔ 9 ➔ 8 Inputs Outputs Processors Parsers Serializers Aggregators © 2019 InfluxData. All rights reserved. 4

V1.10 (May) ➔ Inputs ◆ ◆ ◆ ◆ ◆ ◆ ◆ Google Cloud PubSub Kinesis Consumer Kube Inventory Neptune Apex Nginx Upstream Checks Multifile Stack Driver ➔ Outputs ◆ Google Cloud PubSub ➔ Serializers © 2019 InfluxData. All rights reserved. ◆ ◆ Nowmetric Carbon2 5

V1.11 (June) ➔ Inputs ◆ ◆ ◆ ◆ ◆ ◆ ◆ bind9 Cisco GNMI Cisco MDT ECS & Fargate GitHub OpenWeatherMap PowerDNS ➔ Outputs ◆ ◆ Health Syslog ➔ Serializers ◆ ➔ © 2019 InfluxData. All rights reserved. Wavefront Aggregators ◆ Final 6

V1.12 (September) ➔ Inputs ◆ ◆ ◆ ◆ ◆ ◆ ◆ apcupsd Docker Logs Fireboard Logstash MarkLogic OpenNTPD uWSGI ➔ Outputs ◆ Exec ➔ Parsers ◆ ➔ © 2019 InfluxData. All rights reserved. Form Processors ◆ Date ◆ Pivot ◆ Unpivot ◆ Tag Limit 7

© 2019 InfluxData. All rights reserved. 8

There are 3249 telegraf.conf files on GitHub

© 2019 InfluxData. All rights reserved. 10

I could only get 1000 telegraf.conf files from GitHub

I used a sample of 1000 telegraf.conf files from GitHub

73% Use 10s (Default) 5.6% use 1s Interval 4% use 5s 2% use 1m 1% use 30s

Round Interval 90% True

Jitter 90% None

Omit Hostname 90% False

Output Plugins ➔ ➔ ➔ ➔ 71% 5% 2% .6% 1 2 0 3 Output Outputs Outputs Outputs © 2019 InfluxData. All rights reserved. 17

Output Plugins ➔ ➔ ➔ ➔ ➔ 72% 5% 2% .9% .6% InfluxDB File Prometheus Client Graphite Kafka © 2019 InfluxData. All rights reserved. 18

Input Plugins ➔ ➔ ➔ ➔ ➔ ➔ ➔ ➔ 17% 12% 10% 5% 5% 5% 5% 1 1 9 8 10 11 6 7 56 Input Inputs Inputs Inputs Inputs Inputs Inputs Inputs ?!?! © 2019 InfluxData. All rights reserved. 19

Input Plugins ➔ ➔ ➔ ➔ ➔ 58% 53% 52% 51% 47% CPU Mem Disk System DiskIO ➔ ➔ ➔ ➔ ➔ © 2019 InfluxData. All rights reserved. 47% 40% 31% 28% 23% Swap Process Kernel Docker Net 20

12% Invalid

Multiple Outputs Dual Write to InfluxDB 1 and 2 5.6%

Multiple Outputs [[outputs.influxdb]] urls = [“http://influxdb:8086”] [[outputs.influxdb_v2]] urls = [“http://influxdb2:9999”] © 2019 InfluxData. All rights reserved. 23

Remote Configuration Fetching Configuration at Runtime v1.10

Remote Configuration telegraf —config <http_uri> © 2019 InfluxData. All rights reserved. 25

Remote Configuration telegraf —config https://raw.githubusercontent.com /influxdata/telegraf/master/etc/telegraf.conf © 2019 InfluxData. All rights reserved. 26

Remote Configuration SOME_VAR=abc123 telegraf —config <http_uri> © 2019 InfluxData. All rights reserved. 27

Remote Configuration [agent] interval = “${INTERVAL}” [[outputs.influxdb_v2]] token = “${TOKEN}” © 2019 InfluxData. All rights reserved. 28

Output Resiliency Handling failures without losing metrics v0.10 (2015!)

Output Resiliency Use metric_buffer_limit to handle downtime of your outputs © 2019 InfluxData. All rights reserved. 30

Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf © 2019 InfluxData. All rights reserved. 31

Application Software Dependencies Bare Metal / VMs Buffer Batch Network Telegraf © 2019 InfluxData. All rights reserved. 32

Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf © 2019 InfluxData. All rights reserved. 33

Output Resiliency influxdb_listener http_listener v1.9 / 0.01% / It was me! © 2019 InfluxData. All rights reserved. 34

Output Resiliency influxdb_listener Allows Telegraf to serve as a proxy for the /write endpoint of the InfluxDB HTTP API. © 2019 InfluxData. All rights reserved. 35

[[inputs.influxdb_listener]] influxdb_listener service_address = “:8086”

© 2019 InfluxData. All rights reserved. 37

v1 Client Libraries

Output Resiliency http_listener_v2 Allows Telegraf to accept metrics over HTTP in any supported format © 2019 InfluxData. All rights reserved. 39

[[inputs.http_listener_v2]] http_listener_v2 service_address = “:8080” data_format = “json”

[[inputs.http_listener_v2]] data_format = “json” JSON Docs json_name_key = “name” tag_keys = [“go_version”]

Bring Your Own Telegraf Cloud Native Telegraf

Plugins come at a cost

➜ docker image ls byot-sample telegraf 19.3MB 254MB

Bring Your Own Telegraf FROM rawkode/telegraf:byo AS build FROM alpine:3.7 AS telegraf COPY —from=build /etc/telegraf /etc/telegraf COPY —from=build /binary /bin/telegraf ENTRYPOINT [ “/bin/telegraf” ] © 2019 InfluxData. All rights reserved. 45

Bring Your Own Telegraf BYOT BYOT Example Click Me! Click Me! © 2019 InfluxData. All rights reserved. 46

Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf © 2019 InfluxData. All rights reserved. 47

InfluxDB Application Telegraf © 2019 InfluxData. All rights reserved. Telegraf Software Dependencies Telegraf Bare Metal / VMs Telegraf Network 48

Internal Telegraf Exposes Metrics Too! v1.2 / 0.1%

Internal ➔ Keep an eye on buffer_size ➔ Make sure you alert on metrics_dropped

Health Output Sophisticated Health Checks v1.11 / 0%

@app.route(“/health”) def healthcheck(): return “OK”

@app.route(“/health”) def healthcheck(): return “OK”

Health Output [[outputs.health]] service_address = “http://:5559” namepass = [“web-metrics”] [[outputs.health.compares]] field = “response_time” lt = 0.300 © 2019 InfluxData. All rights reserved. 54

Kafka In, Kafka Out Controlling Ingress

InfluxDB Telegraf Telegraf Software Dependencies Telegraf Bare Metal / VMs Kafka Telegraf Application Telegraf © 2019 InfluxData. All rights reserved. Telegraf Network 56

Demo

We’ve covered less than 10 of the 200 plugins

Thank you @rawkode