The Telegraf Toolbelt

A presentation at Open Source Monitoring Conference in November 2019 in Nuremberg, Germany by David McKay

Slide 1

The Telegraf Toolbelt It can do that, really?

Slide 2

David McKay 🏴󠁧󠁢󠁳󠁣󠁴󠁿 Scottish 🐭 Has 9 Pets Developer Advocate 💙 Esoteric Programming Languages at InfluxData ☸ Kubernetes Release Team @rawkode 2 © 2019 InfluxData. All rights reserved. 🤔 Stoic

Slide 3

3 © 2019 InfluxData. All rights reserved.

Slide 4

Introduction Demo 4 © 2019 InfluxData. All rights reserved.

Slide 5

Telegraf is plugin based 5 © 2019 InfluxData. All rights reserved.

Slide 6

Over 200 Plugins ➔ 169 ➔ 35 ➔ 15 ➔ 14 ➔ 9 ➔ 8 6 Inputs Outputs Processors Parsers Serializers Aggregators © 2019 InfluxData. All rights reserved.

Slide 7

V1.10 (May) ➔ 7 Inputs ◆ Google Cloud PubSub ◆ Kinesis Consumer ◆ Kube Inventory ◆ Neptune Apex ◆ Nginx Upstream Checks ◆ Multifile ◆ Stack Driver © 2019 InfluxData. All rights reserved. ➔ Outputs ◆ ➔ Google Cloud PubSub Serializers ◆ ◆ Nowmetric Carbon2

Slide 8

V1.11 (June) ➔ 8 Inputs ◆ bind9 ◆ Cisco GNMI ◆ Cisco MDT ◆ ECS & Fargate ◆ GitHub ◆ OpenWeatherMap ◆ PowerDNS © 2019 InfluxData. All rights reserved. ➔ Outputs ◆ ◆ ➔ Serializers ◆ ➔ Health Syslog Wavefront Aggregators ◆ Final

Slide 9

V1.12 (September) ➔ 9 Inputs ◆ apcupsd ◆ Docker Logs ◆ Fireboard ◆ Logstash ◆ MarkLogic ◆ OpenNTPD ◆ uWSGI © 2019 InfluxData. All rights reserved. ➔ Outputs ◆ Exec ➔ Parsers ◆ ➔ Form Processors ◆ Date ◆ Pivot ◆ Unpivot ◆ Tag Limit

Slide 10

10 © 2019 InfluxData. All rights reserved.

Slide 11

There are 3249 telegraf.conf files on GitHub 11 © 2019 InfluxData. All rights reserved.

Slide 12

12 © 2019 InfluxData. All rights reserved.

Slide 13

I used a sample of 1000 telegraf.conf files from GitHub 13 © 2019 InfluxData. All rights reserved.

Slide 14

Output Plugins ➔ ➔ ➔ ➔ ➔ 14 72% 5% 2% .9% .6% InfluxDB File Prometheus Client Graphite Kafka © 2019 InfluxData. All rights reserved.

Slide 15

73% Use 10s (Default) 5.6% use 1s Interval 4% use 5s 2% use 1m 1% use 30s 15 © 2019 InfluxData. All rights reserved.

Slide 16

Round Interval 16 © 2019 InfluxData. All rights reserved. 90% True

Slide 17

Jitter 17 © 2019 InfluxData. All rights reserved. 90% None

Slide 18

Omit Hostname 18 © 2019 InfluxData. All rights reserved. 90% False

Slide 19

Output Plugins ➔ ➔ ➔ ➔ 19 71% 5% 2% .6% © 2019 InfluxData. All rights reserved. 1 2 0 3 Output Outputs Outputs Outputs

Slide 20

Input Plugins ➔ ➔ ➔ ➔ ➔ ➔ ➔ ➔ 20 17% 12% 10% 5% 5% 5% 5% 1 © 2019 InfluxData. All rights reserved. 1 9 8 10 11 6 7 56 Input Inputs Inputs Inputs Inputs Inputs Inputs Inputs ?!?!

Slide 21

Input Plugins ➔ ➔ ➔ ➔ ➔ 21 58% 53% 52% 51% 47% © 2019 InfluxData. All rights reserved. CPU Mem Disk System DiskIO ➔ ➔ ➔ ➔ ➔ 47% 40% 31% 28% 23% Swap Process Kernel Docker Net

Slide 22

Multiple Outputs 22 © 2019 InfluxData. All rights reserved.

Slide 23

Multiple Outputs [[outputs.influxdb]] urls = [“http://influxdb:8086”] [[outputs.influxdb_v2]] urls = [“http://influxdb2:9999”] 23 © 2019 InfluxData. All rights reserved.

Slide 24

Remote Configuration 24 © 2019 InfluxData. All rights reserved.

Slide 25

Remote Configuration telegraf —config <http_uri> 25 © 2019 InfluxData. All rights reserved.

Slide 26

Remote Configuration telegraf —config https://raw.githubusercontent.com /influxdata/telegraf/master/etc/telegraf.conf 26 © 2019 InfluxData. All rights reserved.

Slide 27

Remote Configuration SOME_VAR=abc123 telegraf —config <http_uri> 27 © 2019 InfluxData. All rights reserved.

Slide 28

Remote Configuration [agent] interval = “${INTERVAL}” [[outputs.influxdb_v2]] token = “${TOKEN}” 28 © 2019 InfluxData. All rights reserved.

Slide 29

Output Resiliency 29 © 2019 InfluxData. All rights reserved.

Slide 30

Output Resiliency Use metric_buffer_limit to handle downtime of your outputs 30 © 2019 InfluxData. All rights reserved.

Slide 31

Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf 31 © 2019 InfluxData. All rights reserved.

Slide 32

Application Software Dependencies Bare Metal / VMs Buffer Batch Network Telegraf 32 © 2019 InfluxData. All rights reserved.

Slide 33

Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf 33 © 2019 InfluxData. All rights reserved.

Slide 34

Output Resiliency influxdb_listener http_listener 34 © 2019 InfluxData. All rights reserved.

Slide 35

Output Resiliency influxdb_listener Allows Telegraf to serve as a proxy for the /write endpoint of the InfluxDB HTTP API. 35 © 2019 InfluxData. All rights reserved.

Slide 36

[[inputs.influxdb_listener]] influxdb_listener 36 © 2019 InfluxData. All rights reserved. service_address = “:8086”

Slide 37

37 © 2019 InfluxData. All rights reserved.

Slide 38

v1 Client Libraries 38 © 2019 InfluxData. All rights reserved.

Slide 39

Output Resiliency http_listener_v2 Allows Telegraf to accept metrics over HTTP in any supported format 39 © 2019 InfluxData. All rights reserved.

Slide 40

[[inputs.http_listener_v2]] http_listener_v2 40 © 2019 InfluxData. All rights reserved. service_address = “:8080” data_format = “json”

Slide 41

[[inputs.http_listener_v2]] data_format = “json” JSON Docs 41 © 2019 InfluxData. All rights reserved. json_name_key = “name” tag_keys = [“go_version”]

Slide 42

Bring Your Own Telegraf 42 © 2019 InfluxData. All rights reserved.

Slide 43

Plugins come at a cost 43 © 2019 InfluxData. All rights reserved.

Slide 44

➜ docker image ls byot-sample telegraf 44 © 2019 InfluxData. All rights reserved. 15.2MB 254MB

Slide 45

Bring Your Own Telegraf FROM rawkode/telegraf:byo AS build FROM alpine:3.7 AS telegraf COPY —from=build /etc/telegraf /etc/telegraf COPY —from=build /binary /bin/telegraf ENTRYPOINT [ “/bin/telegraf” ] 45 © 2019 InfluxData. All rights reserved.

Slide 46

Bring Your Own Telegraf BYOT BYOT Example Click Me! Click Me! 46 © 2019 InfluxData. All rights reserved.

Slide 47

Application Software Dependencies Bare Metal / VMs InfluxDB Buffer Batch Network Telegraf 47 © 2019 InfluxData. All rights reserved.

Slide 48

InfluxDB Application 48 © 2019 InfluxData. All rights reserved. Telegraf Telegraf Software Dependencies Telegraf Bare Metal / VMs Telegraf Network

Slide 49

Internal 49 © 2019 InfluxData. All rights reserved.

Slide 50

Internal 50 © 2019 InfluxData. All rights reserved. ➔ Keep an eye on buffer_size ➔ Make sure you alert on metrics_dropped

Slide 51

Health Output 51 © 2019 InfluxData. All rights reserved.

Slide 52

@app.route(“/health”) def healthcheck(): return “OK” 52 © 2019 InfluxData. All rights reserved.

Slide 53

@app.route(“/health”) def healthcheck(): return “OK” 53 © 2019 InfluxData. All rights reserved.

Slide 54

Health Output [[outputs.health]] service_address = “http://:5559” namepass = [“web-metrics”] [[outputs.health.compares]] field = “response_time” lt = 0.300 54 © 2019 InfluxData. All rights reserved.

Slide 55

Kafka In, Kafka Out 55 © 2019 InfluxData. All rights reserved.

Slide 56

InfluxDB Telegraf Telegraf Software Dependencies Telegraf Bare Metal / VMs Kafka Telegraf Application 56 © 2019 InfluxData. All rights reserved. Telegraf Telegraf Network

Slide 57

Serverless? 57 © 2019 InfluxData. All rights reserved.

Slide 58

Demo 58 © 2019 InfluxData. All rights reserved.

Slide 59

We’ve covered less than 10 of the 200 plugins 59 © 2019 InfluxData. All rights reserved.

Slide 60

https://speakerdeck.com/rawkode https://github.com/rawkode/influxdb-examples @rawkode