Distributed Tracing: The good, The Bad, and the Ugly
A presentation at Open Source North 2022 in in St Paul, MN, USA by Ricardo Ferreira
Distributed Tracing: The Good, the Bad, and the Ugly Ricardo Ferreira Senior Developer Advocate Amazon Web Services © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Before we begin: Happy hour: it starts at 4:00 PM. New trailer for Thor: Love and Thunder. I welcome your questions anytime. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
What is distributed tracing anyway? © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
It is a type of logging. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Logging 101 © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Stitching additional data is hard © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
© 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Because logging with distributed systems is hard. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Single-machine threading Log{“/customer/find”, 235, 200, 30} API Thread 2 Thread 1 Log{“/api/find”, 840, 200, 35} Log{“/api/find”, 840, 200, 45} Customer API Database Customer Log{“/customer/find”, 235, 200, 42} © 2022, Amazon Web Services, Inc. or its affiliates. Log{“/db/find”, 450, 200, 5} Database Log{“/db/find”, 450, 200, 3} @riferrei
Correlating network distributed threads is hard © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Reason why is called distributed tracing Distributed Transaction Microservice A Microservice B © 2022, Amazon Web Services, Inc. or its affiliates. Microservice C @riferrei
Black-box versus white-box instrumentation Black-box White-box • Code is not changed • Require code changes • Handled by some runtime • Handled by the application • Minimal execution visibility • Better execution visibility © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Ideally; you should use both of them. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
But what about logs and metrics? © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
The famous trio in the SRE world Metrics: hey, something is not right. Traces: I can reveal the culprit for you. Logs: Let me show you what happened. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Best way to get started with this? © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
High-quality, ubiquitous, and portable telemetry to enable effective observability. https://opentelemetry.io © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
OpenTelemetry: SDKs, protocols, and integration © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
AWS Distro for OpenTelemetry https://aws-otel.github.io/download © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Distributed tracing: the good parts 👍 © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Decrease time spent with Incident management 10% 60% 30% • Monitoring metric values • Understanding topologies • Reading logs and events • Catching up with alerts • Isolating the anomalies • Reproducing exceptions • Evaluating trends and changes • Collecting context data • Creating code patches © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Finding bugs between releases V1.12 V1.13 © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Understand what the code (really) does. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Distributed tracing: the bad parts 👎 © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
My language. Your language. Their language. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
UI and frontends give the vendors goosebumps. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Asynchronous programming has its challenges. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
To correlate, or not correlate, that is the question. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
eBPF is the future; but not the present. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Distributed tracing: the ugly stuff 🤮 © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
© 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Native binaries may not come in handy for you. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Black-box is great; but not for everybody. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Sampling is a nightmare to get it right. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Mixing open standards with proprietary stuff. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Be prepared to become a data engineer. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Thank you! Ricardo Ferreira @riferrei © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. @riferrei
Let’s face it: most people talking about observability all end up talking about distributed tracing somehow. It is a technology that radically changes how we identify and solve technical problems. In a world where virtually all applications are born distributed — it seems to be something that you ought to know in more detail.
This talk will provide a pragmatic overview of distributed tracing by clearly articulating its motivation, problems it solves, the challenges, and what technologies you should use to ensure a vendor-agnostic implementation.
It will discuss things like white-box and black-box instrumentation, the pros and cons of each approach, and why most developers prefer one versus the other.
Buzz and feedback
Here’s what was said about this presentation on social media.
