--- title: "Tracing Microservices with OpenTelemetry" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/post/tracing-microservices-opentelemetry --- ![Blog post image for Tracing Microservices with OpenTelemetry - How OpenTelemetry traces a request across distributed services: instrumenting your code, running a collector, and visualizing the resulting spans in Jaeger or Zipkin to find bottlenecks and errors.](/_astro/hero.BOHz8WyH_1yBSiO.webp) [Home](/)›[Devtips](/devtips)›[All Categories](/devtips/categories)›[Observability & Monitoring](/devtips/categories/observability--monitoring) Devtips [Observability & Monitoring](/devtips/categories/observability--monitoring) # Tracing Microservices with OpenTelemetry [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 23 Jun 202503 Mins read04 Mins listen [Markdown for AI(opens in a new tab)](/post/tracing-microservices-opentelemetry/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR How OpenTelemetry traces a request across distributed services: instrumenting your code, running a collector, and visualizing the resulting spans in Jaeger or Zipkin to find bottlenecks and errors. Series [Observability & Monitoring](/series/observability--monitoring)1/2 [NextStructured Logging & Log Aggregation with ELK Stack](/devtips/post/structured-logging-elk-stack) All posts in this series (2) DevTips2 1. [Tracing Microservices with OpenTelemetryYou are here](/devtips/post/tracing-microservices-opentelemetry) 2. [Structured Logging & Log Aggregation with ELK Stack](/devtips/post/structured-logging-elk-stack) ### Tracing Microservices with OpenTelemetry Contents [Why monitor your microservices?](#why-monitor-your-microservices)[The complexity of distributed systems](#the-complexity-of-distributed-systems)[What poor observability costs you](#what-poor-observability-costs-you)[The challenge of microservices observability](#the-challenge-of-microservices-observability)[Debugging distributed systems](#debugging-distributed-systems)[The cascade effect](#the-cascade-effect)[Where logs and metrics stop helping](#where-logs-and-metrics-stop-helping)[The fix: OpenTelemetry distributed tracing](#the-fix-opentelemetry-distributed-tracing)[How distributed tracing works](#how-distributed-tracing-works)[Instrumenting your services](#instrumenting-your-services)[Reading the traces](#reading-the-traces)[Implementation steps](#implementation-steps)[Adding the libraries](#adding-the-libraries)[Configuring the collector](#configuring-the-collector)[Keeping up with it](#keeping-up-with-it)[What tracing gives you](#what-tracing-gives-you)[Faster resolution](#faster-resolution)[Fewer cascading failures](#fewer-cascading-failures)[A real picture of your dependencies](#a-real-picture-of-your-dependencies)[What's your monitoring strategy?](#whats-your-monitoring-strategy)[Community approaches](#community-approaches)[Tool comparisons](#tool-comparisons) ## [Why monitor your microservices?](#why-monitor-your-microservices) ### [The complexity of distributed systems](#the-complexity-of-distributed-systems) If you’re juggling multiple services, it’s hard to track how they work together. OpenTelemetry lets you follow one request end to end and see where it went wrong, which is closer to a map than to a dashboard. ### [What poor observability costs you](#what-poor-observability-costs-you) Without tracing, a microservice architecture is a black box with good uptime graphs. Every incident starts with the same twenty minutes of asking which service is at fault. ## [The challenge of microservices observability](#the-challenge-of-microservices-observability) ### [Debugging distributed systems](#debugging-distributed-systems) With microservices, one slow or broken part drags the rest down with it. Without a clear view of what’s happening, you’re guessing at which service to look at, and you’re guessing while the pager is going off. ### [The cascade effect](#the-cascade-effect) One failing service triggers timeouts in its callers, which trigger retries, which push load onto services that were fine a minute ago. By the time you look, five things are red and none of them is the cause. ### [Where logs and metrics stop helping](#where-logs-and-metrics-stop-helping) Metrics tell you latency went up. Logs tell you what one service did. Neither one connects the log line in your API to the log line in the database wrapper three hops later, and that connection is the thing you actually need. ## [The fix: OpenTelemetry distributed tracing](#the-fix-opentelemetry-distributed-tracing) ### [How distributed tracing works](#how-distributed-tracing-works) OpenTelemetry is a free, open source toolkit that tracks requests as they move through your services. Each service records a span, the spans carry a shared trace ID, and the collector stitches them back into one timeline. ### [Instrumenting your services](#instrumenting-your-services) Add the OpenTelemetry libraries to your code, then run a collector to receive the data and forward it to something like Jaeger or Zipkin. Auto-instrumentation for HTTP clients and database drivers gets you most of the picture before you write a single manual span. ### [Reading the traces](#reading-the-traces) What you get is a waterfall: one bar per span, nested by caller. The slow hop is the wide bar, and the failed hop is the red one. That is usually the whole investigation. ## [Implementation steps](#implementation-steps) ### [Adding the libraries](#adding-the-libraries) - Add OpenTelemetry libraries to your services. - Set up a collector to feed data to a visualization tool. - Check traces regularly to spot and fix issues fast. ### [Configuring the collector](#configuring-the-collector) Point your services at one collector rather than at the backend directly. It handles batching, retries and sampling, and it means switching from Jaeger to a hosted platform later is a collector config change instead of a redeploy of every service. ### [Keeping up with it](#keeping-up-with-it) Look at traces when nothing is broken, not only during incidents. Knowing what a healthy trace looks like is what makes an unhealthy one obvious, and it’s how you notice the retry loop that has been quietly doubling your database load. ## [What tracing gives you](#what-tracing-gives-you) ### [Faster resolution](#faster-resolution) You stop reading five log streams and start reading one timeline. The question changes from “which service is slow” to “why is this span slow”, and that second question has an answer. ### [Fewer cascading failures](#fewer-cascading-failures) Catching the slow dependency early keeps it from turning into timeouts, retries and a system-wide incident. ### [A real picture of your dependencies](#a-real-picture-of-your-dependencies) Traces show the calls your architecture diagram forgot. Every team I’ve watched turn tracing on has found at least one call nobody meant to make. ## [What’s your monitoring strategy?](#whats-your-monitoring-strategy) ### [Community approaches](#community-approaches) How do you keep tabs on your microservices? Got any favorite tools to share? ### [Tool comparisons](#tool-comparisons) Jaeger, Zipkin and DataDog all pull this off, with different amounts of running your own storage. I’d like to hear which trade-off you took and whether you’d take it again. Was this useful? ## Tags [#OpenTelemetry](/devtips/tags/opentelemetry)[#Microservices](/devtips/tags/microservices)[#Distributed Tracing](/devtips/tags/distributed-tracing)[#Observability](/devtips/tags/observability)[#Monitoring](/devtips/tags/monitoring)[#DevOps](/devtips/tags/devops)[#Cloud Native](/devtips/tags/cloud-native) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fdevtips%2Fpost%2Ftracing-microservices-opentelemetry "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Tracing%20Microservices%20with%20OpenTelemetry&url=https%3A%2F%2Fmkabumattar.com%2Fdevtips%2Fpost%2Ftracing-microservices-opentelemetry "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fdevtips%2Fpost%2Ftracing-microservices-opentelemetry&title=Tracing%20Microservices%20with%20OpenTelemetry&summary=How%20OpenTelemetry%20traces%20a%20request%20across%20distributed%20services%3A%20instrumenting%20your%20code%2C%20running%20a%20collector%2C%20and%20visualizing%20the%20resulting%20spans%20in%20Jaeger%20or%20Zipkin%20to%20find%20bottlenecks%20and%20errors.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Tracing%20Microservices%20with%20OpenTelemetry%20https%3A%2F%2Fmkabumattar.com%2Fdevtips%2Fpost%2Ftracing-microservices-opentelemetry "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fdevtips%2Fpost%2Ftracing-microservices-opentelemetry&text=Tracing%20Microservices%20with%20OpenTelemetry "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fdevtips%2Fpost%2Ftracing-microservices-opentelemetry&title=Tracing%20Microservices%20with%20OpenTelemetry "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fdevtips%2Fpost%2Ftracing-microservices-opentelemetry&t=Tracing%20Microservices%20with%20OpenTelemetry "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fdevtips%2Fpost%2Ftracing-microservices-opentelemetry&media=&description=How%20OpenTelemetry%20traces%20a%20request%20across%20distributed%20services%3A%20instrumenting%20your%20code%2C%20running%20a%20collector%2C%20and%20visualizing%20the%20resulting%20spans%20in%20Jaeger%20or%20Zipkin%20to%20find%20bottlenecks%20and%20errors. "Share on Pinterest")[Email]() ## Comments ## You might also enjoy More posts on similar topics [![Structured Logging & Log Aggregation with ELK Stack](/_astro/hero.w_mPuDHM_Z1SSNz9.webp)](/devtips/post/structured-logging-elk-stack) ## [Structured Logging & Log Aggregation with ELK Stack](/devtips/post/structured-logging-elk-stack) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps & Observability](/devtips/categories/devops--observability) Why centralized logging matters When services fail, where do you look first? In a distributed system, logs scatter across servers, containers and regions. One request might touch five service [#Logging](/devtips/tags/logging)[#ELK Stack](/devtips/tags/elk-stack)[#Elasticsearch](/devtips/tags/elasticsearch)+4 tags [read more](/devtips/post/structured-logging-elk-stack) [![Organizing Terraform with Modules](/_astro/hero.5dVJEd3X_Z16rLJ8.webp)](/devtips/post/organizing-terraform-modules) ## [Organizing Terraform with Modules](/devtips/post/organizing-terraform-modules) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud & Infrastructure Automation](/devtips/categories/cloud--infrastructure-automation) Why organize your Terraform code? Where the complexity comes from If you're using Terraform to build out your infrastructure, you know how quickly things get complicated. Every new environmen [#Terraform](/devtips/tags/terraform)[#Infrastructure as Code](/devtips/tags/infrastructure-as-code)[#Modules](/devtips/tags/modules)+4 tags [read more](/devtips/post/organizing-terraform-modules) [![Securing CI/CD with IAM Roles](/_astro/hero.Bl9B2DZz_ZDIuXQ.webp)](/devtips/post/securing-cicd-with-iam-roles) ## [Securing CI/CD with IAM Roles](/devtips/post/securing-cicd-with-iam-roles) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps & DevSecOps](/devtips/categories/devops--devsecops) Why secure your CI/CD pipeline? Why pipeline security matters Your pipeline holds credentials for every environment you deploy to, which makes it one of the most valuable targets you own. A s [#CICD Security](/devtips/tags/cicd-security)[#IAM Roles](/devtips/tags/iam-roles)[#Least Privilege](/devtips/tags/least-privilege)+4 tags [read more](/devtips/post/securing-cicd-with-iam-roles) [![Understanding Kubernetes Services: ClusterIP vs NodePort vs LoadBalancer](/_astro/hero.DBNjupL__148EQW.webp)](/devtips/post/kubernetes-services-clusterip-nodeport-loadbalancer) ## [Understanding Kubernetes Services: ClusterIP vs NodePort vs LoadBalancer](/devtips/post/kubernetes-services-clusterip-nodeport-loadbalancer) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [DevOps & Kubernetes](/devtips/categories/devops--kubernetes) If you're working with Kubernetes, you've probably noticed that Pods come and go, and their IP addresses keep changing. That's where Services come in. They give you a stable way to keep your apps acce [#Kubernetes](/devtips/tags/kubernetes)[#K8s Services](/devtips/tags/k8s-services)[#ClusterIP](/devtips/tags/clusterip)+5 tags [read more](/devtips/post/kubernetes-services-clusterip-nodeport-loadbalancer) [![Managing Terraform at Scale with Terragrunt](/_astro/hero.DUZZoi07_ZRPUOh.webp)](/devtips/post/terraform-terragrunt-wrappers) ## [Managing Terraform at Scale with Terragrunt](/devtips/post/terraform-terragrunt-wrappers) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud & Infrastructure Automation](/devtips/categories/cloud--infrastructure-automation) The problem with Terraform at scale Duplicated code across environments If you're managing infrastructure with Terraform across several environments or projects, you've probably hit the point [#Terraform](/devtips/tags/terraform)[#Terragrunt](/devtips/tags/terragrunt)[#Infrastructure as Code](/devtips/tags/infrastructure-as-code)+4 tags [read more](/devtips/post/terraform-terragrunt-wrappers) [![HashiCorp Pulls the Plug on CDKTF](/_astro/hero.BBIsBB2t_Z22hNwP.webp)](/devtips/post/cdktf-deprecation-hashicorp-terraform) ## [HashiCorp Pulls the Plug on CDKTF](/devtips/post/cdktf-deprecation-hashicorp-terraform) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud & Infrastructure Automation](/devtips/categories/cloud--infrastructure-automation) CDKTF is officially deprecated The deprecation announcement Well, it finally happened. HashiCorp (now owned by IBM) officially deprecated the Cloud Development Kit for Terraform (CDKTF) [#Terraform](/devtips/tags/terraform)[#CDKTF](/devtips/tags/cdktf)[#HashiCorp](/devtips/tags/hashicorp)+6 tags [read more](/devtips/post/cdktf-deprecation-hashicorp-terraform) 6 related posts