I was debugging a production issue last week. A user reported that their order was stuck in “processing” for hours. The logs showed nothing. The database seemed fine. The payment service logs were empty. I spent hours checking each service individually, but the problem was in the handoff between them. That’s when I realized: in a world of microservices, traditional debugging is like trying to solve a puzzle with half the pieces missing. This experience is why I want to talk about distributed tracing. It’s the tool that gives you all the pieces back. If you’re building anything more complex than a simple API, you need this. Let’s build it together.
Think of a user clicking “buy now.” That single click might trigger a dozen services: checking inventory, processing payment, sending a confirmation email, updating a warehouse system. If something goes wrong, which service failed? Was it slow? Did the request even reach it? Distributed tracing answers these questions by giving every request a unique ID and tracking its journey through your entire system.
Why should you care? Because without tracing, you’re flying blind in production. An error in your authentication service might manifest as a timeout in your checkout service. You’ll waste days looking in the wrong place. Tracing shows you the complete picture, not just isolated snapshots.
So, how does it work in practice? Let’s start with the core concepts. A “trace” is the entire journey of one request. Within that trace are “spans.” Each span represents a single operation, like a database query or an API call. Spans have parents and children, showing you the flow of execution. This hierarchy is your map through the system.
We’ll use OpenTelemetry. It’s become the standard for collecting traces, metrics, and logs. It’s vendor-neutral, meaning you can send your data to Jaeger, Zipkin, or commercial tools without changing your code. We’ll pair it with Jaeger, a powerful open-source tool for visualizing traces.
Ready to see some code? Let’s set up a project. We’ll create a simple order processing system with Express.js. First, initialize your project and install the necessary packages.
npm init -y
npm install express @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-jaeger @opentelemetry/instrumentation-express @opentelemetry/instrumentation-http pg axiosNext, we need Jaeger running to receive our traces. The easiest way is with Docker.
# docker-compose.yml
version: '3.8'
services:
jaeger:
image: jaegertracing/all-in-one:1.51
ports:
- "16686:16686" # The UI
- "14268:14268" # Where we send tracesRun docker-compose up -d. Open your browser to http://localhost:16686. You’ll see the Jaeger UI, waiting for data. It’s empty now, but not for long.
The heart of our setup is the tracing configuration. We create a file called tracing.js. This code initializes OpenTelemetry and tells it to send data to our Jaeger instance.
// tracing.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const jaegerExporter = new JaegerExporter({
endpoint: 'http://localhost:14268/api/traces',
});
const sdk = new NodeSDK({
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: 'order-service',
}),
traceExporter: jaegerExporter,
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
console.log('Tracing started for order-service');Import this file at the very top of your main application file (app.js or index.js). This ensures tracing is active before any other code runs. Now, what does this get us? Automatic instrumentation for Express and HTTP calls. Every incoming request will automatically start a trace. Every outgoing HTTP call made with axios or the native http module will create a child span. It works out of the box.
But automatic tracing only gets you so far. The real power comes from custom instrumentation. Let’s say you have a function that processes an order. You want to see how long each step takes.
const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('order-processor');
async function processOrder(orderId) {
// Start a new span for this function
return tracer.startActiveSpan('processOrder', async (span) => {
try {
span.setAttribute('order.id', orderId);
// Step 1: Validate order
await validateOrder(orderId);
// Step 2: Charge payment
const paymentResult = await chargePayment(orderId);
span.setAttribute('payment.success', paymentResult.success);
// Step 3: Update inventory
await updateInventory(orderId);
span.setStatus({ code: trace.SpanStatusCode.OK });
return { success: true };
} catch (error) {
// Record the error on the span
span.setStatus({
code: trace.SpanStatusCode.ERROR,
message: error.message,
});
span.recordException(error);
throw error;
} finally {
// Always end the span
span.end();
}
});
}See what we did? We wrapped our business logic in a span. We added useful attributes (order.id, payment.success). If an error happens, we record it directly on the span. Now, in Jaeger, you can search for all traces where payment.success was false. You can see exactly how long the chargePayment step took. This context turns a generic error into a specific, actionable insight.
What about database calls? The automatic instrumentation might not cover your specific PostgreSQL or MongoDB library. You can easily wrap those queries too.
async function queryUserOrders(userId) {
return tracer.startActiveSpan('database.query.user_orders', async (span) => {
span.setAttribute('db.system', 'postgresql');
span.setAttribute('db.user.id', userId);
span.setAttribute('db.statement', 'SELECT * FROM orders WHERE user_id = $1');
try {
const result = await db.query('SELECT * FROM orders WHERE user_id = $1', [userId]);
span.setAttribute('db.rows.returned', result.rows.length);
return result.rows;
} finally {
span.end();
}
});
}Now you can identify slow database queries instantly. Is the user_orders query taking 2 seconds? That span will show you. You can see if it’s related to a specific user ID.
The magic of distributed systems is that services talk to each other. How does a trace cross from your order-service to your payment-service? Through context propagation. OpenTelemetry handles this by injecting trace information into HTTP headers.
When your service makes an outgoing call, the instrumentation automatically adds headers like traceparent. The receiving service, if also instrumented, picks up these headers and continues the same trace. This creates a single, unified view across all your services. You can see the request flow from the initial API gateway, through the order service, to the payment service, and back.
Let’s look at a practical example. Your order service calls a payment service.
// In order-service
const axios = require('axios');
const { trace, context } = require('@opentelemetry/api');
async function callPaymentService(orderData) {
const span = trace.getActiveSpan(); // Get the current active span
console.log(`This trace's ID is: ${span.spanContext().traceId}`);
// The HTTP instrumentation automatically injects headers.
const response = await axios.post('http://payment-service/charge', orderData);
return response.data;
}In the payment service, the Express instrumentation automatically extracts the trace context from the incoming headers and creates a new span that is a child of the one from the order service. You don’t have to write any extra code for this linkage. It just works.
After running your application and making a few requests, go back to the Jaeger UI at http://localhost:16686. Select your order-service from the dropdown and hit “Find Traces.” You’ll see a list. Click on one.
You’ll see a timeline, often called a waterfall view. Each horizontal bar is a span. You can see the total time of the request and drill down into each operation. You can see which span was the slowest. You can see the exact attributes and logs attached to each span. This visualization is where hours of debugging turn into minutes of understanding.
You might wonder, is this expensive? Will it slow down my app? The overhead is minimal. OpenTelemetry samples traces, meaning it doesn’t record every single request by default. You can control the sampling rate. For high-throughput services, you might sample only 10% of requests. For critical user-facing paths, you might sample 100%. The data is also sent asynchronously in batches, so it doesn’t block your application’s response.
Let’s talk about errors. When an error occurs, the trace becomes your best friend. Instead of a generic “500 Internal Server Error” log, you get a complete story. You see that the request entered the order-service, called the validateOrder function successfully, then called the payment-service, which returned a 402 “Insufficient Funds” error. You see the exact duration of each step. You know immediately where the problem is and what kind of problem it is.
This approach changes how you think about monitoring. You move from asking “Is the service up?” to “Is the user experience good?” You can set alerts based on trace data. For example, alert me if the 95th percentile latency for the processOrder span exceeds 3 seconds. Or, alert me if the error rate for calls to the inventory-service spikes above 5%.
We’ve focused on Jaeger, but OpenTelemetry gives you options. If you outgrow Jaeger’s UI or need more advanced analytics, you can switch the exporter to send data to commercial tools like Datadog or New Relic without changing your application code. The instrumentation stays the same.
Start simple. Add basic tracing to your most critical service. Instrument one key transaction. Look at the traces. You’ll be surprised by what you find—often, performance issues you never knew existed. Once you see the value, gradually add more custom instrumentation and roll it out to all your services.
The goal is clarity. In complex systems, complexity is the enemy. Distributed tracing is a light that cuts through that complexity. It gives you and your team the confidence to deploy changes, knowing you can understand their impact in real time. It turns the chaotic web of microservices into a comprehensible, observable system.
I hope this guide helps you see your systems more clearly. The initial setup is a small investment for a massive return in developer productivity and system reliability. Give it a try. Start with one service. I think you’ll wonder how you ever managed without it.
Was this walkthrough helpful? Did it demystify tracing for you? If you have questions or your own experiences to share, please leave a comment below. If you found this useful, consider sharing it with a teammate who’s also wrestling with microservice debugging. Let’s all build more observable systems.
As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!
101 Books
101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.
Check out our book Golang Clean Code available on Amazon.
Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!
📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!
Our Creations
Be sure to check out our creations:
Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools
We are on Medium
Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva
