Distributed Tracing Explained: Spans, Traces, and Context Propagation

Distributed tracing is how you follow a single request as it travels through a system made of many services. When a checkout call fans out to an auth service, a pricing service, a database, and a payment provider, a stack trace only shows you one process. Distributed tracing stitches all of those hops into one timeline so you can see where the 900 ms actually went. This guide explains what spans and traces are, how trace context propagates across service boundaries, how sampling keeps the cost sane, and how to instrument a service end to end with OpenTelemetry.

This is a supporting page in the Devgains observability cluster. Start with the observability pillar — metrics, logs, and traces for how the three signals fit together, then see the OpenTelemetry Collector guide for shipping this data to a backend. You can browse the whole cluster on the observability category page.

Quick answer: what is distributed tracing?

Distributed tracing records the end-to-end journey of one request across every service it touches and presents it as a single, causally-ordered timeline. It is built from two primitives:

• Span — one unit of work (an HTTP handler, a DB query, an outbound call), with a start time, duration, attributes, and a parent.
• Trace — the full tree of spans that share one trace ID, representing a single request.

The one-line rule: a trace is the request; spans are the steps. Metrics tell you that latency spiked, and a trace tells you which hop caused it.

Why distributed tracing matters

In a monolith, a single stack trace and a log line usually pinpoint a problem. In a distributed system that breaks down for concrete reasons:

• Failures cross process boundaries. A slow request might be fast everywhere except one downstream call. No single log file shows that.
• Latency is additive and hidden. Total time is the sum of many hops plus queueing and serialization. Only a trace attributes the milliseconds to a specific span.
• Fan-out hides the culprit. One API call can trigger a dozen parallel requests. A trace shows which branch of the tree was on the critical path.
• Errors need context. Knowing service C threw is far less useful than knowing service C threw while handling the request that started at the checkout endpoint for this user.

This is the reason tracing is indispensable for anyone running microservices — a topic we cover from the design side in the hidden cost of microservices.

Architecture: spans, traces, and trace context

Every trace is a tree of spans that share a trace ID. Each span carries its own span ID and a pointer to its parent span ID. That parent/child chain is what lets a UI reconstruct the timeline.

The hard part of distributed tracing is not creating spans — it is making the trace ID survive the jump from one service to the next. That is context propagation: the caller serializes the trace context into the outgoing request (usually HTTP headers), and the callee reads it back and continues the same trace instead of starting a new one.

The industry standard is the W3C Trace Context specification, which defines the traceparent header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^  ^                                ^                ^
             |  trace-id (16 bytes)              parent-id        trace-flags (01 = sampled)
             version

When service A calls service B, A writes this header and B parses it. Because both sides agree on the format, a trace flows across languages, frameworks, and vendors without custom glue.

Step-by-step: instrument a service with OpenTelemetry

OpenTelemetry is the vendor-neutral standard for producing traces. Auto-instrumentation gets you most of the way with almost no code.

1. Install the SDK and instrumentations.

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http

2. Start tracing before your app code loads. Auto-instrumentation patches libraries like http and express at require time, so this file must run first (node -r ./tracing.js app.js).

// tracing.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
 
const sdk = new NodeSDK({
  serviceName: "checkout-api",
  traceExporter: new OTLPTraceExporter({
    url: "http://localhost:4318/v1/traces", // OTLP HTTP endpoint (Collector or Tempo)
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
 
sdk.start(); // Incoming requests now produce spans, and outbound calls propagate context.

With this alone, every incoming HTTP request becomes a SERVER span, every outbound call becomes a CLIENT span, and the traceparent header is injected and extracted automatically. Trace context propagation is handled for you.

3. Add a manual span for business logic the auto-instrumentation can't see:

import { trace } from "@opentelemetry/api";
 
const tracer = trace.getTracer("checkout");
 
export async function applyPromo(cartId: string, code: string) {
  return tracer.startActiveSpan("applyPromo", async (span) => {
    span.setAttribute("promo.code", code);
    try {
      const result = await promoService.validate(cartId, code);
      span.setAttribute("promo.valid", result.valid);
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: 2 }); // ERROR
      throw err;
    } finally {
      span.end(); // Always end the span, or the trace never completes.
    }
  });
}

4. Export to a backend. Point the OTLP endpoint at an OpenTelemetry Collector, which forwards to a trace store like Jaeger or Grafana Tempo. See the Collector guide for a production pipeline, and Prometheus for developers for correlating the metrics side.

Sampling: keeping tracing affordable

Recording every span of every request is expensive and mostly redundant. Sampling decides which traces to keep.

The pragmatic default: tail-based sampling that keeps every error and every request over a latency threshold, plus a small percentage of the healthy baseline. You lose noise, not signal.

Best practices

• Propagate context everywhere. A trace that stops at the first hop is worthless. Use libraries that honor W3C Trace Context across HTTP, gRPC, and message queues.
• Put trace_id in your logs. Correlating a log line to its trace is what turns three separate signals into one debugging workflow.
• Use semantic conventions. Name attributes with OpenTelemetry's standard keys (http.method, db.system) so backends and dashboards understand them.
• Keep spans meaningful. One span per logical operation, not per line of code. Over-instrumenting creates noise and cost.
• Sample errors at 100%. Never let a sampling rule throw away the traces you actually need.

Common mistakes

• Broken propagation. A service that starts a fresh trace instead of continuing the incoming one splits your request into disconnected fragments.
• Forgetting to end spans. An unclosed span leaves the trace open and skews durations. Always end() in a finally block.
• Putting high-cardinality data in metrics instead of traces. User IDs and request IDs belong on spans, not metric labels — that distinction is covered in the observability pillar.
• Tracing without sampling. Full-fidelity tracing at scale will blow up your storage bill and add latency to the request path.

Key takeaways

• A trace is one request; spans are the steps within it, linked by trace ID and parent span ID.
• Context propagation via the W3C traceparent header is what makes tracing "distributed."
• OpenTelemetry gives you vendor-neutral instrumentation; a Collector ships spans to Jaeger or Tempo.
• Use tail-based sampling to keep errors and slow requests while dropping healthy noise.
• Tracing answers where latency and errors happen — the question metrics and logs can't.

FAQ

What is distributed tracing? Distributed tracing records the full path of a single request as it moves across multiple services and presents it as one timeline of spans, so you can see exactly where time and errors occurred.

What is the difference between a span and a trace? A span is one unit of work with a duration and attributes. A trace is the complete tree of spans that share a trace ID and represent a single end-to-end request.

How does trace context propagation work? The calling service serializes the trace ID and parent span ID into a header (the W3C traceparent header) on the outgoing request. The receiving service reads that header and continues the same trace instead of starting a new one.

Is distributed tracing the same as logging? No. Logs are discrete event records; a trace is the causal, timed structure of one request across services. They complement each other — correlating logs to traces by trace_id is a best practice.

What tools do I need for distributed tracing? OpenTelemetry to instrument your code, an OpenTelemetry Collector to route the data, and a trace backend such as Jaeger or Grafana Tempo to store and visualize it.

Conclusion

Distributed tracing is the signal that makes distributed systems debuggable. Metrics and logs tell you something is wrong; a trace tells you exactly which service, which call, and which span owns the latency or the error. With OpenTelemetry auto-instrumentation and W3C Trace Context propagation, you get most of that value for very little code — the work is in propagating context reliably and sampling intelligently. Get those two right and every slow request becomes a story you can read from start to finish.

References: W3C Trace Context specification, OpenTelemetry tracing concepts, OpenTelemetry Node.js instrumentation, Jaeger architecture, and Grafana Tempo documentation.