Distributed Tracing in Microservices

Distributed tracing follows a single request as it travels through many microservices and records how long each step took. Every request gets a unique trace ID that is passed from service to service, and each service reports the work it did as timed spans. A tracing tool then joins those spans into one timeline, so you can see exactly where a slow or failed request spent its time.

This tutorial explains traces, spans and context propagation, how tracing works in a Spring Boot system, and how to read a trace. Three runnable Java 21 examples show the W3C traceparent header, building a trace timeline, and why sampling must be consistent. Interview questions follow at the end.

1. What Is Distributed Tracing?

In a monolith, one request runs inside one process, and a profiler or a stack trace shows where time goes. In microservices, one click can pass through an API gateway, three services, a database and a message broker. Each piece only sees its own part.

Distributed tracing connects those parts. It answers questions like "Why did this checkout take 4 seconds?" or "Which service returned the error?", for one specific request, across every service it touched.

2. Why Microservices Need Distributed Tracing

Metrics tell you that something is wrong: checkout latency rose from 300 ms to 2 seconds. Logs tell you what each service printed. Neither shows the path of one request, or which of six calls was slow.

A trace does. It shows the calls in order, how long each took, which ran in parallel, and where the error started. It also reveals problems nobody designed, such as a service calling another one 40 times in a loop, or an unexpected dependency on a slow legacy system.

3. Core Concepts: Traces, Spans and Context

3.1 Trace

A trace is the full journey of one request. It is identified by a trace ID, a random 128-bit value written as 32 hexadecimal characters.

3.2 Span

A span is one timed unit of work within a trace, such as "handle POST /orders", "SELECT from cart" or "call Payment Service". Each span has its own span ID, a name, a start time, a duration, a status (OK or error), and attributes: key-value details such as http.response.status_code=500.

Spans form a tree. Every span except the first, the root span, has a parent span ID pointing to the span that caused it. OpenTelemetry also gives each span a kind: SERVER (handling an incoming call), CLIENT (making an outgoing call), PRODUCER and CONSUMER (messaging), or INTERNAL.

3.3 Context Propagation

For services to add spans to the same trace, the trace ID and the current span ID must travel with each call. This is context propagation. For synchronous calls over HTTP, the standard is the W3C Trace Context traceparent header. For Kafka and other brokers used for asynchronous communication, the same values travel in message headers.

4. How Distributed Tracing Works

1. Instrument. A tracing library creates spans automatically for incoming requests, outgoing HTTP calls, database queries and messages. You can add your own spans for important business steps.
2. Propagate. Every outgoing call carries the traceparent header. The receiving service continues the same trace.
3. Export. When a span ends, the library buffers it and sends it in batches in the background, usually with the OpenTelemetry protocol (OTLP), so the request is not slowed down.
4. Collect and store. An OpenTelemetry Collector receives spans from all services and forwards them to a backend such as Jaeger, Zipkin or Grafana Tempo.
5. View. The backend groups spans by trace ID and draws a timeline, often called a waterfall.

4.1 Distributed Tracing in Spring Boot

Spring Boot 3 and later use Micrometer Tracing, which replaced Spring Cloud Sleuth. Add Spring Boot Actuator, a tracer bridge such as micrometer-tracing-bridge-otel, and an exporter such as opentelemetry-exporter-otlp. Spring then traces incoming requests, RestClient and WebClient calls, and adds traceId and spanId to log lines. Clients must be created from the auto-configured builders; a client built with new is not instrumented.

An alternative is the OpenTelemetry Java agent, started with -javaagent:opentelemetry-javaagent.jar, which instruments many libraries without code changes.

5. Distributed Tracing in Java: Examples

These programs use only the JDK on Java 21, so you can see what a tracing library does inside. In real services, use OpenTelemetry or Micrometer Tracing instead of writing this yourself.

5.1 Creating and Continuing a traceparent Header

The header has four parts: version, trace ID, parent span ID and flags. The lowest bit of the flags means "sampled":

import java.util.HexFormat;
import java.util.Random;
import java.util.regex.Pattern;

public class TraceparentHeader {

    // W3C Trace Context: version-traceId-parentSpanId-flags
    record SpanContext(String traceId, String spanId, boolean sampled) {
        String toTraceparent() {
            return "00-" + traceId + "-" + spanId + "-" + (sampled ? "01" : "00");
        }
    }

    static final Pattern FORMAT = Pattern.compile("([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})");
    static final Random random = new Random(42);          // fixed seed so this output repeats; real tracers use random IDs

    static String randomHex(int bytes) {
        byte[] b = new byte[bytes];
        random.nextBytes(b);
        return HexFormat.of().formatHex(b);
    }

    // A new trace starts at the first service that has no incoming header
    static SpanContext startTrace() {
        return new SpanContext(randomHex(16), randomHex(8), true);
    }

    // The next service keeps the trace ID and creates its own span ID
    static SpanContext continueFrom(String header) {
        var m = FORMAT.matcher(header);
        if (!m.matches() || m.group(1).equals("ff") || m.group(2).matches("0+") || m.group(3).matches("0+")) {
            System.out.println("  invalid header '" + header + "' -> starting a new trace");
            return startTrace();
        }
        boolean sampled = (Integer.parseInt(m.group(4), 16) & 1) == 1;   // lowest bit = sampled flag
        return new SpanContext(m.group(2), randomHex(8), sampled);
    }

    public static void main(String[] args) {
        SpanContext gateway = startTrace();
        System.out.println("gateway sends:  traceparent: " + gateway.toTraceparent());

        SpanContext order = continueFrom(gateway.toTraceparent());
        System.out.println("order sends:    traceparent: " + order.toTraceparent());
        System.out.println("same trace? " + order.traceId().equals(gateway.traceId())
                + ", parent of order span = " + gateway.spanId());

        continueFrom("00-00000000000000000000000000000000-" + order.spanId() + "-01");
    }
}

gateway sends:  traceparent: 00-359d41baf78afe0de1bbe7ae28c0450c-e43c084f4bbb2bf1-01
order sends:    traceparent: 00-359d41baf78afe0de1bbe7ae28c0450c-839dee466d852cb5-01
same trace? true, parent of order span = e43c084f4bbb2bf1
  invalid header '00-00000000000000000000000000000000-839dee466d852cb5-01' -> starting a new trace

The Order Service keeps the trace ID and creates a new span ID, so both services' spans belong to one trace. An all-zero trace ID is invalid by the specification, so the receiver ignores the header and starts a new trace instead of failing the request.

5.2 Building the Waterfall, and Finding the Hidden Gap

A tracing backend links spans through their parent IDs and draws them on a timeline. This program does the same with six spans from one checkout request:

import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;

public class TraceWaterfall {

    // What each service reports: times are milliseconds from the start of the request
    record Span(String id, String parentId, String service, String name, int start, int end) {
        int duration() { return end - start; }
    }

    static final List<Span> SPANS = List.of(
            new Span("a1", null, "gateway", "GET /checkout", 0, 412),
            new Span("b2", "a1", "order", "POST /orders", 6, 405),
            new Span("c3", "b2", "order", "SELECT cart", 10, 28),
            new Span("d4", "b2", "order", "call payment (client)", 30, 398),
            new Span("e5", "d4", "payment", "POST /charge (server)", 41, 115),
            new Span("f6", "e5", "payment", "card network call", 45, 110));

    static void print(Span span, int depth) {
        int scale = 10;                                          // one character = 10 ms
        String bar = " ".repeat(span.start() / scale) + "#".repeat(Math.max(1, span.duration() / scale));
        System.out.printf("%-38s %4d ms |%-42s|%n", "  ".repeat(depth) + span.service() + ": " + span.name(),
                span.duration(), bar);
        SPANS.stream().filter(s -> span.id().equals(s.parentId()))
                .sorted(Comparator.comparingInt(Span::start))
                .forEach(child -> print(child, depth + 1));
    }

    public static void main(String[] args) {
        Span root = SPANS.stream().filter(s -> s.parentId() == null).findFirst().orElseThrow();
        print(root, 0);

        // Time a span spends that its children do not explain
        List<String> gaps = new ArrayList<>();
        for (Span span : SPANS) {
            int childTime = SPANS.stream().filter(s -> span.id().equals(s.parentId())).mapToInt(Span::duration).sum();
            if (childTime > 0 && span.duration() - childTime > 100) {
                gaps.add(span.name() + " has " + (span.duration() - childTime) + " ms not covered by child spans");
            }
        }
        gaps.forEach(g -> System.out.println("Look here: " + g));
    }
}

gateway: GET /checkout                  412 ms |######################################### |
  order: POST /orders                   399 ms |#######################################   |
    order: SELECT cart                   18 ms | #                                        |
    order: call payment (client)        368 ms |   ####################################   |
      payment: POST /charge (server)     74 ms |    #######                               |
        payment: card network call       65 ms |    ######                                |
Look here: call payment (client) has 294 ms not covered by child spans

This is the skill most tutorials skip: reading the gaps. The Order Service's client span lasted 368 ms, but the Payment Service's server span lasted only 74 ms. The missing 294 ms happened between the two services: waiting for a free connection in the HTTP client's pool, a slow DNS lookup, a hidden retry, or a busy proxy. Looking only at the Payment Service, you would conclude it is fast, and you would be right. The problem is in the path between the services.

5.3 Why Sampling Must Follow the Trace

Storing every span is expensive, so systems keep only a share of traces. This is called sampling. If each service decides on its own, traces break:

import java.util.HexFormat;
import java.util.Random;

public class ConsistentSampling {

    static final int SERVICES = 3;           // gateway -> order -> payment
    static final double RATE = 0.10;         // keep 10% of traces

    // Every service decides from the trace ID itself, so they all agree without talking to each other
    static boolean sampledByTraceId(String traceId) {
        long lower = Long.parseUnsignedLong(traceId.substring(16), 16) >>> 1;   // lower 8 bytes, made non-negative
        return lower < (long) (RATE * Long.MAX_VALUE);
    }

    public static void main(String[] args) {
        Random random = new Random(1);
        int traces = 100_000, completeIndependent = 0, partialIndependent = 0, completeConsistent = 0;

        for (int t = 0; t < traces; t++) {
            byte[] id = new byte[16];
            random.nextBytes(id);
            String traceId = HexFormat.of().formatHex(id);

            int kept = 0;                                       // each service flips its own coin
            for (int s = 0; s < SERVICES; s++) {
                if (random.nextDouble() < RATE) kept++;
            }
            if (kept == SERVICES) completeIndependent++;
            else if (kept > 0) partialIndependent++;

            if (sampledByTraceId(traceId)) completeConsistent++;   // the same answer in all three services
        }

        System.out.printf("Each service samples on its own: %d complete traces, %d broken traces%n",
                completeIndependent, partialIndependent);
        System.out.printf("Decision follows the trace:      %d complete traces, 0 broken traces%n", completeConsistent);
    }
}

Each service samples on its own: 99 complete traces, 27070 broken traces
Decision follows the trace:      10003 complete traces, 0 broken traces

With independent 10% decisions, only about 0.1% of requests (10% × 10% × 10%) are fully recorded, and over 27% leave useless fragments. That is why the first service decides, and the others follow the sampled flag in traceparent. This is called head-based sampling, and it is what parent-based samplers in OpenTelemetry do. Deciding from the trace ID, as here, gives the same answer everywhere even without a parent.

6. Logs vs Metrics vs Traces

The three work best together. An alert from metrics leads to an example trace, and the trace ID leads to the exact log lines.

7. Common Mistakes and Pitfalls

• Broken propagation. One uninstrumented HTTP client, or a proxy that drops unknown headers, splits a request into two unrelated traces.
• Losing context on thread hops. Work moved to an ExecutorService or @Async method starts a new trace unless the context is passed along. Spring's context propagation support handles this when configured.
• Sampling everything in production. At high traffic, 100% sampling costs a lot in storage and network. Sampling nothing in development hides problems.
• Only keeping random traces. Head-based sampling decides before knowing whether the request will fail, so rare errors may never be recorded. Tail-based sampling in the OpenTelemetry Collector waits until a trace finishes and can keep all slow or failed ones.
• Sensitive or high-cardinality attributes. Card numbers, emails or full request bodies do not belong in span attributes.
• Span names with IDs in them. Use GET /orders/{id}, not GET /orders/ORD-1001, or the backend cannot group similar spans.

8. Best Practices

• Use OpenTelemetry standards: W3C traceparent for propagation and OTLP for export.
• Start at the edge: the gateway should start or accept the trace and pass it on.
• Rely on automatic instrumentation first, then add custom spans for key business steps, such as "reserve stock".
• Put the trace ID in every log line, so traces and logs link both ways.
• Choose sampling deliberately: parent-based head sampling, plus tail sampling for errors and slow requests.
• Propagate context through messages, and use span links when one consumer processes a batch from many traces.
• Export through a Collector rather than straight to the backend, so you can change backends and sampling without redeploying services.

9. Summary

• A trace follows one request across services; spans are its timed steps, linked by parent IDs.
• Context propagation passes the trace ID and parent span ID in the traceparent header or message headers.
• Services export spans in the background through an OpenTelemetry Collector to Jaeger, Zipkin or Tempo.
• Read the gaps between client and server spans, keep sampling decisions consistent, and watch for lost context.
• Combined with logs and metrics, distributed tracing turns "the system is slow" into "this call, in this service, for this request".

Distributed Tracing: Interview Questions

Common interview questions on distributed tracing in microservices, from basic to scenario-based, with short answers.