00:00

Help Us Improve

We're sorry this tutorial wasn't helpful. Please tell us what we can do better.

Logging, Monitoring and Tracing in System Design: A Practical Guide

Last updated:

Logging, monitoring and tracing are the three ways a running system tells you what it is doing. Logs record individual events, metrics track numbers over time so you can monitor health and raise alerts, and traces follow one request as it moves through many services. Together they are called observability: the ability to understand a system from the outside.

This tutorial explains each signal in plain terms, shows how they fit together in a real pipeline, and builds small Java examples of structured logs, latency percentiles and distributed traces. It also covers traps many guides skip, such as correlation IDs vanishing in thread pools and averages hiding slow requests.

1. What Are Logging, Monitoring and Tracing?

  • A log is a timestamped record of something that happened: "order ORD-1042 received", "payment declined". Logs answer what exactly happened?
  • A metric is a number measured over time, such as requests per second, error rate or memory use. Monitoring means watching metrics on dashboards and alerting when they cross a threshold. Metrics answer is the system healthy?
  • A trace records the path of a single request across services. Each step is a span with a start time and duration. Traces answer where did the time go?

A typical investigation uses all three: an alert from a metric says checkout is slow, a trace shows the payment service is the bottleneck, and the payment service's logs explain why.

2. Why Observability Matters in Distributed Systems

In a single application, you can attach a debugger or read one log file. In a microservices architecture, one click can touch ten services on fifty machines. No single log file tells the whole story, and failures often appear only under real traffic.

Good observability shortens the time from "something is wrong" to "here is the cause", which directly improves recovery time and fault tolerance. It also catches slow degradation, such as a memory leak or a creeping error rate, before users notice.

3. How Logs, Metrics and Traces Work

3.1 Logs

Modern systems write structured logs: one JSON object per line instead of free text, so tools can search and filter by field. Each log line carries a correlation ID, a value shared by every log line belonging to the same request. Java applications usually log through SLF4J with Logback or Log4j 2, using log levels (TRACE, DEBUG, INFO, WARN, ERROR) to control detail.

3.2 Metrics

Metrics are cheap to store because they are numbers, not text. Common types are counters (only go up, such as total requests), gauges (go up and down, such as queue size) and histograms (distributions, such as response times). Two popular checklists help pick what to measure: the RED method (rate, errors, duration) for services, and the four golden signals (latency, traffic, errors, saturation). In Spring Boot, Micrometer and Spring Boot Actuator expose these for Prometheus to collect.

3.3 Traces

When service A calls service B, it passes the trace ID and its own span ID in a header. The standard is the W3C traceparent header: 00-<trace-id>-<parent-span-id>-<flags>. Every service records its span with the same trace ID, so a tracing tool can rebuild the whole request as a timeline. Because tracing every request is expensive, systems usually keep only a sample, such as 10% of requests or every slow and failed one.

3.4 Putting It Together

A typical logging, monitoring and tracing pipeline Java services send logs, metrics and traces over OTLP to an OpenTelemetry Collector. The collector sends logs to a log store such as Elasticsearch or Loki, metrics to Prometheus, and traces to a trace store such as Jaeger or Tempo. Grafana reads all three stores to show dashboards and fire alerts. Java services logs, metrics, traces OTLP OpenTelemetry Collector Log store Elasticsearch or Loki Metrics store Prometheus Trace store Jaeger or Tempo Grafana dashboards and alerts
Figure 1: services send all three signals to a collector, which routes each to its own store; one dashboard tool queries them all, joined by trace ID.

OpenTelemetry is the open standard and toolkit for producing all three signals in one consistent way. Prometheus can also scrape metrics directly from each app's endpoint instead of receiving them from a collector.

4. Logging vs Monitoring vs Tracing

LoggingMonitoring (metrics)Tracing
Question it answersWhat exactly happened?Is the system healthy?Where did the time go?
DataTimestamped events with detailsNumbers over timeSpans of one request across services
CostHigh: large text volumesLow: compact numbersMedium: usually sampled
Best forDebugging a specific errorDashboards and alertsFinding slow or failing hops
Java toolsSLF4J, Logback, Log4j 2Micrometer, ActuatorOpenTelemetry, Micrometer Tracing

5. Logging, Monitoring and Tracing in Java: Code Examples

The examples use only the JDK and need Java 17 or later. Fixed clocks and seeds keep the output repeatable. Real projects use the libraries above, but the mechanics are the same.

5.1 Example 1: Structured Logs with a Correlation ID

StructuredLoggingDemo.java 
package com.javabytechie.observability;

import java.time.Clock;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class StructuredLoggingDemo {

    // The current request's correlation ID, per thread (SLF4J's MDC works the same way)
    private static final ThreadLocal<String> CORRELATION_ID = new ThreadLocal<>();

    // A fixed clock keeps the demo output stable; real code uses the system clock
    private static final Clock CLOCK = Clock.fixed(Instant.parse("2026-11-02T09:15:30Z"), ZoneOffset.UTC);

    // Writes one JSON object per line; fields are key, value pairs
    static void log(String level, String event, Object... fields) {
        StringBuilder json = new StringBuilder()
                .append("{\"ts\":\"").append(Instant.now(CLOCK))
                .append("\",\"level\":\"").append(level)
                .append("\",\"event\":\"").append(event)
                .append("\",\"correlationId\":\"").append(CORRELATION_ID.get()).append('"');
        for (int i = 0; i < fields.length; i += 2) {
            Object value = fields[i + 1];
            json.append(",\"").append(fields[i]).append("\":")
                .append(value instanceof Number ? value : "\"" + value + "\"");
        }
        System.out.println(json.append('}'));
    }

    public static void main(String[] args) throws InterruptedException, ExecutionException {
        CORRELATION_ID.set("req-7f3a");
        log("INFO", "order_received", "orderId", "ORD-1042", "items", 3);

        ExecutorService pool = Executors.newSingleThreadExecutor();

        // Pitfall: the pool thread has its own ThreadLocal, so the ID is lost
        pool.submit(() -> log("WARN", "payment_slow", "latencyMs", 2300)).get();

        // Fix: copy the ID into the task and clean up afterwards
        String id = CORRELATION_ID.get();
        pool.submit(() -> {
            CORRELATION_ID.set(id);
            try {
                log("WARN", "payment_slow", "latencyMs", 2300);
            } finally {
                CORRELATION_ID.remove();
            }
        }).get();

        pool.shutdown();
        CORRELATION_ID.remove();
    }
}
Output 
{"ts":"2026-11-02T09:15:30Z","level":"INFO","event":"order_received","correlationId":"req-7f3a","orderId":"ORD-1042","items":3}
{"ts":"2026-11-02T09:15:30Z","level":"WARN","event":"payment_slow","correlationId":"null","latencyMs":2300}
{"ts":"2026-11-02T09:15:30Z","level":"WARN","event":"payment_slow","correlationId":"req-7f3a","latencyMs":2300}

The correlation ID lives in a ThreadLocal, which is exactly how SLF4J's MDC works. The second line shows a common bug: work handed to a thread pool loses the ID, because the pool thread has its own copy. Copying the ID into the task fixes it; tracing libraries do this automatically for supported executors.

5.2 Example 2: Why Averages Hide Problems

LatencyMetricsDemo.java 
package com.javabytechie.observability;

import java.util.Arrays;
import java.util.Random;

public class LatencyMetricsDemo {

    public static void main(String[] args) {
        Random random = new Random(7);            // fixed seed: the same "traffic" on every run
        int requests = 1_000;
        long[] latencyMs = new long[requests];
        int errors = 0;

        for (int i = 0; i < requests; i++) {
            boolean slowPath = random.nextInt(100) < 3;          // about 3% of requests hit a slow database query
            latencyMs[i] = slowPath ? 1_500 + random.nextInt(1_500) : 20 + random.nextInt(60);
            if (random.nextInt(1_000) < 12) {
                errors++;                                        // about 1.2% of requests fail
            }
        }

        Arrays.sort(latencyMs);
        double average = Arrays.stream(latencyMs).average().orElse(0);

        System.out.printf("requests: %d, errors: %d (%.1f%%)%n", requests, errors, errors * 100.0 / requests);
        System.out.printf("average: %.0f ms%n", average);
        System.out.println("p50:     " + percentile(latencyMs, 50) + " ms");
        System.out.println("p95:     " + percentile(latencyMs, 95) + " ms");
        System.out.println("p99:     " + percentile(latencyMs, 99) + " ms");
        System.out.println("max:     " + latencyMs[requests - 1] + " ms");
    }

    // Nearest-rank percentile on sorted data
    static long percentile(long[] sorted, int p) {
        int rank = (int) Math.ceil(p / 100.0 * sorted.length);
        return sorted[Math.max(rank, 1) - 1];
    }
}
Output 
requests: 1000, errors: 9 (0.9%)
average: 100 ms
p50:     50 ms
p95:     78 ms
p99:     2076 ms
max:     2956 ms

The average of 100 ms looks fine, and even p95, meaning 95% of requests were faster, is only 78 ms. But p99 shows that 1 in 100 users waits over two seconds. Alert on percentiles, not averages; the latency page explains tail latency in more detail.

5.3 Example 3: Passing a Trace Across Services

TracingDemo.java 
package com.javabytechie.observability;

import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
import java.util.Random;

public class TracingDemo {

    record Span(String service, String operation, String traceId, String spanId,
                String parentSpanId, long startMs, long durationMs) { }

    private static final Random RANDOM = new Random(2026);    // fixed seed: stable IDs for the demo
    private static final List<Span> SPANS = new ArrayList<>();

    static String randomHex(int bytes) {
        byte[] b = new byte[bytes];
        RANDOM.nextBytes(b);
        return HexFormat.of().formatHex(b);
    }

    // W3C Trace Context header: version-traceId-parentSpanId-flags (01 = sampled)
    static String traceparent(Span span) {
        return "00-" + span.traceId() + "-" + span.spanId() + "-01";
    }

    // Starts a span, joining the caller's trace if a traceparent header was received
    static Span startSpan(String service, String operation, String incoming, long startMs, long durationMs) {
        String traceId = incoming == null ? randomHex(16) : incoming.split("-")[1];
        String parentSpanId = incoming == null ? null : incoming.split("-")[2];
        Span span = new Span(service, operation, traceId, randomHex(8), parentSpanId, startMs, durationMs);
        SPANS.add(span);
        return span;
    }

    public static void main(String[] args) {
        // Timings are simulated; in a real system each service measures its own span
        Span checkout = startSpan("checkout-service", "POST /checkout", null, 0, 340);
        String toPayment = traceparent(checkout);
        System.out.println("checkout -> payment   traceparent: " + toPayment);

        Span payment = startSpan("payment-service", "POST /payments", toPayment, 20, 250);
        String toFraud = traceparent(payment);
        System.out.println("payment  -> fraud     traceparent: " + toFraud);

        startSpan("fraud-service", "GET /score", toFraud, 40, 180);
        startSpan("inventory-service", "POST /reserve", toPayment, 280, 40);

        System.out.println();
        System.out.println("trace " + checkout.traceId());
        for (Span span : SPANS) {
            String name = "  ".repeat(depth(span)) + span.service() + " " + span.operation();
            String bar = " ".repeat((int) (span.startMs() / 20)) + "#".repeat((int) Math.max(1, span.durationMs() / 20));
            System.out.printf("%-38s %4d ms |%-18s|%n", name, span.durationMs(), bar);
        }
    }

    static int depth(Span span) {
        int depth = 0;
        String parent = span.parentSpanId();
        while (parent != null) {
            depth++;
            String p = parent;
            parent = SPANS.stream().filter(s -> s.spanId().equals(p)).findFirst().map(Span::parentSpanId).orElse(null);
        }
        return depth;
    }
}
Output 
checkout -> payment   traceparent: 00-efb75e9ec5c2a532bbc865d964adcf2d-929a56ca6d1de532-01
payment  -> fraud     traceparent: 00-efb75e9ec5c2a532bbc865d964adcf2d-ea369487b992389c-01

trace efb75e9ec5c2a532bbc865d964adcf2d
checkout-service POST /checkout         340 ms |################# |
  payment-service POST /payments        250 ms | ############     |
    fraud-service GET /score            180 ms |  #########       |
  inventory-service POST /reserve        40 ms |              ##  |

All four spans share one trace ID, and each traceparent carries the caller's span ID as the parent. The timeline makes the bottleneck obvious: fraud scoring takes 180 of the 340 ms.

6. Common Mistakes

  • Logging secrets and personal data: passwords, tokens and card numbers end up in log stores with weak access rules. Mask them at the source.
  • High-cardinality metric labels: adding a user ID or order ID as a label creates a new time series per value and can overwhelm Prometheus. Put those in logs or traces instead.
  • Building log messages eagerly: log.debug("cart " + cart) builds the string even when DEBUG is off; use placeholders such as log.debug("cart {}", cart).
  • Losing context across threads: MDC and trace context do not follow work into thread pools or CompletableFuture unless propagated.
  • Alerting on everything: too many noisy alerts teach people to ignore them. Alert on user-facing symptoms, not every internal blip.
  • Ignoring the cost: verbose logs at full traffic can cost more than the servers they describe.

7. Best Practices

  • Log in structured JSON with a timestamp, level, service name and correlation or trace ID on every line.
  • Track the RED metrics for every service and use histograms, so you can see p95 and p99.
  • Propagate trace context through every HTTP call, message and thread pool, using OpenTelemetry.
  • Sample traces sensibly, keeping all errors and slow requests.
  • Alert on symptoms and link alerts to dashboards and runbooks; pair alerts with circuit breakers and other automatic safeguards.
  • Set retention by value: keep metrics for months, and detailed logs and traces for days.

8. Summary

  • Logs record events, metrics measure health over time, and traces follow one request across services.
  • Correlation and trace IDs tie the three together.
  • Percentiles such as p99 reveal problems that averages hide.
  • OpenTelemetry provides one standard way to produce all three signals.
  • Watch for leaked secrets, high-cardinality labels and lost context in thread pools.

Logging, monitoring and tracing turn a distributed system from a black box into something you can question, so build them in from the first service, not after the first outage. Open the FAQ tab for interview questions on observability.