Monitoring and Logging in Microservices: A Practical Java Guide

Monitoring and logging in microservices means collecting logs, metrics and traces from every service into central tools, so you can see what the whole system is doing and quickly find out why something broke. Logs record individual events, metrics measure behaviour over time, and traces follow a single request as it travels across services.

In a monolith, when something fails, you open one log file on one server. In microservices, a single "Place order" click may pass through five services running on fifteen containers. Without a plan for monitoring and logging, debugging turns into guesswork.

1. Why Microservices Need Special Monitoring

Imagine a food-delivery app. When a customer pays, the Order service calls Payment, then Restaurant, then Delivery. Each service runs as three copies (called instances or pods) behind a load balancer.

A customer complains: "My payment went through but my order never arrived." Where do you look?

• The relevant log lines are spread across 12 different containers.
• Containers are ephemeral: when one restarts, its local files are deleted.
• Each service writes its own log format, so you cannot search them together.
• Nothing links the Payment log line to the Delivery log line for the same order.

The fix: send everything to one place, in one format, tagged with an ID that ties each request together.

2. The Three Pillars of Monitoring and Logging in Microservices

You will often hear the word observability. Monitoring tells you that something is wrong ("error rate is 8%"). Observability is the ability to ask why ("errors come only from Payment pod 2, only for UPI payments"). You get observability from three kinds of data, often called the three pillars.

2.1 Logs

A log is a timestamped record of one event, such as "Order ORD-1042 received". Logs are detailed but expensive to store and search in large volumes.

In microservices, logs should be structured: written as JSON with named fields instead of free text. A log tool can then filter on orderId = ORD-1042 instead of guessing with text search.

2.2 Metrics

A metric is a number measured over time, such as "requests per second" or "memory used". Metrics are cheap because they store totals, not individual events. They power dashboards and alerts.

The three basic metric types are:

• Counter: only goes up (total requests, total errors).
• Gauge: goes up and down (active connections, queue size).
• Histogram / Timer: records how values are spread out, such as response times, so you can calculate percentiles.

A percentile answers "how slow were the slowest requests?" The p99 latency is the time that 99% of requests finished within. The remaining 1% were slower.

2.3 Traces

A trace follows one request across all services. Each step inside it is a span, for example "Payment service: charge card, 120 ms". All spans in a trace share one trace ID, which is passed from service to service in an HTTP header. The modern standard header is the W3C traceparent header.

When the same trace ID is also written into every log line, you can jump from a slow span straight to the exact logs for that request. This is called log correlation, and the ID is often called a correlation ID.

3. How It Works: The Observability Pipeline

Here is what each step does:

1. Logs: the service writes JSON lines to standard output (stdout). The container platform captures them, and a log agent such as Fluent Bit ships them to a log store.
2. Metrics: the service exposes its current metric values at an HTTP endpoint. Prometheus pulls (scrapes) that endpoint at a fixed interval. The service never sends metrics anywhere itself.
3. Traces: a tracing library records spans and pushes them, usually using the OTLP protocol, to an OpenTelemetry Collector, which forwards them to a trace store.
4. Dashboards and alerts: Grafana reads from all three stores. Prometheus sends alerts to Alertmanager, which notifies the on-call engineer.

In Spring Boot 3, the Java side is mostly configuration:

• Spring Boot Actuator adds endpoints such as /actuator/health and /actuator/prometheus.
• Micrometer is the metrics library. Think of it as "SLF4J for metrics": you write code against one API, and a registry exports to Prometheus or another backend.
• Micrometer Tracing creates spans and puts the trace ID into the logging MDC. It replaced Spring Cloud Sleuth, which does not support Boot 3.
• Structured logging is built in from Spring Boot 3.4 with logging.structured.format.console=ecs (or logstash, gelf).

The MDC (Mapped Diagnostic Context) is worth understanding, because Example 1 builds a tiny version of it. It is a per-thread map of values, such as traceId, that the logger automatically adds to every log line. Internally, Logback's MDC is stored in a ThreadLocal, which matters a lot once threads change.

4. Code Examples

These programs use only the JDK (Java 21), so you can see what the frameworks do for you.

4.1 Example 1: Structured Logs with a Trace ID (and Two Hidden Bugs)

import java.time.Clock;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class CorrelationIdLoggingDemo {

    // A tiny MDC: a per-thread map of context values, like SLF4J's MDC
    static final ThreadLocal<Map<String, String>> CONTEXT = ThreadLocal.withInitial(LinkedHashMap::new);
    static final Clock CLOCK = Clock.fixed(Instant.parse("2026-10-28T09:15:00Z"), ZoneOffset.UTC);

    static void log(String level, String message) {
        var json = new StringBuilder("{");
        json.append("\"ts\":\"").append(Instant.now(CLOCK)).append("\",");
        json.append("\"level\":\"").append(level).append("\",");
        json.append("\"service\":\"order-service\",");
        CONTEXT.get().forEach((k, v) -> json.append('"').append(k).append("\":\"").append(v).append("\","));
        json.append("\"msg\":\"").append(message).append("\"}");
        System.out.println(json); // one JSON object per line
    }

    // Copies the caller's context into the worker thread, then cleans up
    static Runnable withContext(Runnable task) {
        Map<String, String> captured = new LinkedHashMap<>(CONTEXT.get());
        return () -> {
            CONTEXT.get().putAll(captured);
            try {
                task.run();
            } finally {
                CONTEXT.remove(); // pooled threads are reused, so always clear
            }
        };
    }

    public static void main(String[] args) throws Exception {
        CONTEXT.get().put("traceId", "7f3a9c2e41d05b88"); // normally read from the request header
        CONTEXT.get().put("orderId", "ORD-1042");
        log("INFO", "Order received");

        try (ExecutorService pool = Executors.newSingleThreadExecutor()) {
            pool.submit(() -> log("INFO", "Reserving stock (plain task)")).get();
            pool.submit(withContext(() -> log("INFO", "Reserving stock (wrapped task)"))).get();

            // A careless task sets context and never clears it ...
            pool.submit(() -> CONTEXT.get().put("traceId", "aa11bb22cc33dd44")).get();
            // ... so a different request on the same pooled thread inherits it
            pool.submit(() -> log("ERROR", "Payment failed for ORD-2077")).get();
        }
        CONTEXT.remove();
    }
}

{"ts":"2026-10-28T09:15:00Z","level":"INFO","service":"order-service","traceId":"7f3a9c2e41d05b88","orderId":"ORD-1042","msg":"Order received"}
{"ts":"2026-10-28T09:15:00Z","level":"INFO","service":"order-service","msg":"Reserving stock (plain task)"}
{"ts":"2026-10-28T09:15:00Z","level":"INFO","service":"order-service","traceId":"7f3a9c2e41d05b88","orderId":"ORD-1042","msg":"Reserving stock (wrapped task)"}
{"ts":"2026-10-28T09:15:00Z","level":"ERROR","service":"order-service","traceId":"aa11bb22cc33dd44","msg":"Payment failed for ORD-2077"}

The output shows two bugs that real teams hit all the time:

• Line 2: the plain task ran on a pool thread, and that thread has its own empty ThreadLocal. The trace ID silently disappeared. Wrapping the task (line 3) copies the context across, which is what Spring's TaskDecorator does for @Async methods.
• Line 4: a task set a trace ID and never cleared it. The pool reused the same thread, so the error for order ORD-2077 is logged with another request's trace ID. Always clear context in a finally block.

4.2 Example 2: Why Averages Lie, and How Labels Explode

import java.util.ArrayList;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

public class LatencyMetricsDemo {

    // Nearest-rank percentile: the value below which p% of samples fall
    static long percentile(List<Long> sorted, double p) {
        int rank = (int) Math.ceil(p / 100.0 * sorted.size());
        return sorted.get(rank - 1);
    }

    public static void main(String[] args) {
        List<Long> latenciesMs = new ArrayList<>();
        for (int i = 1; i <= 100; i++) {
            // Most requests are fast; every 20th one hits a slow database query
            latenciesMs.add(i % 20 == 0 ? 1_800L + i : 30L + (i % 7) * 5);
        }
        Collections.sort(latenciesMs);

        double average = latenciesMs.stream().mapToLong(Long::longValue).average().orElse(0);
        System.out.printf("Average : %.1f ms%n", average);
        System.out.println("p50     : " + percentile(latenciesMs, 50) + " ms");
        System.out.println("p95     : " + percentile(latenciesMs, 95) + " ms");
        System.out.println("p99     : " + percentile(latenciesMs, 99) + " ms");

        // Cardinality: every unique label combination becomes a separate time series
        String[] endpoints = {"/orders", "/payments", "/stock"};
        Set<String> byEndpoint = new HashSet<>();
        Set<String> byEndpointAndUser = new HashSet<>();
        for (int i = 0; i < 10_000; i++) {
            String endpoint = endpoints[i % 3];
            String userId = "user-" + (i % 2_500);
            byEndpoint.add("http_requests_total{endpoint=\"" + endpoint + "\"}");
            byEndpointAndUser.add("http_requests_total{endpoint=\"" + endpoint + "\",user=\"" + userId + "\"}");
        }
        System.out.println("Series with endpoint label      : " + byEndpoint.size());
        System.out.println("Series with endpoint+user labels: " + byEndpointAndUser.size());
    }
}

Average : 135.4 ms
p50     : 45 ms
p95     : 60 ms
p99     : 1880 ms
Series with endpoint label      : 3
Series with endpoint+user labels: 7500

Two lessons:

• The average (135.4 ms) describes nobody. 95 out of 100 requests took 60 ms or less, and 5 took over 1.8 seconds. The p50 and p95 say "fast", and only the p99 reveals the slow requests. Alert on high percentiles, not on averages.
• Labels multiply storage. Prometheus stores one time series per unique combination of metric name and labels. Adding a user label turned 3 series into 7,500. With a million users, your monitoring system becomes the thing that crashes. This problem is called high cardinality.

4.3 Example 3: Liveness vs Readiness Health Checks

Kubernetes asks each pod two different questions. A liveness probe asks "Is this process stuck? Should I restart it?" A readiness probe asks "Can it handle traffic right now? Should I send it requests?"

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.atomic.AtomicBoolean;

public class HealthCheckDemo {

    static void respond(HttpExchange exchange, int status, String body) throws IOException {
        byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
        exchange.getResponseHeaders().set("Content-Type", "application/json");
        exchange.sendResponseHeaders(status, bytes.length);
        try (OutputStream out = exchange.getResponseBody()) {
            out.write(bytes);
        }
    }

    public static void main(String[] args) throws Exception {
        var databaseUp = new AtomicBoolean(true);
        HttpServer server = HttpServer.create(new InetSocketAddress("localhost", 0), 0); // 0 = any free port

        // Liveness: "is this process stuck?" Never check external systems here
        server.createContext("/health/liveness", ex -> respond(ex, 200, "{\"status\":\"UP\"}"));

        // Readiness: "can I serve traffic right now?" Dependencies belong here
        server.createContext("/health/readiness", ex -> {
            if (databaseUp.get()) {
                respond(ex, 200, "{\"status\":\"UP\"}");
            } else {
                respond(ex, 503, "{\"status\":\"OUT_OF_SERVICE\",\"reason\":\"database unreachable\"}");
            }
        });
        server.start();
        String base = "http://localhost:" + server.getAddress().getPort();

        try (HttpClient client = HttpClient.newHttpClient()) { // HttpClient is AutoCloseable since Java 21
            probe(client, base, "liveness");
            probe(client, base, "readiness");

            databaseUp.set(false);
            System.out.println("--- database goes down ---");
            probe(client, base, "liveness");
            probe(client, base, "readiness");
        } finally {
            server.stop(0);
        }
    }

    static void probe(HttpClient client, String base, String probe) throws Exception {
        var request = HttpRequest.newBuilder(URI.create(base + "/health/" + probe)).build();
        var response = client.send(request, HttpResponse.BodyHandlers.ofString());
        System.out.printf("%-9s -> %d %s%n", probe, response.statusCode(), response.body());
    }
}

liveness  -> 200 {"status":"UP"}
readiness -> 200 {"status":"UP"}
--- database goes down ---
liveness  -> 200 {"status":"UP"}
readiness -> 503 {"status":"OUT_OF_SERVICE","reason":"database unreachable"}

When the database goes down, readiness returns 503, so Kubernetes stops routing traffic to this pod. Liveness stays 200, so the pod is not restarted. This is exactly how Spring Boot Actuator's /actuator/health/liveness and /actuator/health/readiness groups are designed. A readiness state of REFUSING_TRAFFIC shows as OUT_OF_SERVICE with HTTP 503.

5. Comparison Table: Logs vs Metrics vs Traces

6. Common Mistakes and Pitfalls

1. Putting database checks in the liveness probe. If liveness fails whenever the database is down, Kubernetes restarts every pod of every service that uses that database at the same time. The restarts fix nothing and add a startup storm on top of the outage. Dependencies belong in readiness, or nowhere.

2. Losing the trace ID across threads. @Async methods, CompletableFuture.supplyAsync() and message listeners run on other threads, so the ThreadLocal-based MDC is empty there unless something copies it, as Example 1 showed.

3. Using IDs as metric labels. User IDs, order IDs and full URLs with path variables (/orders/1042) create unbounded series. Use a route template such as /orders/{id} instead. Spring's HTTP metrics already do this through the uri tag.

4. Logging secrets and personal data. Passwords, card numbers, JWT tokens and phone numbers end up in logs through careless log.info("Request: " + request) calls. Logs are copied to many systems and kept for months, so mask sensitive fields before they are logged.

5. Alerting on everything. Noisy alerts train the team to ignore them. Alert on what users feel: error rate, latency percentiles and availability.

7. Best Practices

• Log to stdout as JSON and let the platform collect it. Do not manage log files inside containers.
• Generate a trace ID at the edge (the API gateway) and propagate it through every call, queue message and log line.
• Use log levels with intent: ERROR for something that needs action, WARN for something unusual, INFO for business events, DEBUG off in production.
• Watch the four golden signals for every service: latency, traffic, errors and saturation.
• Record histograms, not just averages, so percentiles can be calculated across all instances.
• Keep label values bounded, usually under a few hundred per label.
• Expose only the Actuator endpoints you need, and secure them. Endpoints like /actuator/env and /actuator/heapdump can leak secrets.
• Link your monitoring to resilience. Breaker state changes and retry counts from your resilience patterns should appear on the same dashboards.

8. Summary

• Monitoring and logging in microservices brings logs, metrics and traces from many short-lived containers into central tools.
• Logs explain single events, metrics show trends and drive alerts, and traces follow a request across services.
• Write structured JSON logs to stdout and include a trace ID in every line.
• The MDC is ThreadLocal-based: it is lost when work moves to another thread and leaks if a pooled thread is not cleared.
• Averages hide slow requests; alert on p95/p99 latency.
• Every unique label combination is a separate time series, so never use unbounded IDs as labels.
• Liveness checks the process itself; readiness checks dependencies.

Get monitoring and logging in microservices right early, and production problems become questions you can answer in minutes instead of mysteries that take days.

Monitoring and Logging: Interview Questions

Common interview questions on Monitoring and Logging, from basic to scenario-based, with short answers.

Counter.builder("orders.placed").register(registry).increment();
Gauge.builder("queue.size", queue, Queue::size).register(registry);
Timer.builder("payment.latency").register(registry).record(() -> gateway.charge(order));

POST /actuator/loggers/com.shop.payment
{"configuredLevel": "DEBUG"}