Centralized Logging for Microservices

Centralized logging for microservices means collecting the logs of every service instance into one searchable system, instead of leaving them in files on many servers. An agent ships each service's log lines to a central store, where you can search, filter and alert across all services at once. Combined with a shared trace ID, it lets you follow a single request through every service it touched.

This tutorial explains how a centralized logging pipeline works, what makes logs useful once they are centralized, and how to compare the popular tool stacks. Three runnable Java 21 examples show structured JSON logs, correlation IDs and masking sensitive data. Interview questions follow at the end.

1. What Is Centralized Logging?

A log is a record of something that happened in an application: a request arrived, an order was saved, a payment failed. In a monolith, all of these go to one file on one server, and you can read it with a text editor.

Centralized logging moves every log line from every service to one place. That place stores the lines, indexes them (builds a lookup structure so searches are fast) and gives you a web UI to search them. Kibana and Grafana are two common UIs.

2. Why Microservices Need Centralized Logging

Imagine a customer reports: "My order failed at 9:15." One checkout request touched the API gateway, the Order Service, the Payment Service and the Inventory Service. Each runs three instances, so the logs you need are spread across up to twelve places.

It gets worse with containers. In Kubernetes, a crashed pod is replaced by a new one, and the old pod's logs disappear with it, often exactly the logs you need. Instances also scale up and down all day, so there is no fixed list of servers to check.

Centralized logging solves both problems: logs leave the instance almost immediately, and one search covers every service.

3. How Centralized Logging Works

1. Produce. Each service writes one log event per line to standard output (stdout), not to its own files. The platform captures that output; Kubernetes stores it in files on the node under /var/log/pods.
2. Collect. A lightweight agent, such as Fluent Bit, Filebeat or the OpenTelemetry Collector, runs on every node, reads those files and adds labels like the pod and service name. Some teams run the agent as a sidecar next to each service instead.
3. Buffer (optional). Large systems put Kafka between the agents and the store, so a burst of logs or a slow store does not lose data.
4. Store and index. Elasticsearch, OpenSearch or Grafana Loki keep the logs for a set time.
5. Search and alert. Developers search in Kibana or Grafana, and alerts fire on patterns such as a jump in ERROR lines.

Writing to stdout keeps the service simple: it does not need to know where logs go. This follows the Twelve-Factor App rule of treating logs as a stream of events.

4. Core Concepts: Making Logs Searchable

4.1 Structured Logs

A plain text line such as Order ORD-1001 placed for 2499.00 is easy for humans but hard for machines. Structured logs write each event as JSON with named fields, so you can search orderId:"ORD-1001" or level:ERROR AND service:payment-service directly. Spring Boot 3.4 and later can do this without extra libraries: set logging.structured.format.console=ecs (Elastic Common Schema), logstash or gelf.

4.2 Correlation and Trace IDs

A correlation ID, or trace ID, is a unique value created when a request enters the system and passed to every service in a header. Every log line includes it, so searching for one ID returns the request's whole journey. With asynchronous communication, the ID travels in message headers instead. With Micrometer Tracing on the classpath, Spring Boot adds traceId and spanId to every log line for you.

4.3 Log Levels

Levels, from most to least detailed, are TRACE, DEBUG, INFO, WARN and ERROR. Production usually runs at INFO. Use ERROR only for problems someone must act on, otherwise alerts on ERROR become noise.

4.4 Retention

Central storage costs money and grows fast. Decide how long to keep logs, for example 14 days searchable and longer in cheap archive storage if regulations require it.

5. Centralized Logging for Microservices in Java: Examples

These programs use only the JDK on Java 21, so you can see what logging libraries do inside. In real services you would use SLF4J with Logback, which Spring Boot includes by default.

5.1 Writing Structured JSON Logs

import java.math.BigDecimal;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.LinkedHashMap;
import java.util.Map;

public class StructuredJsonLogger {

    static Clock clock = Clock.fixed(Instant.parse("2026-10-28T09:15:30.120Z"), ZoneOffset.UTC); // fixed for repeatable output

    static String escape(String s) {
        return s.replace("\\", "\\\\").replace("\"", "\\\"").replace("\n", "\\n").replace("\t", "\\t");
    }

    // One event = one line of JSON with named fields, so the log store can search by field
    static void log(String level, String service, String message, Map<String, Object> fields) {
        StringBuilder json = new StringBuilder("{");
        json.append("\"@timestamp\":\"").append(Instant.now(clock)).append("\",");
        json.append("\"level\":\"").append(level).append("\",");
        json.append("\"service\":\"").append(service).append("\",");
        json.append("\"message\":\"").append(escape(message)).append("\"");
        fields.forEach((key, value) -> {
            json.append(",\"").append(key).append("\":");
            json.append(value instanceof Number ? value.toString() : "\"" + escape(value.toString()) + "\"");
        });
        System.out.println(json.append("}"));
    }

    public static void main(String[] args) {
        Map<String, Object> fields = new LinkedHashMap<>();
        fields.put("traceId", "7f3a9c");
        fields.put("orderId", "ORD-1001");
        fields.put("amount", new BigDecimal("2499.00"));
        log("INFO", "order-service", "Order placed", fields);

        // A stack trace has many lines; escaped, it stays ONE event instead of 3 unrelated ones
        String error = "PaymentTimeoutException: no reply in 2000 ms\n\tat PaymentClient.charge(PaymentClient.java:42)";
        log("ERROR", "order-service", error, Map.of("traceId", "7f3a9c"));
    }
}

{"@timestamp":"2026-10-28T09:15:30.120Z","level":"INFO","service":"order-service","message":"Order placed","traceId":"7f3a9c","orderId":"ORD-1001","amount":2499.00}
{"@timestamp":"2026-10-28T09:15:30.120Z","level":"ERROR","service":"order-service","message":"PaymentTimeoutException: no reply in 2000 ms\n\tat PaymentClient.charge(PaymentClient.java:42)","traceId":"7f3a9c"}

Look at the second line. A stack trace normally spans many lines, and a line-based agent would split it into separate, unrelated events. Inside JSON the newlines are escaped as \n, so the whole error stays one searchable event. Escaping also blocks log injection, where a user puts a newline in their input to forge a fake log line.

5.2 Correlation IDs and the Thread Hop Problem

SLF4J's MDC (Mapped Diagnostic Context) stores values such as the trace ID per thread, using a ThreadLocal. This example builds the same idea and shows a pitfall many tutorials skip:

import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class CorrelationIdContext {

    // Works like SLF4J's MDC: a per-thread context added to every log line
    static final ThreadLocal<String> traceId = new ThreadLocal<>();

    static void log(String service, String message) {
        System.out.println("[" + service + "] traceId=" + traceId.get() + " " + message);
    }

    // Copy the caller's context into the task, and clean up so pooled threads do not keep it
    static Runnable withContext(Runnable task) {
        String captured = traceId.get();
        return () -> {
            traceId.set(captured);
            try {
                task.run();
            } finally {
                traceId.remove();
            }
        };
    }

    public static void main(String[] args) {
        try (ExecutorService pool = Executors.newSingleThreadExecutor()) {
            traceId.set("7f3a9c");     // set once when the request arrives, e.g. from the X-Trace-Id header
            log("order-service", "Received POST /orders");

            CompletableFuture.runAsync(() -> log("order-service", "Sending confirmation email"), pool).join();
            CompletableFuture.runAsync(withContext(() -> log("order-service", "Sending confirmation email")), pool).join();

            log("order-service", "Responded 201 Created");
            traceId.remove();          // the request is finished; free the thread for the next one
        }
    }
}

[order-service] traceId=7f3a9c Received POST /orders
[order-service] traceId=null Sending confirmation email
[order-service] traceId=7f3a9c Sending confirmation email
[order-service] traceId=7f3a9c Responded 201 Created

The first email log has traceId=null: the work moved to a pool thread, and a ThreadLocal does not follow it. That line will never appear when you search by trace ID. The fix is to copy the context into the task, as withContext does. In Spring, a TaskDecorator or Micrometer's context propagation does this for you.

5.3 Masking Sensitive Data Before It Leaves the Service

Once logs are centralized, many people and tools can read them. Tokens, passwords and card numbers must be removed before a line is written:

import java.util.List;
import java.util.regex.Pattern;

public class LogMasker {

    record Rule(Pattern pattern, String replacement) {}

    static final List<Rule> RULES = List.of(
            new Rule(Pattern.compile("(?i)(authorization: bearer )[A-Za-z0-9._-]+"), "$1****"),
            new Rule(Pattern.compile("\\b(\\d{4})[ -]?\\d{4}[ -]?\\d{4}[ -]?(\\d{4})\\b"), "$1-****-****-$2"),
            new Rule(Pattern.compile("(?i)(\"?password\"?\\s*[:=]\\s*\"?)[^\",\\s]+"), "$1****"),
            new Rule(Pattern.compile("([A-Za-z0-9._%+-])[A-Za-z0-9._%+-]*(@[A-Za-z0-9.-]+)"), "$1***$2"));

    // Mask BEFORE the line leaves the service: once shipped, many people and systems can read it
    static String mask(String line) {
        for (Rule rule : RULES) {
            line = rule.pattern().matcher(line).replaceAll(rule.replacement());
        }
        return line;
    }

    public static void main(String[] args) {
        List<String> lines = List.of(
                "Calling payment-service with Authorization: Bearer eyJhbGciOiJSUzI1NiJ9.abc.def",
                "Charge failed for card 4111 1111 1111 1234, amount 2499.00",
                "Login request {\"user\":\"priya\",\"password\":\"S3cret!\"}",
                "Sent invoice to priya.sharma@example.com");
        lines.forEach(line -> System.out.println(mask(line)));
    }
}

Calling payment-service with Authorization: Bearer ****
Charge failed for card 4111-****-****-1234, amount 2499.00
Login request {"user":"priya","password":"****"}
Sent invoice to p***@example.com

Pattern-based masking is a safety net, not the main defence. The better habit is to never pass secrets to a log statement in the first place.

6. ELK vs EFK vs Grafana Loki

Managed services such as AWS CloudWatch Logs, Google Cloud Logging and Datadog follow the same pipeline, and you do not have to run the store yourself.

7. Common Mistakes and Pitfalls

• Writing log files inside containers. The files vanish with the container and fill its disk. Write to stdout.
• Plain text logs. Without fields, every search becomes a fragile text match.
• No trace ID, or one lost on thread hops, as in section 5.2.
• Trusting timestamps to order events across services. Server clocks drift by milliseconds or more, so a reply can appear to happen before its request. Use the trace ID and trace spans for order across services; timestamps are reliable only within one instance.
• Logging secrets or personal data. A central log store becomes a data leak with a search box.
• DEBUG in production. Log volume and storage cost multiply, and the important lines get buried.
• Using logs as metrics. Counting requests by searching logs is slow and expensive. Use metrics for numbers and logs for details.

8. Best Practices

• Log structured JSON to stdout and let an agent ship it.
• Use the same field names in every service, for example by adopting the ECS format, so one query works everywhere.
• Include the trace ID, service name and environment in every line.
• Log events, not noise: one line per meaningful step, with the IDs needed to investigate.
• Mask sensitive data at the source, and restrict who can search production logs.
• Set retention and volume limits, and alert on error rates rather than single errors.
• Make sure logging never blocks requests: buffer in the agent or Kafka, not in the service's request threads.

9. Summary

• Centralized logging collects logs from every service instance into one searchable store.
• Services write to stdout; agents collect and ship; Elasticsearch or Loki store and index; Kibana or Grafana search.
• Structured JSON logs and a trace ID in every line make the central store useful.
• Watch for context lost on thread hops, clock drift, multi-line stack traces and leaked secrets.
• Done well, centralized logging for microservices turns "search twelve servers" into one query by trace ID.

Centralized Logging: Interview Questions

Common interview questions on centralized logging in microservices, from basic to scenario-based, with short answers.