Service Mesh in Microservices: How It Works, With Java Examples

A service mesh in microservices is a dedicated infrastructure layer that handles communication between services: encryption, retries, timeouts, load balancing, traffic routing and telemetry. It works by placing a small network proxy next to every service, so these features are applied to every call without changing the application code.

With dozens of services written by different teams, copying the same networking logic into every codebase becomes slow and inconsistent. A mesh moves it into the platform.

1. The Problem a Service Mesh Solves

Imagine a ride-hailing platform. The Booking service (Java) calls Pricing (Java), Driver Matching (Go) and Payments (Python). Every one of these calls needs the same things:

• Encryption, so nobody inside the network can read fares or card tokens.
• Timeouts and retries, so one slow service doesn't freeze the others.
• Metrics and traces, so you can see which call is slow.
• Traffic control, so you can send 10% of users to a new Pricing version before everyone gets it.

In Java, you could add Resilience4j, Micrometer and TLS configuration to each service. But the Go and Python teams need their own libraries, versions drift apart, and a security fix means redeploying everything.

2. Core Concepts of a Service Mesh in Microservices

2.1 Sidecar Proxy

A proxy is a program that receives network traffic and forwards it somewhere else. In the classic mesh design, every pod gets its own proxy, running as an extra container next to the application. This helper container is called a sidecar, like the sidecar attached to a motorcycle.

All traffic into and out of the application passes through its sidecar. The application still thinks it is talking directly to http://pricing:8080.

The most widely used proxy is Envoy, which Istio, Consul and Kuma use. Linkerd uses its own lightweight proxy written in Rust.

2.2 Data Plane and Control Plane

• The data plane is the set of all proxies. They carry the real traffic.
• The control plane is the "brain". It doesn't touch your requests. It sends configuration (routes, retry rules, security policies) and security certificates to every proxy. In Istio, the control plane is a single component called istiod.

You change behaviour by applying Kubernetes YAML, and the control plane pushes it to every proxy within seconds.

2.3 Mutual TLS (mTLS)

Normal TLS (as in HTTPS) proves the server's identity to the client. Mutual TLS also proves the client's identity to the server. In a mesh, every workload gets a certificate tied to its Kubernetes service account. The proxies then encrypt traffic and check identities automatically, so you can write rules like "only Booking may call Payments".

2.4 Traffic Management

Mesh routing rules can split traffic by weight (90% to v1, 10% to v2), match on headers, mirror traffic to a test version, inject delays or errors for testing, and eject unhealthy instances.

3. How It Works Internally

Here is what happens when the booking app calls http://pricing:8080/quote:

1. Traffic is captured. When the pod starts, a setup step (an init container or a CNI plugin) adds iptables rules. These are Linux firewall rules that redirect all of the pod's network traffic to the sidecar. In Istio, outgoing traffic goes to port 15001 and incoming traffic to port 15006. The app never knows.
2. The client-side proxy decides. The booking sidecar looks up its routing rules, picks a pricing pod (load balancing), applies the timeout and retry policy, and records metrics.
3. mTLS handshake. The two sidecars exchange certificates, verify each other's identity, and encrypt the connection.
4. The server-side proxy checks. The pricing sidecar checks authorization rules, records metrics, and forwards plain HTTP to the pricing app on localhost.
5. Configuration stays fresh. The control plane streams updates to the proxies over the xDS APIs, Envoy's discovery protocol, whenever you change a rule or a pod is added or removed.

So every call goes through two extra hops, one proxy on each side. This is where both the power and the cost of a mesh come from.

A newer design, ambient mode (generally available since Istio 1.24), removes the per-pod sidecar. A shared proxy on each node, called ztunnel, handles mTLS and basic (Layer 4) traffic, and optional waypoint proxies handle HTTP-level (Layer 7) features such as retries and routing.

4. Code Examples

A real mesh needs Kubernetes, so these JDK-only Java 21 programs act out what the proxies do.

4.1 Example 1: A Sidecar That Retries Without the App Knowing

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.concurrent.atomic.AtomicInteger;

public class SidecarProxyDemo {

    static void send(HttpExchange ex, int status, String body) throws IOException {
        byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
        ex.sendResponseHeaders(status, bytes.length);
        try (OutputStream out = ex.getResponseBody()) {
            out.write(bytes);
        }
    }

    public static void main(String[] args) throws Exception {
        // 1. The business app: plain HTTP, no retry or metrics code at all
        var appCalls = new AtomicInteger();
        HttpServer app = HttpServer.create(new InetSocketAddress("localhost", 0), 0);
        app.createContext("/orders/88", ex -> {
            int call = appCalls.incrementAndGet();
            System.out.println("[order-app] handling " + ex.getRequestURI() + " (call #" + call + ")");
            if (call == 1) {
                send(ex, 503, "warming up"); // first call fails, like a pod that is still starting
            } else {
                send(ex, 200, "{\"orderId\":\"ORD-88\",\"status\":\"CONFIRMED\"}");
            }
        });
        app.start();
        String appUrl = "http://localhost:" + app.getAddress().getPort();

        // 2. The sidecar: sits in front of the app and adds retries, timeouts and metrics
        var requests = new AtomicInteger();
        var retries = new AtomicInteger();
        HttpServer sidecar = HttpServer.create(new InetSocketAddress("localhost", 0), 0);
        try (HttpClient upstream = HttpClient.newHttpClient()) {
            sidecar.createContext("/", ex -> {
                requests.incrementAndGet();
                var forward = HttpRequest.newBuilder(URI.create(appUrl + ex.getRequestURI()))
                        .timeout(Duration.ofSeconds(1)) // per-attempt timeout set by mesh config
                        .build();
                try {
                    HttpResponse<String> resp = null;
                    for (int attempt = 1; attempt <= 3; attempt++) {
                        resp = upstream.send(forward, HttpResponse.BodyHandlers.ofString());
                        System.out.println("[sidecar]   attempt " + attempt + " -> " + resp.statusCode());
                        if (resp.statusCode() != 503) {
                            break;
                        }
                        retries.incrementAndGet(); // retry on 503, invisible to the caller
                    }
                    send(ex, resp.statusCode(), resp.body());
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    send(ex, 504, "upstream interrupted");
                }
            });
            sidecar.start();

            // 3. A caller in another pod only ever talks to the sidecar
            try (HttpClient caller = HttpClient.newHttpClient()) {
                var url = URI.create("http://localhost:" + sidecar.getAddress().getPort() + "/orders/88");
                var resp = caller.send(HttpRequest.newBuilder(url).build(), HttpResponse.BodyHandlers.ofString());
                System.out.println("[caller]    got " + resp.statusCode() + " " + resp.body());
            }
            System.out.println("[metrics]   requests_total=" + requests.get() + " retries_total=" + retries.get());
        } finally {
            sidecar.stop(0);
            app.stop(0);
        }
    }
}

[order-app] handling /orders/88 (call #1)
[sidecar]   attempt 1 -> 503
[order-app] handling /orders/88 (call #2)
[sidecar]   attempt 2 -> 200
[caller]    got 200 {"orderId":"ORD-88","status":"CONFIRMED"}
[metrics]   requests_total=1 retries_total=1

The caller got a clean 200. It never saw the first 503, and neither the caller nor the order app contains any retry code. This is exactly what a mesh retry policy does. Notice the metric too: one request from the caller, but two calls reached the app. If the app also retried on its own, the attempts would multiply.

4.2 Example 2: The One Thing the Mesh Cannot Do for You

Proxies can create and report trace spans, but they cannot connect an incoming request to the outgoing calls your code makes while handling it. Only your application knows that.

import java.util.HashMap;
import java.util.HexFormat;
import java.util.Map;
import java.util.Random;

public class TracePropagationDemo {

    static final Random IDS = new Random(2026); // fixed seed so the IDs are repeatable
    static final HexFormat HEX = HexFormat.of();

    static String newId(int bytes) {
        byte[] b = new byte[bytes];
        IDS.nextBytes(b);
        return HEX.formatHex(b);
    }

    // What every sidecar does with an incoming request
    static Map<String, String> sidecarInbound(String service, Map<String, String> headers) {
        var copy = new HashMap<>(headers);
        String traceparent = copy.get("traceparent");
        if (traceparent == null) {
            traceparent = "00-" + newId(16) + "-" + newId(8) + "-01"; // no context: start a new trace
            copy.put("traceparent", traceparent);
            System.out.println("  [" + service + " sidecar] started NEW trace " + traceparent.substring(3, 11));
        } else {
            System.out.println("  [" + service + " sidecar] joined trace     " + traceparent.substring(3, 11));
        }
        return copy;
    }

    // The order app receives a request and calls the payment service
    static void orderApp(Map<String, String> inbound, boolean forwardTraceHeader) {
        Map<String, String> outbound = new HashMap<>();
        if (forwardTraceHeader) {
            outbound.put("traceparent", inbound.get("traceparent")); // the one line the mesh cannot write for you
        }
        sidecarInbound("payment", outbound);
    }

    public static void main(String[] args) {
        System.out.println("Without forwarding headers:");
        orderApp(sidecarInbound("order", Map.of()), false);

        System.out.println("With forwarding headers:");
        orderApp(sidecarInbound("order", Map.of()), true);
    }
}

Without forwarding headers:
  [order sidecar] started NEW trace efb75e9e
  [payment sidecar] started NEW trace ea369487
With forwarding headers:
  [order sidecar] started NEW trace 6e1d02d8
  [payment sidecar] joined trace     6e1d02d8

Without forwarding, the payment call starts a brand-new trace (ea369487), so your tracing tool shows two unrelated requests instead of one journey. With the traceparent header copied from the incoming request to the outgoing one, both hops share trace 6e1d02d8. In Spring Boot, Micrometer Tracing does this copying for you when you use the auto-configured RestClient, WebClient or RestTemplate builders.

4.3 Example 3: Canary Routing and Sticky Users

A canary release sends a small share of traffic to a new version first. This program mimics a mesh route that sends beta testers to v2 by header, and splits everyone else 90/10.

import java.util.Map;
import java.util.Random;
import java.util.StringJoiner;
import java.util.TreeMap;

public class CanaryRoutingDemo {

    record Request(String userId, Map<String, String> headers) { }

    static boolean isBetaTester(Request r) {
        return "true".equals(r.headers().get("x-beta-tester"));
    }

    // Like a mesh route: header match first, then a 90/10 weighted split per request
    static String routeByWeight(Request r, Random random) {
        if (isBetaTester(r)) {
            return "v2";
        }
        return random.nextInt(100) < 10 ? "v2" : "v1";
    }

    // Same 10% target, but the user ID decides, so a user always lands on one version
    static String routeByUserBucket(Request r) {
        if (isBetaTester(r)) {
            return "v2";
        }
        return Math.floorMod(r.userId().hashCode(), 100) < 10 ? "v2" : "v1";
    }

    public static void main(String[] args) {
        var random = new Random(7); // fixed seed so the output is repeatable
        Map<String, Integer> counts = new TreeMap<>();
        for (int i = 0; i < 1_000; i++) {
            var headers = i % 20 == 0 ? Map.of("x-beta-tester", "true") : Map.<String, String>of();
            counts.merge(routeByWeight(new Request("u-" + i, headers), random), 1, Integer::sum);
        }
        System.out.println("1000 requests (50 beta testers): " + counts);

        var user = new Request("u-42", Map.of());
        var weighted = new StringJoiner(" ");
        var bucketed = new StringJoiner(" ");
        for (int i = 0; i < 10; i++) {
            weighted.add(routeByWeight(user, random));
            bucketed.add(routeByUserBucket(user));
        }
        System.out.println("u-42, per-request weights: " + weighted);
        System.out.println("u-42, user bucket        : " + bucketed);
    }
}

1000 requests (50 beta testers): {v1=849, v2=151}
u-42, per-request weights: v1 v1 v2 v1 v1 v1 v1 v2 v1 v1
u-42, user bucket        : v1 v1 v1 v1 v1 v1 v1 v1 v1 v1

The 1,000 requests split as expected: all 50 beta testers plus 101 of the other 950 requests went to v2, which is close to 10%. But look at user u-42. With per-request weights, they bounced between v1 and v2, and could see an old screen on one click and a new one on the next.

Mesh weight splits are decided per request, and Istio's consistent-hash load balancing works only within one version, not across a weighted split. To keep a user on one version, route on a header or cookie that your gateway sets from the user's ID, like the bucket approach above.

5. Comparison Table: Libraries vs Sidecar Mesh vs Ambient Mesh

6. Common Mistakes and Pitfalls

1. Retrying in both the app and the mesh. As Example 1 showed, the mesh can retry silently. If Resilience4j also retries 3 times and the mesh retries 2 more times on each attempt, one failure becomes 9 calls. Decide which layer owns retries for each route.

2. Believing mTLS is on when it is only "permissive". Istio's default PeerAuthentication mode is PERMISSIVE, meaning a workload accepts both mTLS and plain-text traffic, which eases migration. Until you switch to STRICT, a pod outside the mesh can still call your services unencrypted.

3. The startup race. If the app container starts before its sidecar is ready, its first outgoing calls (for example, to a config server) fail. In Istio, set holdApplicationUntilProxyStarts: true so the app waits for the proxy.

4. Client-side load balancing that bypasses the mesh. If a Spring Cloud LoadBalancer or Eureka client picks pod IPs itself, the mesh's routing rules for that service, such as canary weights, are skipped. In a mesh, call the Kubernetes service name and let the proxy balance.

5. Expecting the mesh to replace business logic. A proxy can return 503 or retry, but it cannot show "bestsellers" instead of personalised recommendations. Fallbacks still belong in your code.

7. Best Practices

• Start with mTLS and telemetry, which need no routing rules.
• Move to STRICT mTLS namespace by namespace, after checking that every caller is inside the mesh.
• Set explicit timeouts on routes. Istio does not apply an HTTP request timeout by default.
• Pick one owner for retries and timeouts per call path, and write it down.
• Keep fallbacks and user-specific logic in the app, using resilience libraries only where business behaviour is needed.
• Forward trace headers in every service, or use a framework that does it automatically.
• Watch the proxies' resource use. With sidecars, hundreds of pods means hundreds of extra containers that need CPU and memory limits.
• Ask whether you need a mesh at all. For five Java services owned by one team, Spring Cloud libraries may be simpler.

8. Summary

• A service mesh in microservices moves networking concerns such as mTLS, retries, timeouts, routing and telemetry out of the code and into proxies.
• The data plane (proxies) carries traffic; the control plane (for example, istiod) pushes configuration and certificates.
• Sidecars capture traffic with iptables rules, so every call takes two extra hops.
• The mesh cannot forward trace context for you; the app must copy traceparent from inbound to outbound requests.
• Weighted canary routing is per request; use header or cookie matching for sticky users.
• Watch for double retries, PERMISSIVE mTLS, sidecar startup races and client-side load balancing that bypasses the mesh.
• Ambient mode replaces per-pod sidecars with shared node proxies and optional waypoints.

A service mesh in microservices is powerful when you have many services and many teams. With the internals above, you can decide when it is worth the extra moving parts and explain your choice in an interview.

Service Mesh: Interview Questions

Common interview questions on Service Mesh, from basic to scenario-based, with short answers.

trafficPolicy:
  outlierDetection:
    consecutive5xxErrors: 5
    interval: 10s
    baseEjectionTime: 30s

http:
- route:
  - destination: { host: pricing, subset: v1 }
    weight: 90
  - destination: { host: pricing, subset: v2 }
    weight: 10

kubectl label namespace shop istio-injection=enabled