Client-Side Load Balancing in Microservices

Client-side load balancing means the calling service itself decides which instance of another service receives each request. The client keeps a list of available instances, usually from a service registry, and picks one per call using an algorithm such as round robin. There is no separate load balancer server in the middle: the request goes straight from the caller to the chosen instance.

This tutorial explains how client-side load balancing works, how it differs from a server-side load balancer, and which algorithms to choose. Three runnable Java 21 examples show round robin, failover to a healthy instance and what happens with a slow instance. Interview questions follow at the end.

1. What Is Client-Side Load Balancing?

Load balancing spreads requests across several copies, or instances, of a service, so no single instance is overloaded and the service keeps working if one copy fails.

With a server-side load balancer, such as NGINX, HAProxy or a cloud load balancer, every request first goes to the balancer, which forwards it to an instance. With client-side load balancing, a small library inside the calling service does the same job. In Spring, that library is Spring Cloud LoadBalancer. It replaced Netflix Ribbon, which went into maintenance in 2018 and was removed from Spring Cloud in release 2020.0.

2. Why Use a Client-Side Load Balancer?

Inside a microservices system, services call each other constantly. Sending every internal call through a central load balancer adds an extra network hop, another component that can fail or become a bottleneck, and more infrastructure to manage.

A client-side balancer removes the hop. Each caller talks to instances directly, and the balancing logic scales with the callers automatically: ten Order Service instances means ten small balancers, not one big one. The client can also make smarter choices, such as preferring instances in its own zone or skipping one that just failed.

3. How Client-Side Load Balancing Works

1. Register. Each Inventory Service instance registers its address with a service registry and sends regular heartbeats. See service registry for details.
2. Fetch and cache. The Order Service downloads the list of Inventory instances and caches it, refreshing every few seconds. This is service discovery.
3. Choose. Your code calls a logical name, http://inventory-service/stock/P-501. The load balancer replaces that name with one real address from the cache.
4. Call directly. The request goes straight to that instance.

In Spring Boot, you mark a client builder with @LoadBalanced, for example a RestClient.Builder or WebClient.Builder bean, and Spring Cloud LoadBalancer resolves service names on every call. OpenFeign clients use it automatically.

4. Load Balancing Algorithms

• Round robin: instances take turns: 1, 2, 3, 1, 2, 3. It is simple and fair when all instances are equal. It is Spring Cloud LoadBalancer's default.
• Random: pick any instance. Over many requests it evens out, with no shared counter needed. Spring offers it as RandomLoadBalancer.
• Weighted: bigger instances get a larger share, for example during a gradual rollout.
• Least outstanding requests: pick the instance with the fewest requests still in progress. Proxies such as Envoy and gRPC clients offer this; it reacts to slow instances.
• Zone preference: prefer instances in the caller's own data centre zone, which is faster and cheaper.

5. Client-Side Load Balancing in Java: Examples

These programs use only the JDK on Java 21, so you can see what a load balancer does inside. In a Spring Boot service you would use Spring Cloud LoadBalancer instead of writing this yourself.

5.1 A Thread-Safe Round-Robin Balancer

Round robin needs a counter shared by all request threads, so it must be an AtomicInteger. There is a hidden trap:

import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

public class RoundRobinBalancer {

    private final List<String> instances;
    private final AtomicInteger counter;          // shared by all request threads, so it must be atomic

    RoundRobinBalancer(List<String> instances, int start) {
        this.instances = instances;
        this.counter = new AtomicInteger(start);
    }

    String brokenChoose() {
        return instances.get(counter.getAndIncrement() % instances.size());   // negative after overflow
    }

    String choose() {
        int position = counter.getAndIncrement() & Integer.MAX_VALUE;    // clear the sign bit: never negative
        return instances.get(position % instances.size());
    }

    public static void main(String[] args) {
        List<String> inventory = List.of("10.0.1.11:8081", "10.0.1.12:8081", "10.0.1.13:8081");

        RoundRobinBalancer lb = new RoundRobinBalancer(inventory, 0);
        for (int i = 1; i <= 4; i++) {
            System.out.println("Request " + i + " -> " + lb.choose());
        }

        // After about 2.1 billion requests the int counter wraps around to a negative number
        RoundRobinBalancer old = new RoundRobinBalancer(inventory, Integer.MAX_VALUE);
        old.brokenChoose();
        try {
            old.brokenChoose();
        } catch (IndexOutOfBoundsException e) {
            System.out.println("Broken version after overflow: " + e.getMessage());
        }

        RoundRobinBalancer fixed = new RoundRobinBalancer(inventory, Integer.MAX_VALUE);
        System.out.println("Fixed version: " + fixed.choose() + ", then " + fixed.choose());
    }
}

Request 1 -> 10.0.1.11:8081
Request 2 -> 10.0.1.12:8081
Request 3 -> 10.0.1.13:8081
Request 4 -> 10.0.1.11:8081
Broken version after overflow: Index -2 out of bounds for length 3
Fixed version: 10.0.1.12:8081, then 10.0.1.11:8081

An int counter wraps to a negative number after Integer.MAX_VALUE, and % of a negative number is negative in Java. A busy service reaches that point after about 2.1 billion calls, then crashes with IndexOutOfBoundsException. Clearing the sign bit with & Integer.MAX_VALUE fixes it, which is the same technique Spring Cloud LoadBalancer uses. Spring also starts the counter at a random position, so that a hundred freshly started clients do not all send their first request to instance 1.

5.2 Skipping a Dead Instance with a Stale Cache

Here payment-2 has crashed, but the cached list still contains it. The client retries once on the next instance, then the cache refreshes:

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;

public class HealthAwareBalancer {

    // The client's cached copy of the registry: it can be a few seconds out of date
    static List<String> cachedInstances = List.of("payment-1", "payment-2", "payment-3");
    static final Set<String> reallyDown = Set.of("payment-2");     // crashed, but still in the cache
    static int next = 0;

    static String send(String instance, String request) {
        if (reallyDown.contains(instance)) throw new IllegalStateException("connection refused");
        return "200 from " + instance;
    }

    // Try the chosen instance; on a connection failure, try ONE other instance
    static String call(String request) {
        List<String> tried = new ArrayList<>();
        for (int attempt = 0; attempt < 2; attempt++) {
            String instance = cachedInstances.get(Math.floorMod(next++, cachedInstances.size()));
            tried.add(instance);
            try {
                return request + ": " + send(instance, request) + "  (tried " + tried + ")";
            } catch (IllegalStateException e) {
                System.out.println("  " + request + ": " + instance + " failed (" + e.getMessage() + "), trying next instance");
            }
        }
        return request + ": all attempts failed " + tried;
    }

    public static void main(String[] args) {
        for (String request : List.of("GET /payments/1", "GET /payments/2", "GET /payments/3")) {
            System.out.println(call(request));
        }

        // The cache refreshes from the registry, and the dead instance disappears
        Map<String, String> registry = Map.of("payment-1", "UP", "payment-3", "UP");
        cachedInstances = registry.keySet().stream().sorted().toList();
        System.out.println("Cache refreshed: " + cachedInstances);
        System.out.println(call("GET /payments/4"));
    }
}

GET /payments/1: 200 from payment-1  (tried [payment-1])
  GET /payments/2: payment-2 failed (connection refused), trying next instance
GET /payments/2: 200 from payment-3  (tried [payment-2, payment-3])
GET /payments/3: 200 from payment-1  (tried [payment-1])
Cache refreshed: [payment-1, payment-3]
GET /payments/4: 200 from payment-1  (tried [payment-1])

Spring Cloud LoadBalancer works the same way: its retry support can move to the next instance, and by default it retries only GET requests, because repeating a POST could create a duplicate.

5.3 Why Round Robin Struggles With a Slow Instance

A dead instance is easy to spot. A slow one is worse, because it still accepts requests. This simulation sends 300 requests to three instances, where order-3 is ten times slower:

import java.util.Arrays;

public class SlowInstanceSimulation {

    static final String[] NAMES = {"order-1", "order-2", "order-3"};
    static final int[] SERVICE_MS = {20, 20, 200};     // order-3 is ten times slower (e.g. a noisy neighbour)

    // One request arrives every 12 ms; together the instances could easily handle that
    static String simulate(String strategy, int requests) {
        long[] busyUntil = new long[3];                // when each instance finishes its current queue
        int[] sent = new int[3];
        long totalLatency = 0, worstLatency = 0;
        for (int r = 0; r < requests; r++) {
            long now = r * 12L;
            int pick;
            if (strategy.equals("round robin")) {
                pick = r % 3;
            } else {                                   // least outstanding work right now
                pick = 0;
                for (int i = 1; i < 3; i++) {
                    if (Math.max(busyUntil[i], now) < Math.max(busyUntil[pick], now)) pick = i;
                }
            }
            long start = Math.max(busyUntil[pick], now);   // wait in that instance's queue
            busyUntil[pick] = start + SERVICE_MS[pick];
            long latency = busyUntil[pick] - now;     // queue wait + service time
            totalLatency += latency;
            worstLatency = Math.max(worstLatency, latency);
            sent[pick]++;
        }
        return String.format("%-18s avg %4d ms, worst %5d ms, requests per instance %s", strategy + ":",
                totalLatency / requests, worstLatency, Arrays.toString(sent));
    }

    public static void main(String[] args) {
        System.out.println("Instances " + Arrays.toString(NAMES) + ", service times " + Arrays.toString(SERVICE_MS) + " ms");
        System.out.println(simulate("round robin", 300));
        System.out.println(simulate("least outstanding", 300));
    }
}

Instances [order-1, order-2, order-3], service times [20, 20, 200] ms
round robin:       avg 2786 ms, worst 16436 ms, requests per instance [100, 100, 100]
least outstanding: avg   20 ms, worst    20 ms, requests per instance [150, 150, 0]

Round robin keeps sending one third of the traffic to order-3. Its queue grows with every request, so some users wait more than 16 seconds although the other two instances have spare capacity. Choosing by outstanding work sends nothing to the slow instance here. This is why the algorithm matters most when instances are not identical.

6. Client-Side vs Server-Side Load Balancing

Most systems use both: a server-side balancer or API gateway at the edge, and client-side balancing between internal services.

7. Common Mistakes and Pitfalls

• Assuming the instance list is always current. Stale caches are normal, so pair the balancer with timeouts and a retry on another instance.
• Retrying non-idempotent requests. Moving a POST /payments to another instance after a timeout can charge twice, because the first instance may have succeeded.
• Forgetting @LoadBalanced. Without it, the client tries to resolve inventory-service through DNS and fails with an unknown host error.
• Using round robin with uneven instances. As section 5.3 shows, one slow instance slows a share of all requests.
• Long-lived connections in Kubernetes. A Kubernetes Service balances connections, not requests. gRPC and HTTP/2 reuse one connection for thousands of requests, so all of them can land on one pod. Balance per request on the client side, using a headless Service and the client's balancing, or a service mesh.

8. Best Practices

• Use Spring Cloud LoadBalancer rather than writing your own, and keep the default round robin until you have a reason to change.
• Set connect and read timeouts on every client, so a dead instance is detected in milliseconds, not minutes.
• Retry once on a different instance, only for idempotent requests, and combine it with a circuit breaker.
• Keep cache refresh intervals short enough for your deployment speed, and deregister instances on graceful shutdown.
• Prefer instances in the same zone when your platform spans several.
• Watch per-instance latency and error rate, so a slow instance is noticed and removed.

9. Summary

• Client-side load balancing lets the caller pick an instance and call it directly, with no proxy in between.
• It depends on service discovery: instances register, and clients cache the instance list.
• Round robin is the default; random, weighted, least outstanding requests and zone preference suit other cases.
• Expect stale lists, avoid counter overflow, retry only idempotent calls, and watch for slow instances.
• Use client-side load balancing for service-to-service calls and a server-side balancer or gateway at the edge.

Client-Side Load Balancing: Interview Questions

Common interview questions on client-side load balancing, from basic to scenario-based, with short answers.