Load Balancing Algorithms Explained
A load balancer has one job on every request: choose a server. The rule it uses to make that choice is the load balancing algorithm. The choice looks small, but it decides whether traffic spreads evenly, whether a slow server gets buried, and whether a user keeps landing on the server that holds their session.
This tutorial explains the algorithms you will meet in real systems and in interviews, shows how each one decides, and gives working Nginx and Java examples. If you are new to the topic, start with the Load Balancer overview.
1. What Is a Load Balancing Algorithm?
A load balancing algorithm is the policy a load balancer uses to select one backend server, from a pool of healthy servers, for each incoming request or connection.
Algorithms fall into two families. Static algorithms follow a fixed rule, such as a rotation or a hash, and never look at how busy a server is. Dynamic algorithms read live signals, such as open connections or response time, before they choose. Static rules are cheap and predictable; dynamic rules adapt when servers or requests are uneven.
2. Where the Algorithm Runs
The algorithm sits inside the load balancer, between the clients and the server pool. It only chooses among servers that pass their health checks, so a crashed instance is never picked, whichever algorithm you use.
3. How a Load Balancing Algorithm Works
Take an online store with three servers: A is a large machine, B and C are smaller. A request for the product page arrives:
- The load balancer receives the request on its public address.
- It removes any server that failed recent health checks from the candidate list.
- It applies the algorithm to the remaining servers. With Least Connections it compares open connections: A has 12, B has 4 and C has 9, so it chooses B.
- It forwards the request to B, increases B's connection count, and decreases it again when the response finishes.
- The next request repeats the same steps with the updated numbers.
4. Types of Load Balancing Algorithms
Every algorithm answers the same question: which healthy server should get the next request? What changes is the information each one uses to answer it. Static algorithms decide from fixed information only, such as a turn order, a weight or a hash of the request. Dynamic algorithms also look at live numbers, such as open connections or recent response times, at the moment the request arrives.
The difference is easiest to see with the same traffic. Below, the three servers from this tutorial (A, B and C, with weights 5, 3 and 2, and 12, 4 and 9 open connections) receive the same five requests under three different algorithms:
Round Robin (static)
Round Robin works like a card dealer giving one card to each player in turn. The load balancer remembers which server is next, sends the request there, and moves the pointer forward. After the last server, it starts again from the first. With servers A, B and C, requests 1 to 6 go to A, B, C, A, B, C.
- Use it when: servers are identical and requests are short and similar, such as a stateless REST API.
- Watch out for: a server that becomes slow still receives its full share, because Round Robin never checks how a server is doing.
Weighted Round Robin (static)
This is Round Robin where stronger servers get more turns. You give each server a weight that reflects its capacity. With weights 5, 3 and 2, every block of ten requests sends five to A, three to B and two to C. Nginx's smooth version spreads those turns out as A, B, C, A, A, B, A, C, B, A, so A never receives five requests in a row.
- Use it when: servers have different sizes, or you want to send a small share of traffic to a new release (weights 19 and 1 send 5% to the new version).
- Watch out for: weights are fixed numbers. If a strong server slows down because of a bad deployment or garbage collection, it still receives its large share.
IP Hash (static)
The load balancer passes the client's IP address through a hash function and maps the result to a server. A hash always gives the same output for the same input, so a user at 203.0.113.25 keeps reaching the same server, say B, for as long as the server list does not change. This is a simple form of "sticky sessions" that needs no cookies.
- Use it when: an older application keeps session data in server memory and cannot share it yet.
- Watch out for: many users behind one office or mobile-network IP all land on one server, and adding or removing a server changes the mapping for most clients. Moving sessions to a shared store such as Redis is usually the better fix.
Consistent Hashing (static)
Imagine a clock face. Each server is hashed to a point on the circle, and each request key, such as a user ID or a cache key, is hashed onto the same circle. To find who owns a key, start at the key's point and walk clockwise until you meet a server.
Compare this with the simple rule hash(key) % N. If one of three servers fails, N changes from 3 to 2 and about two out of every three keys map to a different server, which empties most of a cache at once. On the ring, only the keys owned by the failed server move, roughly one third here.
- Use it when: the server choice must stay stable as servers come and go, as in distributed caches, sharded databases and CDNs.
- Watch out for: with only a few servers, the arcs can be very different in size. Placing each server on the ring many times ("virtual nodes") evens out the load.
Random (static)
The load balancer picks any healthy server at random. Over thousands of requests, each server ends up with a similar share, and no shared counter is needed. This is useful when several load balancer instances run side by side and cannot easily coordinate.
- Use it when: the pool is large and the servers are similar.
- Watch out for: over a short period, one server can get several requests in a row by chance.
Least Connections (dynamic)
This is like choosing the supermarket checkout with the shortest queue. The load balancer keeps a live count of open connections for each server and sends the new request to the lowest one. With A at 12, B at 4 and C at 9, new requests go to B until its count is no longer the lowest (Figure 2). Nginx's least_conn also takes weights into account, so a bigger server can hold more connections before it stops being chosen.
- Use it when: request durations vary a lot, for example file uploads, report generation or WebSocket connections.
- Watch out for: it counts connections, not effort. One heavy request and one trivial request look the same to it.
Least Response Time (dynamic)
Here the load balancer measures how quickly each server has answered recently, usually combined with its open connections, and prefers the one expected to respond fastest. If A has been answering in about 40 ms while B has slowed to 180 ms during a garbage-collection pause, new requests lean towards A until B recovers. Some load balancers offer this as a feature, for example least_time in NGINX Plus.
- Use it when: server speed changes over time because of noisy neighbours, cache warm-up or background jobs.
- Watch out for: it needs constant measurement, and response times can jump around, so the load balancer averages them over a short window.
Power of Two Choices (dynamic)
Instead of checking every server, the load balancer picks two servers at random and sends the request to the less busy of the pair. If it happens to pick A (12 open) and C (9 open), C wins. Checking just two servers is almost as good as checking all of them: a request ends up on a very busy server only when both random picks are busy, which rarely happens. The work also stays tiny, even with thousands of servers. Envoy's least-request balancer uses this approach by default.
- Use it when: the pool is very large, or many load balancer instances share the same servers.
- Watch out for: each single decision is a little less precise than a full Least Connections scan, although the overall spread is very close.
5. Practical Implementation
Nginx upstream configuration
Nginx uses Weighted Round Robin by default. Adding least_conn, ip_hash or hash ... consistent to the upstream block changes the algorithm. The max_fails and fail_timeout parameters add passive health checks.
http {
upstream product_service {
least_conn; # dynamic: fewest active connections
server 10.0.1.11:8080 weight=5 max_fails=3 fail_timeout=30s;
server 10.0.1.12:8080 weight=3 max_fails=3 fail_timeout=30s;
server 10.0.1.13:8080 weight=2 max_fails=3 fail_timeout=30s;
}
upstream session_cache {
hash $request_uri consistent; # consistent hashing on the URL
server 10.0.2.21:6081;
server 10.0.2.22:6081;
}
server {
listen 80;
location /products/ {
proxy_pass http://product_service;
}
location /cache/ {
proxy_pass http://session_cache;
}
}
}
Java implementation of the main algorithms
The program below implements five algorithms behind one Strategy interface. It uses an AtomicInteger for Round Robin so it is safe under concurrent requests, and the smooth weighted algorithm that Nginx uses.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.atomic.AtomicInteger;
public class LoadBalancerDemo {
/** A backend server with a weight and a live count of in-flight requests. */
static final class Server {
final String name;
final int weight;
final AtomicInteger active = new AtomicInteger();
Server(String name, int weight) {
this.name = name;
this.weight = weight;
}
}
interface Strategy {
Server pick(List<Server> servers, String clientIp);
}
/** Round Robin: hand out servers in a fixed circular order. */
static final class RoundRobin implements Strategy {
private final AtomicInteger next = new AtomicInteger();
public Server pick(List<Server> servers, String clientIp) {
int i = Math.floorMod(next.getAndIncrement(), servers.size());
return servers.get(i);
}
}
/** Smooth Weighted Round Robin: spreads heavier servers evenly instead of in bursts. */
static final class WeightedRoundRobin implements Strategy {
private final int[] current;
WeightedRoundRobin(int size) {
current = new int[size];
}
public synchronized Server pick(List<Server> servers, String clientIp) {
int total = 0;
int best = 0;
for (int i = 0; i < servers.size(); i++) {
current[i] += servers.get(i).weight;
total += servers.get(i).weight;
if (current[i] > current[best]) {
best = i;
}
}
current[best] -= total;
return servers.get(best);
}
}
/** Least Connections: choose the server with the fewest in-flight requests. */
static final class LeastConnections implements Strategy {
public Server pick(List<Server> servers, String clientIp) {
Server best = servers.get(0);
for (Server s : servers) {
if (s.active.get() < best.active.get()) {
best = s;
}
}
return best;
}
}
/** IP Hash: the same client IP always maps to the same server (while the list is unchanged). */
static final class IpHash implements Strategy {
public Server pick(List<Server> servers, String clientIp) {
return servers.get(Math.floorMod(clientIp.hashCode(), servers.size()));
}
}
/** Power of Two Choices: sample two servers at random and keep the less busy one. */
static final class PowerOfTwoChoices implements Strategy {
public Server pick(List<Server> servers, String clientIp) {
ThreadLocalRandom r = ThreadLocalRandom.current();
Server a = servers.get(r.nextInt(servers.size()));
Server b = servers.get(r.nextInt(servers.size()));
return a.active.get() <= b.active.get() ? a : b;
}
}
static List<Server> pool() {
List<Server> servers = new ArrayList<>();
servers.add(new Server("A", 5));
servers.add(new Server("B", 3));
servers.add(new Server("C", 2));
return servers;
}
static String sequence(Strategy strategy, List<Server> servers, int requests) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < requests; i++) {
sb.append(strategy.pick(servers, "203.0.113.7").name);
}
return sb.toString();
}
public static void main(String[] args) {
List<Server> servers = pool();
System.out.println("Round Robin : " + sequence(new RoundRobin(), servers, 9));
System.out.println("Weighted Round Robin : " + sequence(new WeightedRoundRobin(servers.size()), servers, 10));
servers.get(0).active.set(12);
servers.get(1).active.set(4);
servers.get(2).active.set(9);
System.out.println("Least Connections : " + new LeastConnections().pick(servers, null).name);
IpHash ipHash = new IpHash();
System.out.println("IP Hash 203.0.113.7 : " + ipHash.pick(servers, "203.0.113.7").name
+ ", again: " + ipHash.pick(servers, "203.0.113.7").name);
System.out.println("IP Hash 198.51.100.4 : " + ipHash.pick(servers, "198.51.100.4").name);
Server chosen = new PowerOfTwoChoices().pick(servers, null);
System.out.println("Power of Two Choices : " + chosen.name + " (random sample, varies per run)");
}
}
Output:
Round Robin : ABCABCABC
Weighted Round Robin : ABCAABACBA
Least Connections : B
IP Hash 203.0.113.7 : C, again: C
IP Hash 198.51.100.4 : A
Power of Two Choices : B (random sample, varies per run)
The weighted sequence contains A five times, B three times and C twice in ten requests, which matches weights 5, 3 and 2. The last line changes between runs because Power of Two Choices samples at random. The code compiles without warnings on Java 17 and Java 21.
6. Advantages and Disadvantages
| Algorithm | Advantage | Disadvantage |
|---|---|---|
| Round Robin | Simplest, no state to track | Ignores server capacity and current load |
| Weighted Round Robin | Handles servers of different sizes | Weights are set by hand and can go stale |
| IP Hash | Same client reaches the same server | Uneven load behind shared IPs; reshuffles when servers change |
| Consistent Hashing | Few keys move when the pool changes | More complex; needs virtual nodes for even spread |
| Least Connections | Adapts to long and short requests | Connection count is not the same as CPU load |
| Least Response Time | Prefers the fastest server | Needs continuous measurement |
| Power of Two Choices | Near-optimal with very little work | Randomness makes single decisions less precise |
7. Choosing an Algorithm
| Situation | Recommended algorithm |
|---|---|
| Identical, stateless servers | Round Robin |
| Mixed server sizes | Weighted Round Robin |
| Requests of very different duration | Least Connections |
| Server speed varies over time | Least Response Time |
| Caches or sharded data | Consistent Hashing |
| Very large pools | Power of Two Choices |
8. Real-World Use Cases
- Web front ends: stateless web servers behind Nginx or a cloud load balancer usually run Round Robin or Least Connections.
- Microservices: clients that use Spring Cloud LoadBalancer pick an instance from the service registry with Round Robin by default. See client-side load balancing.
- Distributed caches: Memcached clients and many CDN tiers use consistent hashing so that adding a node does not empty the whole cache.
- Blue-green and canary releases: weights send a small share of traffic, such as 5%, to a new version before a full rollout.
9. Best Practices
- Start simple: Round Robin or Least Connections covers most stateless services; change only when metrics show a problem.
- Always pair with health checks: the best algorithm still fails if it can pick a dead server.
- Avoid relying on IP Hash for sessions: store sessions in a shared store such as Redis so any server can serve any user.
- Use virtual nodes with consistent hashing: many points per server on the ring spread keys more evenly.
- Watch per-server metrics: track requests, active connections and latency per backend to spot an uneven spread early.
10. Summary
- A load balancing algorithm decides which healthy server handles each request.
- Static algorithms (Round Robin, Weighted Round Robin, IP Hash, consistent hashing) follow fixed rules.
- Dynamic algorithms (Least Connections, Least Response Time, power of two choices) use live load.
- Nginx selects the algorithm with one directive in the
upstreamblock. - Choose by workload: identical requests suit Round Robin; uneven requests suit Least Connections; caches suit consistent hashing.
Next, learn where these decisions happen in the network stack in Layer 4 vs Layer 7 Load Balancing.
Load Balancing Algorithms: Top Interview Questions
These are the load balancing algorithm questions most often asked in system design interviews, with short answers you can explain in your own words.
Q1. How does Weighted Round Robin decide how many requests each server gets?
Each server receives a share proportional to its weight. With weights 5, 3 and 2, every ten requests are split five, three and two. The smooth variant used by Nginx interleaves them, so the heaviest server does not receive its share in one burst.
Q2. When does Round Robin perform badly?
When requests take very different times or servers have different capacity. Round Robin keeps sending a fair share to a server that is already stuck on slow requests, so its queue grows. Least Connections or weights fix this.
Q3. Why is consistent hashing preferred over hash(key) % N?
With hash(key) % N, changing N remaps almost every key. With a ring-based scheme, only the keys between the changed server and its neighbour move, roughly 1/N of them, so cache hit rates stay high while the pool scales.
Q4. What is the power of two choices algorithm?
The load balancer picks two servers at random and sends the request to the less loaded one. This small step avoids most of the imbalance of pure random choice while needing far less work than scanning every server, so it works well for very large pools.
Q5. How would you choose an algorithm for a new service?
Start from the workload. Use Round Robin for identical stateless servers, Weighted Round Robin for mixed sizes, Least Connections for requests with very different durations, and consistent hashing when requests should stick to the node that holds their data. Then confirm the choice with per-server metrics.