Load Balancing: Algorithms and Strategies
In modern applications, thousands of users may use a system at the same time. If every request goes to a single server, that server soon becomes slow, and if it crashes, the whole application goes down. Load balancing solves this problem by spreading incoming requests across several servers, so that no single server is overloaded and users always reach one that is healthy.
This tutorial explains what load balancing is, how a load balancer works, the most common algorithms, health checks, sticky sessions and the main types of load balancer.
1. What Is Load Balancing?
Load balancing is a technique used to distribute incoming client requests across multiple servers so that no single server becomes overloaded. It helps improve application performance, scalability, reliability, and availability.
A load balancer receives client requests and forwards them to backend servers based on a configured load-balancing algorithm, such as Round Robin or Least Connections. It can also monitor server health and redirect traffic away from unavailable servers, ensuring that applications remain accessible even when individual servers fail.
2. Why Load Balancing Is Important
- Better performance: work is shared, so each server stays fast and response times stay low, even at busy times.
- No overloaded servers: a sudden spike in traffic is spread out instead of crushing one machine.
- High availability: if one server crashes, the load balancer sends traffic to the others, and users may not notice anything.
- Easy scaling: you can add servers when traffic grows and remove them when it drops, without changing the address users connect to. This is called horizontal scaling.
- Zero-downtime updates: servers can be updated one at a time while the others keep serving traffic.
- A single place for common tasks: many load balancers also handle HTTPS, compression and basic security checks.
3. How a Load Balancer Works
A load balancer processes incoming client requests through the following steps:
- Receive Request: The client sends a request to the load balancer through its public IP address or domain name.
- Check Server Health: The load balancer uses health-check information to identify available backend servers.
- Select Server: It chooses an appropriate healthy server using a configured load-balancing algorithm.
- Forward Request: The request is routed to the selected backend server for processing.
- Return Response: The backend server processes the request and sends the response to the client, typically through the load balancer.
4. Common Load Balancing Algorithms
Load-balancing algorithms are generally classified into two categories: static and dynamic. Static algorithms distribute requests using predefined rules without considering real-time server load, while dynamic algorithms use current server conditions, such as active connections or resource utilization, to make routing decisions.
Round Robin
The Round Robin algorithm is a simple and widely used static load-balancing technique that distributes incoming client requests across available servers in a sequential, circular order.
Round Robin treats all servers equally without considering their current workload, processing capacity, or active connections. It works best when backend servers have similar capabilities and requests require approximately equal processing resources.
Example: Suppose a load balancer has three backend servers: Server A, Server B, and Server C.
Weighted Round Robin
The Weighted Round Robin algorithm is an enhanced version of the Round Robin algorithm that distributes incoming client requests among backend servers based on their assigned weights. Servers with higher weights receive proportionally more requests than servers with lower weights.
Each server is assigned a weight according to its processing capacity or configured priority. The load balancer uses these weights to determine how frequently each server should receive requests.
For every 10 requests, the load balancer distributes approximately:
- Server A: 5 requests
- Server B: 3 requests
- Server C: 2 requests
The exact request order depends on the Weighted Round Robin implementation.
IP Hash
IP Hash turns the client's IP address into a number using a hash function and uses that number to pick a server. Because the same IP always produces the same result, a given client keeps reaching the same server, as long as the server list does not change.
It is useful when a server keeps some information about the user in memory, but it can spread traffic unevenly, for example when many users share one IP address behind a company network.
Consistent Hashing
Simple hashing has a problem: if a server is added or removed, almost every client may be moved to a different server. Consistent hashing places servers and keys on an imaginary ring, so adding or removing one server only moves a small share of the keys. It is widely used for distributed caches and databases, where moving keys is expensive.
Random and "Power of Two Choices"
The Random algorithm simply picks a server at random, which evens out over many requests. A clever improvement is the power of two choices: pick two servers at random and send the request to the less busy of the two. It is cheap to compute yet spreads load almost as well as checking every server.
Least Connections
The Least Connections algorithm is a dynamic load-balancing technique that directs incoming client requests to the backend server with the fewest active connections.
Unlike the Round Robin algorithm, which distributes requests in a fixed sequence, Least Connections evaluates the current connection count of each available server before selecting one to handle a new request.
This approach helps distribute connections more evenly across servers, reducing the risk of overloading heavily connected servers while making better use of servers with fewer active connections.
For Example, If Server A has 10 active connections, Server B has 5 and Server C has 8, the next request goes to Server B.
Least Response Time
The Least Response Time algorithm is an another dynamic load-balancing technique that directs incoming client requests to the backend server with the lowest response time, typically considering the number of active connections as well.
It evaluates server performance using response-time measurements and connection counts before selecting a server to handle a new client request.
This approach helps reduce request latency, improve application performance, and distribute traffic efficiently by prioritizing servers that respond quickly and have fewer active connections.
For Example, If Server A answers in 180 milliseconds on average, Server B in 45 and Server C in 90, the next request goes to Server B, which is likely to respond fastest.
5. Comparing the Algorithms
| Algorithm | Type | Best for | Watch out for |
|---|---|---|---|
| Round Robin | Static | Identical servers, similar requests | Ignores how busy servers are |
| Weighted Round Robin | Static | Servers of different sizes, gradual rollouts | Weights must be set and updated by hand |
| IP Hash | Static | Keeping a client on the same server | Uneven load when many users share an IP |
| Consistent Hashing | Static | Caches and data that live on specific servers | More complex to implement |
| Least Connections | Dynamic | Requests with very different durations | Connection count is not the same as actual load |
| Least Response Time | Dynamic | Servers whose speed varies | Needs constant measurement |
| Power of Two Choices | Dynamic | Very large pools of servers | Slightly less precise than checking every server |
If you are not sure where to start, Round Robin for identical, stateless servers and Least Connections for uneven workloads cover most applications.
6. Load Balancing Strategies
6.1 Server-Side Load Balancing
In server-side load balancing, a dedicated load balancer sits between clients and servers. Clients only know the load balancer's address; the load balancer chooses the server. This is the most common setup for public websites and APIs, because clients stay simple and the server list can change freely.
6.2 Client-Side Load Balancing
In client-side load balancing, the calling application gets a list of available server instances, usually from a service registry, and picks one itself. There is no extra network hop through a separate load balancer, which makes it popular for calls between microservices inside a private network.
6.3 DNS-Based Load Balancing
DNS-based load balancing returns different IP addresses for the same domain name, so different users connect to different servers. It is simple and needs no extra hardware, but DNS answers are cached by browsers and internet providers, so traffic can keep going to a failed server until the cache expires.
6.4 Global Server Load Balancing (GSLB)
Global server load balancing spreads traffic across data centres or cloud regions around the world. Users are sent to the nearest healthy region, which reduces delay, and if a whole region fails, its traffic moves to another. It is usually built on smart DNS or on anycast networking, where many locations share the same IP address.
7. Health Checks and Failover
A load balancer must never send users to a broken server, so it keeps checking the health of every server in the pool.
- Active health checks: the load balancer regularly calls a health endpoint, such as
/health, every few seconds. If a server fails several checks in a row, it is removed from the pool; when it passes again, it is added back. - Passive health checks: the load balancer watches real traffic. If requests to a server keep failing or timing out, it stops using that server for a while.
- Connection draining: before a server is removed for maintenance, the load balancer stops sending it new requests but lets current requests finish, so no user is cut off.
A good health endpoint checks that the application can really do its job, for example that it can reach its database, but stays fast and cheap so the checks themselves do not add load.
8. Session Persistence (Sticky Sessions)
Some applications store user data, such as a shopping cart, in the memory of one server. If the next request goes to another server, that data is missing. Sticky sessions fix this by sending all of a user's requests to the same server, usually by setting a cookie or by using IP Hash.
Sticky sessions are easy to switch on, but they have drawbacks: load can become uneven, and if that server fails, its users lose their session. The better long-term solution is to make servers stateless and keep session data in a shared store, such as Redis or a database, so any server can handle any request. Read more in Stateful vs Stateless Architecture.
9. Hardware, Software and Cloud Load Balancers
| Type | Examples | Advantages | Disadvantages |
|---|---|---|---|
| Hardware | Dedicated appliances, for example from F5 or Citrix | Very high performance, specialised features | Expensive, fixed capacity, harder to scale |
| Software | Nginx, HAProxy, Envoy, Traefik | Free or low cost, flexible, runs anywhere | You install, configure and run it yourself |
| Cloud (managed) | AWS Elastic Load Balancing, Google Cloud Load Balancing, Azure Load Balancer | No servers to manage, scales automatically, built-in health checks | Pay per use, tied to one cloud provider |
In container platforms, load balancing is often built in. For example, a Kubernetes Service spreads traffic across the pods behind it, and an Ingress controller adds Layer 7 routing.
10. Avoiding a Single Point of Failure
A load balancer protects your servers, but what protects the load balancer? If you run only one and it fails, the whole application is unreachable, so the load balancer itself must be redundant:
- Active-passive pair: two load balancers share a virtual (floating) IP address. The standby watches the active one and takes over the address within seconds if it fails. Tools such as keepalived implement this.
- Active-active: several load balancers handle traffic at the same time, with DNS or anycast spreading users between them.
- Managed cloud load balancers: these are built from many machines across several data centres, so redundancy is handled for you.
11. Best Practices
- Keep servers stateless and store sessions in a shared store, so any server can handle any request.
- Configure meaningful health checks that confirm the application really works, and keep them fast.
- Choose the algorithm to match the workload: Round Robin for similar requests, Least Connections for uneven ones, hashing when data lives on specific servers.
- Run at least two load balancers, or use a managed cloud load balancer, to avoid a single point of failure.
- Use connection draining during deployments so users are never cut off mid-request.
- Terminate HTTPS at the load balancer when it suits your security needs, so certificates are managed in one place.
- Set sensible timeouts so slow servers cannot hold connections forever.
- Monitor request rates, error rates, response times and the health of each server, and alert when the pool is shrinking.
- Combine with auto-scaling so servers are added automatically when traffic rises.
12. Common Mistakes to Avoid
| Mistake | What goes wrong | Better approach |
|---|---|---|
| Running a single load balancer | It becomes a single point of failure for the whole system | Use a redundant pair or a managed service |
| Health checks that only test "is the process running?" | Broken servers keep receiving traffic | Check real dependencies, such as the database |
| Storing sessions in server memory | Users are logged out when they move to another server | Use a shared session store or stateless tokens |
| Using Round Robin for very uneven requests | Some servers are overloaded while others are idle | Use Least Connections or Least Response Time |
| Long DNS cache times with DNS load balancing | Traffic keeps going to failed servers for minutes or hours | Use short TTLs, or a real load balancer for failover |
| Removing servers without draining | Requests in progress fail during deployments | Enable connection draining |
13. Summary
- Load balancing spreads traffic across several servers to improve performance, availability and scalability.
- A load balancer receives each request, picks a healthy server using an algorithm and forwards the request.
- Common algorithms include Round Robin, Weighted Round Robin, Least Connections, Least Response Time, IP Hash, Consistent Hashing and Power of Two Choices.
- Strategies include server-side, client-side, DNS-based and global server load balancing.
- Health checks and connection draining keep users away from broken or restarting servers.
- Stateless servers with shared session storage work better than sticky sessions in the long run.
- The load balancer itself must be redundant to avoid a single point of failure.
By choosing the right algorithm and strategy, and pairing load balancing with health checks and auto-scaling, applications can handle heavy traffic smoothly and give every user a fast, reliable experience. Open the Interview Questions tab to practise the load balancing questions most often asked in system design interviews.
Load Balancing: Top Interview Questions
These are the load balancing questions most often asked in system design interviews, with short answers you can explain in your own words.
Q1. What is a load balancer, and why do we need one?
A load balancer sits in front of a group of servers and spreads incoming requests across them. Clients see one address, while the work is shared by many machines.
- Scalability: you can add servers behind it as traffic grows.
- Availability: it stops sending traffic to a server that fails.
- Performance: no single server becomes a bottleneck.
- Easier maintenance: servers can be updated one at a time without downtime.
Q2. What is the difference between Layer 4 and Layer 7 load balancing?
- Layer 4 (transport): decides using only IP addresses and ports. It does not read the request content, so it is very fast and works for any TCP or UDP traffic.
- Layer 7 (application): reads the HTTP request, such as the URL path, headers and cookies. It can route
/apiand/imagesto different servers, but it does more work per request.
A common setup is a Layer 4 balancer at the network edge and Layer 7 balancers or gateways closer to the services.
Q3. What is the difference between static and dynamic load balancing algorithms?
Static algorithms follow fixed rules and ignore the current state of the servers, for example Round Robin, Weighted Round Robin and IP Hash. They are simple and cheap.
Dynamic algorithms look at live data before choosing a server, such as active connections or response time, for example Least Connections and Least Response Time. They adapt to uneven load but need constant monitoring.
Q4. When would you choose Least Connections over Round Robin?
Choose Least Connections when requests take very different amounts of time. Round Robin only counts turns, so a server busy with several slow requests, such as file uploads or reports, still receives its full share and falls behind.
Least Connections sends new requests to the server with the fewest active connections, so busy servers naturally receive less. For identical servers and short, similar requests, Round Robin is simpler and works just as well.
Q5. How do health checks work in a load balancer?
The load balancer regularly checks each server and only sends traffic to healthy ones.
- Active checks: it calls an endpoint such as
/healthevery few seconds and expects a success response. - Passive checks: it watches real traffic and marks a server unhealthy after several errors or timeouts.
A server is usually removed only after several failed checks in a row and added back after several successes, so one slow response does not cause it to flap in and out.
Q6. What are sticky sessions, and why should you avoid them where possible?
Sticky sessions (session affinity) send every request from the same user to the same server, usually by setting a cookie or hashing the client IP. They are needed when a server keeps user data, such as a shopping cart, in its own memory.
The downsides are uneven load, lost sessions when that server fails, and harder scaling. A better design is to make servers stateless and keep session data in a shared store such as Redis or a database, so any server can handle any request.
Q7. How do you prevent the load balancer from becoming a single point of failure?
- Run at least two load balancers, either active-passive or active-active.
- Use a floating (virtual) IP, for example with VRRP and Keepalived, so the standby takes over the address if the active one fails.
- Use DNS with several load balancer addresses, or a managed cloud load balancer that is already replicated across zones.
- Spread load balancers and servers across multiple availability zones.
Q8. What is the difference between server-side and client-side load balancing?
In server-side load balancing, clients call one address and a separate load balancer, such as NGINX, HAProxy or a cloud load balancer, chooses the server.
In client-side load balancing, the client gets the list of healthy instances from a service registry and chooses one itself. This is common inside microservices, for example with Spring Cloud LoadBalancer or a service mesh. It removes an extra network hop, but every client needs the balancing logic.
Q9. What is the difference between a reverse proxy, a load balancer and an API gateway?
- Reverse proxy: sits in front of servers and forwards requests to them, often adding HTTPS, caching and compression. It may front only one server.
- Load balancer: a reverse proxy whose main job is to spread traffic across several servers and remove unhealthy ones.
- API gateway: a single entry point for APIs that adds features such as authentication, rate limiting, request routing and response aggregation.
One product, such as NGINX or Envoy, can play all three roles.
Q10. Why is consistent hashing better than simple hashing for load balancing?
With simple hashing, such as hash(key) % N, changing the number of servers N changes the result for almost every key, so nearly all clients or cached items move to a different server.
Consistent hashing places servers and keys on a ring, and each key goes to the next server clockwise. Adding or removing one server only moves the keys next to it, roughly 1/N of them. That is why it is used for distributed caches and databases.
Q11. What is connection draining?
Connection draining (also called deregistration delay) lets a server finish its in-flight requests before it is removed. The load balancer stops sending it new requests but keeps existing connections open for a set time, such as 30 seconds.
It is essential during deployments and scale-down, so users do not see failed requests when a server is taken out of the pool.
Q12. Where should TLS (HTTPS) be terminated: at the load balancer or at the servers?
- At the load balancer (TLS termination): certificates are managed in one place, servers do less work, and a Layer 7 balancer can read the request to route it. Traffic behind the balancer is plain HTTP unless you re-encrypt it.
- Re-encryption: the balancer decrypts, inspects and encrypts again to the servers. This is common when internal traffic must stay encrypted for security or compliance.
- TLS passthrough: the balancer forwards encrypted traffic untouched at Layer 4, and the servers decrypt it. It is the most private option, but the balancer cannot route by URL or headers.
Q13. How does DNS load balancing work, and what are its limitations?
DNS load balancing returns different IP addresses for the same domain name, so different clients connect to different servers or data centres. Global server load balancing (GSLB) extends this by choosing the closest or healthiest region.
Its main limitation is caching: clients and resolvers keep a DNS answer until its TTL expires, so traffic can keep going to a failed server for a while. DNS also cannot see real server load, so it is usually combined with regular load balancers in each region.
Q14. How do you scale the load balancer layer itself for very high traffic?
- Run several load balancers in parallel and spread traffic to them with DNS or anycast, where many sites share one IP address.
- Use a fast Layer 4 tier in front of a Layer 7 tier, so the expensive work is spread over many machines.
- Use managed cloud load balancers, which scale automatically.
- Keep the load balancers stateless where possible, so any of them can handle any connection.
Q15. Scenario: one server gets far more traffic than the others. How would you investigate?
Check the common causes one by one:
- Sticky sessions or IP Hash: many users behind one company network or mobile carrier can share an IP and all land on one server.
- Long-lived connections: WebSockets or HTTP keep-alive connections stay on the server that accepted them, even after new servers are added.
- Wrong weights: a weight may be set too high for that server.
- Health check problems: other servers may be failing checks and dropping out of the pool.
- Algorithm choice: Round Robin with uneven request durations can overload a slower server.
Look at per-server request counts, active connections and health check logs, then fix the cause, for example by moving sessions to a shared store or switching to Least Connections.