Scalability: Horizontal vs Vertical Scaling

Scalability is a system's ability to handle more work, more users, more data or more requests, by adding resources, without a drop in performance or a redesign. A scalable system serving 1,000 users today can serve 1,000,000 tomorrow mainly by adding machines or capacity, not by being rewritten.

This tutorial explains the different dimensions of scalability, compares vertical scaling (bigger machines) with horizontal scaling (more machines), shows what an application needs before it can scale out, how databases are scaled step by step, how auto-scaling works, and the limits described by Amdahl's law, with a runnable Java example. It ends with a typical scaling journey from one server to millions of users, common mistakes and best practices.

1. What Is Scalability?

Think of a small tea stall that becomes popular. At first, one person can serve everyone. As the queue grows, the owner can either buy a bigger stove and a faster kettle (scale up) or hire more people and open more counters (scale out). A scalable business can keep serving customers quickly as demand grows, without shutting down to rebuild the shop.

In system design, scalability means:

• Performance stays acceptable as load grows: latency and error rates do not get worse.
• Capacity can be added in proportion to demand, ideally automatically.
• Cost grows reasonably, roughly in line with usage rather than exponentially.

Scalability is not the same as performance. Performance is how fast the system is for one user; scalability is how well it keeps that performance as users multiply. A system can be fast for 10 users and collapse at 10,000.

2. Dimensions of Scalability

3. Vertical Scaling (Scale Up)

Vertical scaling means making one machine more powerful: more CPU cores, more memory, faster disks or a faster network card. In the cloud, it is often just choosing a larger instance type.

Advantages:

• No code changes: the application simply gets more resources.
• No distributed-system complexity: one machine, one copy of the data, simple transactions.
• Often the quickest fix, especially for databases.

Disadvantages:

• Hard ceiling: there is always a largest machine you can buy.
• Rising cost: the biggest machines cost disproportionately more than mid-sized ones.
• Single point of failure: one machine still means one failure takes everything down.
• Downtime to upgrade: resizing often requires a restart.

4. Horizontal Scaling (Scale Out)

Horizontal scaling means adding more machines and spreading the work across them, usually with a load balancer in front. This is how large internet services handle enormous traffic: not with one giant computer, but with thousands of ordinary ones.

Advantages:

• Nearly unlimited growth: keep adding machines as demand grows.
• Built-in redundancy: if one machine fails, the others keep serving.
• Elasticity: add machines at peak times and remove them when quiet, paying only for what you use.
• Zero-downtime upgrades: replace machines one at a time.

Disadvantages:

• The application must be designed for it, typically stateless.
• More moving parts: load balancers, service discovery, monitoring of many instances.
• Data becomes harder: distributing a database across machines brings replication, sharding and consistency challenges.

5. Vertical vs Horizontal Scaling: Comparison Table

In practice, most systems use both: application servers scale horizontally, while databases are first scaled vertically and later horizontally with replicas and shards.

6. What an Application Needs Before It Can Scale Out

• Stateless servers: no user sessions or uploaded files stored on a particular server. Sessions go to a shared store such as Redis or into signed tokens; files go to object storage. See Stateful vs Stateless Architecture.
• Externalised configuration: every instance gets the same settings from environment variables or a configuration service.
• Idempotent operations: requests may be retried on another instance, so repeating them must be safe.
• Coordinated background jobs: scheduled tasks must not run once per instance; use a distributed lock or a dedicated scheduler.
• Health checks and graceful shutdown: load balancers must know when an instance is ready, and instances must finish in-flight requests before stopping.
• Sized connection pools: ten instances each opening 100 database connections is 1,000 connections; make sure the database can handle the total.

7. Scaling the Database

Application servers are usually easy to scale out. The database is where scaling gets hard, so teams typically move through these steps in order, adding complexity only when needed:

1. Optimise and scale up: add missing indexes, fix slow queries and give the database a bigger machine. This alone often buys years of growth.
2. Add caching: serve frequent reads from Redis or an in-process cache, cutting database load dramatically. See Caching.
3. Add read replicas: send read-only queries to copies of the database, and keep writes on the primary. See Database Replication.
4. Partition and shard: when write traffic or data size outgrows one primary, split data across several databases by a shard key. See Database Sharding.

Alternatives include moving specific workloads to specialised stores (search engines, time-series databases, key-value stores) and using distributed databases that shard automatically.

8. A Typical Scaling Journey

Most products grow through similar stages. Each step solves the bottleneck of the previous one:

The key lesson is to scale in response to measured bottlenecks, not imagined ones. Each stage adds operational cost, so move to the next only when monitoring shows you need to.

9. Auto-Scaling

Auto-scaling adds and removes instances automatically based on demand, so the system handles peaks without paying for idle capacity at quiet times. It relies on:

• Metrics: CPU utilisation, request rate, latency, or queue length for background workers.
• Policies: for example, keep average CPU around 65%, with minimum and maximum instance counts.
• Fast, reliable startup: new instances must start quickly and pass health checks before receiving traffic.
• Scheduled scaling for predictable peaks, such as a sale starting at noon, because reactive scaling takes minutes.

In Kubernetes, the Horizontal Pod Autoscaler does this for a deployment:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  minReplicas: 3            # never fewer than 3, for redundancy across zones
  maxReplicas: 30           # protect the database and the budget
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65

Remember that auto-scaling the application tier can overload shared dependencies. Thirty instances may send three times more queries to the database than ten did, so scale and protect the data tier too.

10. The Limits of Scaling: Amdahl's Law

Adding machines does not speed things up forever. Amdahl's law says that if part of the work cannot be done in parallel, that part limits the maximum speedup:

Speedup = 1 ÷ ((1 − P) + P ÷ N), where P is the fraction of work that can run in parallel and N is the number of machines or cores.

package com.javabytechie.scalability;

/** Speedup predicted by Amdahl's law when part of the work cannot be parallelised. */
public class AmdahlsLaw {

    static double speedup(double parallelFraction, int servers) {
        return 1.0 / ((1 - parallelFraction) + parallelFraction / servers);
    }

    public static void main(String[] args) {
        int[] serverCounts = {1, 2, 4, 8, 16, 64, 1024};
        double[] fractions = {0.50, 0.90, 0.95, 0.99};

        System.out.print("servers ");
        for (double f : fractions) {
            System.out.printf("| %3.0f%% parallel ", f * 100);
        }
        System.out.println();
        for (int n : serverCounts) {
            System.out.printf("%7d ", n);
            for (double f : fractions) {
                System.out.printf("| %10.2fx    ", speedup(f, n));
            }
            System.out.println();
        }
    }
}

servers |  50% parallel |  90% parallel |  95% parallel |  99% parallel 
      1 |       1.00x    |       1.00x    |       1.00x    |       1.00x    
      2 |       1.33x    |       1.82x    |       1.90x    |       1.98x    
      4 |       1.60x    |       3.08x    |       3.48x    |       3.88x    
      8 |       1.78x    |       4.71x    |       5.93x    |       7.48x    
     16 |       1.88x    |       6.40x    |       9.14x    |      13.91x    
     64 |       1.97x    |       8.77x    |      15.42x    |      39.26x    
   1024 |       2.00x    |       9.91x    |      19.64x    |      91.18x

Even when 95% of the work is parallel, 1,024 machines give less than a 20x speedup, because the remaining 5% (for example a single database lock, a shared queue or a coordinator) caps it. With only half the work parallel, you can never get more than 2x. In real systems, coordination between machines also adds overhead, and beyond a point adding nodes can even reduce throughput. The Universal Scalability Law models this extra coordination cost.

The practical lesson: to scale well, remove serial bottlenecks and shared contention points (global locks, single-writer tables, central coordinators) rather than only adding hardware.

11. Measuring Scalability

• Load testing: use tools such as JMeter, Gatling or k6 to increase traffic step by step and record throughput, latency percentiles and error rates.
• Find the knee: the point where latency starts to rise sharply shows the real capacity of the current setup.
• Scale and repeat: double the instances and check whether throughput roughly doubles. If not, find the shared bottleneck.
• Watch every tier: CPU and memory of app servers, database connections and locks, cache hit rate, queue depth and network bandwidth.
• Test realistic data volumes: a query that is fast on 10,000 rows may be slow on 100 million.

Read Throughput and Latency for more on these measurements.

12. Making a Spring Boot Service Scale-Ready

These settings help a Spring Boot service run safely as many instances behind a load balancer or in Kubernetes:

server:
  shutdown: graceful                 # finish in-flight requests before stopping
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s
  datasource:
    hikari:
      maximum-pool-size: 20          # per instance: 20 x number of instances must fit the database limit
      connection-timeout: 3000       # fail fast (ms) instead of queueing forever for a connection
  session:
    timeout: 30m                     # with spring-session-data-redis, sessions live in Redis, not in memory
management:
  endpoint:
    health:
      probes:
        enabled: true                # /actuator/health/liveness and /readiness for the orchestrator
  endpoints:
    web:
      exposure:
        include: health,prometheus

Combine this with stateless controllers, files in object storage, idempotent APIs and a distributed lock (for example ShedLock) for scheduled jobs, and the service can be scaled out simply by increasing the replica count.

13. Common Mistakes to Avoid

• Scaling too early: building a complex distributed architecture for traffic that does not exist yet.
• Keeping state on app servers, which breaks as soon as a second instance is added.
• Scaling only the application tier, and overwhelming the database with connections and queries.
• Ignoring serial bottlenecks such as global locks, single counters or a single-threaded consumer.
• No load testing, so capacity limits are discovered by customers during a peak.
• Assuming the cloud scales automatically; auto-scaling needs correct metrics, limits and fast startup.
• Forgetting cost: scaling without monitoring cost per request can make growth unprofitable.

14. Best Practices

• Design application services to be stateless from the start.
• Scale vertically first where it is cheap and simple, especially for databases, but plan for horizontal growth.
• Cache aggressively, use a CDN and offload slow work to queues.
• Scale the database step by step: optimise, cache, replicate, then shard.
• Remove serial bottlenecks and shared locks that limit parallelism.
• Use auto-scaling with sensible minimums, maximums and scheduled scaling for known peaks.
• Load test regularly with realistic data and traffic, and track throughput, latency percentiles and cost per request.
• Spread instances across availability zones so scaling also improves resilience.

15. Summary

• Scalability is the ability to handle growth in traffic, data, regions, features and teams while keeping performance acceptable.
• Vertical scaling adds power to one machine: simple, but limited and a single point of failure.
• Horizontal scaling adds machines: nearly unlimited and fault tolerant, but requires stateless design and distributed data.
• Databases scale in steps: optimise, cache, read replicas, then sharding.
• Amdahl's law shows that serial bottlenecks cap the benefit of more machines, so remove contention as well as adding capacity.

Build for today's load, measure continuously, and keep the design ready to scale out when the numbers say it is time. Open the FAQ tab for answers to common questions about scalability.

Frequently Asked Questions (FAQ)