Throughput in System Design

In system design, throughput is the amount of work a system completes in a given amount of time. It describes the system's processing capacity. Put simply, throughput answers the question: "How many requests, tasks or operations can the system finish per second?"

This tutorial explains how throughput is measured, how it differs from latency, the simple law that connects them, what limits throughput, and practical ways to increase it, with Java examples you can run.

1. What Is Throughput?

Throughput is a rate: the number of completed units of work divided by the time taken. If a server finishes 6,000 requests in one minute, its throughput is 6,000 / 60 = 100 requests per second.

A higher throughput means the system can serve more users at the same time. A throughput that stays flat while traffic grows is a sign of a bottleneck.

2. How Throughput Is Measured

Only successful work should count. A server that returns 1,000 errors per second does not have a throughput of 1,000 useful requests per second.

3. Why Throughput Is Important

Throughput tells designers whether an application can support real-world usage. An e-commerce site must handle thousands of orders during a sale, and a payment system must process every transaction quickly, even at peak times.

• Capacity planning: knowing the throughput of one server tells you how many servers you need.
• User experience: when demand exceeds throughput, requests queue up, slow down and eventually fail.
• Cost: higher throughput per server means fewer servers for the same traffic.
• Finding bottlenecks: comparing throughput across components shows which part is holding the system back.

Throughput is usually considered together with latency (how fast one request is), scalability (how well the system grows) and availability (how reliably it stays online).

4. Throughput vs Latency

The two are related but not the same. A system can have high throughput and high latency at the same time, for example when it processes huge batches: many items finish per second, but each item waits for its batch. Likewise, a single very fast server can have low latency but limited throughput.

5. Little's Law: Connecting the Two

A simple formula called Little's Law links throughput, latency and the number of requests being handled at the same time (concurrency):

Concurrency = Throughput x Latency
Throughput  = Concurrency / Latency

Example: a server has 50 worker threads and each request takes 100 ms (0.1 s)
Maximum throughput = 50 / 0.1 = 500 requests per second

This shows the two main ways to raise throughput: handle more requests at once (more threads, more servers) or make each request faster (lower latency). It also explains why a slow database call that holds a thread for a long time reduces how many requests a server can finish.

6. Example: A Food Delivery App

Imagine the backend of a food delivery app that processes orders placed by users. If the system can process 200 orders per second, its throughput is:

Throughput = 200 orders/second

Now consider two situations:

• Normal hours: the app receives 100 orders per second. That is only half its capacity, so everything works smoothly.
• Lunch rush: the app receives 300 orders per second. The system can finish only 200 per second, so 100 extra orders pile up every second. Waiting times grow, and soon requests time out or fail.

Planning Capacity for the Lunch Rush

Peak demand                  = 300 orders/second
Throughput of one app server = 50 orders/second (measured in a load test)
Servers needed at peak       = 300 / 50 = 6
Add ~30% headroom            = 6 x 1.3 ≈ 8 servers

Running at 100% capacity is risky because any spike or server failure causes delays. Keeping headroom, or using auto-scaling, keeps the app smooth during the rush. This shows that throughput directly affects user experience and system reliability.

7. What Limits Throughput

A system's throughput is limited by its slowest part, like the narrowest section of a pipe. Common bottlenecks are:

• CPU: heavy calculations, encryption or inefficient code keep processors busy.
• Memory: too little memory causes swapping or long garbage-collection pauses.
• Disk I/O: slow reads and writes, especially on unindexed database tables.
• Network: limited bandwidth or many round trips between services.
• Database connections: a small connection pool makes requests wait for a free connection.
• Thread pools: when all worker threads are busy waiting on I/O, new requests must queue.
• Locks and contention: many requests competing for the same row, file or lock run one at a time.
• Slow dependencies: a third-party API that handles 50 requests per second caps everything that depends on it.

8. How to Improve Throughput

• Horizontal scaling: add more servers so the work is shared. See Scalability.
• Load balancing: spread requests evenly so no server is overloaded while others are idle. See Load Balancing.
• Caching: serve frequently read data from memory instead of the database. See What Is Caching.
• Asynchronous processing: put slow work (emails, reports, image processing) on a queue and let background workers handle it.
• Batching: group many small operations into one, such as inserting 500 rows in one database round trip.
• Efficient database access: add indexes, avoid N+1 queries, and use read replicas or sharding for heavy loads.
• Connection pooling: reuse database and HTTP connections instead of opening new ones each time.
• More concurrency for I/O work: larger thread pools or Java virtual threads let a server wait on many I/O calls at once.
• Smaller payloads: compress responses, send only needed fields and use efficient formats.
• CDN for static content: let a CDN serve images and files so your servers focus on dynamic requests.

9. Measuring Throughput

• Load testing: tools such as Apache JMeter, Gatling and k6 send increasing traffic and report requests per second and errors.
• Find the knee: increase load step by step. Throughput rises until the system is saturated; after that, latency jumps while throughput stays flat or drops. That point is your real capacity.
• Production metrics: count completed requests with Micrometer and graph the rate in Prometheus and Grafana.
• Watch it together with latency and errors: high throughput is meaningless if p99 latency is several seconds or many requests fail.

10. Throughput in Java

Concurrency and Throughput: Threads vs Virtual Threads

This program runs 200 simulated I/O calls (10 ms each) three ways and prints the resulting throughput. It needs Java 21 or later.

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.stream.IntStream;

public class ThroughputDemo {

    private static final int TASKS = 200;

    // Simulates an I/O call, such as a database query, that takes about 10 ms.
    private static void handleRequest() {
        try {
            Thread.sleep(10);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

    private static void measure(String label, ExecutorService executor) {
        long start = System.nanoTime();
        try (executor) { // close() waits until every task has finished
            IntStream.range(0, TASKS).forEach(i -> executor.submit(ThroughputDemo::handleRequest));
        }
        double seconds = (System.nanoTime() - start) / 1_000_000_000.0;
        System.out.printf("%-17s %d tasks in %.2f s -> %,.0f tasks/second%n",
                label, TASKS, seconds, TASKS / seconds);
    }

    public static void main(String[] args) {
        measure("1 thread:", Executors.newSingleThreadExecutor());
        measure("20 threads:", Executors.newFixedThreadPool(20));
        measure("Virtual threads:", Executors.newVirtualThreadPerTaskExecutor());
    }
}

1 thread:         200 tasks in 2.40 s -> 83 tasks/second
20 threads:       200 tasks in 0.12 s -> 1,652 tasks/second
Virtual threads:  200 tasks in 0.02 s -> 8,844 tasks/second

Each task still takes about 10 ms, so latency is the same in all three runs. Only the number of tasks running at the same time changes, and throughput grows with it, exactly as Little's Law predicts.

Batching Database Writes

Sending rows one by one costs a network round trip each time. A JDBC batch sends many rows together, which can multiply write throughput:

import java.math.BigDecimal;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.List;
import javax.sql.DataSource;

public class OrderBatchWriter {

    private static final int BATCH_SIZE = 500;
    private static final String INSERT_SQL =
            "INSERT INTO orders (id, customer_id, amount) VALUES (?, ?, ?)";

    private final DataSource dataSource;

    public OrderBatchWriter(DataSource dataSource) {
        this.dataSource = dataSource;
    }

    public void saveAll(List<Order> orders) throws SQLException {
        try (Connection connection = dataSource.getConnection();
             PreparedStatement statement = connection.prepareStatement(INSERT_SQL)) {

            connection.setAutoCommit(false);
            try {
                int count = 0;
                for (Order order : orders) {
                    statement.setLong(1, order.id());
                    statement.setLong(2, order.customerId());
                    statement.setBigDecimal(3, order.amount());
                    statement.addBatch();

                    if (++count % BATCH_SIZE == 0) {
                        statement.executeBatch(); // send 500 rows in one round trip
                    }
                }
                statement.executeBatch();         // send the remaining rows
                connection.commit();
            } catch (SQLException ex) {
                connection.rollback();
                throw ex;
            }
        }
    }

    public record Order(long id, long customerId, BigDecimal amount) {
    }
}

With MySQL, also add rewriteBatchedStatements=true to the JDBC URL so the driver sends the batch as a single multi-row insert.

Tuning a Spring Boot Service

For I/O-heavy services on Spring Boot 3.2+ and Java 21, virtual threads let each request wait on I/O without blocking a scarce platform thread:

spring:
  threads:
    virtual:
      enabled: true            # handle each request on a virtual thread
  datasource:
    hikari:
      maximum-pool-size: 30    # database connections are still limited; size the pool carefully

server:
  compression:
    enabled: true              # smaller responses use less network bandwidth

Virtual threads remove the thread-pool limit, but the database pool and the database itself still cap throughput, so measure before and after any change.

Measuring Throughput with Micrometer

package com.javabytechie.delivery;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Service;

@Service
public class OrderProcessor {

    private final Counter processedOrders;

    public OrderProcessor(MeterRegistry registry) {
        this.processedOrders = Counter.builder("orders.processed")
                .description("Number of orders processed successfully")
                .register(registry);
    }

    public void process(FoodOrder order) {
        // ... validate, save and dispatch the order ...
        processedOrders.increment(); // count only successful work
    }
}

In Prometheus, the query rate(orders_processed_total[1m]) shows orders processed per second, averaged over the last minute. That is your live throughput.

11. Trade-offs

• Throughput vs latency: batching and queuing raise throughput but make each item wait a little longer.
• Throughput vs cost: more servers and bigger databases increase capacity but also the bill.
• Throughput vs consistency: asynchronous processing and caching increase throughput but data may be briefly out of date.
• Throughput vs simplicity: sharding, queues and caches add moving parts that must be operated and monitored.

12. Best Practices

1. Define a throughput target from expected peak traffic, not average traffic.
2. Load test regularly and find the point where latency starts to rise sharply.
3. Keep 20–30% headroom, or use auto-scaling, for unexpected spikes.
4. Find the real bottleneck before optimising anything else.
5. Keep services stateless so you can scale them out easily.
6. Cache hot data and move slow work to background queues.
7. Batch database writes and avoid one query per item.
8. Size thread pools and connection pools based on measurements.
9. Protect the system with rate limiting so overload does not cause total failure.
10. Always monitor throughput together with latency and error rate.

13. Summary

Throughput is a key performance metric in system design that measures how much work a system completes in a fixed amount of time. It is different from latency, but Little's Law connects the two: throughput equals concurrency divided by latency. A system's throughput is set by its slowest part, so finding and fixing bottlenecks is essential. Horizontal scaling, load balancing, caching, asynchronous processing, batching and the right amount of concurrency all raise throughput. By designing and testing for the throughput you need at peak, systems become more reliable, scalable and ready for real-world usage.

Frequently Asked Questions (FAQ)

Concurrency = Throughput x Latency
50 threads / 0.1 s per request = 500 requests per second