Latency in System Design: Causes, Percentiles and How to Reduce It

Latency is the time between asking for something and getting it: from the moment a user taps a button to the moment the result appears, or from the moment a service sends a request to the moment the reply arrives. It is usually measured in milliseconds. Low latency makes applications feel instant; high latency makes them feel broken, even if they are technically working.

This tutorial explains where latency comes from, the latency numbers every engineer should know, why averages hide problems and percentiles reveal them, why tail latency matters in large systems, how latency explodes when servers get busy, and practical ways to reduce it. It includes a runnable Java example that computes percentiles and a Spring Boot configuration for measuring latency in production.

1. What Is Latency?

Imagine ordering food at a restaurant. Latency is how long you wait from placing your order until the plate arrives. It includes the waiter walking to the kitchen, the time your order waits behind others, the cooking itself and the walk back. Making any one of those steps faster reduces your wait.

In computing, latency can describe different spans of time:

• Network latency: how long data takes to travel between two machines. Often measured as round-trip time (RTT), there and back.
• Server (processing) latency: how long a server spends handling a request, including database queries and calls to other services.
• End-to-end latency: the total time the user experiences, from action to visible result.

Latency is different from throughput. Latency is how long one request takes; throughput is how many requests the system completes per second. A highway can carry thousands of cars per hour (high throughput) while each car still takes an hour to reach the city (high latency).

2. Where Latency Comes From

Every request's latency is the sum of several delays:

• Propagation delay: signals travel through fibre at roughly two-thirds of the speed of light, about 200 km per millisecond. A round trip between India and the United States therefore costs well over 100 ms no matter how fast the servers are.
• Transmission delay: the time to push the bytes onto the wire, which grows with payload size and shrinks with bandwidth.
• Processing delay: time spent in application code, serialisation, encryption, database queries and calls to other services.
• Queuing delay: time a request waits behind others, in network buffers, thread pools, connection pools or database locks. This is the part that grows dramatically under load.
• Connection setup: DNS lookup, TCP handshake and TLS handshake before the first byte of the request is even sent.

3. Latency Numbers Every Engineer Should Know

Exact values depend on hardware and networks, but the orders of magnitude are what matter for design decisions:

Two lessons follow. First, memory is vastly faster than disk or network, which is why caching is so effective. Second, physical distance sets a floor that no code optimisation can remove, which is why CDNs and multi-region deployments exist.

4. Measuring Latency: Averages vs Percentiles

Average latency is a misleading number. If 98 requests take 40 ms and 2 take 1.5 seconds, the average looks fine, yet some users are having a terrible experience. Engineers therefore use percentiles:

• p50 (median): half of requests are faster than this. It describes the typical experience.
• p90 / p95: 90% or 95% of requests are faster than this.
• p99: only 1 in 100 requests is slower than this. It describes the bad experience that frequent users meet regularly.
• p99.9 and max: the extreme tail, often caused by garbage-collection pauses, cold caches, retries or lock contention.

The program below simulates 10,000 requests where 2% hit a slow path, then compares the average with percentiles:

package com.javabytechie.latency;

import java.util.Arrays;
import java.util.Random;

/** Simulates 10,000 request latencies and shows why averages hide slow requests. */
public class LatencyPercentiles {

    static double percentile(double[] sorted, double p) {
        int index = (int) Math.ceil(p / 100.0 * sorted.length) - 1;          // nearest-rank method
        return sorted[Math.max(0, Math.min(index, sorted.length - 1))];
    }

    public static void main(String[] args) {
        Random random = new Random(7);
        double[] latencies = new double[10_000];
        for (int i = 0; i < latencies.length; i++) {
            double ms = 40 + random.nextGaussian() * 8;                       // most requests: about 40 ms
            if (random.nextDouble() < 0.02) {
                ms = 400 + random.nextDouble() * 1_200;                       // 2% hit a slow path: GC pause, cold cache
            }
            latencies[i] = Math.max(5, ms);
        }
        Arrays.sort(latencies);

        double average = Arrays.stream(latencies).average().orElse(0);
        System.out.printf("average : %7.1f ms%n", average);
        System.out.printf("p50     : %7.1f ms%n", percentile(latencies, 50));
        System.out.printf("p90     : %7.1f ms%n", percentile(latencies, 90));
        System.out.printf("p99     : %7.1f ms%n", percentile(latencies, 99));
        System.out.printf("p99.9   : %7.1f ms%n", percentile(latencies, 99.9));
        System.out.printf("max     : %7.1f ms%n", latencies[latencies.length - 1]);

        // Tail amplification: a page that calls 100 back-end services in parallel
        double chanceAllFast = Math.pow(0.99, 100);
        System.out.printf("If each call has a 1%% chance of being slower than p99, a page making 100 calls%n"
                + "waits for at least one slow call %.1f%% of the time.%n", (1 - chanceAllFast) * 100);
    }
}

average :    60.5 ms
p50     :    40.1 ms
p90     :    51.2 ms
p99     :  1087.5 ms
p99.9   :  1543.1 ms
max     :  1594.3 ms
If each call has a 1% chance of being slower than p99, a page making 100 calls
waits for at least one slow call 63.4% of the time.

The average (about 60 ms) suggests a healthy service, but the p99 is over one second. Monitoring only the average would hide the problem completely. Always track p50, p95 and p99, and set objectives on high percentiles, such as "p99 under 300 ms".

5. Why Tail Latency Matters

The last line of the output shows tail latency amplification. Modern pages and APIs often call many back-end services or shards in parallel and must wait for the slowest one. If each call has a 1% chance of being slow, a request that makes 100 such calls will be slowed down 1 − 0.99¹⁰⁰ ≈ 63% of the time. A rare problem in each component becomes the common experience for users.

Techniques to tame the tail:

• Reduce fan-out: call fewer services per request, or precompute results.
• Hedged requests: if a reply has not arrived after, say, the p95 time, send a duplicate request to another replica and use whichever answers first (only for safe, idempotent reads).
• Timeouts with fallbacks: return partial results rather than waiting for the slowest component.
• Remove sources of pauses: tune garbage collection, warm caches, avoid lock contention and noisy neighbours.

6. Latency vs Throughput vs Bandwidth

They influence each other. Batching many items together raises throughput but makes each item wait longer. Running a server close to full capacity maximises throughput but, as the next section shows, sharply increases latency.

7. Latency Under Load: Why Busy Servers Get Slow

When a server is lightly loaded, a new request is handled immediately. As utilisation rises, requests increasingly arrive while others are being processed, so they wait in a queue. Queuing theory shows that waiting time grows roughly in proportion to 1 ÷ (1 − utilisation): at 50% utilisation, latency is about double the unloaded time; at 90%, about ten times; near 100%, it shoots towards infinity.

Practical consequences:

• Plan capacity with headroom so peak traffic does not push servers into the steep part of the curve.
• Auto-scale on latency or queue length, not only on CPU.
• Limit queue sizes and shed excess load early; a fast "please retry" is better than a request that waits 30 seconds.
• Watch connection and thread pools: a pool that is too small creates hidden queues inside the application.

8. How to Reduce Latency

Always measure first. Profile real requests, find the biggest contributor and fix that; optimising a 2 ms step in a 400 ms request rarely helps.

9. Measuring Latency in Spring Boot

Spring Boot Actuator with Micrometer records the duration of every HTTP request automatically in the http.server.requests metric. A few properties add percentile histograms and service-level objective buckets, which monitoring systems such as Prometheus and Grafana can chart:

management:
  endpoints:
    web:
      exposure:
        include: health,prometheus
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true            # lets the monitoring system compute any percentile
      slo:
        http.server.requests: 100ms, 300ms, 1s  # counts of requests faster than each threshold

For a specific operation, such as a call to a payment provider, wrap it in a timer:

package com.javabytechie.latency;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import org.springframework.stereotype.Component;

@Component
public class PaymentGatewayClient {

    private final Timer chargeTimer;

    public PaymentGatewayClient(MeterRegistry registry) {
        this.chargeTimer = Timer.builder("payment.gateway.charge")
                .description("Latency of charge calls to the payment provider")
                .publishPercentileHistogram()
                .register(registry);
    }

    public String charge(String orderId, long amountInPaise) {
        return chargeTimer.record(() -> callProvider(orderId, amountInPaise));
    }

    private String callProvider(String orderId, long amountInPaise) {
        // real HTTP call to the provider, with a timeout
        return "PAY-" + orderId;
    }
}

Combine these metrics with distributed tracing (Micrometer Tracing with OpenTelemetry) to see exactly which service or query adds the most time to a slow request.

10. Latency Budgets and SLOs

A latency budget splits an end-to-end target among the components of a request, so every team knows its share. For a product page that must load its data within 300 ms at p95:

Turn the target into a service-level objective, such as "95% of product page API requests complete in under 300 ms over 30 days", and alert when the error budget is being consumed too quickly.

11. Common Mistakes to Avoid

• Reporting only the average, which hides the slow requests users complain about.
• Averaging percentiles across servers; percentiles must be computed from combined data or histograms.
• Calling services one after another when they could be called in parallel.
• N+1 database queries, such as loading 50 orders and then making 50 more queries for their items.
• Running servers close to 100% utilisation, where queuing makes latency unpredictable.
• Ignoring geography: serving global users from one distant region.
• Missing timeouts, so one slow dependency makes every request slow.
• Optimising without measuring, spending effort on parts of the request that do not matter.

12. Best Practices

• Measure p50, p95 and p99 for every important endpoint and dependency.
• Set latency SLOs on high percentiles and split them into budgets per component.
• Cache aggressively at the right layers, and serve static content from a CDN.
• Reuse connections and keep services that talk often close to each other.
• Parallelise independent calls and move slow work to background jobs.
• Keep utilisation below the steep part of the queuing curve, and auto-scale on latency signals.
• Use distributed tracing to find where time is actually spent.
• Set timeouts everywhere and return partial results rather than waiting forever.

13. Summary

• Latency is the time one request takes; it is the sum of propagation, transmission, processing, queuing and connection-setup delays.
• Memory is thousands of times faster than disk or network, and physical distance sets a hard lower limit.
• Use percentiles, not averages; tail latency grows into the common case when requests fan out.
• Latency rises steeply as utilisation approaches 100%, so plan headroom and shed excess load.
• Reduce latency with caching, CDNs, connection reuse, database tuning, parallel calls, asynchronous work and smaller payloads, guided by measurement and latency budgets.

Users judge a system by how fast it feels, so treat latency as a feature with clear targets. Open the FAQ tab for answers to common questions about latency.

Frequently Asked Questions (FAQ)