00:00

Rate Limiting Pattern in Microservices

Last updated:

In a microservices architecture, many independent services and clients send requests to each other all day long. Keeping that traffic stable and fair is essential. One of the most important patterns for this is rate limiting. It acts as a traffic officer for your APIs, preventing overload and keeping the experience good for every user.

This guide explains how rate limiting works, compares the main algorithms, and shows working Java examples: a token bucket and sliding window in plain Java, a Spring Boot filter, Spring Cloud Gateway with Redis, Resilience4j, and distributed limits shared across many instances.

1. What Is the Rate Limiting Pattern?

In microservices architecture, the Rate Limiting Pattern controls the number of requests a service can handle within a specific time window. It prevents service overload, manages traffic spikes, and maintains system stability and availability. Since microservices often depend on one another, rate limiting helps prevent excessive requests from overwhelming individual services and reduces the risk of cascading failures across the system.

2. Why Rate Limiting Is Important in Microservices

In microservices, many clients and services share the same back ends, so one bad caller can affect everyone. Rate limiting helps in these ways:

  • Prevents overload: A service accepts only as many requests as it can handle, so it stays up during traffic spikes.
  • Stops cascading failures: When one service slows down under heavy load, the services that call it slow down too. Limits stop the problem from spreading.
  • Protects shared resources: No single client can use up all the CPU, memory, threads or database connections.
  • Ensures fair usage: Every client gets a fair share, and you can offer different limits for free and premium plans.
  • Improves security: It slows down brute-force login attempts, scrapers and simple denial-of-service attacks.
  • Controls cost: In the cloud, every request costs money. Limits stop a buggy or abusive client from creating a huge bill.
  • Protects downstream services: Your service does not flood slower dependencies or exceed the quota of a paid third-party API.
  • Keeps performance stable: Response times stay consistent for all users, even when traffic is uneven.

3. Where Can Rate Limiting Be Implemented in Microservices?

A request passes through several layers before it reaches your business code, and a limit can be placed at any of them. Each layer sees different information and stops traffic at a different cost, so real systems often combine two or three layers:

LayerTypical toolsBest used for
Edge (CDN or WAF)Cloudflare, AWS WAF, AkamaiBlocking floods and abusive IP addresses before they reach your network
API gatewaySpring Cloud Gateway, Kong, NGINX, AWS API GatewayOne central place for per-client and per-API-key limits on every public route
Service mesh or sidecarIstio, Envoy, LinkerdLimiting traffic between internal services without changing application code
Inside the serviceServlet filter, Spring interceptor, Bucket4jLimits that depend on business data, such as the user's plan or the cost of an operation
Outgoing calls (client side)Resilience4j RateLimiter, Guava RateLimiterKeeping your service within the quota of a downstream service or third-party API
Message consumersConsumer concurrency, prefetch and poll settings in Kafka or RabbitMQControlling how fast queued work is processed, so the database is not overloaded

A common approach is a coarse limit at the edge or gateway to absorb bulk traffic, plus finer limits inside the services where business rules are known. This tutorial builds the in-service version in section 7, the gateway version in section 8 and the outgoing-call version in section 9.

4. Microservices Rate Limiting Algorithms

Every rate limiter answers one question: should this request go through right now? The algorithm decides how that answer is calculated. The five common algorithms differ in how they handle bursts, how accurate they are and how much memory they need.

4.1 Token Bucket

  • How it works: A bucket holds tokens up to a fixed capacity, and new tokens are added at a steady rate.
  • Each request takes one token. If no token is left, the request is rejected.
  • Pros: Allows short bursts up to the bucket size while keeping the average rate under control. It needs only two values per client: the token count and the last refill time.
  • Cons: A full bucket lets a burst through at once, so the downstream service must handle that burst.
  • Best for: Most public APIs. It is the most widely used algorithm. Section 5 builds one in plain Java.
Token bucket algorithm Tokens are added to a bucket at a fixed rate up to its capacity. Each incoming request takes one token and is allowed. When the bucket is empty, the request is rejected with 429. Refill: 1 token per second 3 tokens left Bucket capacity: 5 tokens Request Allowed a token is available, take 1 Rejected (429) the bucket is empty
Token bucket: each request spends one token, and tokens refill at a steady rate.

4.2 Leaky Bucket

  • How it works: Incoming requests join a queue of fixed size.
  • Requests leave the queue, or "leak", at a constant rate, no matter how fast they arrive. When the queue is full, new requests are rejected.
  • Pros: The service behind it receives a perfectly smooth, predictable flow.
  • Cons: No bursts are allowed, and queued requests wait longer, which adds latency.
  • Best for: Protecting systems that need an even load, such as a legacy database or a payment provider with a strict rate.
Leaky bucket algorithm A burst of requests enters a queue with a fixed capacity. Requests leave the queue at a constant rate, so the service receives an even load. When the queue is full, new requests are rejected. Queue (capacity 4) Burst of requests fixed rate 10 requests/s Service steady, even load Queue full → 429
Leaky bucket: requests wait in a queue and leave it at a constant rate, however fast they arrive.

4.3 Fixed Window Counter

  • How it works: Time is split into fixed windows, such as each calendar minute, with one counter per window.
  • Each request increases the counter. When the counter reaches the limit, further requests are rejected until the next window starts and the counter resets to zero.
  • Pros: The simplest algorithm, with very low memory use. It is easy to build with a single Redis INCR and an expiry.
  • Cons: Bursts at a window boundary can let through up to twice the limit in a short time, as the diagram shows.
  • Best for: Simple limits where an occasional burst is acceptable, such as daily quotas.
Fixed window counter algorithm Two one-minute windows, each with a limit of 5 requests. A client sends 5 requests at the end of the first window and 5 more at the start of the second. Both are allowed, so 10 requests pass in about 2 seconds. A sixth request in the second window is rejected. Limit: 5 requests per minute 12:00:00 – 12:00:59 12:01:00 – 12:01:59 count = 5 / 5 count = 5 / 5 6th → 429 10 requests in about 2 seconds: double the limit
Fixed window counter: the count resets at each window boundary, so a burst around the boundary can pass twice the limit.

4.4 Sliding Window Log

  • How it works: The limiter stores the timestamp of every request.
  • For each new request, it removes timestamps older than the window and counts the rest. The request is allowed only if the count is below the limit.
  • Pros: Exact. There is no boundary problem, because the window moves with every request.
  • Cons: Memory grows with traffic, because every request is stored until it expires.
  • Best for: Low-traffic, high-value endpoints where accuracy matters, such as login or OTP requests.
Sliding window log algorithm A timeline stores one timestamp per request. The window always covers the last 60 seconds up to now. Timestamps older than the window are removed, and the remaining four are counted against the limit. Window: the last 60 seconds now − 60 s now (new request) expired: removed from the log 4 timestamps counted
Sliding window log: only the timestamps inside the last 60 seconds count. The new request is allowed if that count is below the limit.

4.5 Sliding Window Counter

  • How it works: The limiter keeps only two counters: the previous window and the current window.
  • It estimates the rolling count by adding the current count to a share of the previous count. The share equals the part of the previous window that still falls inside the sliding window.
  • Pros: Almost as accurate as the log, with the low memory use of the fixed window.
  • Cons: It is an estimate, because it assumes requests in the previous window were spread evenly.
  • Best for: High-traffic APIs that need accurate limits at low cost. Section 6 builds one in Java.
Sliding window counter algorithm The previous minute had 8 requests and the current minute has 3 so far. It is now 15 seconds into the current minute, so 45 of the last 60 seconds fall in the previous minute. The estimate is 3 plus 8 times 45 over 60, which is 9. With a limit of 10, the request is allowed. Sliding window: the last 60 seconds Previous minute: 8 Current minute: 3 now (15 s into the current minute) 3 + 8 × 45/60 = 9 → 9 < 10, allowed 45/60 is the part of the previous minute still inside the window
Sliding window counter: two counters and a weighted sum give a close estimate of a true rolling window. Limit: 10 requests per minute.

Quick Comparison

AlgorithmAllows bursts?AccuracyMemory per client
Token bucketYes, up to the bucket sizeGoodVery low
Leaky bucketNo, output is smoothGoodGrows with queue size
Fixed window counterYes, at window edgesLow near boundariesVery low
Sliding window logNoExactHigh
Sliding window counterVery smallClose estimateVery low

5. Building a Token Bucket in Plain Java

The token bucket is easy to understand and to build. The bucket starts full. Tokens are added back continuously, and each request takes one:

TokenBucket.java
package com.example.ratelimit;

import java.time.Duration;

public class TokenBucket {

    private final long capacity;          // maximum burst size
    private final double tokensPerNano;   // refill speed
    private double availableTokens;
    private long lastRefillTime;

    public TokenBucket(long capacity, long refillTokens, Duration refillPeriod) {
        this.capacity = capacity;
        this.tokensPerNano = (double) refillTokens / refillPeriod.toNanos();
        this.availableTokens = capacity;
        this.lastRefillTime = System.nanoTime();
    }

    public synchronized boolean tryConsume() {
        refill();
        if (availableTokens >= 1) {
            availableTokens -= 1;
            return true;    // request allowed
        }
        return false;       // bucket empty: reject
    }

    private void refill() {
        long now = System.nanoTime();
        double newTokens = (now - lastRefillTime) * tokensPerNano;
        availableTokens = Math.min(capacity, availableTokens + newTokens);
        lastRefillTime = now;
    }
}
Usage
// 100 requests per minute, with bursts of up to 100
TokenBucket bucket = new TokenBucket(100, 100, Duration.ofMinutes(1));

if (bucket.tryConsume()) {
    // handle the request
} else {
    // respond with 429 Too Many Requests
}

The bucket size controls how big a burst can be, and the refill rate controls the long-term average. A bucket of 20 refilled with 100 tokens per minute allows short bursts of 20 while still averaging 100 per minute.

6. Building a Sliding Window Counter

The sliding window counter keeps only two numbers per client: the count for the current window and the count for the previous one. It estimates the requests in the last full window like this:

Sliding window formula
estimated = previousCount x (1 - elapsedInCurrentWindow / windowLength)
          + currentCount

Example: limit 100/minute, 15 seconds into the current minute
  previousCount = 80, currentCount = 30
  estimated = 80 x (1 - 15/60) + 30 = 60 + 30 = 90  -> allowed
SlidingWindowCounter.java
package com.example.ratelimit;

import java.time.Duration;

public class SlidingWindowCounter {

    private final int limit;
    private final long windowMillis;
    private long currentWindowStart;
    private int currentCount;
    private int previousCount;

    public SlidingWindowCounter(int limit, Duration window) {
        this.limit = limit;
        this.windowMillis = window.toMillis();
        this.currentWindowStart = System.currentTimeMillis();
    }

    public synchronized boolean tryAcquire() {
        long now = System.currentTimeMillis();
        long elapsed = now - currentWindowStart;

        if (elapsed >= windowMillis) {
            // Move forward; if more than one full window passed, the previous count is 0
            previousCount = (elapsed < 2 * windowMillis) ? currentCount : 0;
            currentCount = 0;
            currentWindowStart = now - (elapsed % windowMillis);
            elapsed = now - currentWindowStart;
        }

        double previousWeight = 1.0 - (double) elapsed / windowMillis;
        double estimated = previousCount * previousWeight + currentCount;

        if (estimated < limit) {
            currentCount++;
            return true;
        }
        return false;
    }
}

These plain-Java classes are great for learning and for a single instance. For production, libraries such as Bucket4j, Resilience4j or your API gateway’s built-in limiter add tested implementations, metrics and distributed storage.

7. A Rate Limiting Filter in Spring Boot

A servlet filter can apply the token bucket from section 5 to every incoming request, with one bucket per client:

RateLimitFilter.java
package com.example.product.ratelimit;

import java.io.IOException;
import java.time.Duration;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;

import com.example.ratelimit.TokenBucket;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;

@Component
public class RateLimitFilter extends OncePerRequestFilter {

    private static final int LIMIT_PER_MINUTE = 100;

    private final Map<String, TokenBucket> buckets = new ConcurrentHashMap<>();

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                    HttpServletResponse response,
                                    FilterChain chain) throws ServletException, IOException {

        String clientKey = resolveClientKey(request);
        TokenBucket bucket = buckets.computeIfAbsent(clientKey,
                key -> new TokenBucket(LIMIT_PER_MINUTE, LIMIT_PER_MINUTE, Duration.ofMinutes(1)));

        if (bucket.tryConsume()) {
            chain.doFilter(request, response);
            return;
        }

        response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
        response.setHeader("Retry-After", "1");
        response.setHeader("X-RateLimit-Limit", String.valueOf(LIMIT_PER_MINUTE));
        response.setContentType("application/json");
        response.getWriter().write(
                "{\"error\":\"too_many_requests\",\"message\":\"Please slow down.\"}");
    }

    // Prefer an API key; fall back to the caller's IP address
    private String resolveClientKey(HttpServletRequest request) {
        String apiKey = request.getHeader("X-API-Key");
        return (apiKey != null) ? "key:" + apiKey : "ip:" + request.getRemoteAddr();
    }

    @Override
    protected boolean shouldNotFilter(HttpServletRequest request) {
        // Never rate limit health checks
        return request.getRequestURI().startsWith("/actuator/health");
    }
}
Limitations: This map grows with every new client; use a cache with expiry, such as Caffeine, in real code. The limits also live in the memory of one instance, so with three instances a client effectively gets three times the limit. Section 10 solves this with Redis.

8. Rate Limiting at the API Gateway

Enforcing limits at the API gateway gives one consistent place for all services. Spring Cloud Gateway has a built-in, Redis-backed token bucket limiter. Add the reactive Redis starter (spring-boot-starter-data-redis-reactive), then configure a route:

application.yml (gateway)
spring:
  cloud:
    gateway:
      server:
        webflux:
          routes:
            - id: product-service
              uri: lb://product-service
              predicates:
                - Path=/api/products/**
              filters:
                - StripPrefix=1
                - name: RequestRateLimiter
                  args:
                    redis-rate-limiter.replenishRate: 2      # tokens per second (about 120/minute)
                    redis-rate-limiter.burstCapacity: 20     # maximum short burst
                    redis-rate-limiter.requestedTokens: 1    # tokens per request
                    key-resolver: "#{@apiKeyResolver}"
RateLimitKeyConfig.java
package com.example.gateway.config;

import org.springframework.cloud.gateway.filter.ratelimit.KeyResolver;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import reactor.core.publisher.Mono;

@Configuration
public class RateLimitKeyConfig {

    // One bucket per API key; requests without a key fall back to the IP address
    @Bean
    public KeyResolver apiKeyResolver() {
        return exchange -> {
            String apiKey = exchange.getRequest().getHeaders().getFirst("X-API-Key");
            if (apiKey != null) {
                return Mono.just("key:" + apiKey);
            }
            return Mono.justOrEmpty(exchange.getRequest().getRemoteAddress())
                    .map(address -> "ip:" + address.getAddress().getHostAddress())
                    .defaultIfEmpty("ip:unknown");
        };
    }
}

Because the counters live in Redis, every gateway instance shares the same limits. Rejected requests get 429, and the gateway adds headers such as X-RateLimit-Remaining automatically.

9. Limiting Outgoing Calls with Resilience4j

Rate limiting also protects the services you call. A paid third-party API may allow only 10 requests per second; going over it might cost money or get your account blocked. Resilience4j’s rate limiter caps outgoing calls:

application.yml
resilience4j:
  ratelimiter:
    instances:
      currency-api:
        limit-for-period: 10          # 10 calls...
        limit-refresh-period: 1s      # ...per second
        timeout-duration: 200ms       # wait up to 200 ms for permission
CurrencyClient.java
package com.example.order.client;

import java.math.BigDecimal;

import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

import io.github.resilience4j.ratelimiter.RequestNotPermitted;
import io.github.resilience4j.ratelimiter.annotation.RateLimiter;

@Component
public class CurrencyClient {

    private final RestClient restClient;
    private final ExchangeRateCache cache;

    public CurrencyClient(RestClient.Builder builder, ExchangeRateCache cache) {
        this.restClient = builder.baseUrl("https://api.currency.example").build();
        this.cache = cache;
    }

    @RateLimiter(name = "currency-api", fallbackMethod = "cachedRate")
    public BigDecimal getRate(String from, String to) {
        BigDecimal rate = restClient.get()
                .uri("/rates?from={from}&to={to}", from, to)
                .retrieve()
                .body(BigDecimal.class);
        cache.put(from, to, rate);
        return rate;
    }

    // Used when our own limit is reached
    private BigDecimal cachedRate(String from, String to, RequestNotPermitted ex) {
        return cache.get(from, to);
    }
}

The annotation requires the resilience4j-spring-boot3 dependency and Spring AOP. Note that this limit applies per application instance; with several instances, divide the provider’s limit between them or use a shared limiter.

10. Distributed Rate Limiting with Redis

Microservices usually run as several instances behind a load balancer. If each instance keeps its own counters, a client spread across three instances gets three times the intended limit. The solution is a shared store, most often Redis, that every instance updates.

Here is a simple fixed-window limiter using Redis’s atomic INCR command with Spring Data Redis:

RedisRateLimiter.java
package com.example.ratelimit;

import java.time.Duration;
import java.time.Instant;

import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;

@Component
public class RedisRateLimiter {

    private static final int LIMIT = 100;
    private static final Duration WINDOW = Duration.ofMinutes(1);

    private final StringRedisTemplate redis;

    public RedisRateLimiter(StringRedisTemplate redis) {
        this.redis = redis;
    }

    public boolean tryAcquire(String clientKey) {
        // One key per client per minute, for example "rate:key:abc123:29852160"
        long window = Instant.now().getEpochSecond() / WINDOW.toSeconds();
        String redisKey = "rate:" + clientKey + ":" + window;

        // INCR is atomic, so concurrent requests on all instances are counted correctly
        Long count = redis.opsForValue().increment(redisKey);

        if (count != null && count == 1) {
            // First request in this window: let Redis delete the key afterwards
            redis.expire(redisKey, WINDOW.plusSeconds(5));
        }
        return count != null && count <= LIMIT;
    }
}
  • All instances share the same counter, so the limit is exact across the cluster.
  • Keys expire automatically, so Redis does not fill up with old windows.
  • For a token bucket or sliding window in Redis, run the logic as a small Lua script, so the read and update happen atomically in one step. Spring Cloud Gateway and Bucket4j already do this for you.
  • Decide what happens if Redis is down: usually allow requests (“fail open”) so a Redis problem does not take down your API, and alert on it.

11. Choosing the Rate Limiting Key

KeyGood forWatch out for
API keyPublic and partner APIs; supports per-plan limitsKeys can be shared or leaked
User ID (from the login token)Fair limits for logged-in usersNot available before login
IP addressAnonymous traffic, login pages, basic abuse protectionMany users can share one IP (offices, mobile networks); behind proxies, read the client IP carefully
EndpointProtecting expensive operations, such as search or reportsCombine with a client key so one client cannot use the whole budget
CombinationFor example user + endpoint, or IP + endpoint for loginMore keys to store

Use different limits for different endpoints: stricter for expensive or sensitive ones (search, login, password reset) and relaxed or none for cheap ones (health checks, static data).

12. Best Practices

  1. Use the API gateway for consistent, central limits across all services, and add service-level limits for special cases.
  2. Choose the right key: API key, user ID, IP address or a combination, based on your security and business needs.
  3. Communicate limits clearly with 429, Retry-After and rate limit headers, and document them.
  4. Differentiate between endpoints: stricter limits for expensive operations such as search, relaxed ones for cheap requests.
  5. Use shared storage, such as Redis, when services run as several instances.
  6. Allow reasonable bursts with a token bucket, so normal users clicking quickly are not punished.
  7. Offer tiers: different limits for free, paid and internal clients.
  8. Never limit health checks or monitoring traffic.
  9. Monitor and adapt: track rejections per client and endpoint, and adjust limits to real traffic and capacity.
  10. Combine with other patterns such as circuit breakers, bulkheads and timeouts for complete protection.

13. Common Mistakes to Avoid

MistakeWhat goes wrongBetter approach
In-memory counters with many instancesClients get several times the intended limitStore counters in Redis or use the gateway’s shared limiter
Limiting only by IP addressWhole offices or mobile networks are blocked togetherPrefer API keys or user IDs where possible
Returning 500 or 503 instead of 429Clients cannot tell overload from a real errorReturn 429 with Retry-After
Clients retrying immediatelyRejected traffic multiplies and makes overload worseRespect Retry-After and use exponential backoff
Limits that are too strictNormal users hit errors during regular useBase limits on measured traffic and allow bursts
One limit for every endpointCheap endpoints are blocked, expensive ones are under-protectedSet limits per endpoint group
Rate limiting health checksLoad balancers think healthy instances are downExclude health and monitoring endpoints
Unbounded per-client mapsMemory grows with every new clientUse caches with expiry or external storage

14. Summary

The rate limiting pattern is a foundation of resilient, secure and fair microservices. It controls how many requests each client may make in a time window, protects services from spikes, bugs and attacks, keeps cloud costs predictable and enables fair, tiered plans. Choose an algorithm that fits your needs, usually a token bucket or sliding window; enforce limits centrally at the API gateway with shared Redis storage; protect outgoing calls with Resilience4j; and always answer with 429 and clear headers so clients can back off politely. With thoughtful limits in place, your services stay available and fast for everyone, even under unexpected load. Continue with the Bulkhead pattern to learn how to limit concurrent calls as well.