Bulkhead Pattern in Microservices

Building resilient microservices with watertight compartments. In a microservices system, one failing component should never sink the whole application. A ship uses bulkheads, watertight walls, so that a leak in one compartment does not flood the entire hull. Software architects use the Bulkhead pattern in the same way: to isolate failures and keep the rest of the system running.

This guide explains the pattern in simple terms, shows the problem it solves, and walks through working Java examples, from a plain-Java bulkhead to Resilience4j with Spring Boot, along with sizing, monitoring and best practices.

1. What Is the Bulkhead Pattern?

The Bulkhead pattern is a resilience pattern that prevents a failure in one part of a system from spreading to other parts. It does this by dividing the system into isolated compartments, each with its own limited resources and its own failure boundary.

The core idea is simple: do not put all your eggs in one basket. When resources are separated, a compartment that fails or becomes overloaded can only use up its own resources. The other compartments keep working normally, which improves the fault tolerance and availability of the whole system.

2. From Ships to Software

Imagine a ship’s hull as one big open space. A single hole would let water flood everything, and the ship would sink. Real ships are built with several watertight compartments separated by bulkheads. If one compartment is breached, the water stays there and the ship stays afloat.

 Without bulkheads            With bulkheads
+-------------------+      +-----+-----+-----+
|~~~~~~~~~~~~~~~~~~~|      |~~~~~|     |     |
|~~~~ flooded ~~~~~~|      |~~~~~|  OK |  OK |
+-------------------+      +-----+-----+-----+
     ship sinks          one compartment floods,
                            the ship floats

In software, the “water” is usually resource exhaustion: threads, database connections, CPU or memory being used up. A single slow or misbehaving dependency can consume all of a shared resource and bring down other, perfectly healthy features. The Bulkhead pattern builds walls that keep this exhaustion contained.

3. The Problem: Resource Exhaustion and Cascading Failures

Most services handle work with limited, shared resources. A web server has a fixed number of request threads; an application has a limited database connection pool. Problems start when one dependency becomes slow, which is often worse than being completely down:

1. A downstream service starts responding slowly, for example in 10 seconds instead of 100 milliseconds.
2. Every request that calls it holds a thread while waiting.
3. New requests keep arriving, and soon all threads are waiting on the slow service.
4. Requests for completely unrelated features, which do not even use that service, now have no threads left and fail too.
5. Other services that call this one start waiting as well, and the failure cascades through the system.

A fast failure, such as “connection refused”, releases resources immediately. A slow dependency holds them, which is why bulkheads are so valuable: they cap how much any single dependency can take.

4. Types of Bulkhead Isolation

The first three work inside one service (fine-grained bulkheads). The last two work at the infrastructure level (coarse-grained bulkheads): a hardware failure or severe outage in one cluster does not affect services running in another.

5. A Practical Example: E-Commerce Application

Consider a simplified e-commerce application with three microservices:

• Order service: processes customer orders.
• Inventory service: checks and updates product stock.
• Recommendation service: suggests related products to users.

5.1 Scenario without bulkheads

The order service calls both the inventory service and the recommendation service, using one shared thread pool for all outgoing HTTP calls.

1. During a flash sale, the inventory service becomes slow and starts timing out under the load.
2. Threads waiting for inventory responses pile up until the shared pool is exhausted.
3. When the order service now tries to call the recommendation service, there are no free threads left.
4. The recommendation service is perfectly healthy, yet it becomes unreachable. The whole order flow is degraded.

5.2 Scenario with bulkheads

We refactor the order service to use two dedicated bulkheads:

• Bulkhead A: only for calls to the inventory service.
• Bulkhead B: only for calls to the recommendation service.

Each bulkhead has a fixed number of threads and a small, bounded waiting queue.

              Order Service
       +------------+------------+
       |                         |
 [ Bulkhead A ]            [ Bulkhead B ]
   10 threads                5 threads
   (all busy!)               (still free)
       |                         |
       v                         v
 Inventory Service      Recommendation Service
   (slow)                    (healthy)

The result: during the same flash sale, the slow inventory service fills up Bulkhead A, but Bulkhead B is untouched. The order service can still call the recommendation service and show product suggestions. The failure is contained, and the system degrades gracefully instead of failing completely.

6. Building a Simple Bulkhead in Plain Java

To understand how bulkheads work, it helps to build a tiny one yourself. A semaphore holds a fixed number of permits; each call takes one permit and returns it when finished. When no permits are left, extra calls are rejected quickly instead of waiting forever:

package com.example.resilience;

import java.time.Duration;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;

public class SimpleBulkhead {

    private final String name;
    private final Semaphore permits;
    private final Duration maxWait;

    public SimpleBulkhead(String name, int maxConcurrentCalls, Duration maxWait) {
        this.name = name;
        this.permits = new Semaphore(maxConcurrentCalls);
        this.maxWait = maxWait;
    }

    public <T> T call(Supplier<T> task, Supplier<T> fallback) {
        boolean acquired;
        try {
            // Wait a short time for a free slot, then give up
            acquired = permits.tryAcquire(maxWait.toMillis(), TimeUnit.MILLISECONDS);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            return fallback.get();
        }

        if (!acquired) {
            System.out.println("Bulkhead '" + name + "' is full - using fallback");
            return fallback.get();
        }

        try {
            return task.get();
        } finally {
            permits.release();   // always give the slot back
        }
    }
}

SimpleBulkhead inventoryBulkhead =
        new SimpleBulkhead("inventory", 10, Duration.ofMillis(100));
SimpleBulkhead recommendationBulkhead =
        new SimpleBulkhead("recommendations", 5, Duration.ofMillis(50));

Stock stock = inventoryBulkhead.call(
        () -> inventoryClient.getStock(productId),
        () -> Stock.unknown(productId));

List<Product> suggestions = recommendationBulkhead.call(
        () -> recommendationClient.forProduct(productId),
        List::of);   // no suggestions is better than no page

Thread pool isolation works in a similar way with a separate ExecutorService per dependency. Always use a bounded queue; Executors.newFixedThreadPool() uses an unbounded queue, which only moves the pile-up from threads to memory:

package com.example.resilience;

import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

public final class BulkheadExecutors {

    private BulkheadExecutors() {
    }

    public static ThreadPoolExecutor create(int threads, int queueCapacity) {
        return new ThreadPoolExecutor(
                threads, threads,
                60, TimeUnit.SECONDS,
                new ArrayBlockingQueue<>(queueCapacity),    // bounded queue
                new ThreadPoolExecutor.AbortPolicy());      // reject when full
    }
}

This is a learning example. In real projects, use a proven library such as Resilience4j, which adds configuration, metrics and integration with other resilience patterns.

7. Semaphore Bulkhead with Resilience4j

Resilience4j is the standard resilience library for modern Java and Spring Boot applications. Its semaphore bulkhead limits how many calls to a dependency can run at the same time.

7.1 Dependencies

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
</dependency>
<!-- The annotations work through Spring AOP -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

Add the Resilience4j version explicitly, or import its BOM. Spring Cloud Circuit Breaker with Resilience4j is an alternative that brings the same features.

7.2 Configuration

resilience4j:
  bulkhead:
    instances:
      inventory:
        max-concurrent-calls: 10      # at most 10 calls at the same time
        max-wait-duration: 100ms      # wait briefly for a free slot, then reject
      recommendations:
        max-concurrent-calls: 5
        max-wait-duration: 0          # reject immediately when full

7.3 Protecting a client with @Bulkhead

package com.example.order.client;

import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

import io.github.resilience4j.bulkhead.BulkheadFullException;
import io.github.resilience4j.bulkhead.annotation.Bulkhead;

@Component
public class InventoryClient {

    private final RestClient restClient;

    public InventoryClient(RestClient.Builder builder) {
        this.restClient = builder.baseUrl("http://inventory-service").build();
    }

    @Bulkhead(name = "inventory", fallbackMethod = "stockUnavailable")
    public StockStatus getStock(String productId) {
        return restClient.get()
                .uri("/stock/{id}", productId)
                .retrieve()
                .body(StockStatus.class);
    }

    // Called when the bulkhead is full (or the call fails)
    private StockStatus stockUnavailable(String productId, BulkheadFullException ex) {
        return StockStatus.unknown(productId);
    }
}

When 10 inventory calls are already running, the 11th waits up to 100 ms for a free slot. If none frees up, Resilience4j throws a BulkheadFullException, and the fallback returns “stock unknown” at once. The order page stays fast instead of joining the queue of stuck requests.

8. Thread Pool Bulkhead with Resilience4j

The thread pool bulkhead runs calls on a dedicated pool of threads. The caller’s thread is not blocked, so the protected method must return a CompletableFuture (or another CompletionStage):

resilience4j:
  thread-pool-bulkhead:
    instances:
      recommendations:
        core-thread-pool-size: 2
        max-thread-pool-size: 4
        queue-capacity: 10
        keep-alive-duration: 20ms

package com.example.order.client;

import java.util.List;
import java.util.concurrent.CompletableFuture;

import org.springframework.core.ParameterizedTypeReference;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

import io.github.resilience4j.bulkhead.annotation.Bulkhead;

@Component
public class RecommendationClient {

    private final RestClient restClient;

    public RecommendationClient(RestClient.Builder builder) {
        this.restClient = builder.baseUrl("http://recommendation-service").build();
    }

    @Bulkhead(name = "recommendations",
              type = Bulkhead.Type.THREADPOOL,
              fallbackMethod = "noRecommendations")
    public CompletableFuture<List<Product>> forProduct(String productId) {
        List<Product> products = restClient.get()
                .uri("/recommendations?productId={id}", productId)
                .retrieve()
                .body(new ParameterizedTypeReference<List<Product>>() { });
        return CompletableFuture.completedFuture(products);
    }

    private CompletableFuture<List<Product>> noRecommendations(String productId, Throwable ex) {
        return CompletableFuture.completedFuture(List.of());
    }
}

The body of the method runs on one of the 2–4 threads reserved for recommendations. When all threads are busy and the queue of 10 is full, new calls go straight to the fallback, and the page simply shows no suggestions.

9. Semaphore vs Thread Pool Bulkhead

Rule of thumb: start with the semaphore bulkhead. It is simpler and cheaper, and it is the better fit for reactive code and Java virtual threads, where creating threads is cheap but the downstream service’s capacity is still limited.

10. How to Size a Bulkhead

A bulkhead that is too small rejects good traffic; one that is too large does not protect anything. A simple starting point is Little’s law:

concurrent calls needed = requests per second x average response time (seconds)

Example:
  50 requests/second x 0.2 seconds = 10 concurrent calls
  Add headroom for spikes (about 50%)  -> max-concurrent-calls = 15

• Measure real traffic and response times, ideally at peak hours, using your metrics.
• Respect the downstream limit: never allow more concurrent calls than the other service can handle.
• Keep the total sensible: the sum of all bulkheads should fit within your service’s thread and connection limits.
• Prioritize: give critical paths, such as checkout, more room than optional features, such as recommendations.
• Load test and adjust: confirm in a test environment that a slow dependency fills only its own bulkhead.

11. Combining Bulkheads with Other Resilience Patterns

Bulkheads limit how much a dependency can take, but they do not decide how long to wait or when to stop calling. Combine them with other patterns:

@Retry(name = "inventory")
@CircuitBreaker(name = "inventory", fallbackMethod = "stockUnavailable")
@Bulkhead(name = "inventory")
public StockStatus getStock(String productId) {
    return restClient.get()
            .uri("/stock/{id}", productId)
            .retrieve()
            .body(StockStatus.class);
}

By default, Resilience4j applies the annotations in this order, from outside to inside: Retry → CircuitBreaker → RateLimiter → TimeLimiter → Bulkhead. The bulkhead is closest to the actual call, so each retry attempt also needs a free slot. Always configure timeouts on the HTTP client as well, so a slot is never held for too long.

12. Bulkheads Beyond Code: Connection Pools and Deployments

12.1 Separate connection pools

Database connections are a classic shared resource. A heavy reporting query can take every connection and block customer checkouts. Give each workload its own pool:

app:
  datasource:
    orders:
      jdbc-url: jdbc:postgresql://db:5432/shop
      maximum-pool-size: 20      # customer-facing work
    reporting:
      jdbc-url: jdbc:postgresql://db-replica:5432/shop
      maximum-pool-size: 5       # heavy reports cannot take more than 5

Each pool is created as its own DataSource bean, bound with @ConfigurationProperties. The same idea applies to HTTP clients: give each downstream service its own connection pool with its own limits.

12.2 Separate deployments

At the infrastructure level, run critical and non-critical workloads on separate instances, node pools or namespaces, each with fixed resource limits:

spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: checkout-service
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

Resource limits stop one container from starving its neighbors on the same machine, and separate deployments mean a crash or memory leak in a batch job cannot take checkout down with it.

13. Monitoring and Testing Bulkheads

With Spring Boot Actuator and Micrometer, Resilience4j publishes bulkhead metrics automatically. Useful ones include:

Set alerts when a bulkhead is often full or rejects many calls: it means either the dependency is in trouble or the bulkhead is too small.

To test a bulkhead, simulate a slow dependency, for example with WireMock adding a fixed delay, send many requests in parallel, and check that:

• Calls beyond the limit are rejected quickly and reach the fallback.
• Calls to other dependencies still succeed at normal speed.
• The bulkhead recovers as soon as the slow dependency speeds up again.

14. Key Benefits of the Bulkhead Pattern

• Failure containment: a problem in one dependency cannot bring down the entire system.
• Graceful degradation: parts of the system keep working while others fail, for example a product page without recommendations.
• Better resource allocation: critical features can be guaranteed capacity based on their importance.
• Higher availability: preventing cascading failures improves overall uptime.
• Fair sharing: one noisy client or tenant cannot use up capacity meant for everyone.
• Faster failures: rejected calls fail in milliseconds instead of hanging, so users get a quick answer.

15. Drawbacks and Trade-offs

16. Best Practices

1. Identify critical boundaries: isolate dependencies based on how critical they are and who uses them.
2. Set sensible limits: size each bulkhead (concurrent calls, threads, queue size) from performance tests and capacity planning.
3. Always provide a fallback, such as cached data, a default value or a friendly message.
4. Keep waiting short: a small max-wait-duration or queue keeps rejections fast.
5. Combine with other patterns: bulkheads are most powerful together with timeouts, circuit breakers, retries and fallbacks.
6. Protect the most important paths first, such as checkout and payment.
7. Monitor and alert on bulkhead usage and rejections.
8. Isolate at several levels: threads and connections in code, plus separate deployments for critical services.

You do not need to build bulkheads from scratch. Resilience4j (Java) provides semaphore and thread pool bulkheads. Hystrix, which popularized thread pool isolation, is now in maintenance mode but remains conceptually important. Polly offers a bulkhead policy for .NET, and service meshes such as Istio can limit connections and pending requests per service.

17. Common Mistakes to Avoid

18. Summary

The Bulkhead pattern is a fundamental technique for building robust, fault-tolerant microservices. Like the watertight compartments of a ship, it separates resources, such as threads, connections and service instances, so that a slow or failing dependency can only exhaust its own share. With Resilience4j you can add semaphore or thread pool bulkheads to Spring Boot with a few lines of configuration, provide fallbacks for a graceful experience, and monitor them through Actuator. Size bulkheads from real measurements, combine them with timeouts and circuit breakers, and isolate critical workloads at the infrastructure level too. In the unpredictable world of distributed systems, these compartments are not just a best practice; they are essential for keeping your application stable and available. Continue with the Circuit Breaker Pattern to complete your resilience toolkit.

19. Frequently Asked Questions (FAQ)

@Bulkhead(name = "inventory", fallbackMethod = "stockUnavailable")
public Mono<StockStatus> getStock(String productId) {
    return webClient.get()
            .uri("/stock/{id}", productId)
            .retrieve()
            .bodyToMono(StockStatus.class);
}

max concurrent calls = peak requests per second x typical response time (seconds) x 1.5