# Bulkhead: Split Shared Resources So One Failure Stays in Its Compartment

> Bulkhead splits threads, connections, and service copies into isolated groups, so one slow part cannot drain what the rest of the system needs.

**In short:** Bulkhead splits threads, connections, and service copies into isolated groups, so one slow part cannot drain the resources the rest of the system needs.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/bulkhead

## What You Will Learn

This lesson explains Bulkhead, a design idea for keeping one failing part of a system from taking down the others. By the end you should be able to explain why shared resources cause trouble, name ways to split them, and weigh what each split costs. The name comes from ship design. A ship's hull is divided into sealed sections, so water entering one section does not sink the whole vessel.

- **Bulkhead:** A boundary that splits a system's resources into separate groups, so trouble in one group stays in that group.
- **Resource:** Anything a system has only a limited amount of, such as threads, connections, memory, or server capacity.
- **Thread:** A worker that runs part of a program. A server can only run a limited number of threads at the same time.
- **Pool:** A fixed group of reusable items, such as threads or connections, that code borrows and gives back.
- **Connection:** An open link, for example to a database or another service, over which requests travel.
- **Downstream service:** Another service that your code calls to get data or do work.
- **Cascading failure:** One failure that spreads from part to part, until healthy parts also stop working.
- **Noisy consumer:** A user, feature, or job that uses far more than its fair share of a shared resource.

## Why Do Shared Pools Spread Failure?

Many systems start with one pool that serves every request. Each request takes a thread or a connection from that pool and returns it when finished. This is simple and cheap while everything is fast. The trouble starts when one kind of request slows down. Suppose a checkout page calls a payment service. If that service suddenly answers very slowly, every thread waiting on it stays busy. Requests that only need healthy services now find the pool empty, so they wait or fail too.

### How Does One Slow Part Spread?

Think of a reporting feature that runs slow database queries. Each query holds a connection for a long time. Those connections are then missing for login, product pages, and payments, even though none of those features is slow. This is a cascading failure. A noisy consumer can do the same thing with no bug at all, simply by asking for more than its share. The core problem is that unrelated features quietly depend on the same resources.

## How Do You Apply Bulkhead?

Applying Bulkhead means dividing shared resources into partitions, which are separate groups, and giving each group a fixed limit. Decide the dividing line first. Three common lines are below.

- **By consumer:** Give each kind of caller its own group. Interactive user requests, background jobs, and admin tools each get a separate share, so a batch job cannot starve people using the site.
- **By dependency:** Give each downstream service its own connection pool or thread pool. A slow analytics service then fills only its own group.
- **By criticality:** Separate important work from work that can wait. Payment usually deserves protection before a recommendations widget does.

### What Are Concrete Forms of Isolation?

Bulkhead is an idea, so it can take several shapes. Separate connection pools per downstream service keep network links apart. Separate thread pools keep worker threads apart. Dedicated service instances mean one part of the app runs on its own servers, where an instance is one running copy of a service. Separate containers with CPU and memory limits cap how much one workload can use on a shared machine. A container is a sealed package that runs one program, and CPU and memory are the processing power and working space it can draw on. Stronger walls cost more, so pick the lightest form that protects the part you care about.

### How Do You Size Each Partition?

A partition that is too small becomes a bottleneck during normal load, so healthy requests queue behind each other. A partition that is too large can still use enough CPU, memory, or network to hurt its neighbours. Base each size on the expected peak load for that job, plus some headroom, which is spare room for bursts. Also set a short wait limit, so callers learn quickly that a pool is full instead of hanging. A timeout is the maximum time a caller will wait before giving up.

> **Tip:** Start from measured peak load, not a guess. Write down the reason for each size, so the next engineer knows why a pool holds that number.

## Worked Example: An Online Store Checkout

Here is one realistic setup, step by step. A store's checkout page calls three services: payment, recommendations, and reviews. In the first version, all three calls share one connection pool. Now suppose the recommendations service stops answering, so its calls keep their connections open and never give them back.

1. **Map the calls:** List every downstream service the checkout page calls. Here there are three: payment, recommendations, and reviews.
2. **Give each service its own pool:** Create one connection pool per service, each with a fixed limit. Only calls to that service can borrow from it.
3. **Set a short wait limit:** When a pool is full, a new call waits only a brief time and then fails. The page can then use a fallback, which is a backup answer such as an empty list of suggestions.
4. **Watch the failure stay put:** The recommendations service hangs. Its calls keep taking connections until its own pool is full. New recommendation calls are refused quickly.
5. **Check the payment path:** Payment still has free connections in its own pool. Checkout keeps calling payment and completes the order, and the page simply shows no suggestions.

The key point is that each pool is separate, so a hung service can only empty its own pool. The next part shows the same idea in a tiny program.

### Simulate the Pools

This program uses semaphores. A semaphore is a counter that hands out a fixed number of permits, and a caller must get a permit before it can proceed. Here each pool is a semaphore with two permits. The three recommendation calls never release their permits, which imitates calls that hang. The third one is refused at once. Payment has its own permits, so its calls still succeed.

```python
import threading

pools = {
    'recommendations': threading.Semaphore(2),
    'payment': threading.Semaphore(2),
}

def call(name):
    if pools[name].acquire(blocking=False):
        return 'accepted'
    return 'rejected'

# These calls hang and never release their permits.
stuck = [call('recommendations') for _ in range(3)]
print(stuck)
print('payment:', call('payment'))
print('payment:', call('payment'))

```

Output:

```text
['accepted', 'accepted', 'rejected']
payment: accepted
payment: accepted

```

## Trade-offs and How to Judge Them

Bulkhead is not free. Every wall makes the system a little less efficient and a little harder to run. Judge each split by asking what it costs and what it protects.

- **Lost efficiency:** Idle capacity in one partition cannot be borrowed by another. If analytics sits quiet at night, its connections still stay reserved for it.
- **More complexity:** You now have many pools to size, name, document, and change. Each one can also be set wrong, and a wrong size can cause the very bottleneck you wanted to avoid.
- **Need for monitoring:** Partitions help only if you can see them. Track how full each pool gets, how many calls are refused, and how long callers wait.

### Decide Which Parts Deserve Isolation

Not every part needs a wall. Isolate the parts where a failure would hurt most, or where the neighbour is least reliable. Two common splits are critical versus non-critical workloads, and premium versus free tiers. A tier is a level of service, such as a paid plan or a free plan. Paying customers might get their own pool, so a flood of free-tier traffic cannot push them out. A best-effort analytics feed can share a smaller group, because its outages are acceptable. Do not wall off everything, though. Many tiny pools waste capacity and add work for the team.

### Check Your Understanding

**Quick check:** A web app uses one shared pool of 100 threads to call three backends: search, billing, and a slow analytics service. Analytics starts timing out, and soon the whole app stops responding. Which change keeps billing and search working, and what does it cost?

A. Raise the shared pool to 200 threads, so analytics has room to time out without starving the others.
B. Give each backend its own bounded thread pool, so analytics can only fill its own pool. The cost is that idle threads in one pool cannot help the others, and three pools must be sized and monitored.
C. Remove the timeouts, so slow analytics calls wait patiently instead of giving up and returning their threads.
D. Move the analytics calls onto the billing pool, so the slowest calls share a group with the most important one.

**Answer:** B. A bigger shared pool still lets slow analytics calls eventually take every thread. Separate bounded pools confine the damage to the slow backend, and the price is less flexible use of idle threads plus more to manage.

## Bulkhead in Real Libraries

Resilience4j is a Java library that offers two bulkhead styles. A semaphore bulkhead caps how many calls can run at once, and the calls run on the caller's own thread. A thread pool bulkhead runs calls on a separate, bounded set of threads with its own queue, where a queue is a line that holds work waiting its turn. The thread pool style isolates more strongly, but it costs more.

```
resilience4j:
  bulkhead:
    instances:
      reportService:
        max-concurrent-calls: 10   # at most 10 calls at once
        max-wait-duration: 100ms   # a caller waits up to 100 ms, then fails

```

- **Semaphore style:** Light, since it runs on the caller's own thread. It adds no extra threads, so it suits most blocking calls to a single dependency.
- **Thread pool style:** Stronger separation, useful when a call can hang even with timeouts. It needs its own threads to size and watch, so the overhead is higher.

## Common Mistakes

- **Unbounded queues:** A thread pool with an unlimited queue never says no, so the wall does nothing. Keep queues bounded, so extra work is refused early.
- **A hidden shared part:** Teams often build separate thread pools per dependency, then send them all through one HTTP client or one database pool with a single global limit. The shared part becomes the weak link. Trace each call path down to the connections.
- **Pools nobody watches:** A pool you cannot see filling up is one you will only notice during an outage. Add a metric for fullness and refusals before you go live.

## Key Takeaways

Bulkhead splits shared resources into fixed groups, so one slow or busy part can only use its own share. Divide by consumer, dependency, or criticality. Size each group from real load, set short wait limits, and keep queues bounded. Accept the cost of idle capacity and extra complexity, and watch every group with monitoring. Other lessons cover the neighbouring ideas, such as circuit breakers, retries, and throttling, which work well alongside a bulkhead.

## Sources

- [Bulkhead pattern (Wikipedia)](https://en.wikipedia.org/wiki/Bulkhead_pattern)
- [The Bulkhead Pattern: Isolating Failures in Distributed Systems, James Ross Jr.](https://www.jamesrossjr.com/blog/bulkhead-pattern-resilience)
- [Implementing Bulkhead with Resilience4j, Reflectoring](https://reflectoring.io/bulkhead-with-resilience4j/)
- [Resilience Patterns: Retries, Circuit Breakers, Bulkheads, codefarm backend guide](https://codefarm.in/guides/backend-engineer/07-microservices-integration/resilience-patterns)
- [The Bulkhead Pattern: Sinking One Compartment, Not the Whole Ship, The Daily Byte](https://thedailybyte.substack.com/p/the-bulkhead-pattern-sinking-one)
- [Bulkhead Pattern: Isolating Failures So One Service Can't Sink the Fleet, DEV Community](https://dev.to/merolhack/bulkhead-pattern-isolating-failures-so-one-service-cant-sink-the-fleet-1d94)
- [Resilient Microservices with Resilience4j: Circuit Breakers, Retries, and Bulkheads, DevDomain on Medium](https://medium.com/devdomain/resilient-microservices-with-resilience4j-circuit-breakers-retries-and-bulkheads-d6091b44ed4d)
