Bulkhead: Split Shared Resources So One Failure Stays in Its Compartment
Bulkhead splits threads, connections, and service copies into isolated groups, so one slow part cannot drain what the rest of the system needs.
In short
Bulkhead splits threads, connections, and service copies into isolated groups, so one slow part cannot drain the resources the rest of the system needs.
9 min readFree
What You Will Learn
This lesson explains Bulkhead, a design idea for keeping one failing part of a system from taking down the others. By the end you should be able to explain why shared resources cause trouble, name ways to split them, and weigh what each split costs. The name comes from ship design. A ship's hull is divided into sealed sections, so water entering one section does not sink the whole vessel.
Key words
- Bulkhead
- A boundary that splits a system's resources into separate groups, so trouble in one group stays in that group.
- Resource
- Anything a system has only a limited amount of, such as threads, connections, memory, or server capacity.
- Thread
- A worker that runs part of a program. A server can only run a limited number of threads at the same time.
- Pool
- A fixed group of reusable items, such as threads or connections, that code borrows and gives back.
- Connection
- An open link, for example to a database or another service, over which requests travel.
- Downstream service
- Another service that your code calls to get data or do work.
- Cascading failure
- One failure that spreads from part to part, until healthy parts also stop working.
- Noisy consumer
- A user, feature, or job that uses far more than its fair share of a shared resource.
How Does One Slow Part Spread?
Think of a reporting feature that runs slow database queries. Each query holds a connection for a long time. Those connections are then missing for login, product pages, and payments, even though none of those features is slow. This is a cascading failure. A noisy consumer can do the same thing with no bug at all, simply by asking for more than its share. The core problem is that unrelated features quietly depend on the same resources.
How Do You Apply Bulkhead?
Applying Bulkhead means dividing shared resources into partitions, which are separate groups, and giving each group a fixed limit. Decide the dividing line first. Three common lines are below.
By consumer
Give each kind of caller its own group. Interactive user requests, background jobs, and admin tools each get a separate share, so a batch job cannot starve people using the site.
By dependency
Give each downstream service its own connection pool or thread pool. A slow analytics service then fills only its own group.
By criticality
Separate important work from work that can wait. Payment usually deserves protection before a recommendations widget does.
What Are Concrete Forms of Isolation?
Bulkhead is an idea, so it can take several shapes. Separate connection pools per downstream service keep network links apart. Separate thread pools keep worker threads apart. Dedicated service instances mean one part of the app runs on its own servers, where an instance is one running copy of a service. Separate containers with CPU and memory limits cap how much one workload can use on a shared machine. A container is a sealed package that runs one program, and CPU and memory are the processing power and working space it can draw on. Stronger walls cost more, so pick the lightest form that protects the part you care about.
How Do You Size Each Partition?
A partition that is too small becomes a bottleneck during normal load, so healthy requests queue behind each other. A partition that is too large can still use enough CPU, memory, or network to hurt its neighbours. Base each size on the expected peak load for that job, plus some headroom, which is spare room for bursts. Also set a short wait limit, so callers learn quickly that a pool is full instead of hanging. A timeout is the maximum time a caller will wait before giving up.
Tip
Start from measured peak load, not a guess. Write down the reason for each size, so the next engineer knows why a pool holds that number.
Worked Example: An Online Store Checkout
Here is one realistic setup, step by step. A store's checkout page calls three services: payment, recommendations, and reviews. In the first version, all three calls share one connection pool. Now suppose the recommendations service stops answering, so its calls keep their connections open and never give them back.
- 1
Map the calls
List every downstream service the checkout page calls. Here there are three: payment, recommendations, and reviews.
- 2
Give each service its own pool
Create one connection pool per service, each with a fixed limit. Only calls to that service can borrow from it.
- 3
Set a short wait limit
When a pool is full, a new call waits only a brief time and then fails. The page can then use a fallback, which is a backup answer such as an empty list of suggestions.
- 4
Watch the failure stay put
The recommendations service hangs. Its calls keep taking connections until its own pool is full. New recommendation calls are refused quickly.
- 5
Check the payment path
Payment still has free connections in its own pool. Checkout keeps calling payment and completes the order, and the page simply shows no suggestions.
The key point is that each pool is separate, so a hung service can only empty its own pool. The next part shows the same idea in a tiny program.
Simulate the Pools
This program uses semaphores. A semaphore is a counter that hands out a fixed number of permits, and a caller must get a permit before it can proceed. Here each pool is a semaphore with two permits. The three recommendation calls never release their permits, which imitates calls that hang. The third one is refused at once. Payment has its own permits, so its calls still succeed.
import threading
pools = {
'recommendations': threading.Semaphore(2),
'payment': threading.Semaphore(2),
}
def call(name):
if pools[name].acquire(blocking=False):
return 'accepted'
return 'rejected'
# These calls hang and never release their permits.
stuck = [call('recommendations') for _ in range(3)]
print(stuck)
print('payment:', call('payment'))
print('payment:', call('payment'))
Output
['accepted', 'accepted', 'rejected'] payment: accepted payment: accepted
Trade-offs and How to Judge Them
Bulkhead is not free. Every wall makes the system a little less efficient and a little harder to run. Judge each split by asking what it costs and what it protects.
Lost efficiency
Idle capacity in one partition cannot be borrowed by another. If analytics sits quiet at night, its connections still stay reserved for it.
More complexity
You now have many pools to size, name, document, and change. Each one can also be set wrong, and a wrong size can cause the very bottleneck you wanted to avoid.
Need for monitoring
Partitions help only if you can see them. Track how full each pool gets, how many calls are refused, and how long callers wait.
Decide Which Parts Deserve Isolation
Not every part needs a wall. Isolate the parts where a failure would hurt most, or where the neighbour is least reliable. Two common splits are critical versus non-critical workloads, and premium versus free tiers. A tier is a level of service, such as a paid plan or a free plan. Paying customers might get their own pool, so a flood of free-tier traffic cannot push them out. A best-effort analytics feed can share a smaller group, because its outages are acceptable. Do not wall off everything, though. Many tiny pools waste capacity and add work for the team.
Bulkhead in Real Libraries
Resilience4j is a Java library that offers two bulkhead styles. A semaphore bulkhead caps how many calls can run at once, and the calls run on the caller's own thread. A thread pool bulkhead runs calls on a separate, bounded set of threads with its own queue, where a queue is a line that holds work waiting its turn. The thread pool style isolates more strongly, but it costs more.
resilience4j:
bulkhead:
instances:
reportService:
max-concurrent-calls: 10 # at most 10 calls at once
max-wait-duration: 100ms # a caller waits up to 100 ms, then fails
Semaphore style
Light, since it runs on the caller's own thread. It adds no extra threads, so it suits most blocking calls to a single dependency.
Thread pool style
Stronger separation, useful when a call can hang even with timeouts. It needs its own threads to size and watch, so the overhead is higher.
Common Mistakes
Unbounded queues
A thread pool with an unlimited queue never says no, so the wall does nothing. Keep queues bounded, so extra work is refused early.
A hidden shared part
Teams often build separate thread pools per dependency, then send them all through one HTTP client or one database pool with a single global limit. The shared part becomes the weak link. Trace each call path down to the connections.
Pools nobody watches
A pool you cannot see filling up is one you will only notice during an outage. Add a metric for fullness and refusals before you go live.
Key Takeaways
Bulkhead splits shared resources into fixed groups, so one slow or busy part can only use its own share. Divide by consumer, dependency, or criticality. Size each group from real load, set short wait limits, and keep queues bounded. Accept the cost of idle capacity and extra complexity, and watch every group with monitoring. Other lessons cover the neighbouring ideas, such as circuit breakers, retries, and throttling, which work well alongside a bulkhead.
Sources
- Bulkhead pattern (Wikipedia)
- The Bulkhead Pattern: Isolating Failures in Distributed Systems, James Ross Jr.
- Implementing Bulkhead with Resilience4j, Reflectoring
- Resilience Patterns: Retries, Circuit Breakers, Bulkheads, codefarm backend guide
- The Bulkhead Pattern: Sinking One Compartment, Not the Whole Ship, The Daily Byte
- Bulkhead Pattern: Isolating Failures So One Service Can't Sink the Fleet, DEV Community
- Resilient Microservices with Resilience4j: Circuit Breakers, Retries, and Bulkheads, DevDomain on Medium
Last updated on 18 November 2026
Related lessons
- Design systemsAvatar: The Shared Person Picture That Stays Readable When Photos Are Missing
- System designCQRS: Split the Code That Changes Data From the Code That Reads It
- System designGateway Offloading: Move Shared Jobs to One Gateway, Not Every Service
- System designCompute Resource Consolidation: Share One Unit to Cut Cost, Accept Less Isolation
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.