# Noisy Neighbor: When One Shared Customer Slows Everyone Else

> Learn how one heavy tenant can slow down innocent users in a shared system, how to spot it with per-tenant metrics, and how to limit the damage.

**In short:** A noisy neighbor is one heavy customer on a shared system who slows everyone else, and per-tenant metrics help you spot and limit it.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/noisy-neighbor

## What You Will Learn

This lesson explains a problem called Noisy Neighbor. You will learn how it starts, what it looks like to the people it hurts, how to find the cause, and how to limit the damage.

Noisy Neighbor is a performance antipattern. An antipattern is a common design mistake that looks fine at first and causes harm later. The name is a picture: in a block of flats, one loud neighbor can make life hard for everyone else, even when the others are quiet and polite.

- **Tenant:** One customer, team, or group of users that uses a shared system alongside others.
- **Multi-tenant system:** One running system that serves many tenants at the same time.
- **Shared resource:** Something many tenants use together, such as a server's CPU, memory, a disk, a network link, a database, or a queue.
- **CPU:** The part of a computer that does the work of running programs.
- **Disk I/O:** Reading and writing data on storage. I/O stands for input and output.
- **Queue depth:** How many jobs are waiting in line to be handled.
- **Quota:** A fixed allowance for one tenant, such as a number of calls or a budget of work over a period.
- **Rate limit:** A cap on how many requests one tenant can send in a set time window.
- **Throttling:** Slowing down or refusing extra work once a limit is reached.
- **Percentile:** A speed value where a set share of requests finish at or below that time. The p99 is the time that 99 out of 100 requests beat.

## How Does the Problem Happen?

Many systems are built to be shared, and for good reason. When capacity sits in one common pool, a provider can serve many customers for less money than it would cost to give each customer a private set of everything.

Sharing has a cost. A shared system has a fixed amount of capacity, and every tenant draws from it. If one tenant pulls too much, the others have less left. These places are often shared:

- **Servers:** One machine's CPU and memory are split between every tenant's requests and background jobs.
- **Databases:** One database has a limited number of connections, which are open links that queries use, and a limited amount of query work it can do at once.
- **Queues:** One line of waiting jobs can hold work from every tenant, so one flood fills the line for everyone.
- **Network links:** One pipe carries all the traffic, so a large transfer from one tenant can crowd out the rest.

The problem is not always one greedy tenant. Azure's guidance points out that it can also appear when each tenant uses only a small part of the system. Together, the load of many tenants can reach a peak above what the system can handle.

Nothing in a basic shared system stops this on its own. Without a limit, the first work to arrive or the heaviest work gets served first, and everyone else waits. Azure's guidance also says that most tenants do not intend harm, and may not even know their work is the cause. A nightly export that feels normal to one customer can be a problem for everyone else.

> **Tip:** When you plan a shared system, ask one question first: which resource do all tenants use? That is where Noisy Neighbor risk will show up first.

### What Are the Symptoms for Innocent Users?

The people who suffer most are not the heavy tenant. They are the tenants who changed nothing. Their own traffic looks the same as it did last week, yet their pages get slow or stop loading. This is what makes the problem confusing for support teams.

- **Latency spikes:** Latency means how long a request takes to finish. Pages take much longer than usual, but only at certain hours or during certain tenant activity.
- **Random timeouts:** A timeout is when a request gives up after waiting too long. The same button works one minute and times out the next.
- **Intermittent errors:** Errors come and go, so bug reports are hard to repeat.
- **No local change:** The affected tenant's request count, data size, and settings all look normal.

The key sign is the mismatch. A slow system with no change in the victim's behavior suggests that something else, often another tenant, is using the shared capacity.

## How Do You Detect It?

Detection starts with two questions. Who is using the shared resource? And who is being hurt? If you only watch the whole system, you can see that things are slow, but not why. Without per-tenant visibility, the problem looks like random slowness, and teams waste time guessing.

Azure's guidance recommends watching both the overall resource use and the use by each tenant. It also suggests alerts for sudden spikes, so the team learns about a problem before customers write in.

### What Are Per-tenant Metrics?

A metric is a number measured over time, such as how many requests arrived in the last minute. A per-tenant metric is the same number, split by tenant. This shows which tenant is taking the most, and who is paying for it.

- **Requests:** How many calls each tenant makes per minute. A sudden jump is a common first clue.
- **CPU and memory:** How much processing and memory each tenant uses on shared machines.
- **Disk I/O and database time:** How much reading, writing, and query time each tenant causes.
- **Queue depth:** How many of each tenant's jobs are waiting. A long line for one tenant can hold up the others.

> **Tip:** Tag every request with a tenant ID at the entry point. If the tag travels with the request into logs, queues, and the database, you can trace a slow page back to its cause.

### What Are Latency Percentiles?

Many teams first look at average latency, but the average hides slow requests. A few very slow requests barely move it, so the dashboard looks fine while some users suffer.

Percentiles give a clearer picture. The p50 is the median, the value that half of requests beat. The p99 is the value that 99 out of 100 requests beat, so the slowest 1 percent sit above it.

```python
latencies = [10, 12, 11, 13, 900, 12, 11, 10, 14, 12]
ordered = sorted(latencies)
print('mean', sum(latencies) / len(latencies))
print('median', ordered[len(ordered) // 2])
print('slowest', ordered[-1])

```

Output:

```text
mean 100.5
median 12
slowest 900

```

In this small sample, the mean is 100.5 ms, which describes no typical request. The median is 12 ms, which shows the normal experience. The slowest request took 900 ms. One slow request is enough to ruin the average, which is why percentiles are the better choice.

A useful pattern: if the median stays flat but the high percentiles rise, only some requests are slowing down. That often means one tenant is competing with others for the same resource.

> **Tip:** Compare each tenant's percentiles with the system-wide percentiles. If one tenant's volume jumps in the same window where other tenants' p99 rises, you have a strong suspect.

### Quick Check: Detection

**Quick check:** A reporting app runs 200 customers on one shared server. Support says pages time out for different customers at different times. The average response time looks normal. Per-tenant metrics show that one customer's request count rose twenty times during the same windows. What is the best next step?

A. Restart the database every hour to clear its memory
B. Set a per-tenant rate limit for that customer and watch its usage
C. Move all 200 customers to a new region at once
D. Ask the customers who see timeouts to use the app less

**Answer:** B. The per-tenant numbers point to one heavy tenant. A rate limit caps that tenant without punishing the others. The affected customers did nothing wrong, so asking them to cut back is the wrong fix.

## How to Mitigate It

You cannot remove every Noisy Neighbor risk from a shared system, but you can make it rarer and reduce the harm. The main tools are limits, isolation, throttling, and more capacity. Most real designs use several together.

Azure's guidance treats the problem as resource governance. That means deciding how much each tenant may use, and enforcing that rule in the system, rather than hoping tenants behave.

### Rate Limits and Quotas

A rate limit caps how many requests one tenant can send in a time window. A quota is a larger allowance, such as a set amount of work over a day or a month. Both are set per tenant. A heavy customer hits its own wall while other customers keep getting served.

This lets short bursts pass while still capping the long-term rate. Some teams also limit concurrency, which is how many jobs one tenant may run at the same time.

1. **Tag each request:** Read the tenant ID from the request, such as an account key, and keep it with the request.
2. **Count per tenant:** Track how many requests or jobs each tenant used in the current window.
3. **Reject or delay the excess:** When a tenant goes past its limit, refuse the extra call with a clear message, or hold it in that tenant's own queue.
4. **Reset and review:** When the window ends, start counting again, and review the limits against real usage.

### Resource Isolation

Isolation means giving some tenants their own part of a shared resource, so their load cannot reach everyone else. It is stronger than a limit, because the separation is built into the design. Azure's guidance says dedicated resources keep each tenant's compute separated from the others, which helps prevent contention.

- **Separate pools:** Give heavy tenants their own database connection pool, a set of reserved connections, so they cannot use up the connections other tenants need.
- **Separate queues:** Send heavy or slow work, such as bulk imports, into its own queue. Other tenants' jobs stay in a different line.
- **Dedicated instances:** Run a heavy tenant on its own server, database, or cluster. This gives the strongest protection.

If a tenant keeps going past its limits, the next step is often to move it to dedicated resources.

### Throttling and Autoscaling

Throttling protects the shared resource by slowing or refusing work once a limit is reached. Azure's guidance links quota enforcement to throttling and rate limiting, and suggests that clients handle throttling and retry transient failures.

Some operations can be made safer by changing their shape. Azure suggests limiting how many records one query may return, setting a time limit on queries, or moving heavy work into a background job that runs at off-peak times, when few people are using the system.

Autoscaling means the system adds capacity when usage rises and removes it when usage falls. It helps when total load is too high for current machines. Autoscaling works best alongside limits.

### The Trade-Off: Isolation Versus Cost

Isolation costs money and effort. Separate pools, queues, and databases mean more machines to buy, run, and watch. A fully shared design is cheaper and simpler, but it leaves everyone exposed to the heaviest tenant.

Azure's guidance on databases shows the trade. A shared database serves many tenants at a low cost per tenant, while separate databases may be worth the extra cost when tenants need stronger isolation.

> **Tip:** Start with limits and good metrics. Move a tenant to dedicated resources only when limits are not enough, and the extra cost is clearly worth the protection.

## Worked Example: A Shared Analytics Dashboard

This example uses a sample product, modeled on a common kind of software-as-a-service (SaaS) analytics dashboard. SaaS means software that customers use over the web. Each customer sees charts built from its own data. The company has 500 customers, and all of them share one database.

1. **Normal day:** Each dashboard page runs small queries. They finish in a few milliseconds, and every customer sees fast charts.
2. **The export starts:** Customer 42 clicks Export All to download three years of raw events. The app sends one very large query to the shared database.
3. **The database fills up:** The large query takes most of the database's CPU time and most of its connections. Other dashboard queries wait in line behind it.
4. **Innocent users suffer:** Customers 7, 118, and 301 see their pages time out. None of them changed anything. Their request counts look normal, but their p99 latency jumps.
5. **Detection finds the cause:** Per-tenant query-time metrics show that customer 42 uses most of the database time. Alerts fire for slow pages across many tenants at the same moment.

The team applies two fixes. The first is a per-tenant query limit, which caps how many heavy queries one customer may run at once. The second is a separate export queue, so exports wait in their own line and run on a small set of workers, ideally off-peak.

1. **Add a per-tenant limit:** Each customer gets a cap on concurrent heavy queries. Customer 42 is held to its own allowance.
2. **Move exports to their own queue:** Export jobs go into a separate queue with a small worker pool. Dashboard queries never wait behind them.
3. **Check the result:** Customer 42's export takes longer to finish, but the dashboards of customers 7, 118, and 301 load on time again.

The code below shows the core idea of a per-tenant limit. Each tenant gets the same allowance of three calls in a window. The heavy tenant is rejected after its third call, while the small tenant is still served.

```python
from collections import defaultdict

LIMIT = 3  # calls each tenant may make in one window

def handle(tenant, calls):
    calls[tenant] += 1
    if calls[tenant] > LIMIT:
        return 'rejected'
    return 'ok'

calls = defaultdict(int)
for tenant in ['big', 'big', 'big', 'big', 'small', 'big', 'small']:
    print(tenant, handle(tenant, calls))

```

Output:

```text
big ok
big ok
big ok
big rejected
small ok
big rejected
small ok

```

The big tenant is blocked once it passes its allowance, and the small tenant is never blocked, even though its calls arrive in the same stream. Real systems use time windows and shared stores, but the logic is the same.

### Quick Check: Mitigation

**Quick check:** Your app serves 500 customers from one shared database. Suddenly all customers report slow pages. You see that one customer is running a bulk import of millions of rows. What is this problem called, and which two steps would stop it from hurting the other customers?

A. A cache stampede, fixed by making cache entries last longer and adding a second cache
B. A Noisy Neighbor problem, fixed by a per-tenant limit on the import and a separate queue for imports
C. A split brain, fixed by choosing a new leader database and restarting every server
D. A single point of failure, fixed by adding a second database and a load balancer

**Answer:** B. One tenant's heavy work slows everyone sharing the database, which is the Noisy Neighbor pattern. A per-tenant limit caps the import, and a separate queue keeps its work away from other tenants' requests.

**Quick check:** A team runs 2,000 small customers on one shared database. Three very heavy customers often slow everyone down. Which plan fits best?

A. Move all 2,000 small customers to their own dedicated databases
B. Give the three heavy customers dedicated resources, and keep the rest shared with per-tenant limits
C. Remove all limits so every customer can run at full speed
D. Turn off the per-tenant metrics so the team stops seeing the problem

**Answer:** B. Isolating the few heavy tenants protects the many small ones, and shared resources with limits keep costs low. Dedicating resources to all 2,000 customers would cost far more than the problem needs.

## Key Takeaways

- **What it is:** One tenant's heavy use of a shared resource slows or breaks other tenants, even though those tenants did nothing wrong.
- **How to find it:** Use per-tenant metrics for requests, CPU, I/O, and queue depth, and read percentiles rather than averages.
- **How to limit it:** Use per-tenant rate limits and quotas, isolate heavy work with separate pools, queues, or instances, throttle extra work, and add capacity when needed. Weigh each step against its cost.

## Sources

- [Azure Architecture Center: Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor)
- [Azure Architecture Center: Noisy Neighbor antipattern (UK English page)](https://learn.microsoft.com/en-gb/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor)
- [Azure Architecture Center: Multitenancy and Azure SQL Database](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/service/sql-database)
- [Azure Architecture Center: Architectural Approaches for Compute in Multitenant Solutions](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/approaches/compute)
- [Azure Architecture Center: Architectural Approaches for Messaging in Multitenant Solutions](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/approaches/messaging)
- [Azure Architecture Center: Architectural Approaches for a Multitenant Solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/approaches/overview)
- [Inngest blog: Fixing noisy neighbor problems in multi-tenant queueing](https://www.inngest.com/blog/fixing-multi-tenant-queueing-concurrency-problems)
- [Mark Heath: Mitigating the Noisy Neighbour Multitenancy Problem](https://markheath.net/post/noisy-neighbour-multitenancy)
- [Design Gurus: How do you isolate noisy neighbors in multi-tenant systems?](https://www.designgurus.io/answers/detail/how-do-you-isolate-noisy-neighbors-in-multitenant-systems)
- [Zerofilter (Medium): Noisy Neighbor Problem in Multi-Tenant Systems Explained Briefly](https://zerofilter.medium.com/noisy-neighbor-problem-in-multi-tenant-systems-explained-briefly-3788ae5e9d5b)
- [ClickHouse: Percentiles vs averages](https://clickhouse.com/resources/engineering/percentiles-vs-averages)
- [DEV Community: p50, p95, p99 Explained](https://dev.to/kevinrawal/p50-p95-p99-explained-a-simple-guide-to-latency-metrics-p90)
- [QA Skills: Performance Percentiles: p50, p95, p99 Explained](https://qaskills.sh/blog/performance-test-percentiles-p95-p99-guide)
