# Synchronous I/O: Why Waiting Threads Stall a Service

> A thread that sits idle while a slow call finishes cannot serve anyone else. Here is how that stalls a service, how to spot it, and when waiting is still fine.

**In short:** Synchronous I/O means a thread sits idle while a slow call finishes, so it cannot serve anyone else and the service can stall.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/synchronous-i-o

## What You Will Learn

This lesson explains synchronous I/O, a common reason web services slow down under load. You will see how a waiting thread holds back other work, how to spot the problem in a running system, and how to fix it with non-blocking code. You will also learn when waiting is acceptable, so you do not rewrite code that does not need it.

- **I/O:** Input and output. Any time a program reads or writes data outside its own memory, such as a database query, a network call, or a file read.
- **Thread:** A worker inside a program that runs instructions. A web server often gives each incoming request a thread.
- **Blocking:** Stopping and waiting until a call finishes before the thread can do anything else.
- **Synchronous:** Happening in step with the caller. The caller waits for the answer before it moves on.
- **Thread pool:** A set of threads a server keeps ready and reuses for new requests.
- **Latency:** How long one request takes from start to finish.
- **Throughput:** How much work a system finishes in a given amount of time.
- **CPU:** The processor that does the actual computing.
- **Asynchronous (async) I/O:** A call that starts, lets the thread go, and delivers the result later when it is ready.
- **Callback:** A function you hand over so it runs when a result arrives.
- **Event loop:** A loop that watches many pending operations and runs the right follow-up code as each one finishes.
- **Downstream service:** Another service your code calls, such as a payment provider or an address lookup.

## What Does Synchronous I/O Mean?

A synchronous I/O call is a request for data, or a write of data, where the calling thread stops and waits for the reply. The thread stays attached to that one job and does nothing useful during the wait. Common examples include a database query, a call to another web service, sending or reading a message on a queue, and reading or writing a local file. Each of these can be slow for reasons outside your code, such as a busy database or a distant network.

The waiting is the real problem. The thread still belongs to your server, so it cannot pick up another visitor's request while it is parked. The wait uses almost no CPU, which is why the problem often hides in plain sight. Your servers look lightly loaded, yet they cannot take more work.

## How Does Blocking Exhaust the Thread Pool?

Most web servers keep a thread pool, and it can create new threads to serve incoming requests when a spike drains it. When a request arrives, the server takes a free thread from the pool and gives it the request. When the reply is sent, the thread goes back into the pool for the next request. The pool is a limited resource, so every thread that is stuck waiting is one less thread for a new visitor.

A single slow call can also hold up a whole chain. If service A waits on service B, and B waits on a slow database, every thread in the chain stays busy until the database answers. Azure's guidance on this antipattern makes the same point: one blocking call can stall an entire call chain.

Under a spike, the pool drains faster than it refills. New requests then queue up, and when the queue is full or a wait runs too long, users see timeouts or errors. The fix is not usually more CPU. The bottleneck is the number of threads that are waiting.

> **Tip:** Think of threads as checkout lanes in a shop. A cashier who is waiting for a slow card reader still blocks the lane, even though the cashier is not busy.

Why does the pool not just grow? Some runtimes add threads slowly on purpose.

Blocking has a cost beyond the pool. Each thread uses memory for its stack, so a huge number of threads is not a free fix either. Blocking is a bad fit for systems with many concurrent users and many slow calls.

## How Do You Spot Synchronous I/O?

Synchronous I/O has a recognizable set of symptoms. They are easy to miss if you only look at the CPU chart, so check these signs together.

- **High latency with low throughput:** Each request takes longer than it should, and the system finishes fewer requests per minute than its hardware should allow.
- **Low CPU with many waiting threads:** The processor is mostly idle, but the thread count is high and most threads are in a waiting state, not a running one.
- **Latency that tracks the slowest dependency:** Response times climb whenever one downstream service slows down, even when your own code has not changed.
- **Queues, timeouts, and error codes at the edge:** Users see timeouts or server errors. Load balancers and servers report long request queues and HTTP 503 (Service Unavailable) responses.

Users often notice the problem before the dashboards do. The app feels frozen at random times, and the same page loads fine when only one person is using it. That gap is a clue.

### How Do You Find It?

1. **Watch the system under real load:** Check whether worker threads are blocked while requests wait. Look at thread counts, queue lengths, and CPU use side by side.
2. **Profile where threads spend their time:** A profiler measures how long each request spends in each function. If most of the time is spent waiting on a database or network call, you have found synchronous I/O.
3. **Review the code path:** List every call that reaches outside the process, such as a query, a web request, or a file read. Mark the ones that block the request thread.
4. **Load test each suspect:** Run controlled tests that hit one slow operation at a time, and see whether the total throughput drops as that operation slows.

> **Tip:** A single test run often looks fine. The problem shows up only when many requests run at once.

## How Do You Fix It with Async I/O?

The main fix is to replace blocking calls with non-blocking ones. The thread starts the call, then goes back to the pool to serve other requests. When the result arrives, a follow-up step finishes the work, often on whichever thread is free. Many libraries already offer both a blocking and a non-blocking version of the same method, so the change is often small. Azure's guidance is to use the asynchronous versions whenever possible.

### Async/Await, Callbacks, and Event Loops

- **Async and await:** You mark the method as async and write await before the call. The method hands control back while it waits, and the code after the await runs once the result is ready.
- **Callbacks:** You pass in a function that runs when the call completes. The thread is free in the meantime, and a later thread runs your function with the result.
- **Event loops:** One loop watches many pending operations at once. When one finishes, the loop runs its follow-up code. A small number of threads can handle a large number of waiting calls this way.

> **Tip:** Async syntax alone does not make code non-blocking. If an async method calls a blocking function inside it, the thread still sits idle. Check the calls, not just the keywords.

### Worked Example: A Checkout Page

Imagine an online store whose checkout handler calls three services: a payment provider, an inventory service, and an email service. The handler makes the calls one after another on a single request thread. Here is how the team finds and fixes the problem.

1. **Map the current path:** The handler calls the payment provider, waits for the reply, then calls inventory, waits, then calls email, and waits again. The request thread is held for all three waits.
2. **See what a sale does:** During a big sale, many shoppers check out together. Each slow reply keeps one thread busy. The pool empties, and new shoppers queue or get errors, even though the servers' CPUs are mostly idle.
3. **Switch the calls to async:** Use the async version of each client library. Await each call, so the thread is released while the provider or inventory service works.
4. **Send the confirmation email in the background:** The shopper does not need the email to finish before seeing the order page. Hand the email to a background worker, which other lessons cover in detail, and return the page without waiting for it.
5. **Measure again:** Run the same load test. The threads now return to the pool sooner, so the same servers can accept far more shoppers before the queue builds up.

The short program below uses only the standard library. The fake service call uses a zero-length sleep as a stand-in for a network wait. The first function waits for each call in turn, like the old checkout. The second starts all three calls and waits for all of them together, like the improved version.

```python
import asyncio

async def call_service(name):
    await asyncio.sleep(0)
    return name + " done"

async def one_by_one():
    done = []
    for name in ["payment", "inventory", "email"]:
        done.append(await call_service(name))
    return done

async def at_once():
    names = ["payment", "inventory", "email"]
    return await asyncio.gather(*(call_service(n) for n in names))

async def main():
    print(await one_by_one())
    print(await at_once())

asyncio.run(main())

```

Output:

```text
['payment done', 'inventory done', 'email done']
['payment done', 'inventory done', 'email done']

```

Both functions print the same results, because the program only shows what each call returns. The difference is in timing. With real network calls, the second style lets all three waits overlap, and the request thread is free while they run. This is the same change the store would make in its real checkout code.

## When Synchronous I/O Is Fine

Not every blocking call is a bug. Going async adds complexity, so use it where waiting causes harm. Here are the common cases where a simple synchronous call is a sound choice.

- **Short, cheap operations:** Azure notes that very short I/O, such as reading a small file from a fast local drive, can be quicker as a plain synchronous call, because handing it to another thread has its own cost.
- **Low traffic:** When few requests arrive at once, there are plenty of free threads.
- **The answer is needed right now:** Sometimes the next line of code cannot run without the result, and the call is rare. In that case, waiting is simple and correct, and the extra code of async would add risk without benefit.
- **One-off jobs and scripts:** A script that runs once a day, alone, on its own machine has no pool of shared threads to protect. Blocking there is usually harmless.

### The Trade-Offs of Going Async

Plan for this before you change anything.

Error handling also changes. When a call fails later, the error arrives at a different point from where the call started. Every await or callback needs a clear rule for timeouts, retries, and partial failures. Forgetting one leaves a request hanging or silently lost.

Some async wrappers are not free. Wrapping a blocking library call in a task can hide the wait while using an extra thread and extra overhead. Azure's guidance urges care with this approach.

Freeing threads can move the bottleneck. Once more requests get through, more calls reach shared resources such as a database, which may then slow down or start throttling. Plan for that next limit, and be ready to scale out or split the data.

> **Tip:** Async is mainly about handling many waiting requests at once, which is scalability. It is not a general speed boost for a single request.

## Check Your Understanding

Try these two short scenarios. Each one needs the ideas from this lesson, not just the definitions.

**Quick check:** A ticket-booking API uses about 8% CPU, but its responses slow down whenever the seat-map service is slow. The server's thread pool is fully used. What is most likely happening, and what should you change first?

A. The CPU is the limit, so the team should buy faster processors first.
B. Threads are blocked while they wait on the seat-map service, so the team should make that call async so threads are released while waiting.
C. The server has a memory leak, so the team should restart it on a schedule.
D. The failed seat-map calls should be retried immediately to speed up responses.

**Answer:** B. Low CPU with a full thread pool points to threads that are waiting, not working. Making the slow call async frees those threads while the seat-map service responds. The other options treat symptoms that the data does not show.

**Quick check:** An internal admin tool reads one short settings file from a local solid-state drive each time an engineer opens a page. At most ten engineers use the tool, and they open it now and then. Should the team rewrite that read as async?

A. Yes, right away, because every blocking call is a bug.
B. No. The read is short, the traffic is tiny, and a simple synchronous read is fine here.
C. Yes, and the team should also add more threads to the pool.
D. No, but the read should move into a background job that runs every minute.

**Answer:** B. Short reads with very low traffic cause no pool pressure, so the extra complexity of async would add risk without a benefit. The other options add work or code that the situation does not need.

## Key Takeaways

Synchronous I/O holds a thread while a slow call finishes. Under load, those held threads fill the pool, and the server stalls even when its CPU is nearly idle. Spot it by looking at high latency, low throughput, low CPU, and many waiting threads, then check which calls block. Fix it by making slow calls async or by handing slow work off, and keep blocking where the work is short, the traffic is low, or the answer is needed right now. Each change brings more complexity, so test it under load before and after.

## Sources

- [Synchronous I/O antipattern, Azure Architecture Center (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/synchronous-io/)
- [Performance testing and antipatterns for cloud applications, Azure Architecture Center](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/)
- [Why async exists: the cost of blocking a thread, Meziantou's blog](https://www.meziantou.net/why-async-exists-the-cost-of-blocking-a-thread.htm)
- [There is no thread: OS async I/O and the .NET thread pool, Meziantou's blog](https://www.meziantou.net/there-is-no-thread-os-async-i-o-and-the-dotnet-thread-pool.htm)
- [(Quite) A Few Words About Async, Yoric's blog](https://yoric.github.io/post/quite-a-few-words-about-async/)
- [On Asynchronous, Non-Blocking I/O, Entzik (Medium)](https://entzik.medium.com/on-asynchronous-non-blocking-i-o-4a2ac0af5c50)
- [Thread Pool Exhaustion: How It Works, ADHDecode](https://adhdecode.com/performance-engineering/performance-anti-patterns-and-optimization/thread-pool-exhaustion/)
- [Why Your .NET App Hangs: A Beginner's Guide to ThreadPool Starvation, C# Corner](https://www.c-sharpcorner.com/article/why-your-net-app-hangs-a-beginners-guide-to-threadpool-starvation)
- [End-to-End Application Cloning for Distributed Cloud Microservices with Ditto (arXiv paper)](https://arxiv.org/pdf/2212.13867)
