esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Queue-Based Load Leveling: Smooth Sudden Traffic Bursts With a Queue

A queue sits between the sender and the worker, so a sudden rush of requests is stored and handled at a steady pace.

In short

Queue-based load leveling puts a queue between the sender and the worker, so a sudden burst of requests is stored and handled at a steady pace.

12 min readFree

What You Will Learn

Some systems get a rush of work all at once. A sale opens, a deadline arrives, or a post goes viral and sends many people to one page. If the part of the system that does the work can only handle a steady amount, the rush can slow it down or crash it. In this lesson you will learn Queue-Based Load Leveling. It is a design pattern, which means a known fix for a problem that shows up again and again. The fix puts a queue between the sender and the worker, so the work is stored first and then handled at a steady pace.

Key words

Pattern
A tried solution to a common design problem, reused across many systems.
Load
The amount of work a system receives over a period of time.
Spike
A sudden jump in load that lasts for a short time.
Producer
The part of a system that creates tasks and sends them away.
Consumer
The part that takes tasks from a queue and does the work.
Queue
A line that holds messages in order until a consumer takes them.
Buffer
A holding area that absorbs a difference in speed between two parts.
Decouple
To separate two parts so that one does not depend on the other's timing.
Asynchronous
Work where the sender does not wait for the result before moving on.
Backlog
Messages that are waiting in a queue and have not been handled yet.

What Is the Problem with Bursts Hitting Services Hard?

A service is built to handle a certain amount of work in each period of time. When work arrives faster than that, requests pile up inside the service. Memory fills up, database connections run out, and replies slow down. Some requests time out, and users press the button again, which adds even more load. The service can fail at the exact moment people need it most.

The danger is not only the average load. A service can sit mostly idle for days and still fail during a ten-minute window. Designers who size a service only for the average are often surprised by bursts.

Why Do Waiting Callers Suffer?

The sender feels the burst too. If the sender waits for a reply, its own workers stay tied up while they wait. That sender can then slow down or fail, and the failure can travel from one part of the system to the next.

How Does a Queue Level the Load?

The pattern adds a message queue between the producer and the consumer. A message is a small package of data that describes one piece of work. The producer writes the message into the queue and moves on. The queue stores it. The consumer reads messages from the queue at the rate it can manage. Many producers can write to the same queue, and their messages wait in line.

  1. 1

    Producer creates a task

    A user action or a scheduled job creates a task. The producer packs the data that the work needs into a message.

  2. 2

    Producer posts the message

    The message goes into the queue. The queue sends back a quick acknowledgement, which means a short confirmation that it has the message. The producer does not wait for the work itself.

  3. 3

    Queue holds the message

    The queue keeps the message safe until someone takes it. During a spike, the line gets longer.

  4. 4

    Consumer pulls at its own pace

    The consumer takes the next message only when it is ready. It receives messages at a manageable rate.

  5. 5

    Consumer finishes and saves the result

    After the work is done, the consumer saves the result somewhere the user can check later.

Think of a busy shop with a waiting room. The door lets people in as fast as they arrive, and the cashier serves them one at a time. The waiting room is the buffer. It takes the rush so the cashier is never swamped.

Decoupling the Two Timelines

Decoupling means the producer and the consumer no longer depend on each other's timing. If the consumer goes down for an hour, the producer keeps posting messages and they wait in the queue. When the consumer returns, it works through the backlog. The producer only needs the queue's address, not the consumer's. Each side can also be changed, scaled, or restarted on its own.

Tip

A queue is only as safe as the storage behind it. Pick a queue service that writes messages to disk or to copies kept on several machines. A queue that keeps messages only in memory can lose them if it crashes.

Worked Example: A Concert Ticket Sale

Imagine a ticketing website. Tickets for a popular concert go on sale at a set time. Thousands of people click Buy in the same minute. Without a queue, each click would call the payment and inventory service directly. That service checks stock, charges the card, and updates the seat count. It would be overwhelmed. Here is the same site with the pattern in place.

  1. 1

    Customer clicks Buy

    The web app checks the basic form data, such as the seat type and whether the session is valid.

  2. 2

    Web app writes an order message

    The web app places the order into the queue instead of calling the payment service.

  3. 3

    Customer sees Order received

    The web app replies with a clear note such as Order received, plus a status page the customer can check later.

  4. 4

    Queue absorbs the rush

    Thousands of order messages line up. The queue grows during the first minutes of the sale, and it can keep orders safe only if it persists messages.

  5. 5

    Payment and inventory take orders steadily

    The payment and inventory service reads one order at a time, at a rate it can sustain. It checks stock and charges the card.

  6. 6

    Customer gets a confirmation

    When an order is complete, the service sends a confirmation email, and the status page shows the result.

Tip

Be clear on the screen. Say that the order is being processed, and do not promise a seat until the payment step has finished.

How Do You Size and Tune the Consumer?

A key choice is how large to make the consumer. Because the queue holds the extra work, the consumer can be sized for average load instead of peak load. That saves money, since you do not pay for machines that sit idle most of the week. The queue makes this safe, because it absorbs the gap between the arrival rate and the processing rate.

The rule behind this is simple. If messages arrive faster than the consumer finishes them over a long period, the backlog keeps growing. The formula below shows the idea in one line.

Bn=max⁡(0, Bn−1+An−Sn)B_n = \max(0,\ B_{n-1} + A_n - S_n)

In words: the backlog after period n equals the backlog before it, plus the messages that arrived, minus the messages finished, and it never drops below zero. Here B is the backlog, A is the number of arrivals, and S is the number finished.

Three Signals That Show It Keeps Up

  • Queue length

    Also called queue depth, this is the number of waiting messages. If the length rises during a spike and falls afterward, the consumer keeps up. If it rises all day, it does not.

  • Processing delay

    This is the time between a message entering the queue and its work being done. Track it by hour, because a delay that grows at peak times is an early warning.

  • Autoscaling on queue depth

    Autoscaling adds or removes copies of the consumer automatically. Scaling up when the queue grows helps the backlog drain. Set safe upper and lower limits, so extra copies do not overload a database behind them.

The short program below imitates a burst. Five orders arrive in the first minute, and the consumer finishes two orders each minute.

Python
from collections import deque

queue = deque()
arrivals = [5, 0, 0, 0]
rate = 2
done = 0
for minute, arriving in enumerate(arrivals):
    for _ in range(arriving):
        queue.append(minute)
    for _ in range(min(rate, len(queue))):
        queue.popleft()
        done += 1
    print(f'minute {minute}: waiting={len(queue)} done={done}')

Output

minute 0: waiting=3 done=2
minute 1: waiting=1 done=4
minute 2: waiting=0 done=5
minute 3: waiting=0 done=5

Read the output. The backlog of three at minute 0 is cleared by minute 2, and no new work arrives after that. If the consumer finished only one order a minute, the same backlog would take longer to clear.

Quick check

A photo-printing app's order service crashes every Monday morning, when thousands of orders arrive within minutes. The service sits idle the rest of the week. Which change fits best, and what will customers notice?

What Are the Trade-offs and When Should You Use It?

Every pattern has a price. With a queue, the main price is time. Once the order goes into the queue, the work becomes asynchronous, which means the sender does not wait for it to finish. The user sees a delay before the final result, so the screen must show progress.

  • Delay is the cost

    Messages wait in line, so the time from request to result grows during a spike. Users need a status page, an email, or a notice that work is in progress.

  • Good fit

    Work that can wait is a good fit. Examples are sending emails, building reports, resizing photos, and processing orders after the customer already has a receipt.

  • Poor fit

    Work that needs an answer now is a poor fit. Examples are a login check, a price shown on a product page, or a balance shown before a payment. Queuing these makes the screen wait for an answer that is not there yet.

A queue also carries messages in one direction. The sender does not get the result back on its own. A second path, such as a status endpoint or a notification, must bring the outcome back to the user.

Quick check

A team wants to move three features behind a queue: a login check, sending a receipt email, and generating a yearly sales report. Which should go behind the queue?

Handling Messages That Fail

Some messages cannot be completed. The data may be badly formed, a record may be missing, or a bug may make the handler crash every time. Two safeguards help. First, a consumer may receive the same message twice. Many queues promise at-least-once delivery, which means a message arrives one time or more. So the work must be safe to repeat. A task that is safe to repeat is called idempotent, because running it twice gives the same result as running it once. Second, a message that keeps failing must not block the whole line.

  • Retries

    The consumer tries the message again a few times, in case the problem was short, such as a brief network drop.

  • Dead-letter queue

    A dead-letter queue is a separate line for messages that failed too many times. Messages move there so the main line keeps flowing. A person can inspect them, fix the cause, and send them back.

  • Watching the dead-letter queue

    A growing dead-letter queue is a warning sign. Monitor it as closely as the main queue.

Tip

Give each order a unique ID. The payment step can then recognize a repeated message and skip the second charge.

Quick check

A consumer receives a message about an order that was deleted yesterday. Each attempt fails, and the message returns to the front of the queue, blocking the messages behind it. What is the best next step?

Key Takeaways

Queue-Based Load Leveling places a buffer between a producer and a consumer. Bursts are stored in the queue and handled at a steady rate, so the consumer can be sized for average load. The cost is delay, which means users need status feedback. The pattern suits work that can wait, and it needs a plan for failed messages, such as retries and a dead-letter queue.

Tip

When you design a queue, ask three questions. Can this work wait? What happens to a message that keeps failing? How will I know the backlog is growing before users do?

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.