esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Competing Consumers: Many Workers, One Queue, Each Message Handled Once

How several identical workers share one message queue, why they must be safe to repeat work, and when this pattern is the wrong choice.

In short

Competing consumers means several identical workers share one message queue so each message is handled once, and the workers must be safe to repeat work.

13 min readFree

What You Will Learn

Imagine an online shop that gets a sudden rush of orders during a sale. If each order waits while its confirmation email is sent, checkout slows down for everyone. Competing Consumers is a messaging pattern that avoids this. Several identical workers read from one shared queue, and each message goes to only one of them. When work piles up you add workers, and when it drops you remove them. The part that sends the messages does not change. This lesson explains how the pattern works, what workers must do to stay safe, where it breaks down, and when to choose something else.

Key words

Message
A small piece of data that describes one job, such as 'send this email to this customer'.
Queue
A waiting line that holds messages until a worker takes them. Most queues hand out messages in the order they arrived.
Producer
The part of a system that creates messages and places them on a queue.
Consumer
A program that takes messages off a queue and does the work they describe.
Worker
One running copy of a consumer. A pool of workers is several copies running at once.
Broker
The software that stores the queue and hands each message out to a consumer.
Throughput
How much work gets finished in a set amount of time.
Horizontal scaling
Handling more load by adding more copies or machines, instead of making one machine stronger.
Stateless
A worker keeps no memory between jobs that another worker would need to finish them.
Idempotent
Safe to repeat. Running the same job twice leaves the same result as running it once.
Acknowledgement (ack)
A reply from a worker that says 'I finished this message', so the broker can delete it.
Visibility timeout
A set time during which a message a worker has picked up is hidden from other workers. If no ack arrives in time, the message becomes visible again.
Redelivery
The broker handing the same message to a worker a second time.
Dead-letter queue
A separate queue where messages go after they keep failing, so they can be checked by a person.
Poison message
A message that makes a worker fail every time it is tried.

How Does the Pattern Work?

The pattern has three roles. Producers add messages to one queue. Consumers, which run as several identical workers, pull messages from that queue. The broker makes sure each message is given to just one worker. Because the workers are interchangeable, any free worker can take the next job.

  • The producer side

    The producer, such as a web checkout, writes a message that describes the job and sends it to the queue. It then returns to the user without waiting for the work to finish.

  • The shared queue

    The queue holds every waiting message. It acts as a buffer, so a short burst of messages sits safely in line instead of overwhelming the workers.

  • The consumer pool

    Identical workers pull messages from the same queue. Each worker finishes one message, sends an acknowledgement, and then asks for the next one.

The pattern is about spreading work, not copying it. The pattern treats the workers as replicas of one another. If one worker crashes, the others keep taking messages, so one failure does not stop the whole queue.

Why Does One Message Go to One Worker?

This pattern is often confused with a design where every listener receives every message. In Competing Consumers, the broker gives each message to exactly one worker. Several workers do compete for the same message, but only the winner processes it. Because of this, adding a worker adds capacity without doubling any work. The broker hides a message while a worker holds it, and it removes the message only after that worker confirms success.

How Do You Scale by Adding Workers?

The main reason to use this pattern is horizontal scaling. You do not rewrite the code to go faster. You start more copies of the same worker program, and they all draw from the same queue. The queue's length tells you whether the pool is keeping up.

  1. 1

    Watch the backlog

    Track how many messages are waiting in the queue and how old the oldest one is. A growing backlog means the workers are behind.

  2. 2

    Start more workers

    Launch more copies of the same worker. Each new copy connects to the same queue and begins pulling messages straight away.

  3. 3

    Let the queue share the work

    A faster or idle worker asks for messages more often, so it naturally takes a larger share. No central dispatcher has to assign jobs.

  4. 4

    Scale back down

    When the backlog clears, stop the extra workers. The producer never needs to know that the number of workers changed.

Scaling workers does not remove limits elsewhere. If every worker writes to the same database or calls the same outside service, that shared resource can become the new limit. Add workers with the database and other services in mind.

Worked Example: Order Confirmation Emails

Here is a realistic design for an online store. The store's checkout places each 'send order confirmation email' message on a queue. A pool of email workers competes to pick them up. This example is illustrative. It shows how the idea is applied, and it is not data from a real company.

  1. 1

    Checkout saves the order

    When a customer pays, checkout saves the order and writes one message with the order number. It then shows the confirmation page at once, without waiting for any email to be sent.

  2. 2

    Two workers share the queue

    On a normal day, two email workers pull messages. Each message goes to one of them, so no order is emailed by both at the same time.

  3. 3

    The flash sale arrives

    Orders spike. The queue grows, and the team raises the pool from 2 workers to 20. The producer, checkout, is unchanged.

  4. 4

    A worker crashes mid-task

    One worker sends an email, then fails before its ack reaches the broker. The visibility timeout ends, the message becomes visible again, and another worker picks it up.

  5. 5

    The repeat is harmless

    Before sending, each worker checks a record that stores which order numbers already have a sent email. If the order is recorded, the worker skips it and acknowledges the message.

  6. 6

    The result

    The team's goal is that every customer gets exactly one email. Scaling adds speed, and the idempotent check makes the repeat safe. The team still watches the dead-letter queue for orders that never send.

Tip

The check in step five must be saved in a durable store that every worker can read. If each worker kept its own list in memory, a restarted worker would forget what was already sent.

A Small Simulation in Python

The code below imitates the worked example with a plain list used as a queue. Two workers take turns. Then one message is delivered a second time, as redelivery would do, and the idempotent check stops it from being handled again. The program uses only the standard library and always prints the same result.

competing_consumers.py
from collections import deque

queue = deque()
for order_id in range(1, 7):
    queue.append(f'order-{order_id}')

sent = set()
handled = {'worker-1': [], 'worker-2': []}

def work(name):
    message = queue.popleft()
    if message not in sent:
        sent.add(message)
        handled[name].append(message)

turn = 0
while queue:
    name = 'worker-1' if turn % 2 == 0 else 'worker-2'
    work(name)
    turn += 1

queue.append('order-3')
work('worker-2')

print('worker-1:', handled['worker-1'])
print('worker-2:', handled['worker-2'])
print('emails sent:', len(sent))

Output

worker-1: ['order-1', 'order-3', 'order-5']
worker-2: ['order-2', 'order-4', 'order-6']
emails sent: 6

Read the output closely. Six messages were shared between two workers, three each. The redelivered order-3 was picked up by worker-2, but the sent set already held it, so no second email was recorded. A real system would store this record outside the program, but the logic is the same.

What Must Workers Do to Stay Safe?

The pattern works only if workers can handle a message more than once. Queues usually guarantee that each message arrives at least once, which means a duplicate is possible. A worker can crash after doing its job and before the ack. The broker then assumes the job failed and gives it to someone else. Four rules keep this safe.

Keep Workers Stateless

A stateless worker keeps nothing that another worker would need. The message holds all the details of the job, and any shared facts live in a database or other shared store. Then any worker can pick up any message, and a crashed worker can be replaced without loss. A worker that stores progress only in its own memory cannot be safely replaced.

Make Handling Idempotent

An idempotent handler gives the same result however many times it sees the same message. A common method is to keep a record of each message ID that was already done. Before doing the work, the worker checks that record. If the ID is there, it skips the work and acknowledges the message. Microsoft's guidance adds that these records must live longer than the window in which redelivery can happen. If they are deleted too early, a late duplicate gets through.

Some duplicates arrive at the same time, not one after another. When a lock expires, a second worker can take the message while the first is still running. A unique key in the shared store helps here, because only one worker can create the record.

Acknowledgements and Visibility Timeouts

A worker should send its ack only after the work is truly finished. If it acks early and then crashes, the message is lost. The visibility timeout controls how long a picked-up message stays hidden. If the timeout is shorter than the job, the message reappears while the first worker still holds it. Then two workers do the same job, and the message's failure count rises toward the dead-letter limit. Set the timeout above the normal processing time. For long jobs, extend it while the work is running.

Retries and the Dead-Letter Queue

Some failures are temporary, such as a brief database outage. Retrying after a short, growing wait often succeeds. Other failures are permanent, such as a malformed order. Retrying those wastes time forever. After a set number of failed deliveries, the broker moves the message to a dead-letter queue. The main queue keeps moving, and a person can inspect the message, fix the cause, and send it back.

A dead-letter queue only helps if someone watches it. Put an alert on it, so that a new message there is noticed the same day. A dead-letter queue that nobody reads is a slow way to lose messages.

Tip

Write down three numbers for every queue: the normal job time, the visibility timeout, and the maximum delivery count. If the timeout is shorter than the job, duplicates will appear. If the count is too high, a poison message will run for a long time before it is moved aside.

Trade-offs and Limits

Competing Consumers buys speed and resilience, and it pays for them in other ways. Knowing these costs in advance helps you decide whether the pattern fits. Three limits matter most.

  • Order is not kept

    Because many workers run in parallel, messages can finish in a different order from the one in which they were sent. Two messages about the same customer may be handled out of sequence. If order matters, the design must add a separate mechanism for it.

  • Poison messages

    A poison message fails every time it is tried. Without a retry limit and a dead-letter queue, it keeps returning to workers and uses up their time. The fix is a maximum delivery count, followed by a move to a dead-letter queue.

  • The queue can become a bottleneck

    A single queue can be overwhelmed by very high message volume. In large systems, teams split traffic across several queues, or give each producer its own queue, so no single queue carries all the load.

The pattern also adds operational work. Teams must watch the queue length, the age of the oldest message, the redelivery rate, and the dead-letter queue. Tracing one job from the producer through the queue to the worker that handled it also needs good logs.

When Does the Pattern Fit?

The pattern is a strong choice when the work can be split into separate tasks and the load changes a lot over time. It is a poor choice when tasks depend on each other or must run in a fixed sequence.

  • Good fit: independent tasks

    Sending emails, resizing images, and creating thumbnails are tasks that do not depend on each other. Any worker can do any one of them, so the pool can grow freely.

  • Good fit: bursty load

    When traffic comes in spikes, the pool can grow during a peak and shrink afterwards. Work waits safely in the queue while workers catch up.

  • Poor fit: strict ordering

    If messages for one account must be applied in the exact order they were sent, workers running in parallel will break that order. Use a design that keeps related messages together, which is covered in another lesson.

  • Poor fit: the user must wait

    If the user's screen cannot continue until the work is done, a queue will only add a delay. Keep that step synchronous.

Check Your Understanding

Try these two short scenarios. Each one needs the ideas above to answer it.

Quick check

A photo app puts a 'resize image' message on a queue for each upload. During evening peaks, one worker cannot keep up. What should the team change, and what must be true of each worker so that no photo is resized wrongly or lost if a worker crashes mid-task?

Quick check

A payment worker pulls a 'charge card' message from a shared queue. It charges the card, then crashes before sending its ack. Another worker receives the same message. What protects the customer from being charged twice?

Key Takeaways

Competing Consumers lets many identical workers share one queue, with each message given to one worker. You scale by adding workers, and the producer stays the same. The pattern needs stateless, idempotent workers, acknowledgements after real success, sensible visibility timeouts, and a dead-letter queue watched by someone. It gives up strict ordering and can strain a single queue, so use it for independent tasks with changing load, and choose another design when order or waiting users matter.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.