Task Queues: Doing Slow Work in the Background, Reliably
Learn how a task queue hands slow or non-urgent work to background workers so the user gets a quick reply, and how to keep that work reliable.
In short
A task queue hands slow or non-urgent work to background workers, so the user gets a quick reply and the work still finishes reliably.
10 min readFree
What You Will Learn
Some work takes too long to make a user wait for it. Sending an email, resizing a photo, or building a monthly report can take seconds or minutes. A task queue lets the app accept the request, place the slow job in a line, and finish that job later in a separate process. In this lesson you will learn the parts of a task queue, the path a task takes, which jobs belong in one, and how to keep those jobs reliable when things fail.
Key words
- Task
- One unit of work plus the small amount of data it needs, such as 'send the receipt for order 1042'.
- Task queue
- A line that holds tasks until a worker is ready to run them. It is a waiting area for work.
- Producer
- The part of your app that creates a task and adds it to the queue. Usually this is the web code that handles a user's request.
- Worker
- A separate program that takes tasks from the queue and runs them. Workers run outside the web request.
- Broker
- The program that stores the queue, such as Redis or RabbitMQ. Many task queues use one behind the scenes.
- Enqueue
- To add a task to the queue. Think of it as joining the end of a line.
- Retry
- Running a failed task again, usually after a short wait.
- Dead-letter queue
- A separate place where tasks go after they have failed too many times, so a person can look at them.
- Idempotent
- Safe to run twice. Running the task again does not cause a second effect, such as a second charge to a card.
How Does the Task Queue Flow Work?
The flow has four parts that happen in order. The user's web request covers only the first part. The rest happens after the user already has a reply.
- 1
Producer adds the task
When a user clicks a button, the web code writes a task with a name and a small amount of data, such as an order ID. It then sends the task to the queue. Passing an ID is better than passing a large object, because the worker can then read the latest data.
- 2
Queue stores the task
The queue keeps the task safe until a worker takes it. The task stays in the line even when no worker is free at that moment.
- 3
Worker runs the task
A worker process checks the queue, takes the next task, and runs the code for it. This happens in a different program from the web server, so the slow work never blocks the page.
- 4
Worker reports the result
When the task succeeds, the worker tells the queue it is done, and the queue removes the task. This report is called an acknowledgement. If the worker crashes before it reports, some queues give the task to another worker, while others, such as Sidekiq, can lose it.
Many guides describe the same shape. The web handler responds at once, the job waits in the queue, and a separate process does the work. Because of this split, the web servers and the workers can be changed and scaled on their own.
Why Does the User Not Wait?
The user's request ends as soon as the task is saved. The reply says that the request was received, not that all the work is finished. The page feels fast, and the slow work still gets done a moment later.
What Belongs in a Task Queue?
Not every job belongs in a queue. Ask two questions. Does the user need the answer on the next screen? And can the work wait, or run again safely if it fails?
Slow jobs
Work that takes seconds or minutes, such as resizing images, building a PDF, or generating a report. Nobody should stare at a spinner while this runs.
Retryable jobs
Work that can fail for a short time, such as sending an email when the mail server is briefly down. The queue can try again later instead of showing an error.
Scheduled jobs
Work that should run later, such as a reminder in 30 minutes or a nightly report. Many task tools can delay a task or repeat it on a timetable.
Payments are a useful edge case. A charge can be queued, but the page should show a pending status until the result is known. The charge job must be written so that it never charges the card twice.
Work the User Needs Right Away
Checking a password, showing a product price, or confirming that an item is in stock are needed for the very next screen. Keep these out of the queue. If you queue them, the user may see an answer that is not ready yet.
How Do You Run Tasks Reliably?
A task queue is only useful if tasks finish correctly. A worker can crash in the middle of a job, a network call can time out, and a task can sometimes run twice when the queue is not sure it finished. Plan for all three cases from the start.
How Do Retries with Limits Work?
A retry runs a failed task again. Set a limit, such as three or five tries, so a broken task does not loop forever. Waiting a little longer before each try, called backoff, gives a struggling service time to recover.
Tip
Retry only errors that might go away, such as a timeout. A bad email address will fail on every try, so retrying it only wastes worker time.
Safe to Run Twice
Because a task can run more than once, design it to be idempotent. Running it twice should give the same result as running it once. For example, 'mark the receipt as sent' is safer than 'add one to the sent count', because the second version counts twice if the task runs twice.
Dead-Letter Queues
When a task has used all its tries, move it to a dead-letter queue. Nothing is lost. A person can read the task, fix the cause, and send it back. Without this step, broken tasks either vanish or keep blocking the line.
Scaling by Adding Workers
If tasks start piling up, add more workers. Workers are separate programs, so you can run several of them on several machines at once. The web servers do not need to change. A line that keeps getting longer is the sign that you need more workers.
Check the Retry Plan
Quick check
A photo app sends a resize task. The image file is corrupt, so every try fails with the same error. The team set five tries with backoff. What should happen to the task after the fifth failure?
Worked Example: A Photo Upload
Here is a simplified picture of how a photo app could use a task queue. This is not Instagram's real internal design. It shows the idea step by step.
- 1
User sends the photo
The user taps Share. The upload request reaches the upload service, which saves the original photo first.
- 2
Producer enqueues the job
The upload service adds a task such as 'make versions for photo 8731'. The task carries the photo ID, not the whole image.
- 3
App confirms right away
The service returns 'Upload received' to the phone. The user can keep scrolling or write a caption while the rest happens.
- 4
Workers build the versions
Image workers take the task and create a resized feed image, a filtered copy, and a small thumbnail. Each file is saved when it is ready.
- 5
Post goes live
When the versions are ready, the post is updated. If a worker fails, the task is retried. If it keeps failing, it moves to a dead-letter queue for a person to check.
Tip
Look at what the user never waits for: the resizing, filtering, and thumbnail work. Only saving the original and adding the task happen before the reply.
Tools You Will Meet
You do not need to build a queue from scratch. Several popular tools provide one for each language. They differ in details, but they share the same basic shape.
Celery for Python
A full-featured task queue often used with Django or Flask. Your app sends tasks to a broker such as Redis or RabbitMQ, and worker processes run them.
BullMQ for Node.js
A queue for Node.js apps that stores jobs in Redis. It offers retries, delays, rate limits, and control over how many jobs run at once.
Sidekiq for Ruby
A background job tool for Ruby and Rails apps, also backed by Redis. It is a common choice for Rails teams.
Tip
Defaults differ between tools. Some retry failed jobs automatically, and others do nothing on failure until you set a retry count. Always check the retry, timeout, and crash settings before you trust a tool.
Try a Tiny Worker
This script copies the main ideas in plain Python. It has a queue, a worker loop, a limit of three tries, and a list for dead letters. It uses only the standard library, so it always prints the same lines.
from collections import deque
queue = deque()
dead_letters = []
MAX_TRIES = 3
def send_email(order_id):
if order_id == 2:
raise ConnectionError('email service down')
return 'email sent for order %d' % order_id
queue.append({'order': 1, 'tries': 0})
queue.append({'order': 2, 'tries': 0})
print('placed orders: 2')
while queue:
task = queue.popleft()
task['tries'] += 1
try:
print(send_email(task['order']))
except ConnectionError:
if task['tries'] < MAX_TRIES:
queue.append(task)
print('order %d failed on try %d, requeued' % (task['order'], task['tries']))
else:
dead_letters.append(task)
print('order %d moved to dead letters' % task['order'])
print('dead letters: %d' % len(dead_letters))
Output
placed orders: 2 email sent for order 1 order 2 failed on try 1, requeued order 2 failed on try 2, requeued order 2 moved to dead letters dead letters: 1
Read the output in order. Order 1 succeeds on its first try. Order 2 fails three times, then lands in the dead-letter list. The web reply was already sent before any of this ran.
Checkout Page Scenario
Now apply the ideas to a real problem. Read the scenario and decide what the team should change.
Quick check
A shop's checkout page takes 8 seconds. Before it shows 'Order placed', it sends a confirmation email and builds a PDF invoice. Later, the email service is down when the worker runs the email task. What is the best plan?
Key Takeaways
Reply first
Save the request, return a quick answer, and let a worker finish the slow part later.
Queue the slow part
Use a queue for slow, retryable, or scheduled jobs. Keep anything the next screen needs out of it.
Expect failure
Workers can crash and tasks can run twice. Use retry limits, safe-to-repeat tasks, dead-letter queues, and more workers when the line grows.
A task queue is a simple idea with careful details. Once you can follow a task from the producer to the worker, and say what happens when it fails, you understand the core of the pattern.
Sources
- Fly.io guide: background job queues with Celery, BullMQ and Fly Machines
- Fly.io Learn: background workers with web, worker, Redis and cron
- DevOpsness: Job Queue Patterns for Sidekiq, Celery and BullMQ in Production
- Back4App glossary: Background Jobs and Task Schedulers
- Layrs: Background Jobs in System Design, Queues and Workers
- GeeksforGeeks: Task Queues in System Design
- TechInterview.org: System Design Interview, Job Scheduler and Task Queue
- Vicky Bytes: Background Jobs Explained, How Backend Apps Process Tasks
- DEV Community: What Really Happens When You Upload a Photo to Instagram (simplified pipeline)
Last updated on 17 November 2026
Related lessons
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.