esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Schedule Driven: Running Background Work at Fixed Times

Learn how jobs that start on a clock work, when to pick them, and how to avoid the common mistakes.

In short

A schedule-driven job is background work that starts because the clock says so, run by a scheduler that watches the time and a worker that does the job.

10 min readFree

What you will learn

Some work in a software product should not wait for someone to press a button. Reports must be built, old data must be cleared, and backups must be taken, all on a steady rhythm. A schedule-driven job is a background job that starts because the clock says so, for example every night at 2 a.m. or every 5 minutes. By the end of this lesson you will know how these jobs run, when to choose them over jobs that react to events, and which mistakes to guard against.

Key words

Schedule-driven job
Work that starts because a timer or rule says it is time, not because a user clicked something.
Background job
Work that runs behind the scenes, so the user does not sit and wait for it to finish.
Scheduler
The part that watches the clock and starts a job at the times you set. Cron is a well-known example.
Worker
The program that does the real work once the scheduler starts it.
Batch job
A job that handles many records in one run, such as all of yesterday's orders.
Idempotent
Safe to run more than once. Running it again gives the same result and creates no duplicates.
Retry
Trying a failed step again, often after a short wait.
Alert
A message that tells a person something needs attention.
Off-peak hours
Quiet times, such as the middle of the night, when few people use the system.

How does a schedule-driven job work?

A schedule-driven job has three parts: a rule that says when it runs, a scheduler that watches the clock, and a worker that does the work. The steps below show how these parts hand off to each other.

  1. 1

    Write the rule

    Someone sets a time pattern, such as 2 a.m. every day. Cron, a common scheduler on Unix-like systems, writes these rules in a short text form. Cron schedules use five fields: minute, hour, day of the month, month, and day of the week.

  2. 2

    The scheduler waits

    The scheduler does nothing but watch the clock until the next time in the rule arrives. A scheduled job is known as a cron job when cron is the scheduler.

  3. 3

    The scheduler starts a worker

    When the time comes, the scheduler starts the job. The job can run on the same machine or on a separate worker.

  4. 4

    The worker does the job

    The worker reads the data it needs, does the work, and saves the results to a database or file. The job does not run inside the page a user is looking at.

  5. 5

    Results are stored for later

    The user sees the outcome later, for example as a report waiting in the app. Whether and how the job reports back is a separate topic.

A simple crontab line shows the idea. This line starts a summary script at 7:00 every day. The five fields come first, then the command.

Text
# minute hour day-of-month month day-of-week  command
0 7 * * * /usr/bin/python3 /opt/shop/send_sales_summary.py

This simple pattern is only part of the story. The scheduler is not the same as the worker, and the rule is not the same as the job's logic. Keeping those roles separate makes the system easier to test and to fix.

What is typical scheduled work?

Scheduled work often has a recurring pattern that people can predict. Examples include nightly data updates, monthly sales reports, periodic health checks, and backups. Azure's guidance lists routine work like daily reports, data retention cleanup, and data consistency checks as typical fits for a timer.

Why do users never wait?

A user should never have to wait for a scheduled job. Many applications need background work that runs apart from the user interface. The job runs in its own process, so the page the user is on stays fast. The user gets a finished result later, instead of waiting on a spinner while the work runs.

When should you choose a schedule-driven job?

The key question is what decides the moment the work should happen. If the clock decides, use a schedule. If a specific thing that happened decides, use a trigger that reacts to that thing. The three cards below show the signs that point to a schedule.

  • Recurring and predictable

    The work repeats on a known pattern, such as every morning or on the first day of each month. A cron job is a good fit for time-triggered, predictable, bounded tasks.

  • Batch style

    The work handles many records together, like building a monthly report or cleaning up old rows. Batch work often suits a scheduler, which starts one run that covers the whole set.

  • A small delay is fine

    A cron schedule runs at fixed times, so there is always some gap between when something happens and when the job handles it. If that gap is acceptable, a schedule is usually simpler and cheaper.

Schedule or event?

Jobs that must react right away belong on the other side. A payment receipt sent right after a customer pays should not wait for the next scheduled run. Those jobs are triggered by an event, such as a message arriving or a user action. The Event-Driven lesson covers that style in detail, so here we only compare the two.

Tip

Ask two questions. Does the clock decide when this should run? Can the result wait until the next scheduled run without harm? If both answers are yes, a schedule-driven job is a strong choice.

Quick check

An online store wants to delete expired shopping carts and email a sales summary to managers every morning at 7 a.m. Should it use a schedule-driven job or run the work on each user action, and why?

Worked example: a bank's monthly statement

Here is how a bank's mobile app might use a schedule-driven job to make your monthly statement PDF. The bank's own internal design is not published in the sources used for this lesson, so treat this as an illustration of the idea. The job runs at 1 a.m. on the first day of each month, so the statement is already waiting when you open the app.

  1. 1

    The rule is set

    The bank sets a rule for 1 a.m. on the first day of every month. Nobody has to start the job by hand.

  2. 2

    The scheduler starts a worker

    At 1 a.m. the scheduler starts a statement worker. Your app is not involved at this point.

  3. 3

    The worker gathers the month's activity

    The worker reads last month's transactions for each account and totals them.

  4. 4

    The worker builds the PDF

    The worker writes each statement as a PDF and saves it where the app can fetch it.

  5. 5

    You open the app later

    When you open the app that morning, the statement is already there. You never waited on a spinner while it was built.

Tip

Night hours are a smart choice here. Few customers are online at 1 a.m., so the heavy PDF work is less likely to slow the app for anyone else.

Checking the design

Now picture a failure. If the same statement job ran twice, a customer might get two copies, or worse, two charges. The design needs a rule that a statement for one account and one month is made only once, even if the job starts again. Keep that rule in mind for the pitfalls below.

What are the common pitfalls and safeguards?

Scheduled jobs look simple, but three problems show up again and again. Jobs can overlap or run twice, jobs can fail without anyone noticing, and heavy jobs can slow the whole system at busy times. Each problem has a known fix.

  • Overlap and double runs

    A job can run twice, or two runs can overlap, if the previous run is still busy when the next one starts. Schedulers can also start extra copies when more than one scheduler is running.

  • Silent failure

    A job that never starts, or that stops halfway, can fail with no error on screen. Nobody notices until someone sees stale or missing data.

  • Heavy work at peak time

    A large job that runs during the busiest hours can slow down the pages and services that customers use.

Make jobs idempotent

The first safeguard is to make each job safe to repeat. Azure's guidance says scheduled tasks should be idempotent, so running the same task twice does not create duplicate results. A job that can run more than once without causing harm is far less risky when a scheduler misfires or a retry happens.

The sample below shows the idea in a small, runnable form. The cleanup rule removes carts older than 30 days. Running it a second time changes nothing, because the carts that were already removed stay removed.

Python
def run_cleanup(carts):
    # Keep only carts that are 30 days old or newer.
    return {cid: age for cid, age in carts.items() if age <= 30}

carts = {"cart-1": 45, "cart-2": 3, "cart-3": 31}
once = run_cleanup(carts)
twice = run_cleanup(once)
print(once)
print(twice)
print(once == twice)

Output

{'cart-2': 3}
{'cart-2': 3}
True

Tip

Before a job writes anything, check whether the work is already done. For example, look for an existing statement for this account and month. A database rule that forbids duplicate rows is another strong guard.

Add retries and alerts

The second safeguard is retries and alerts. A plain cron job does not retry a failed run on its own, so the job or its tooling has to handle that. A retry tries the failed step again, often after a short wait. Pair retries with a heartbeat: the job sends a short signal when it finishes, and a monitor raises an alert when that signal does not arrive on time. This catches jobs that never ran at all, which a failing job cannot report by itself.

Run heavy jobs off-peak

The third safeguard is timing. Run heavy jobs in off-peak hours, when fewer people use the system. Give each big job a clear time limit, so a stuck run does not hold resources all night. If a job must still run during busy hours, split it into smaller batches so it does not compete with customers for all the capacity at once.

Quick check

Every night a scheduler starts a job that emails invoices. Two scheduler servers are running by mistake, so some customers receive the same invoice twice. What is the best first fix?

Recap

A schedule-driven job starts because the clock says so. A scheduler watches the time, a worker does the work, and the user is never kept waiting. Choose this style for recurring, predictable, batch-style work, and choose an event-driven trigger when something must react at once. Protect every scheduled job by making it safe to repeat, adding retries and alerts, and running heavy work off-peak.

Tip

Before you launch any scheduled job, write down three things: its exact time, what happens if it runs twice, and who gets alerted if it never reports success.

Sources

How is this lesson?

Last updated on 17 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.