# Scheduling Agent Supervisor: Finishing Multi-Step Jobs Across Services

> A scheduler runs the steps, agents call each service, and a supervisor finds stuck steps and recovers them.

**In short:** The Scheduling Agent Supervisor pattern finishes a multi-step job across services: a scheduler runs the steps, agents call each service, and a supervisor recovers stuck steps.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/scheduling-agent-supervisor

## What you will learn

Many jobs take more than one step, and each step may call a program on another machine. A trip booking, a shop order, or a monthly report can touch three or four systems. Any of them can fail partway through. This lesson teaches the Scheduling Agent Supervisor pattern, a common way to run such a job so that it either finishes or is cleaned up. You will learn the three roles, how recovery works, a worked example, and when the pattern is worth its cost.

Some catalogs list this pattern as 'Scheduler Agent Supervisor'. It is the same pattern, so this lesson covers it once.

- **Job:** A task made of several steps, such as booking a whole trip.
- **Step:** One piece of work inside a job, such as reserving a hotel room.
- **Remote service:** A program on another machine that your code calls over a network.
- **Partial failure:** Some steps finish, then a later step fails, so the job is only half done.
- **Orchestrate:** Decide which step runs next and in what order, like a conductor leading an orchestra.
- **State store:** A durable place, such as a database, that keeps its data even after a crash, where the job records each step's status.
- **Idempotent:** Safe to run more than once. Running it twice gives the same final result as running it once.
- **Compensating action:** A new step that undoes the effect of an earlier step, such as cancelling a booking.

## What is the problem with a job that spans several services?

Suppose your code makes one call to one service and the call fails. You can often just try again. That is the simplest kind of recovery. The harder case is a job with several steps across several services. Step one succeeds, step two succeeds, and step three never answers. Now your system holds two finished pieces and one unknown. If nothing recorded the progress, nobody knows what to repair.

Microsoft's guidance on this pattern says that simple, short faults can be handled by retrying. When faults last longer, or when the code that runs the job crashes, the system must rebuild a correct state from saved records. The Scheduler Agent Supervisor pattern is built for that second case. It aims to make the whole job either succeed or be undone.

> **Tip:** Ask yourself: if the job stopped right now, could someone look at a record and say which steps finished? If the answer is no, the job needs a saved record of its progress before it needs anything else.

## What are the three roles?

The pattern splits the work into three logical roles. Logical means they are ideas for how to divide the code. One program can hold all three, or each can run as its own service. The roles are easiest to learn as a set of cards.

- **Scheduler:** Runs the job. It decides the step order, starts each step, and writes each step's status into the state store. It is responsible for arranging and orchestrating the steps.
- **Agent:** Does the outside work for one step. It wraps one call to a remote service or one access to a remote resource, and reports back to the Scheduler.
- **Supervisor:** Watches the saved step statuses on a schedule. When it finds a step that timed out or failed, it asks for recovery. It does not run the steps itself.

Keep the split clear in your head. The Scheduler decides what happens next. The Agent talks to the outside world. The Supervisor only checks the records and asks for repairs. Each role has one main job, and that is what makes failures easier to find.

### The Scheduler and the state store

The Scheduler records each step as it moves through three stages: not yet started, running, and completed. Each record also holds a complete-by time, which is the latest moment the step is allowed to finish. These records live in the state store, a durable data store. Durable means the data survives a crash. Without this store, a restarted Scheduler would have no memory of what it had already done.

### Agents wrap each outside call

An Agent usually wraps one service or one resource. It handles the error checks and retries for that call, but only within the complete-by time the Scheduler set. When the Agent retries, it sends the same stable identifier each time. The receiving service can then spot a repeat and ignore it. Two steps can use two different Agents, because which Agent handles which step is an implementation detail.

### How does the Supervisor scan for stuck steps?

The Supervisor runs periodically, at a frequency the workload specifies. Each run it reads the state store. A step that timed out or failed still shows as running, but its complete-by time has passed. The Supervisor finds those records and starts recovery. It cannot tell a timeout from a crash, and it does not need to. In both cases the step is overdue.

## How does recovery work?

Recovery follows a clear order. The Supervisor can extend the complete-by time and retry, or it can ask the Scheduler to undo the whole task. It never performs the fix itself. It asks the Scheduler or an Agent to do the work.

1. **Spot the overdue step:** The Supervisor finds a record that still says running after its complete-by time has passed.
2. **Count the retries:** The Supervisor keeps a retry count for each step in the state store. If the count is past a set limit, it waits longer before the next try.
3. **Ask for a retry:** The Supervisor asks the Scheduler to run the step again, often with a new complete-by time. The Scheduler ignores any late reply from the first attempt.
4. **Undo the whole job if retries fail:** If the step still cannot finish, the Supervisor asks the Scheduler to undo the job. The Scheduler runs compensating actions for the steps that already completed, in reverse order.

### Retry first, then undo the job

A retry is the cheap option, because it keeps the work already done. Undoing is the expensive option, because it throws that work away. A good supervisor tries the cheap option first. It also knows that undoing is not always clean. If a compensating action fails too, the job is neither finished nor fully undone. The state store must be able to show that case, so that a person can find it and fix it.

### Why must steps be idempotent?

A retry can cause a repeat. The first attempt may have worked, but its reply got lost, so the Supervisor cannot know. When the retry runs, the same work might happen twice. If the step is idempotent, the second run changes nothing. For example, 'set the room to booked for guest 42' is safe to repeat, while 'add one more room' is not. Designing steps this way means a repeated request does no harm.

> **Tip:** When you design a step, write down what happens if it runs twice. If the answer is a double charge or a double booking, add a stable request identifier so the other service can recognise the repeat.

### When the Scheduler itself crashes

The Scheduler can crash too. When it starts again, it reads the state store, finds the job, and works out which step was running. It then resumes from that point. The Supervisor does not restart the Scheduler or the Agents. Restarting them is the job of the hosting system, such as the platform that runs your servers. The Supervisor's job is only to notice the overdue record and ask for recovery.

## Worked example: booking a trip

Imagine a travel booking site that reserves a flight, a car, and a hotel. Each one is a separate service with its own agent. The scheduler runs the steps in the order flight, car, hotel. Here is how the pattern handles a hotel that never answers.

1. **Save the job:** When the customer clicks book, the Scheduler writes the job and its three steps to the state store, all marked not yet started.
2. **Run the flight and the car:** The Scheduler sends the flight request to the flight Agent. The record shows running with a complete-by time. The reply arrives, so the record becomes completed. The car step then follows the same path.
3. **The hotel step stalls:** The hotel Agent sends its request, but no reply ever comes back. The hotel record still says running, and its complete-by time passes at minute 5.
4. **The Supervisor notices:** At minute 10, the Supervisor runs its scan. It finds the hotel record overdue and sees that the retry count is still zero.
5. **Retry the hotel step:** The Supervisor asks the Scheduler to try the hotel step again. The hotel Agent sends the same booking identifier, so the hotel can ignore a duplicate if the first try had worked.
6. **Undo if the retry fails:** The retry also fails. The Supervisor asks the Scheduler to cancel the car and the flight through their compensating actions. The customer sees one clear result: no partial trip is left behind.

The toy program below is not the real booking system. It only shows the core check the Supervisor makes: a step is stuck when it is still running and its complete-by time has passed.

```python
now = 10  # minutes since the booking began
state_store = {
    "flight": ("completed", 3),
    "car": ("completed", 8),
    "hotel": ("running", 5),
}
for step, (state, complete_by) in state_store.items():
    if state == "running" and complete_by < now:
        print(step, "timed out, ask scheduler to retry")
    else:
        print(step, "is", state)

```

Output:

```text
flight is completed
car is completed
hotel timed out, ask scheduler to retry

```

The output shows that only the hotel step is stuck. The flight and car are finished, so the Supervisor leaves them alone. Only the hotel step needs a retry, and only the completed steps would need undoing if the retry failed.

## When to use it, and when not to

- **Good fit:** Long workflows that touch several services, where a failure in the middle is likely. Bookings, orders, and payments are common examples. The job must end in a clear result, either done or undone.
- **Overkill:** A single call to a single service, or a short call that only needs a simple retry. For those cases a plain retry is usually enough. Adding a scheduler, a supervisor, and a state store there only adds work.
- **Costs to accept:** You must run a durable state store, keep a Supervisor running on a schedule, and write an undo step for each action. Each extra part can fail, so you also need to monitor it.

> **Tip:** Start with the question: if this job fails halfway, would a customer be harmed by a partial result? If yes, the pattern is worth its cost. If no, a simple retry may be enough.

## Check your understanding

**Quick check:** An online shop's checkout charges the card, then reserves stock, then books shipping. The server crashes after the card is charged but before stock is reserved. Which role finds the half-finished order, what must it be able to read, and what can it do next?

A. The Agent finds the order and charges the card again so that checkout can finish.
B. The Supervisor reads the state store records with expired complete-by times, then asks the Scheduler to retry the stock step or undo the charge.
C. The Supervisor reads the bank's statements and cancels the shipping label.
D. The Scheduler restarts the server and deletes all records so the order starts over.

**Answer:** B. The Supervisor only checks saved step records. It finds the stock step still overdue and asks for recovery. Charging the card again would be a double charge, and deleting the records would hide the problem. The Scheduler, not the Supervisor, carries out the retry or the undo.

**Quick check:** In the trip booking, the hotel step has failed two retries, and the limit is two. The Supervisor scans again and finds the hotel step still overdue. Which action fits the pattern?

A. Retry every second with no limit until the hotel answers.
B. Wait a longer period before the next retry, and if the job still cannot finish, ask the Scheduler to undo the completed steps.
C. Delete the hotel record so the Supervisor stops seeing it, then mark the trip as booked.
D. Tell the hotel Agent to ignore its complete-by time and keep going.

**Answer:** B. The pattern can limit repeated retries by waiting longer after the limit is passed, and undoing the task is an alternative strategy. Deleting the record or ignoring the deadline would hide the failure, and an endless retry loop never ends.

## What this lesson leaves out

This lesson covers the pattern as a whole, so several related topics get their own lessons. Retry policies, which set how long to wait between tries, belong to a retry lesson. Designing the undo steps in detail belongs to a compensating transaction lesson. Systems with no central scheduler, where services react to events on their own, belong to a choreography lesson. Choosing one leader among several running Supervisors belongs to a leader election lesson. Stopping calls to a failing service belongs to a circuit breaker lesson, and checking service health belongs to a health endpoint lesson. Queue mechanics, such as how messages are held and handed out, belong to queue lessons.

> **Tip:** Before you build this pattern, write three things down: the state each step can be in, the record that shows it, and the undo for each step. Those three notes do most of the design work.

## Sources

- [Scheduler Agent Supervisor pattern, Azure Architecture Center (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/architecture/patterns/scheduler-agent-supervisor)
- [Minimize coordination, Azure Architecture Center (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/architecture/guide/design-principles/minimize-coordination)
- [Architecture design patterns that support reliability, Microsoft Azure Well-Architected Framework](https://learn.microsoft.com/en-us/azure/well-architected/reliability/design-patterns)
