# Claim Check: Send a Small Ticket Through the Queue, Keep the Big Payload Outside

> Claim Check stores a large message payload in outside storage and sends only a small reference token through the messaging system.

**In short:** Claim Check stores a large message payload in outside storage and sends only a small reference token through the messaging system.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/claim-check

## What You Will Learn

Some messages are far too big to send through a queue. Claim Check is a design pattern, which means a reusable answer to a common problem. It keeps the big part of a message outside the queue and sends only a small pointer. In this lesson you will learn why big messages cause trouble, how the pattern fixes that, what it costs, and how to handle the hard cases. You do not need to know any cloud product to follow along.

- **Message:** A small package of data that one program sends to another program.
- **Message broker:** The service that holds messages and passes them to the programs that need them.
- **Queue:** A line where messages wait until a worker is ready to read them.
- **Payload:** The main content of a message, such as a file or a large record.
- **Blob storage:** A storage service built to hold big files as separate objects, each with its own ID.
- **Claim check:** A short token, such as an ID or an address, that points to a stored payload.
- **Producer:** The program that sends a message.
- **Consumer:** The program that receives a message and does the work.
- **Idempotent:** Describes an action that gives the same result even when it runs more than once.
- **Orphaned payload:** Stored data that no live message points to, so nothing uses it and nothing removes it.
- **Lifecycle policy:** A storage rule that deletes or moves stored objects after a set time.

## What Is the Problem with Big Messages?

Message brokers are built to move many small messages quickly. Most of them set a maximum message size. Large messages cause trouble in several ways, as Microsoft's Azure Architecture Center explains in its description of this pattern.

- **Rejected at the door:** A broker that receives a message above its limit refuses it. The sender gets an error, and the work never starts.
- **Slow for everyone:** A big message takes up memory and disk inside the broker. It also takes longer to copy and read, which slows the other messages that share the broker.
- **Expensive to run:** Some brokers charge by message size or by capacity tier. Sending large payloads through them can push a team into a higher and more costly tier.

Raising the size limit is not always the answer. Limits differ from product to product and from tier to tier. Even a high limit still means that big data travels through the broker every single time.

## How Does the Claim Check Work?

The pattern splits the work into two paths. The payload goes to a storage service built for big objects. Only a short reference, the claim check, goes through the broker. The receiver uses that reference to get the full payload back. Before sending, the producer can first check whether the payload is large enough to need this treatment.

1. **Save the payload:** The sender writes the full payload to an external store, such as blob storage.
2. **Confirm the save:** The sender waits until the store reports that the write succeeded.
3. **Build the claim:** The sender creates the token, such as a unique ID or a storage address for the object.
4. **Send the small message:** The sender publishes a message that holds the token and only the small details needed for routing.
5. **Read the token:** The receiver takes the message off the queue and reads the token inside it.
6. **Fetch and process:** The receiver uses the token to download the payload from the store, then does its work.

The order of the first four steps matters. If the token reaches the queue before the payload is saved, a receiver may hold a token that points to nothing. So the sender publishes the token only after the save has succeeded.

## Worked Example: A Video Upload

Video platforms are a common picture of this pattern. A video file is large, and it must be converted into several versions before people can watch it. The write-ups below describe a design, not YouTube's private internals. One public guide says the video goes to storage and a message with its storage location goes into a queue. Another notes that some details of YouTube's own setup are not publicly confirmed. So treat this example as one clear way the pattern can be used.

Read the steps below in order. The video file never travels through the queue. Only a small message does.

1. **Storage takes the file:** The creator's upload goes into original storage, where the full video is kept.
2. **Storage gives back an ID:** Once the file is saved, the system has an ID and a location for it.
3. **Queue gets a tiny message:** A message holding only the ID, the location, and basic details goes onto the transcoding queue.
4. **Worker takes the message:** A transcoding worker, a program that converts video into other resolutions, reads the message from the queue.
5. **Worker fetches the file:** The worker uses the ID to download the original video from storage and starts converting it.
6. **Results go back to storage:** The converted versions are written to storage, and a completion message is queued for other services.

> **Tip:** When you review a pipeline, check each arrow and ask one question: does this message carry the big data, or only a pointer to it? If it carries big data, a claim check may be missing.

## What Trade-offs Must You Accept?

The pattern is not free. It trades some simplicity for a broker that stays lean. The team must accept a few new costs, which the Azure guidance lists among the points to think about.

- **More moving parts:** Each payload now needs two systems, the store and the broker, plus one extra write and one extra read.
- **The store must stay up:** If the storage service fails, receivers with valid tokens cannot fetch their payloads, so their work stalls.
- **A little extra waiting:** Each receiver makes one more network trip before it can start, which matters for very fast jobs.

Use the pattern when payloads often exceed the size the broker accepts, or when big messages slow the whole system. For small messages, sending the data inline is simpler and faster. Some brokers also support larger messages in higher tiers, so check the real limit before you design around it.

### How Do You Clean Up Stored Payloads?

Stored payloads pile up unless someone removes them. Decide early which component owns deletion, and how it will know that a payload is no longer needed. Many teams use a lifecycle policy, which deletes or archives stored objects after a set time. The timing matters. If a payload is deleted while its message still waits in the queue, the claim points to nothing. If nothing is ever deleted, the payload becomes an orphaned payload that costs money and that no message uses.

> **Tip:** Keep the retention time longer than the longest a message can wait in the queue, including time spent on retries.

### Who Should Be Able to Read Payloads?

A stored payload may hold private data, such as invoices or customer records. The storage service should apply its own access rules, so that only approved services can read the objects. Keep secrets out of the claim itself. A token that carries a password can leak through logs or queue monitoring tools. If a client needs short, direct access to storage, the Valet Key pattern covers that case, and it has its own lesson.

## When Things Go Wrong

Two failures are common. Both come from the gap between the store and the broker. The payload and the message are written to different systems, so one write can succeed while the other fails. Good designs plan for both cases.

### When the Payload Is Missing

Sometimes a receiver gets a claim, but the payload has expired, was deleted, or never finished saving. The receiver should not guess or invent data. It should report the missing item, mark the message for review, or move it to a separate place for messages that could not be handled, where a person can look at them.

### When the Same Message Arrives Twice

A queue may deliver the same message more than once, for example after a worker crashes before it confirms that it finished. The receiver must therefore be safe to repeat. An idempotent operation gives the same result each time it runs. One simple method is to record which claims were already handled and skip them. A content hash, which is a short fingerprint of the file, can also show that the payload was not changed.

> **Tip:** Test your receiver by sending the same claim twice. The second run should not create a second record.

## A Small Code Sketch

The sketch below uses a Python dictionary as a stand-in for blob storage and a list as a stand-in for the queue. It shows the shape of the pattern, not a real cloud setup. Notice that the message holds only a short key, while the large text stays in the store.

```python
payload = 'scanned invoice page ' * 500
store = {}
queue = []

store['inv-1001'] = payload
queue.append({'claim': 'inv-1001'})

message = queue.pop(0)
fetched = store[message['claim']]

print('message:', message)
print('payload size:', len(fetched))
```

Output:

```text
message: {'claim': 'inv-1001'}
payload size: 10500

```

In a real system, the store would be a cloud storage service and the queue would be a broker. The steps stay the same.

## Check Your Understanding

Try these three short scenarios. Each one needs the idea from this lesson to answer.

**Quick check:** An order system puts 20 MB scanned PDF invoices directly into queue messages. The broker rejects them as too large. What change fixes this, and what should each queue message contain afterward?

A. Raise the broker's size limit as high as it goes, then keep sending the PDFs inside the messages.
B. Save each PDF in blob storage first, then send a small message holding the invoice ID and the file's storage location.
C. Cut each PDF into small chunks, send every chunk as its own message, and let the receiver rebuild the file.
D. Remove the scanned images and send only the invoice total, so the message stays small.

**Answer:** B. The store holds the big file, and the queue holds only a pointer. Raising the limit still sends big data through the broker, and chunking keeps the data inside the queue.

**Quick check:** A receiver handles a claim, saves its result, and then crashes before it confirms the message. The broker delivers the same claim again. Which design choice prevents a duplicate invoice record?

A. Delete the stored PDF as soon as the first read finishes, so the second copy has nothing to open.
B. Make the receiver check whether it already handled this claim, so repeating the work gives the same result.
C. Copy the PDF contents into the message on every retry, so the second delivery skips storage.
D. Ask the sender to publish the claim a second time, so the broker keeps only the newest copy.

**Answer:** B. Duplicates happen when delivery is repeated. A receiver that is safe to repeat makes the second run harmless. Deleting the file would break the second read instead of fixing the problem.

**Quick check:** A team deletes stored payloads after one day. Some messages wait in the queue for three days because their consumers were offline. What happens when those messages are finally read?

A. Nothing breaks, because the message still carries the full payload.
B. The receiver gets a claim that points to nothing, so it must report the failure instead of processing it.
C. The broker automatically copies the payload back from the sender.
D. The broker rejects the message the moment its payload is deleted.

**Answer:** B. The claim is only a pointer, so it is useless once the stored data is gone. The fix is to keep payloads longer than the longest time a message can wait.

## Key Takeaways

- **Keep big data out:** Put large payloads in a storage service so the broker only handles small messages.
- **Send a claim:** The message carries a short token, and the receiver uses it to fetch the payload when it is ready to work.
- **Plan for gaps:** Plan for cleanup, access control, missing payloads, and repeated messages before you launch.

## Sources

- [Claim Check pattern - Azure Architecture Center (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/architecture/patterns/claim-check)
- [Design YouTube - ByteByteGo system design course](https://bytebytego.com/courses/system-design-interview/design-youtube)
- [YouTube System Design - Grokking the System Design guide](https://grokkingthesystemdesign.com/guides/youtube-system-design/)
- [YouTube Tech Stack: How 2.5B Users Stream Video at Scale - VDOCipher blog](https://www.vdocipher.com/blog/youtube-tech-stack-architecture/)
- [Claim-Check Pattern - commons-os pattern notes](https://commons-os.github.io/patterns/claim-check-pattern/)
- [Using the Business Process Model and Notation for Modeling Enterprise Integration Patterns (arXiv)](https://arxiv.org/pdf/1403.4053)
