esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Pipes & Filters: Chain Small Steps and Scale the Slow One

Split one heavy processing job into small filters joined by pipes, so each step can be reused, tested, and scaled on its own.

In short

Pipes and Filters splits one heavy processing job into small steps called filters, joined by pipes, so each step can be reused, tested, and scaled on its own.

10 min readFree

What You Will Learn

This lesson teaches Pipes & Filters, a design pattern for turning one heavy job into a chain of smaller jobs. You will see what the two building blocks do, why teams use the pattern, how a video upload moves through it, and which risks to plan for. A short Python demo at the end lets you run the idea yourself.

Key words

Pattern
A tested design idea for a common problem, so you do not have to invent a fix from scratch each time.
Pipeline
A chain of steps that data passes through in a set order.
Filter
One self-contained step that reads input, changes it once, and writes output.
Pipe
The channel that carries data from one filter to the next.
Queue
A line of messages that waits until a worker takes each one, usually in order.
Stream
A continuous flow of items that are handled one after another.
Monolith
One large block of code that does many jobs in one place.
Bottleneck
The single step that slows the whole line because it cannot keep up.
Scale
To add more capacity, such as more copies of a step, so the system handles more work.
Stateless
A step that remembers nothing from one job to the next.
Idempotent
Safe to repeat: running it twice gives the same result as running it once.
Decoupled
Loosely connected, so one part can change without breaking the others.
Transcoding
Converting a video into another format, size, or quality level.

What Is the Problem with One Big Block?

Imagine an app that takes uploaded photos and does three jobs inside one function. It resizes the picture, adds a watermark, and scans the result for unsafe content. This style is a monolith. It is easy to start with, but it becomes hard to manage as the app grows.

The trouble shows up under load. If resizing is the slowest job, you cannot add extra power only for resizing. You must copy the whole function, scan and watermark included. Changing one job also means retesting all three. A second team that needs only the resize code has to copy it, and the two copies drift apart over time.

What Are the Two Building Blocks?

Pipes & Filters has only two building blocks, but each one has clear rules. Together they form a pipeline. Data enters at a source, which is where the work starts. It flows through filters joined by pipes, and it leaves at a sink, which gathers the data at the end of the pipeline, such as a file or display device.

Filters: One Job Each

A filter is a self-contained step. It reads input from the pipe before it, does one transformation, and writes output to the pipe after it. A filter may also test data against a rule and decide whether to pass it on. Even so, each filter should handle one kind of work, not a mix of several.

A filter only needs to know the shape of the data it reads and the shape of the data it writes. It does not need to know what the steps before or after it do. This is why a resize filter can sit next to a watermark filter without either one changing.

Pipes: The Channels Between Steps

A pipe is the channel that moves data from one filter to the next. A pipe only moves data. It does not pick a destination or change the content, so the filters keep all the decisions. A pipe is also a buffer, a holding area that keeps items between two neighbors. That buffer lets a fast filter and a slow filter work at different speeds.

  • Message queue

    A broker holds each message until a worker finishes it. A chain of queues can carry a whole pipeline from its first step to its last.

  • Byte stream

    In Unix command shells, a pipe passes raw bytes from one program to the next, with no fixed structure.

  • Bounded buffer

    A pipe with a fixed amount of space in shared memory. When it is full, the fast filter must wait for room.

Why Split the Work?

Splitting one block into filters changes what you can do with the code. Three benefits matter most for students and new designers.

  • Reuse

    Because each filter is self-contained, you can assemble new flows from filters you already have. A watermark filter written once can serve many different upload flows.

  • Testing

    Each filter can be checked alone with sample input. Small units are simpler to verify than one large function that does everything.

  • Separate scaling

    Only the slow filter needs more copies. It can run on its own machines, while cheaper filters stay on modest hardware.

How Do You Scale Only the Bottleneck?

A pipeline is only as fast as its slowest filter. That slow filter is the bottleneck. Because filters are separate, you can run several copies of the bottleneck while the other filters stay as they are. More copies handle more jobs in total, and the rest of the line does not need a rewrite. Running several workers on one queue is a separate pattern with its own lesson, so here it is only a way to scale one filter.

Tip

Measure before you add copies. Find which filter is slow first. Copying a fast filter costs money and changes nothing about the delay.

Worked Example: A Video Upload

Video platforms show the idea clearly. The steps below come from third-party write-ups and teaching designs, not from official YouTube documentation. YouTube does not publish its full internal design, so read this as a model of how a pipeline can be built. Each step is a filter, and the steps are joined by queues.

  1. 1

    Receive and store

    The uploaded file lands in storage as the raw original. Write-ups describe this raw store as the start of processing, followed by an event that announces the new file.

  2. 2

    Check safety and policy

    A check filter scans the file for malware and rule breaks before other work begins. A teaching design can place this first, so bad files stop early and cost nothing more.

  3. 3

    Read the basics

    An analysis filter reads details such as resolution, codec (the method used to compress video), and frame rate. Later filters use these details to choose their settings.

  4. 4

    Transcode in parallel

    Write-ups describe a queue that holds these jobs, with stateless workers that take them one at a time.

  5. 5

    Make thumbnails and captions

    Thumbnail and caption filters take the same raw file and produce small extra outputs. Because they are separate filters, they can run beside transcoding instead of waiting for it.

  6. 6

    Publish in stages

    Viewers can watch once the first versions are ready. Higher-quality versions finish in the background and replace the early ones as they arrive.

Notice what the design buys. Each filter has one job, so each can be tested, replaced, or copied without touching the others.

Why Transcoding Gets Scaled Out

Transcoding is the slow filter in this flow. Re-encoding video takes a lot of computing power and time. Running the encodes on many machines at once raises how much video the system can handle.

Because transcoding has its own filter, a platform can add transcoding workers during a traffic spike. The check, analysis, thumbnail, and caption filters do not change. The pipes keep the order of the work, so nothing else needs to be rewritten. This is the main point of the example: you scale the bottleneck alone.

What Are the Trade-offs and Design Cautions?

Pipes & Filters is not free. Its benefits come with costs, and a good design plans for them from the start.

Keep Filters Stateless or Idempotent

A filter that keeps no memory between jobs is easy to copy and restart. Keeping memory out of filters makes them simpler to build and easier to scale. Designers should aim for stateless filters whenever they can.

Some work cannot avoid state, such as writing a row to a database. In that case, make the filter idempotent. If the same message arrives twice, the second run must not add a second copy of the result. Duplicates happen when a worker crashes after doing the work but before it reports success, so the next worker may repeat it.

Tip

Give every job a unique ID and check for it before you write. A repeated job then becomes a harmless no-op.

Plan for Failures and Duplicates

Messages can go wrong in two directions. If a queue promises at-least-once delivery, you should expect the occasional repeat. Azure's pipes-and-filters guidance says filters should be idempotent. You must also make sure the infrastructure does not lose data while it sits in a pipe.

Every filter also needs an error rule. It must decide whether a bad input fails the whole pipeline or is set aside for later. Pipes need a plan too. Backpressure is the name for a slow step pushing back on the steps before it. When a slow filter stops taking work, the pipe before it fills up, so the producer must wait, buffer with a limit, or drop work on purpose.

Avoid Too Many Tiny Stages

Each extra filter adds one more hop, and each hop adds waiting time. Every pipe also needs monitoring and a restart plan. Ten tiny filters can be harder to run than three well-sized ones. Start with steps that have different speeds or different scaling needs. Join steps that always change together, and keep filters that must scale together in one process.

Try It: A Tiny Pipeline in Python

This sample uses three filters written as generators. A generator is a Python function that hands out one item at a time with the yield keyword, so each filter passes items along as they arrive. The output is the same every time you run it.

Python
def strip_spaces(lines):
    for line in lines:
        yield line.strip()

def to_upper(lines):
    for line in lines:
        yield line.upper()

def add_brackets(lines):
    for line in lines:
        yield '[' + line + ']'

raw = ['  photo one ', ' photo two  ']
for item in add_brackets(to_upper(strip_spaces(raw))):
    print(item)

Output

[PHOTO ONE]
[PHOTO TWO]

No filter knows about the others. You can remove, swap, or add a filter by changing one line in the nesting, which is the reuse benefit in miniature.

Check Yourself

Quick check

Your app resizes, watermarks, and scans uploaded photos inside one function. During traffic spikes, the resize step is the bottleneck. Which restructuring fits Pipes & Filters best?

Quick check

A video pipeline has four filters: decode, scan, transcode, and publish. Transcode is the slow step. A new rule says every upload must be scanned before it is published. Which change fits the pattern best?

Key Takeaways

Pipes & Filters splits one heavy job into filters, linked by pipes. Each filter does one job and knows only its input and output shape. You gain reuse, easier testing, and the ability to scale just the bottleneck. You pay with more moving parts, so keep filters stateless or idempotent, plan for repeats and failures, and do not split the work into too many tiny steps.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.