esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Monitoring: Watching a Running System So Problems Surface Early

Collect signals such as speed, errors, and resource use, then turn them into dashboards and alerts that catch trouble before users complain.

In short

Monitoring is watching a running system by collecting signals such as speed, errors, and resource use, then turning them into dashboards and alerts.

12 min readFree

What you will learn

Software that works on a developer's laptop can still fail in front of thousands of real users. Monitoring is how a team keeps a steady eye on a running system, so it knows when something is wrong and, ideally, before users do. In this lesson you will learn what teams watch, how those numbers become dashboards and alerts, and how to pick alert limits that catch real problems without constant false alarms. A worked example follows an online shop's checkout service through a bad release.

Key words

Monitoring
The ongoing practice of collecting and watching data about a running system, such as its speed, errors, and resource use.
Metric
A number measured again and again at regular times, such as how many requests failed in the last minute.
Log
A timestamped record of one event, such as a failed payment, often with an error message attached.
Trace
The full path one request takes through several services, with the time each step took.
Latency
How long a request takes to finish.
Saturation
How full a resource is, such as memory, disk space, or a waiting line of jobs.
Percentile
A cut-off point in a list of timings. The 99th percentile is the time that 99 out of 100 requests beat.
Dashboard
A screen that shows a service's most important numbers together, usually as charts over time.
Alert
A notice sent to a person or a system when a rule decides that something needs attention.
Threshold
The limit a value must cross before an alert rule counts it as a problem.
Page
An alert that reaches a person right away, such as a phone call, so they can act now.
Root cause
The real defect behind a failure. Fixing it should stop the same failure from coming back.

Why does monitoring matter?

Systems fail in ways nobody planned for. A new release can slow one database query. A disk can fill up at night. A partner service can return errors for a few minutes and then recover on its own. You cannot list every failure ahead of time, so you need a steady view of how the system behaves all the time.

Waiting for complaints is a weak plan. By the time many users write in, some have already left, and the team starts from zero, guessing where the trouble began. Monitoring lets a system tell people when it is broken, or when it is about to break. When the system cannot fix itself, a person investigates, reduces the damage, and then looks for the root cause.

Teams also use the same data for other jobs. They spot slow long-term growth, compare this week with last week, and dig into what changed around a sudden spike.

Tip

Start monitoring before you need it. Data that was never collected cannot be checked after an incident has already begun.

What gets watched?

The term golden signals comes from chapter 6 of Google's Site Reliability Engineering book, a chapter called Monitoring Distributed Systems. The chapter says that if you can measure only four things about a user-facing system, you should pick four basic signals. They are latency, traffic, errors, and saturation.

  • Latency

    How long a request takes to finish. Keep the timing of successful and failed requests apart, because a slow failure can be worse than a fast one. Watch the slow end of the range, such as the 99th percentile, not only the average.

  • Traffic

    How much demand the system receives, counted in a unit that fits the service, such as requests per second. A sudden drop can matter as much as a spike, because it can mean users cannot reach you at all.

  • Errors

    How often requests fail. Track the share of requests that fail, not only the raw count, because a count rises whenever traffic rises too.

  • Saturation

    How full a resource is, such as memory, disk, or a queue of waiting jobs. Saturation is often the most forward-looking of the four signals.

The four signals describe what users feel. They are symptoms. A team can see that checkout is failing before it knows why, and then the team moves on to the cause.

The signals are only a starting point. Most teams also collect machine data and detailed records, and they use three kinds of telemetry, which is the word for data a system sends out about itself.

Metrics, logs, and traces (explained below) are the three kinds of telemetry that most teams use together.

What are metrics, logs, and traces?

  • Metrics

    Numbers measured at regular intervals. They are cheap to store and easy to chart, so they suit dashboards and alerts.

  • Logs

    Detailed records of single events, with a timestamp and details such as an error message or a request ID.

  • Traces

    The path of one request as it moves through several services, with the time each step took.

A simple way to remember the roles is to ask three questions in order. Metrics answer whether something is off. Traces answer which step is off. Logs answer what that step reported.

Tip

Begin with user-focused measures, such as whether a checkout finished, before you add internal machine numbers. Machine numbers help with diagnosis, but they rarely prove that customers are hurt.

Watch what users feel first

Machine numbers such as CPU and memory use are still useful. The trap is treating them as proof of health. A server's CPU can look calm while users wait on a slow database call or on a payment provider. User-facing numbers catch that pain, and then the team uses machine numbers to dig deeper.

How does monitoring turn into action?

Collecting data does not help anyone by itself. Monitoring turns numbers into action in two main ways. People look at dashboards to understand trends, and alerts pull a person in when something needs attention now.

Alerts for waking someone up

An alert is a rule that sends a notice when a value crosses a limit. Alerts go to different places. Some go to a ticket queue for work that can wait until the next day. A page goes to a person right away, and it interrupts their work or sleep. Paging is expensive, so it should be kept for problems that need action now. When pages arrive too often, people start skimming them, and a real emergency can be missed in the noise.

Choosing thresholds without constant false alarms

A fixed threshold is simple, but it is brittle. A fixed limit does not adapt to normal daily peaks, new releases, or seasonal shopping rushes, so it can fire when nothing is broken. Constant false alarms lead to alert fatigue, the state where people stop trusting their alerts and miss the real ones.

A practical rule is to page on what users feel, such as slow checkouts or failed payments, rather than on every internal part that looks busy. Measure speed with percentiles, and measure errors as a rate. The 99th percentile reveals the slowest tail of requests. The 95th percentile can show a broader slowdown sooner. A rate of failures compared with all requests stays fair when traffic changes, while a raw count keeps growing with traffic.

Also require a bad condition to last for a while before paging. A brief spike is often harmless. The short program below uses made-up numbers to show the same idea. It sets a 2 percent limit and pages only after two bad minutes in a row.

Python
samples = [(1000, 5), (1000, 4), (1000, 6), (1000, 45), (1000, 52), (1000, 3)]
threshold = 0.02
streak = 0
for minute, (total, failed) in enumerate(samples, start=1):
    rate = failed / total
    if rate > threshold:
        streak += 1
    else:
        streak = 0
    if streak >= 2:
        state = 'PAGE'
    elif streak == 1:
        state = 'watch'
    else:
        state = 'ok'
    print(f'minute {minute}: {rate:.1%} {state}')

Output

minute 1: 0.5% ok
minute 2: 0.4% ok
minute 3: 0.6% ok
minute 4: 4.5% watch
minute 5: 5.2% PAGE
minute 6: 0.3% ok

Setting a first threshold is a small process you can repeat.

  1. 1

    Pick one user symptom

    Choose something users feel, such as checkout speed at the 99th percentile.

  2. 2

    Study normal history

    Look at how the number behaves on ordinary days, including its busiest hours.

  3. 3

    Set the limit above normal

    Place the limit above the usual peaks, so it fires only when something is clearly wrong.

  4. 4

    Add a waiting time

    Require the bad state to last for a set time before anyone is paged.

  5. 5

    Review every page

    After each alert, ask whether someone had to act. If not, change the rule or remove it.

Tip

A threshold is only a guess until real traffic tests it. Review your alerts on a regular schedule and keep the ones that lead to action.

Worked example: a checkout dashboard

This example uses an online shop whose checkout service calls an outside payment provider. The dashboard is a Grafana-style layout invented for teaching. The numbers and limits are example settings, not measured data. The top row has four panels: request rate for traffic, error rate, 99th percentile checkout time for latency, and connection pool use for saturation. Below them sit a trace panel that lists slow checkouts and a log panel for error messages.

  1. 1

    Baseline

    On a normal day the request rate follows its usual daily pattern, checkout time stays steady, and the error rate stays low. The dashboard looks calm.

  2. 2

    Release

    A new version goes live. Within minutes, checkout time climbs well above its usual level and the error rate rises. Traffic stays normal and connection pool use stays low, so the trouble is not extra demand or a full resource.

  3. 3

    Alert

    The alert rule says to page if checkout time at the 99th percentile stays above 2 seconds for 5 minutes, or if the error rate stays above 2 percent for 5 minutes. The rule goes pending, then fires, and the on-call engineer receives a page.

  4. 4

    Triage

    The engineer opens the trace panel. Most of the slow time sits in one step: the call to the payment provider. The log lines for that step show timeout errors that began right after the release.

  5. 5

    Action

    The team rolls back the release. On the dashboard, checkout time and error rate return to their normal levels, which confirms the fix. Later, the team repairs the payment call and records what it learned.

The pattern on the dashboard did most of the work. Latency and errors rose together while traffic and saturation stayed flat. That pointed the team at the new code, not at a need for more servers. The alert told them when, the trace told them where, and the logs told them what.

Quick check

A new version of checkout goes live. Traffic is normal and the CPU graph looks fine, but the error rate and checkout time both jumped. What should the on-call engineer check first?

Common mistakes to avoid

New teams tend to repeat a few mistakes. Watch for these four.

  • Paging on machine numbers alone

    High CPU can be harmless. Page on user pain, and keep machine numbers on dashboards for diagnosis.

  • Counting errors, not rates

    A raw count keeps rising as traffic grows. A failure rate gives a fairer picture of health.

  • Too many alerts

    Each extra alert trains people to ignore alerts. Remove any alert that nobody acts on.

  • Alerts with no next step

    An alert should name the problem and suggest a first place to look. A short note with a first check saves time in the middle of the night.

Applying the idea: slow pages and a calm CPU graph

Imagine users say that pages feel slow. The server's CPU graph looks normal, and you have no other data. The cause could be a slow database query, a slow outside service, or a long network trip. CPU alone cannot show any of these. Next time, you would want page latency by percentile, error rates for each page, traces for page requests, and a simple outside check that loads a page the way a user would. Black-box checks test a service from the outside, as a user would see it, so they confirm what customers feel.

You would also want to be told in a useful way. A sustained slowdown should reach the team in a normal channel, and only a severe, lasting one should page someone.

Quick check

Users say pages feel slow, but the server's CPU graph looks normal and you have no other data. Which setup would find the cause fastest next time?

Where to go next

This lesson gave you the big picture. Later lessons cover specific kinds of monitoring, such as health, availability, performance, security, and usage. Others explain how to add measurement code to an application and how to build charts and alert rules in depth. As you read them, keep the core idea in mind: collect the signals that match what users feel, and let people act only on the alerts that matter.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.