# Monitoring: Watching a Running System So Problems Surface Early

> Collect signals such as speed, errors, and resource use, then turn them into dashboards and alerts that catch trouble before users complain.

**In short:** Monitoring is watching a running system by collecting signals such as speed, errors, and resource use, then turning them into dashboards and alerts.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/monitoring

## What you will learn

Software that works on a developer's laptop can still fail in front of thousands of real users. Monitoring is how a team keeps a steady eye on a running system, so it knows when something is wrong and, ideally, before users do. In this lesson you will learn what teams watch, how those numbers become dashboards and alerts, and how to pick alert limits that catch real problems without constant false alarms. A worked example follows an online shop's checkout service through a bad release.

- **Monitoring:** The ongoing practice of collecting and watching data about a running system, such as its speed, errors, and resource use.
- **Metric:** A number measured again and again at regular times, such as how many requests failed in the last minute.
- **Log:** A timestamped record of one event, such as a failed payment, often with an error message attached.
- **Trace:** The full path one request takes through several services, with the time each step took.
- **Latency:** How long a request takes to finish.
- **Saturation:** How full a resource is, such as memory, disk space, or a waiting line of jobs.
- **Percentile:** A cut-off point in a list of timings. The 99th percentile is the time that 99 out of 100 requests beat.
- **Dashboard:** A screen that shows a service's most important numbers together, usually as charts over time.
- **Alert:** A notice sent to a person or a system when a rule decides that something needs attention.
- **Threshold:** The limit a value must cross before an alert rule counts it as a problem.
- **Page:** An alert that reaches a person right away, such as a phone call, so they can act now.
- **Root cause:** The real defect behind a failure. Fixing it should stop the same failure from coming back.

## Why does monitoring matter?

Systems fail in ways nobody planned for. A new release can slow one database query. A disk can fill up at night. A partner service can return errors for a few minutes and then recover on its own. You cannot list every failure ahead of time, so you need a steady view of how the system behaves all the time.

Waiting for complaints is a weak plan. By the time many users write in, some have already left, and the team starts from zero, guessing where the trouble began. Monitoring lets a system tell people when it is broken, or when it is about to break. When the system cannot fix itself, a person investigates, reduces the damage, and then looks for the root cause.

Teams also use the same data for other jobs. They spot slow long-term growth, compare this week with last week, and dig into what changed around a sudden spike.

> **Tip:** Start monitoring before you need it. Data that was never collected cannot be checked after an incident has already begun.

## What gets watched?

The term golden signals comes from chapter 6 of Google's Site Reliability Engineering book, a chapter called Monitoring Distributed Systems. The chapter says that if you can measure only four things about a user-facing system, you should pick four basic signals. They are latency, traffic, errors, and saturation.

- **Latency:** How long a request takes to finish. Keep the timing of successful and failed requests apart, because a slow failure can be worse than a fast one. Watch the slow end of the range, such as the 99th percentile, not only the average.
- **Traffic:** How much demand the system receives, counted in a unit that fits the service, such as requests per second. A sudden drop can matter as much as a spike, because it can mean users cannot reach you at all.
- **Errors:** How often requests fail. Track the share of requests that fail, not only the raw count, because a count rises whenever traffic rises too.
- **Saturation:** How full a resource is, such as memory, disk, or a queue of waiting jobs. Saturation is often the most forward-looking of the four signals.

The four signals describe what users feel. They are symptoms. A team can see that checkout is failing before it knows why, and then the team moves on to the cause.

The signals are only a starting point. Most teams also collect machine data and detailed records, and they use three kinds of telemetry, which is the word for data a system sends out about itself.

Metrics, logs, and traces (explained below) are the three kinds of telemetry that most teams use together.

### What are metrics, logs, and traces?

- **Metrics:** Numbers measured at regular intervals. They are cheap to store and easy to chart, so they suit dashboards and alerts.
- **Logs:** Detailed records of single events, with a timestamp and details such as an error message or a request ID.
- **Traces:** The path of one request as it moves through several services, with the time each step took.

A simple way to remember the roles is to ask three questions in order. Metrics answer whether something is off. Traces answer which step is off. Logs answer what that step reported.

> **Tip:** Begin with user-focused measures, such as whether a checkout finished, before you add internal machine numbers. Machine numbers help with diagnosis, but they rarely prove that customers are hurt.

### Watch what users feel first

Machine numbers such as CPU and memory use are still useful. The trap is treating them as proof of health. A server's CPU can look calm while users wait on a slow database call or on a payment provider. User-facing numbers catch that pain, and then the team uses machine numbers to dig deeper.

## How does monitoring turn into action?

Collecting data does not help anyone by itself. Monitoring turns numbers into action in two main ways. People look at dashboards to understand trends, and alerts pull a person in when something needs attention now.

### Dashboards for seeing trends

A good dashboard answers basic questions about a service. Is traffic normal? Are requests fast? Are many of them failing? Is any resource close to its limit? Because charts show change over time, dashboards are also where you spot slow growth, such as a database that gains a little more data every week.

### Alerts for waking someone up

An alert is a rule that sends a notice when a value crosses a limit. Alerts go to different places. Some go to a ticket queue for work that can wait until the next day. A page goes to a person right away, and it interrupts their work or sleep. Paging is expensive, so it should be kept for problems that need action now. When pages arrive too often, people start skimming them, and a real emergency can be missed in the noise.

### Choosing thresholds without constant false alarms

A fixed threshold is simple, but it is brittle. A fixed limit does not adapt to normal daily peaks, new releases, or seasonal shopping rushes, so it can fire when nothing is broken. Constant false alarms lead to alert fatigue, the state where people stop trusting their alerts and miss the real ones.

A practical rule is to page on what users feel, such as slow checkouts or failed payments, rather than on every internal part that looks busy. Measure speed with percentiles, and measure errors as a rate. The 99th percentile reveals the slowest tail of requests. The 95th percentile can show a broader slowdown sooner. A rate of failures compared with all requests stays fair when traffic changes, while a raw count keeps growing with traffic.

Also require a bad condition to last for a while before paging. A brief spike is often harmless. The short program below uses made-up numbers to show the same idea. It sets a 2 percent limit and pages only after two bad minutes in a row.

```python
samples = [(1000, 5), (1000, 4), (1000, 6), (1000, 45), (1000, 52), (1000, 3)]
threshold = 0.02
streak = 0
for minute, (total, failed) in enumerate(samples, start=1):
    rate = failed / total
    if rate > threshold:
        streak += 1
    else:
        streak = 0
    if streak >= 2:
        state = 'PAGE'
    elif streak == 1:
        state = 'watch'
    else:
        state = 'ok'
    print(f'minute {minute}: {rate:.1%} {state}')
```

Output:

```text
minute 1: 0.5% ok
minute 2: 0.4% ok
minute 3: 0.6% ok
minute 4: 4.5% watch
minute 5: 5.2% PAGE
minute 6: 0.3% ok

```

Setting a first threshold is a small process you can repeat.

1. **Pick one user symptom:** Choose something users feel, such as checkout speed at the 99th percentile.
2. **Study normal history:** Look at how the number behaves on ordinary days, including its busiest hours.
3. **Set the limit above normal:** Place the limit above the usual peaks, so it fires only when something is clearly wrong.
4. **Add a waiting time:** Require the bad state to last for a set time before anyone is paged.
5. **Review every page:** After each alert, ask whether someone had to act. If not, change the rule or remove it.

> **Tip:** A threshold is only a guess until real traffic tests it. Review your alerts on a regular schedule and keep the ones that lead to action.

## Worked example: a checkout dashboard

This example uses an online shop whose checkout service calls an outside payment provider. The dashboard is a Grafana-style layout invented for teaching. The numbers and limits are example settings, not measured data. The top row has four panels: request rate for traffic, error rate, 99th percentile checkout time for latency, and connection pool use for saturation. Below them sit a trace panel that lists slow checkouts and a log panel for error messages.

1. **Baseline:** On a normal day the request rate follows its usual daily pattern, checkout time stays steady, and the error rate stays low. The dashboard looks calm.
2. **Release:** A new version goes live. Within minutes, checkout time climbs well above its usual level and the error rate rises. Traffic stays normal and connection pool use stays low, so the trouble is not extra demand or a full resource.
3. **Alert:** The alert rule says to page if checkout time at the 99th percentile stays above 2 seconds for 5 minutes, or if the error rate stays above 2 percent for 5 minutes. The rule goes pending, then fires, and the on-call engineer receives a page.
4. **Triage:** The engineer opens the trace panel. Most of the slow time sits in one step: the call to the payment provider. The log lines for that step show timeout errors that began right after the release.
5. **Action:** The team rolls back the release. On the dashboard, checkout time and error rate return to their normal levels, which confirms the fix. Later, the team repairs the payment call and records what it learned.

The pattern on the dashboard did most of the work. Latency and errors rose together while traffic and saturation stayed flat. That pointed the team at the new code, not at a need for more servers. The alert told them when, the trace told them where, and the logs told them what.

**Quick check:** A new version of checkout goes live. Traffic is normal and the CPU graph looks fine, but the error rate and checkout time both jumped. What should the on-call engineer check first?

A. Add more servers, because the shop must be busier than usual
B. Use traces and logs to find which step changed after the release
C. Mute all alerts until the next business day
D. Wait for customer emails to describe the problem

**Answer:** B. Normal traffic and low saturation mean extra servers will not help. The trace and the logs for the changed step point to the real cause.

## Common mistakes to avoid

New teams tend to repeat a few mistakes. Watch for these four.

- **Paging on machine numbers alone:** High CPU can be harmless. Page on user pain, and keep machine numbers on dashboards for diagnosis.
- **Counting errors, not rates:** A raw count keeps rising as traffic grows. A failure rate gives a fairer picture of health.
- **Too many alerts:** Each extra alert trains people to ignore alerts. Remove any alert that nobody acts on.
- **Alerts with no next step:** An alert should name the problem and suggest a first place to look. A short note with a first check saves time in the middle of the night.

## Applying the idea: slow pages and a calm CPU graph

Imagine users say that pages feel slow. The server's CPU graph looks normal, and you have no other data. The cause could be a slow database query, a slow outside service, or a long network trip. CPU alone cannot show any of these. Next time, you would want page latency by percentile, error rates for each page, traces for page requests, and a simple outside check that loads a page the way a user would. Black-box checks test a service from the outside, as a user would see it, so they confirm what customers feel.

You would also want to be told in a useful way. A sustained slowdown should reach the team in a normal channel, and only a severe, lasting one should page someone.

**Quick check:** Users say pages feel slow, but the server's CPU graph looks normal and you have no other data. Which setup would find the cause fastest next time?

A. Keep only the CPU graph, since it shows the servers are healthy
B. Restart the servers every hour to clear any slowness
C. Track page latency by percentile, add error rates and traces, and alert on slowdowns that last
D. Ask users each week which pages feel slow

**Answer:** C. CPU measures one machine. Latency by percentile and traces show what users feel and which step is slow, and an outside check confirms the problem from the user's side.

## Where to go next

This lesson gave you the big picture. Later lessons cover specific kinds of monitoring, such as health, availability, performance, security, and usage. Others explain how to add measurement code to an application and how to build charts and alert rules in depth. As you read them, keep the core idea in mind: collect the signals that match what users feel, and let people act only on the alerts that matter.

## Sources

- [Google SRE book, Chapter 6: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)
- [Google SRE book, Chapter 10: Practical Alerting](https://sre.google/sre-book/practical-alerting/)
- [Web-Alert: Four Golden Signals of Monitoring Explained](https://web-alert.io/blog/four-golden-signals-monitoring-explained)
- [Sherlocks: The Four Golden Signals of SRE](https://www.sherlocks.ai/blog/four-golden-signals-of-sre)
- [incident.io: SRE alerting best practices](https://incident.io/blog/sre-alerting-best-practices)
- [fluidify: Prometheus Alerting Best Practices for SRE Teams](https://fluidify.ai/blog/prometheus-alerting-best-practices)
- [DevOps AI Toolkit: Golden Signals Monitoring for SREs](https://devopsaitoolkit.com/blog/role-of-golden-signals-monitoring/)
- [IBM: Three Pillars of Observability, Logs, Metrics and Traces](https://www.ibm.com/think/insights/observability-pillars)
- [System Design, but Simple: Observability (Metrics, Logs, and Traces)](https://www.systemdesignbutsimple.com/p/observability-metrics-logs-and-traces)
- [OpenObserve: Navigating Observability, Logs, Metrics, and Traces Explained](https://openobserve.ai/blog/observability-logs-metrics-traces-explained/)
- [Dash0: OpenTelemetry Signals Overview, Logs vs Metrics vs Traces](https://www.dash0.com/knowledge/logs-metrics-and-traces-observability)
