# Performance Monitoring: Measure Speed Under Real Load to Find Slowdowns Early

> Learn which speed, traffic, error, and resource numbers to track, why averages hide slow requests, and how to trace a slowdown to one part of a system.

**In short:** Performance monitoring keeps measuring the real system under real traffic, day after day, so slowdowns show up early and can be traced to one part.

From "System design", a free course by Venkat R. Murty: https://designervenkat.online/learn/system-design/performance-monitoring

## What You Will Learn

Performance monitoring means measuring how fast and how well a system answers requests, all the time, while real people use it. A car has a speed gauge on the dashboard for the same reason. You do not wait for the engine to stop before you look at it. In this lesson you will learn which numbers to watch, how to decide what normal looks like, and how to use those numbers to find the part of the system that is slow.

- **Performance monitoring:** Continuous measuring of how fast and how efficiently a system responds, using live traffic.
- **Latency:** The time from when a request arrives to when its response goes out.
- **Percentile:** A cut-off point. The p99 is the time that 99 out of every 100 requests finish within.
- **Tail:** The slowest few percent of requests. Averages often hide them.
- **Throughput:** How much work a system finishes in a set time, such as requests per second.
- **Error rate:** The share of requests that end in a failure, out of all requests.
- **Saturation:** How full a resource is, and how much work is waiting in line for it.
- **Baseline:** A saved record of normal numbers that you compare new numbers against.
- **SLI (service level indicator):** A measured value that shows how the service is doing, such as the share of fast requests.
- **SLO (service level objective):** A target for an SLI, set over a fixed time window.
- **Bottleneck:** The one part of a system that limits how fast the whole thing can go.

## What Does Performance Monitoring Mean?

A one-time speed test tells you how a system behaved on one day, with one fake load. Performance monitoring is different. It keeps measuring the real system with real traffic, day after day. This matters because speed changes. A page that is fast with ten users can crawl with ten thousand. The slowdown may start small, at busy hours, and grow later.

Monitoring has two jobs. The first is to spot a slowdown early, before users write complaints. The second is to point at the cause. A good monitor does not only say the site is slow. It shows which part is slow, and when that started. Many teams use a short list of signals to do this. Google's site reliability engineers made four of them popular: latency, traffic, errors, and saturation. Most of this lesson is about those four signals and how to read them together.

Monitoring is not the same as the fixes. Once you find a slow part, you may use caching, better queries, or other changes. Those fixes are taught in their own lessons. Here, the goal is to know where the problem is and whether it is getting worse.

## What Are the Core Metrics to Track?

A metric is a number that you measure again and again over time. Good performance monitoring picks a few metrics that show how users feel and how the system is holding up. The main ones are latency, throughput, error rate, and resource saturation. Each one answers a different question, so you need all four to get the full picture.

### What Are Latency and Percentiles?

Latency is the most watched number, because users feel it directly. The simplest way to summarize latency is an average, which adds up all the times and divides by how many there were. The average is easy to read, but it has a serious weakness. A few very slow requests can be swamped by many fast ones, so the average looks healthy while some users wait a long time.

A percentile fixes this. The p50 is the median: half of requests are faster, and half are slower. The p95 is the time that 95 out of every 100 requests finish within. The p99 is the time that 99 out of every 100 requests finish within, so only the slowest 1 percent are above it. Teams often track p50, p95, and p99 together. If p50 is low and p99 is high, most users are fine, but a small group has a bad time. A healthy system usually has these numbers fairly close together.

Percentiles need enough data to be trusted. If you only have 100 requests, the p99 depends on a single request. Averages of percentiles are also a trap. If one server has a p99 of 10 ms and another has a p99 of 1000 ms, averaging those two gives 505 ms, which describes neither server. Good tools keep histograms of response times, so they can calculate true percentiles across all servers.

```python
import math

values = [50] * 97 + [300, 400, 6000]

def percentile(data, p):
    ordered = sorted(data)
    rank = (p * len(ordered) + 99) // 100
    return ordered[rank - 1]

print("average:", sum(values) / len(values))
for p in (50, 95, 99):
    print("p" + str(p) + ":", percentile(values, p))
```

Output:

```text
average: 115.5
p50: 50
p95: 50
p99: 400

```

The sample program above makes 100 fake requests. Ninety-seven take 50 ms, and three are slow. The average is 115.5 ms, which looks fine. The p99 is 400 ms, which shows the slow group. The biggest request, at 6000 ms, is hidden in the average. The program uses whole-number math for the rank, so the same input always gives the same output.

### Throughput and Error Rate

- **Throughput: work done per second:** Throughput counts how many requests the system handles in a set time, often per second or per minute. It shows how much load the system is carrying. A sudden drop can mean that something has broken, and a sudden spike can mean heavy traffic. Read throughput next to latency. If throughput rises while latency rises too, the system may be running out of room.
- **Error rate: how often requests fail:** Error rate is the share of requests that fail, out of all requests. Failures include server errors, exceptions, and timeouts. Count the failures that are obvious, such as an HTTP 500 response.

> **Tip:** Measure error rate and latency for each important endpoint, not only for the whole service. One broken endpoint can hide inside a healthy-looking total.

### Resource Saturation

Saturation asks how full each resource is. A resource is any part of the machine or the network that does work. Utilization is the share of time a resource is busy. Saturation is how much of a resource is in use and how close the system is to its capacity limit. Watch these resources and their queues:

- **CPU:** The processor does the calculations. When CPU stays near its limit, requests spend longer waiting for their turn to run.
- **Memory:** Memory holds the data a program is working on. When memory runs low, the system may slow down or start swapping data to disk.
- **Disk:** Disk is where data is stored and read. Slow or busy disks make every read and write take longer.
- **Network:** The network carries data between users, servers, and other services. A full network link adds delay before a response even starts.
- **Connection pools:** A connection pool is a set of open links, for example to a database. If every link is in use, new requests must wait in line for one to free up.

Resource numbers work best when read with the user numbers. If latency and CPU both rise together, the CPU is a likely cause. If latency rises while CPU stays low, the wait may be somewhere else, such as a database or an outside service.

## Baselines and Targets

A number means little on its own. Is a p99 of 800 ms bad? It depends on what the system did before and what users need. That is why you first set a baseline, which is a record of normal behavior. With a baseline, you can tell normal from degraded. You can also set targets, which are the speeds you want the system to meet.

### Setting a Baseline

1. **Choose the key paths:** Pick the pages and API calls that matter most to users, such as sign-in, search, or checkout. Do not measure every endpoint at first.
2. **Record the four signals:** For each path, save latency at p50, p95, and p99, plus throughput, error rate, and the main resource numbers.
3. **Collect over normal days:** Gather numbers across several ordinary days, not only one quiet hour. This shows the usual daily shape of the traffic.
4. **Save the result as the baseline:** Write the normal values down. Later, you compare new numbers with this record to decide whether something has changed.

### Using SLIs and SLOs

An SLI is a measured value. A common SLI is the share of requests that finish under a set time, counted as good events divided by total events over a window. An SLO is the target that the SLI should meet over a set window. For example, a team might want 99 percent of checkout requests to finish within a chosen time, measured over 30 days. The SLI is the measured share. The SLO is the goal. A service level agreement, or SLA, is a formal promise to customers. Teams set SLOs as internal goals, and an SLA is usually looser or set by contract.

> **Tip:** Pick SLIs from what users notice. Start with speed and failure, such as latency at p95 and the share of requests that succeed.

### Watching Trends Under Load

One snapshot can mislead you. A system may look fine at 9 a.m. and fail at 8 p.m., when traffic peaks. Plot latency against load, not only against time. If p99 stays flat while traffic grows, the system has room to spare. If p99 starts to climb at a certain traffic level, you have found a practical limit. That point tells you when to add capacity or fix the slow part, long before users complain.

> **Tip:** Compare the same hour on different days.

## How Do You Find the Bottleneck?

Once a metric says something is slow, the next question is where the time goes. A request passes through many parts before the user sees a result. Each part adds its own delay. The part that adds the most extra delay, compared with its baseline, is the likely bottleneck. Finding it means breaking the total time into pieces and comparing each piece with normal.

### Breaking Time Down by Component

- **Client:** The user's device, such as a phone or browser. Slow rendering or heavy scripts can add delay here.
- **Network:** The path between the user and your servers. Distance, congestion, and lost packets all add time.
- **Web server:** The front door that accepts requests and passes them on.
- **App:** The program that runs the business rules. Slow code or waiting on locks shows up here.
- **Database:** The store that holds the data. Slow queries and waiting for locks show up here.
- **External calls:** Calls to other companies or services, such as payment providers. Their delay is outside your control, but you still see it in the breakdown.

Many monitoring tools record time for each part of a request, so you can compare one slow request with the baseline. The code below shows the idea with fixed sample numbers, in milliseconds.

```python
baseline = {"network": 40, "web server": 10, "app": 60, "database": 100}
slow = {"network": 45, "web server": 12, "app": 65, "database": 5000}

for part in slow:
    extra = slow[part] - baseline[part]
    print(part + ": " + str(slow[part]) + " ms, " + str(extra) + " ms over baseline")

print("total: " + str(sum(slow.values())) + " ms")
```

Output:

```text
network: 45 ms, 5 ms over baseline
web server: 12 ms, 2 ms over baseline
app: 65 ms, 5 ms over baseline
database: 5000 ms, 4900 ms over baseline
total: 5122 ms

```

In the sample, only the database part is far above its baseline. The other parts are close to normal. That gap points straight at the database, even though the total time is what the user felt.

### Worked Example: A Slow Checkout Page

Imagine an online store whose checkout page feels slow. Support has received complaints, but the team's average response time looks fine. Here is how the team reads its dashboard, one step at a time.

1. **Read the percentiles:** The checkout page shows a p50 latency of 300 ms, so the typical shopper is served quickly. The p99 is 6 seconds. The slowest 1 percent of checkouts take a long time, and this is the group that complains.
2. **Check errors and throughput:** The error rate stays flat and throughput rises through the afternoon. The problem is speed, not failure, and it grows with traffic.
3. **Compare with the baseline:** The saved baseline for the checkout p99 was much lower. The change is real, not normal noise, so the team starts looking for the cause.
4. **Break down a slow request:** One slow checkout is split into parts. Around 5 seconds sit in a single database query. The network, web server, and app parts are close to their normal values, as the earlier sample showed.
5. **Check the trend under load:** The team finds that this query is fast at quiet hours and slows down only at peak traffic. That pattern suggests the database is running out of capacity when many shoppers arrive at once. The team now knows where to look next. The fix itself is taught in other lessons.

The average would have hidden all of this. The p50 looked healthy, the average was close to normal, and only the p99 and the breakdown showed the real story.

## Check Your Understanding

**Quick check:** Your API's average response time is a healthy 200 ms. Yet users say the app sometimes hangs for several seconds. Which metric should you check to confirm the problem, and why did the average miss it?

A. Average CPU across all servers, because hangs always come from a full processor
B. The error count per hour, because every hang is a failed request
C. The p99 latency, because a small group of very slow requests barely moves the average
D. The p50 median alone, because the middle of the range shows the hangs

**Answer:** C. The p99 shows the slowest 1 percent of requests. A few requests that hang for seconds add little to an average taken over many fast ones, so the average stays healthy while those users wait.

**Quick check:** A sign-up page's p95 latency jumps from 250 ms to 4 seconds, but only in the evening. A breakdown shows the web server and app parts are near their baselines, while one database query grew from 80 ms to 3.9 seconds. Where is the most likely bottleneck?

A. The user's browser, because the client is always the last step
B. The database query, because it is the one part far above its baseline at peak hours
C. The web server, because it receives every request
D. The average CPU of the app, because a rise of 1 percent would matter

**Answer:** B. Comparing each part with its baseline points to the part with the biggest extra delay. Here that is the one query that slows down under evening load.

**Quick check:** A team writes: 99 percent of checkout requests must finish within 800 ms, measured over 30 days. This week the measured share is 99.4 percent. Which item is the target, and which is the measurement?

A. The target is the SLO, and the measured 99.4 percent is the SLI
B. The target is the SLI, and the measured 99.4 percent is the SLO
C. Both are SLAs, because customers can see them
D. Both are baselines, because they describe normal behavior

**Answer:** A. The SLO is the goal set over a window. The SLI is the measured value that is checked against that goal. A baseline records normal behavior, and an SLA is a formal promise with consequences.

## Sources

- [Dynatrace: What are golden signals?](https://www.dynatrace.com/knowledge-base/golden-signals/)
- [Logz.io: What Is APM? Application Performance Monitoring Explained](https://logz.io/glossary/apm/)
- [SigNoz: APM Metrics Explained](https://signoz.io/guides/apm-metrics/)
- [Redis: P99 Latency: What It Means & How to Fix It](https://redis.io/blog/p99-latency/)
- [Cartesia: What is P95 and P99 latency? How to measure it](https://www.cartesia.ai/learn/p95-p99-latency)
- [LightTrace: P95 vs P99: Why Tail Latency Matters](https://light-trace.robomiri.com/blog/p95-vs-p99-latency/)
- [Saif Hatem: Why Average Latency Is a Dangerous Lie](https://saifhatem.substack.com/p/why-average-latency-is-a-dangerous)
- [Zoyla: P95, P99, and Why Averages Lie](https://www.zoyla.app/resources/latency-percentiles-guide)
- [Layrs: Performance Monitoring: Latency, Throughput & Errors](https://layrs.me/course/hld/10-performance-monitoring/performance-monitoring/)
- [Alfy: 5 Key Metrics to Monitor for Optimal Application Health](https://alfy.xyz/posts/5-key-metrics-to-monitor-for-optimal-application-health)
- [Catchpoint: API Performance Monitoring: Key Metrics and Best Practices](https://www.catchpoint.com/api-monitoring-tools/api-performance-monitoring)
- [ClickHouse: SLOs, SLIs, and SLAs: definitions, differences, and worked examples](https://clickhouse.com/resources/engineering/slo-sli-sla)
- [Atlassian: What are Service-Level Objectives (SLOs)?](https://www.atlassian.com/incident-management/kpis/sla-vs-slo-vs-sli)
- [PagerDuty: What's the Difference Between SLAs, SLOs and SLIs?](https://www.pagerduty.com/resources/digital-operations/learn/what-is-slo-sla-sli/)
- [Safeguard: What is a service level objective (SLO) and how do you set one?](https://safeguard.sh/resources/blog/service-level-objective-slo-explained)
- [Wikipedia: Service-level objective](https://en.wikipedia.org/wiki/Service-level_objective)
