Performance Monitoring: Measure Speed Under Real Load to Find Slowdowns Early
Learn which speed, traffic, error, and resource numbers to track, why averages hide slow requests, and how to trace a slowdown to one part of a system.
In short
Performance monitoring keeps measuring the real system under real traffic, day after day, so slowdowns show up early and can be traced to one part.
13 min readFree
What You Will Learn
Performance monitoring means measuring how fast and how well a system answers requests, all the time, while real people use it. A car has a speed gauge on the dashboard for the same reason. You do not wait for the engine to stop before you look at it. In this lesson you will learn which numbers to watch, how to decide what normal looks like, and how to use those numbers to find the part of the system that is slow.
Key words
- Performance monitoring
- Continuous measuring of how fast and how efficiently a system responds, using live traffic.
- Latency
- The time from when a request arrives to when its response goes out.
- Percentile
- A cut-off point. The p99 is the time that 99 out of every 100 requests finish within.
- Tail
- The slowest few percent of requests. Averages often hide them.
- Throughput
- How much work a system finishes in a set time, such as requests per second.
- Error rate
- The share of requests that end in a failure, out of all requests.
- Saturation
- How full a resource is, and how much work is waiting in line for it.
- Baseline
- A saved record of normal numbers that you compare new numbers against.
- SLI (service level indicator)
- A measured value that shows how the service is doing, such as the share of fast requests.
- SLO (service level objective)
- A target for an SLI, set over a fixed time window.
- Bottleneck
- The one part of a system that limits how fast the whole thing can go.
What Does Performance Monitoring Mean?
A one-time speed test tells you how a system behaved on one day, with one fake load. Performance monitoring is different. It keeps measuring the real system with real traffic, day after day. This matters because speed changes. A page that is fast with ten users can crawl with ten thousand. The slowdown may start small, at busy hours, and grow later.
Monitoring has two jobs. The first is to spot a slowdown early, before users write complaints. The second is to point at the cause. A good monitor does not only say the site is slow. It shows which part is slow, and when that started. Many teams use a short list of signals to do this. Google's site reliability engineers made four of them popular: latency, traffic, errors, and saturation. Most of this lesson is about those four signals and how to read them together.
Monitoring is not the same as the fixes. Once you find a slow part, you may use caching, better queries, or other changes. Those fixes are taught in their own lessons. Here, the goal is to know where the problem is and whether it is getting worse.
What Are the Core Metrics to Track?
A metric is a number that you measure again and again over time. Good performance monitoring picks a few metrics that show how users feel and how the system is holding up. The main ones are latency, throughput, error rate, and resource saturation. Each one answers a different question, so you need all four to get the full picture.
What Are Latency and Percentiles?
Latency is the most watched number, because users feel it directly. The simplest way to summarize latency is an average, which adds up all the times and divides by how many there were. The average is easy to read, but it has a serious weakness. A few very slow requests can be swamped by many fast ones, so the average looks healthy while some users wait a long time.
A percentile fixes this. The p50 is the median: half of requests are faster, and half are slower. The p95 is the time that 95 out of every 100 requests finish within. The p99 is the time that 99 out of every 100 requests finish within, so only the slowest 1 percent are above it. Teams often track p50, p95, and p99 together. If p50 is low and p99 is high, most users are fine, but a small group has a bad time. A healthy system usually has these numbers fairly close together.
Percentiles need enough data to be trusted. If you only have 100 requests, the p99 depends on a single request. Averages of percentiles are also a trap. If one server has a p99 of 10 ms and another has a p99 of 1000 ms, averaging those two gives 505 ms, which describes neither server. Good tools keep histograms of response times, so they can calculate true percentiles across all servers.
import math
values = [50] * 97 + [300, 400, 6000]
def percentile(data, p):
ordered = sorted(data)
rank = (p * len(ordered) + 99) // 100
return ordered[rank - 1]
print("average:", sum(values) / len(values))
for p in (50, 95, 99):
print("p" + str(p) + ":", percentile(values, p))Output
average: 115.5 p50: 50 p95: 50 p99: 400
The sample program above makes 100 fake requests. Ninety-seven take 50 ms, and three are slow. The average is 115.5 ms, which looks fine. The p99 is 400 ms, which shows the slow group. The biggest request, at 6000 ms, is hidden in the average. The program uses whole-number math for the rank, so the same input always gives the same output.
Throughput and Error Rate
Throughput: work done per second
Throughput counts how many requests the system handles in a set time, often per second or per minute. It shows how much load the system is carrying. A sudden drop can mean that something has broken, and a sudden spike can mean heavy traffic. Read throughput next to latency. If throughput rises while latency rises too, the system may be running out of room.
Error rate: how often requests fail
Error rate is the share of requests that fail, out of all requests. Failures include server errors, exceptions, and timeouts. Count the failures that are obvious, such as an HTTP 500 response.
Tip
Measure error rate and latency for each important endpoint, not only for the whole service. One broken endpoint can hide inside a healthy-looking total.
Resource Saturation
Saturation asks how full each resource is. A resource is any part of the machine or the network that does work. Utilization is the share of time a resource is busy. Saturation is how much of a resource is in use and how close the system is to its capacity limit. Watch these resources and their queues:
CPU
The processor does the calculations. When CPU stays near its limit, requests spend longer waiting for their turn to run.
Memory
Memory holds the data a program is working on. When memory runs low, the system may slow down or start swapping data to disk.
Disk
Disk is where data is stored and read. Slow or busy disks make every read and write take longer.
Network
The network carries data between users, servers, and other services. A full network link adds delay before a response even starts.
Connection pools
A connection pool is a set of open links, for example to a database. If every link is in use, new requests must wait in line for one to free up.
Resource numbers work best when read with the user numbers. If latency and CPU both rise together, the CPU is a likely cause. If latency rises while CPU stays low, the wait may be somewhere else, such as a database or an outside service.
Baselines and Targets
A number means little on its own. Is a p99 of 800 ms bad? It depends on what the system did before and what users need. That is why you first set a baseline, which is a record of normal behavior. With a baseline, you can tell normal from degraded. You can also set targets, which are the speeds you want the system to meet.
Setting a Baseline
- 1
Choose the key paths
Pick the pages and API calls that matter most to users, such as sign-in, search, or checkout. Do not measure every endpoint at first.
- 2
Record the four signals
For each path, save latency at p50, p95, and p99, plus throughput, error rate, and the main resource numbers.
- 3
Collect over normal days
Gather numbers across several ordinary days, not only one quiet hour. This shows the usual daily shape of the traffic.
- 4
Save the result as the baseline
Write the normal values down. Later, you compare new numbers with this record to decide whether something has changed.
Using SLIs and SLOs
An SLI is a measured value. A common SLI is the share of requests that finish under a set time, counted as good events divided by total events over a window. An SLO is the target that the SLI should meet over a set window. For example, a team might want 99 percent of checkout requests to finish within a chosen time, measured over 30 days. The SLI is the measured share. The SLO is the goal. A service level agreement, or SLA, is a formal promise to customers. Teams set SLOs as internal goals, and an SLA is usually looser or set by contract.
Tip
Pick SLIs from what users notice. Start with speed and failure, such as latency at p95 and the share of requests that succeed.
Watching Trends Under Load
One snapshot can mislead you. A system may look fine at 9 a.m. and fail at 8 p.m., when traffic peaks. Plot latency against load, not only against time. If p99 stays flat while traffic grows, the system has room to spare. If p99 starts to climb at a certain traffic level, you have found a practical limit. That point tells you when to add capacity or fix the slow part, long before users complain.
Tip
Compare the same hour on different days.
How Do You Find the Bottleneck?
Once a metric says something is slow, the next question is where the time goes. A request passes through many parts before the user sees a result. Each part adds its own delay. The part that adds the most extra delay, compared with its baseline, is the likely bottleneck. Finding it means breaking the total time into pieces and comparing each piece with normal.
Breaking Time Down by Component
Client
The user's device, such as a phone or browser. Slow rendering or heavy scripts can add delay here.
Network
The path between the user and your servers. Distance, congestion, and lost packets all add time.
Web server
The front door that accepts requests and passes them on.
App
The program that runs the business rules. Slow code or waiting on locks shows up here.
Database
The store that holds the data. Slow queries and waiting for locks show up here.
External calls
Calls to other companies or services, such as payment providers. Their delay is outside your control, but you still see it in the breakdown.
Many monitoring tools record time for each part of a request, so you can compare one slow request with the baseline. The code below shows the idea with fixed sample numbers, in milliseconds.
baseline = {"network": 40, "web server": 10, "app": 60, "database": 100}
slow = {"network": 45, "web server": 12, "app": 65, "database": 5000}
for part in slow:
extra = slow[part] - baseline[part]
print(part + ": " + str(slow[part]) + " ms, " + str(extra) + " ms over baseline")
print("total: " + str(sum(slow.values())) + " ms")Output
network: 45 ms, 5 ms over baseline web server: 12 ms, 2 ms over baseline app: 65 ms, 5 ms over baseline database: 5000 ms, 4900 ms over baseline total: 5122 ms
In the sample, only the database part is far above its baseline. The other parts are close to normal. That gap points straight at the database, even though the total time is what the user felt.
Worked Example: A Slow Checkout Page
Imagine an online store whose checkout page feels slow. Support has received complaints, but the team's average response time looks fine. Here is how the team reads its dashboard, one step at a time.
- 1
Read the percentiles
The checkout page shows a p50 latency of 300 ms, so the typical shopper is served quickly. The p99 is 6 seconds. The slowest 1 percent of checkouts take a long time, and this is the group that complains.
- 2
Check errors and throughput
The error rate stays flat and throughput rises through the afternoon. The problem is speed, not failure, and it grows with traffic.
- 3
Compare with the baseline
The saved baseline for the checkout p99 was much lower. The change is real, not normal noise, so the team starts looking for the cause.
- 4
Break down a slow request
One slow checkout is split into parts. Around 5 seconds sit in a single database query. The network, web server, and app parts are close to their normal values, as the earlier sample showed.
- 5
Check the trend under load
The team finds that this query is fast at quiet hours and slows down only at peak traffic. That pattern suggests the database is running out of capacity when many shoppers arrive at once. The team now knows where to look next. The fix itself is taught in other lessons.
The average would have hidden all of this. The p50 looked healthy, the average was close to normal, and only the p99 and the breakdown showed the real story.
Check Your Understanding
Quick check
Your API's average response time is a healthy 200 ms. Yet users say the app sometimes hangs for several seconds. Which metric should you check to confirm the problem, and why did the average miss it?
Quick check
A sign-up page's p95 latency jumps from 250 ms to 4 seconds, but only in the evening. A breakdown shows the web server and app parts are near their baselines, while one database query grew from 80 ms to 3.9 seconds. Where is the most likely bottleneck?
Quick check
A team writes: 99 percent of checkout requests must finish within 800 ms, measured over 30 days. This week the measured share is 99.4 percent. Which item is the target, and which is the measurement?
Sources
- Dynatrace: What are golden signals?
- Logz.io: What Is APM? Application Performance Monitoring Explained
- SigNoz: APM Metrics Explained
- Redis: P99 Latency: What It Means & How to Fix It
- Cartesia: What is P95 and P99 latency? How to measure it
- LightTrace: P95 vs P99: Why Tail Latency Matters
- Saif Hatem: Why Average Latency Is a Dangerous Lie
- Zoyla: P95, P99, and Why Averages Lie
- Layrs: Performance Monitoring: Latency, Throughput & Errors
- Alfy: 5 Key Metrics to Monitor for Optimal Application Health
- Catchpoint: API Performance Monitoring: Key Metrics and Best Practices
- ClickHouse: SLOs, SLIs, and SLAs: definitions, differences, and worked examples
- Atlassian: What are Service-Level Objectives (SLOs)?
- PagerDuty: What's the Difference Between SLAs, SLOs and SLIs?
- Safeguard: What is a service level objective (SLO) and how do you set one?
- Wikipedia: Service-level objective
Last updated on 18 November 2026
Related lessons
- System designPerformance Antipatterns: Common Design Mistakes That Slow Systems Under Load
- System designSecurity Monitoring: Watching Signals to Spot Attacks Early
- System designHealth Monitoring: Catch Failing Parts Before Users Notice
- System designAvailability Monitoring: Checking Whether a System Is Up and Usable
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.