esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Performance Monitoring: Measure Speed Under Real Load to Find Slowdowns Early

Learn which speed, traffic, error, and resource numbers to track, why averages hide slow requests, and how to trace a slowdown to one part of a system.

In short

Performance monitoring keeps measuring the real system under real traffic, day after day, so slowdowns show up early and can be traced to one part.

13 min readFree

What You Will Learn

Performance monitoring means measuring how fast and how well a system answers requests, all the time, while real people use it. A car has a speed gauge on the dashboard for the same reason. You do not wait for the engine to stop before you look at it. In this lesson you will learn which numbers to watch, how to decide what normal looks like, and how to use those numbers to find the part of the system that is slow.

Key words

Performance monitoring
Continuous measuring of how fast and how efficiently a system responds, using live traffic.
Latency
The time from when a request arrives to when its response goes out.
Percentile
A cut-off point. The p99 is the time that 99 out of every 100 requests finish within.
Tail
The slowest few percent of requests. Averages often hide them.
Throughput
How much work a system finishes in a set time, such as requests per second.
Error rate
The share of requests that end in a failure, out of all requests.
Saturation
How full a resource is, and how much work is waiting in line for it.
Baseline
A saved record of normal numbers that you compare new numbers against.
SLI (service level indicator)
A measured value that shows how the service is doing, such as the share of fast requests.
SLO (service level objective)
A target for an SLI, set over a fixed time window.
Bottleneck
The one part of a system that limits how fast the whole thing can go.

What Does Performance Monitoring Mean?

A one-time speed test tells you how a system behaved on one day, with one fake load. Performance monitoring is different. It keeps measuring the real system with real traffic, day after day. This matters because speed changes. A page that is fast with ten users can crawl with ten thousand. The slowdown may start small, at busy hours, and grow later.

Monitoring has two jobs. The first is to spot a slowdown early, before users write complaints. The second is to point at the cause. A good monitor does not only say the site is slow. It shows which part is slow, and when that started. Many teams use a short list of signals to do this. Google's site reliability engineers made four of them popular: latency, traffic, errors, and saturation. Most of this lesson is about those four signals and how to read them together.

Monitoring is not the same as the fixes. Once you find a slow part, you may use caching, better queries, or other changes. Those fixes are taught in their own lessons. Here, the goal is to know where the problem is and whether it is getting worse.

What Are the Core Metrics to Track?

A metric is a number that you measure again and again over time. Good performance monitoring picks a few metrics that show how users feel and how the system is holding up. The main ones are latency, throughput, error rate, and resource saturation. Each one answers a different question, so you need all four to get the full picture.

What Are Latency and Percentiles?

Latency is the most watched number, because users feel it directly. The simplest way to summarize latency is an average, which adds up all the times and divides by how many there were. The average is easy to read, but it has a serious weakness. A few very slow requests can be swamped by many fast ones, so the average looks healthy while some users wait a long time.

A percentile fixes this. The p50 is the median: half of requests are faster, and half are slower. The p95 is the time that 95 out of every 100 requests finish within. The p99 is the time that 99 out of every 100 requests finish within, so only the slowest 1 percent are above it. Teams often track p50, p95, and p99 together. If p50 is low and p99 is high, most users are fine, but a small group has a bad time. A healthy system usually has these numbers fairly close together.

Percentiles need enough data to be trusted. If you only have 100 requests, the p99 depends on a single request. Averages of percentiles are also a trap. If one server has a p99 of 10 ms and another has a p99 of 1000 ms, averaging those two gives 505 ms, which describes neither server. Good tools keep histograms of response times, so they can calculate true percentiles across all servers.

Python
import math

values = [50] * 97 + [300, 400, 6000]

def percentile(data, p):
    ordered = sorted(data)
    rank = (p * len(ordered) + 99) // 100
    return ordered[rank - 1]

print("average:", sum(values) / len(values))
for p in (50, 95, 99):
    print("p" + str(p) + ":", percentile(values, p))

Output

average: 115.5
p50: 50
p95: 50
p99: 400

The sample program above makes 100 fake requests. Ninety-seven take 50 ms, and three are slow. The average is 115.5 ms, which looks fine. The p99 is 400 ms, which shows the slow group. The biggest request, at 6000 ms, is hidden in the average. The program uses whole-number math for the rank, so the same input always gives the same output.

Throughput and Error Rate

  • Throughput: work done per second

    Throughput counts how many requests the system handles in a set time, often per second or per minute. It shows how much load the system is carrying. A sudden drop can mean that something has broken, and a sudden spike can mean heavy traffic. Read throughput next to latency. If throughput rises while latency rises too, the system may be running out of room.

  • Error rate: how often requests fail

    Error rate is the share of requests that fail, out of all requests. Failures include server errors, exceptions, and timeouts. Count the failures that are obvious, such as an HTTP 500 response.

Tip

Measure error rate and latency for each important endpoint, not only for the whole service. One broken endpoint can hide inside a healthy-looking total.

Resource Saturation

Saturation asks how full each resource is. A resource is any part of the machine or the network that does work. Utilization is the share of time a resource is busy. Saturation is how much of a resource is in use and how close the system is to its capacity limit. Watch these resources and their queues:

  • CPU

    The processor does the calculations. When CPU stays near its limit, requests spend longer waiting for their turn to run.

  • Memory

    Memory holds the data a program is working on. When memory runs low, the system may slow down or start swapping data to disk.

  • Disk

    Disk is where data is stored and read. Slow or busy disks make every read and write take longer.

  • Network

    The network carries data between users, servers, and other services. A full network link adds delay before a response even starts.

  • Connection pools

    A connection pool is a set of open links, for example to a database. If every link is in use, new requests must wait in line for one to free up.

Resource numbers work best when read with the user numbers. If latency and CPU both rise together, the CPU is a likely cause. If latency rises while CPU stays low, the wait may be somewhere else, such as a database or an outside service.

Baselines and Targets

A number means little on its own. Is a p99 of 800 ms bad? It depends on what the system did before and what users need. That is why you first set a baseline, which is a record of normal behavior. With a baseline, you can tell normal from degraded. You can also set targets, which are the speeds you want the system to meet.

Setting a Baseline

  1. 1

    Choose the key paths

    Pick the pages and API calls that matter most to users, such as sign-in, search, or checkout. Do not measure every endpoint at first.

  2. 2

    Record the four signals

    For each path, save latency at p50, p95, and p99, plus throughput, error rate, and the main resource numbers.

  3. 3

    Collect over normal days

    Gather numbers across several ordinary days, not only one quiet hour. This shows the usual daily shape of the traffic.

  4. 4

    Save the result as the baseline

    Write the normal values down. Later, you compare new numbers with this record to decide whether something has changed.

Using SLIs and SLOs

An SLI is a measured value. A common SLI is the share of requests that finish under a set time, counted as good events divided by total events over a window. An SLO is the target that the SLI should meet over a set window. For example, a team might want 99 percent of checkout requests to finish within a chosen time, measured over 30 days. The SLI is the measured share. The SLO is the goal. A service level agreement, or SLA, is a formal promise to customers. Teams set SLOs as internal goals, and an SLA is usually looser or set by contract.

Tip

Pick SLIs from what users notice. Start with speed and failure, such as latency at p95 and the share of requests that succeed.

How Do You Find the Bottleneck?

Once a metric says something is slow, the next question is where the time goes. A request passes through many parts before the user sees a result. Each part adds its own delay. The part that adds the most extra delay, compared with its baseline, is the likely bottleneck. Finding it means breaking the total time into pieces and comparing each piece with normal.

Breaking Time Down by Component

  • Client

    The user's device, such as a phone or browser. Slow rendering or heavy scripts can add delay here.

  • Network

    The path between the user and your servers. Distance, congestion, and lost packets all add time.

  • Web server

    The front door that accepts requests and passes them on.

  • App

    The program that runs the business rules. Slow code or waiting on locks shows up here.

  • Database

    The store that holds the data. Slow queries and waiting for locks show up here.

  • External calls

    Calls to other companies or services, such as payment providers. Their delay is outside your control, but you still see it in the breakdown.

Many monitoring tools record time for each part of a request, so you can compare one slow request with the baseline. The code below shows the idea with fixed sample numbers, in milliseconds.

Python
baseline = {"network": 40, "web server": 10, "app": 60, "database": 100}
slow = {"network": 45, "web server": 12, "app": 65, "database": 5000}

for part in slow:
    extra = slow[part] - baseline[part]
    print(part + ": " + str(slow[part]) + " ms, " + str(extra) + " ms over baseline")

print("total: " + str(sum(slow.values())) + " ms")

Output

network: 45 ms, 5 ms over baseline
web server: 12 ms, 2 ms over baseline
app: 65 ms, 5 ms over baseline
database: 5000 ms, 4900 ms over baseline
total: 5122 ms

In the sample, only the database part is far above its baseline. The other parts are close to normal. That gap points straight at the database, even though the total time is what the user felt.

Worked Example: A Slow Checkout Page

Imagine an online store whose checkout page feels slow. Support has received complaints, but the team's average response time looks fine. Here is how the team reads its dashboard, one step at a time.

  1. 1

    Read the percentiles

    The checkout page shows a p50 latency of 300 ms, so the typical shopper is served quickly. The p99 is 6 seconds. The slowest 1 percent of checkouts take a long time, and this is the group that complains.

  2. 2

    Check errors and throughput

    The error rate stays flat and throughput rises through the afternoon. The problem is speed, not failure, and it grows with traffic.

  3. 3

    Compare with the baseline

    The saved baseline for the checkout p99 was much lower. The change is real, not normal noise, so the team starts looking for the cause.

  4. 4

    Break down a slow request

    One slow checkout is split into parts. Around 5 seconds sit in a single database query. The network, web server, and app parts are close to their normal values, as the earlier sample showed.

  5. 5

    Check the trend under load

    The team finds that this query is fast at quiet hours and slows down only at peak traffic. That pattern suggests the database is running out of capacity when many shoppers arrive at once. The team now knows where to look next. The fix itself is taught in other lessons.

The average would have hidden all of this. The p50 looked healthy, the average was close to normal, and only the p99 and the breakdown showed the real story.

Check Your Understanding

Quick check

Your API's average response time is a healthy 200 ms. Yet users say the app sometimes hangs for several seconds. Which metric should you check to confirm the problem, and why did the average miss it?

Quick check

A sign-up page's p95 latency jumps from 250 ms to 4 seconds, but only in the evening. A breakdown shows the web server and app parts are near their baselines, while one database query grew from 80 ms to 3.9 seconds. Where is the most likely bottleneck?

Quick check

A team writes: 99 percent of checkout requests must finish within 800 ms, measured over 30 days. This week the measured share is 99.4 percent. Which item is the target, and which is the measurement?

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.