esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Visualization & Alerts: Turning Monitoring Data Into Dashboards and Pages

Learn how to build dashboards people can read in seconds, and alerts that reach the right person only when users are harmed.

In short

Dashboards and alerts turn monitoring data into something people act on: small readable dashboards, and alerts that reach the right person only when users are harmed.

14 min readFree

What You Will Learn

A running system produces a steady stream of numbers: how many requests arrived, how long each one took, and how many failed. Raw numbers are hard to act on. Visualization turns them into charts a reader can scan in seconds. Alerts watch the same numbers and notify a person when something needs action. Together, they turn monitoring data into decisions.

By the end of this lesson you will know how to pick a small set of signals for a dashboard, how to write alerts that point at harm to users, and how to keep alerts from becoming noise. A worked example follows a checkout service from start to finish.

Key words

Metric
A number a system records over time, such as requests per second or the share of failed requests.
Dashboard
A page of charts that shows a service's key numbers so a person can check its health quickly.
Panel
One chart or number box inside a dashboard.
Threshold
The level at which a number stops being normal. Crossing it can change a chart's colour or start an alert.
Percentile
A value that a set share of requests stay under. The p95 value is the time that 95 out of 100 requests beat.
Error rate
The share of requests that fail, usually shown as a percentage of all requests.
HTTP and 5xx
HTTP is the web protocol that browsers and servers use. A 5xx status is an HTTP answer with a code from 500 to 599, which means the server failed.
Saturation
How full a resource is, such as processor time, memory, or a queue (a line of waiting work).
Symptom
Something a user feels, such as a slow page or a failed payment.
Cause
An event inside the system that may or may not hurt users, such as one server restarting.
Alert
An automatic notification, sent to a person or a queue, when a rule's condition is true.
Page
An alert urgent enough to interrupt a person right away, usually by phone.
On-call
The person whose turn it is to respond to alerts during a given shift.
Runbook
A written guide that tells a responder what to check and do for one alert.
Alert fatigue
The state where people ignore alerts because there are too many of them, or many are false.
Deduplication
Merging repeated copies of the same alert into one, so one problem does not create many notices.

How Does Raw Data Become Decisions?

Systems produce three common kinds of telemetry, which means data about how they behave. Metrics are numbers recorded over time. Logs are written records of single events. Traces follow one request as it moves through several parts. This lesson works mainly with metrics, because most dashboards and alert rules are built from them. How to collect this data from your code belongs to another lesson.

  • A dashboard is for looking

    It gives a person one place to ask how a service is doing and get a fast answer. Its job is to answer a question, not to display every number the system has.

  • An alert is for acting

    An alert pushes a notice to a person, a ticket queue or an email list when a rule's condition holds. Its purpose is to start a response, so an alert that starts nothing is a weak alert.

  • Both should agree

    A dashboard panel and the alert built on it should describe the same fact about users. When they disagree, people stop trusting both.

How Do You Design a Dashboard?

A useful dashboard is small. Dozens of panels with no question behind them make a reader hunt for the answer. Start with one question, such as whether shoppers can complete a purchase right now, and then choose only the charts that answer it.

Pick a Few Signals

Site Reliability Engineering, or SRE, is the discipline of keeping services running reliably. Google's SRE book gives a short starting list. If you can measure only four things about a user-facing system, measure latency, traffic, errors and saturation. People call these the four golden signals.

  • Latency

    How long a request takes to be served. Keep successful and failed requests apart, because a fast error can pull an average down and make things look better than they are.

  • Traffic

    How much demand the system receives, such as requests per second. Traffic gives the other charts context: a latency rise during a sale means something different from one on a quiet Sunday.

  • Errors

    The rate of requests that fail, usually shown as a percentage of all requests.

  • Saturation

    How full a resource is.

Charts, Time Ranges and Thresholds

Each signal needs the right chart. Latency and error rate suit a line chart over time, because a climb or a drop is easy to see. A single big number suits a current status, but it hides the trend, so pair it with a line.

  • Chart type

    Use lines for anything that changes over time. Reserve big single numbers for the current value.

  • Time range

    Show the last few hours by default so a spike is visible, and offer a longer range for trends. A range that is too short hides slow damage, while one that is too long hides a five-minute outage.

  • Threshold line

    Draw a flat line at the level where the team agrees the service is unhealthy. A reader can then tell healthy from unhealthy in seconds, without doing any arithmetic.

Tip

Give each service its own row, and order the rows the way a request flows through them. A reader then follows the data in the same order it travels.

Quick check

Your checkout dashboard shows an average latency of 120 ms all day. Support says some shoppers wait more than five seconds at peak times. What change would show the problem best?

How Do You Design Alerts?

An alert should exist only when a person can do something useful about it. Good alert design answers three questions: what the alert watches, when it fires, and who receives it.

Why Alert on Symptoms, Not Causes?

A symptom is something a user feels. A cause is an internal event that may or may not hurt users. Grafana's guidance says alerts should mainly detect failures that users can see, because a restarted container matters only when the app slows down or fails. Alert on symptoms, and keep causes on dashboards where engineers can investigate them.

CPU, the share of processor time in use, is the classic trap. A CPU spike can be harmless, and a service can break while its CPU looks calm. A sustained CPU rise may help explain a failure, but it rarely deserves a page on its own. Cause-based alerts tend to produce noise, while a rule such as checkout latency staying high for minutes points at real user harm.

Thresholds, Durations and Severity

A threshold is the value that separates trouble from normal, such as an error rate above 2%. A duration is how long the value must stay past that threshold before the alert fires. Without a duration, one noisy minute can wake someone for nothing.

Severity sorts alerts by urgency. Teams that make every alert critical soon find that nothing stands out, because when everything is urgent, nothing is.

Tip

Before you turn paging on for a new rule, run it quietly and check how often it would have fired. Confirm how it recovers and how repeats are grouped before the first real page.

Routing to an On-Call Owner

Every alert needs a named owner, the team that maintains the rule and answers it. The on-call person is whoever holds the pager for the current shift. Routing decides where an alert goes. Escalation decides who gets it next if the first person does not respond. To acknowledge an alert means the responder tells the system they have taken it.

  • Owner

    Route alerts by service ownership, not to one catch-all rotation that nobody knows well.

  • Escalation

    Escalate as confidence in user impact grows, notifying more responders as sustained or correlated signals build.

  • Quiet channel

    Send low-priority information to a dashboard, an alert list or a team chat channel, where people can read it without being woken.

Before you add any page, ask whether something specific must happen right now if it fires at three in the morning. If the honest answer is no, do not page anyone. Move the alert to a quieter route or remove it.

Worked Example: A Checkout Service

Imagine a checkout service that takes payments for an online shop. The team wants a Grafana dashboard that shows whether shoppers can pay, and an alert that pages the on-call engineer through PagerDuty, a paging service, only when real trouble lasts. Grafana is an open-source tool for charts and alerts. The steps below use ideas that most dashboard tools share.

  1. 1

    Add the two user panels

    Create a latency panel that plots p95 latency over time. Create an error panel that plots the 5xx error rate over time. A 5xx response means the server failed the request.

  2. 2

    Draw the threshold lines

    On the latency panel, draw a line at the response time the team promised shoppers, such as 2 seconds. On the error panel, draw a line at 2%.

  3. 3

    Write the alert condition

    Set the rule to fire when the checkout error rate stays above 2%. Set the pending period to five minutes, so one bad minute cannot page anyone.

  4. 4

    Add the context

    Write an annotation, which is a note attached to the alert. It should say what the alert means and link to the checkout dashboard and to the runbook.

  5. 5

    Route it to the on-call engineer

    Send the rule to PagerDuty with critical severity, so the on-call engineer is paged. Latency stays on the dashboard for context, and the page depends on the error rule alone.

Text
Alert: checkout-5xx-high
Condition: 5xx error rate above 2 percent of checkout requests
Pending period: 5 minutes
Severity: critical
Route: PagerDuty, checkout on-call
Annotations: what it means, link to checkout dashboard, link to runbook

The next two Python samples copy the logic behind the latency panel and the alert rule. They use only the standard library and print fixed results, so you can run them with python3 -I. They do not connect to Grafana. The first computes a p95 value with the nearest-rank method: sort the values, then take the value at the 95% mark.

Python
latencies_ms = [120, 180, 95, 210, 150, 130, 900, 110, 105, 140,
                125, 160, 190, 99, 101, 115, 133, 147, 158, 170]

def p95(values):
    ordered = sorted(values)
    rank = (95 * len(ordered) + 99) // 100
    return ordered[rank - 1]

print("p95 latency (ms):", p95(latencies_ms))

Output

p95 latency (ms): 210

Only the 900 ms request is slower than the p95 value. An average of the same numbers would be pulled around by that one request, but the p95 line states plainly how slow the slowest tail is.

The second sample checks the duration rule. It walks through one error rate per minute and fires only after five minutes in a row above 2%.

Python
error_rate_pct = [0.4, 2.5, 3.1, 2.2, 2.8, 2.6, 0.9, 2.4, 2.3, 2.1, 2.7]
THRESHOLD = 2.0
NEEDED_MINUTES = 5

run = 0
fired_at = None
for minute, rate in enumerate(error_rate_pct, start=1):
    if rate > THRESHOLD:
        run += 1
    else:
        run = 0
    if run == NEEDED_MINUTES and fired_at is None:
        fired_at = minute
print("alert fired at minute:", fired_at)

Output

alert fired at minute: 6

The first spike, at minute 2, fires nothing on its own. The rule waits until five minutes in a row are above the threshold, which is why the alert fires at minute 6. The duration protects the on-call engineer from short spikes that heal themselves.

How Do You Avoid Alert Fatigue?

Alert fatigue sets in when a team gets so many alerts that people stop reacting to them. A missed alert is bad, but an alert everyone ignores can be worse, because the real page gets skipped along with the noise. When an alert keeps firing with no action, people learn to tune it out, and the important page gets tuned out too.

Tune, Group and Deduplicate

Three techniques cut noise without hiding real problems. Each one works on a different part of the problem.

  • Tune thresholds

    Lengthen a duration or move a threshold when a rule fires on harmless blips. A good threshold catches real harm early and still avoids false alarms.

  • Group related alerts

    When one failure breaks many services, send one grouped incident instead of a separate page for each service.

  • Deduplicate repeats

    A condition that keeps firing should produce one open alert, not a new page every minute. Attach a fixed key to each alert so repeats of the same problem merge.

Tip

Count how many pages each incident creates. If one outage creates ten pages, grouping or deduplication is missing.

Attach Runbooks

A runbook is a written guide for one alert. It explains what the alert means, how to confirm that users are affected, which safe first steps exist, and who to call next. A page that links to its runbook gives the on-call engineer a place to start instead of a blank screen.

Put the essentials in every alert: what happened, which service is affected, the severity, the impact, and links to the dashboard and the runbook.

Tip

If you cannot write a runbook for a noisy alert, pause the rule until you can document what it should do.

Review Alerts That Fired With No Action

Alert quality decays over time, so review it on a schedule. Look at the alerts that paged someone in the last few weeks and ask whether each one led to real work, such as a rollback or a fix. Each noisy alert then ends in one of four ways: it is removed, moved to a lower level, merged into a group, or adjusted.

  1. 1

    Pull the pages

    List every page sent during the review period, with the owner of each alert.

  2. 2

    Check for action

    For each page, ask whether the responder did something real. If nothing was done, mark the alert as noise.

  3. 3

    Choose a fate

    Remove the alert, move it to a dashboard or a ticket, merge it into a group, or change its threshold or duration.

  4. 4

    Watch the rules too

    A rule that fails to run sends nothing, which looks exactly like a healthy system. Check rule health regularly, and consider alerting when a rule stops evaluating.

Putting It Together

Visualization and alerts work as a pair. The dashboard gives people a fast picture of user health, and the alert pulls a person in only when that picture crosses a line that matters. Keep the signals few, the thresholds honest, the routes clear, and the review regular.

This lesson stops short of several neighbouring topics. Collecting the data is covered in Instrumentation. Choosing what to measure for speed, uptime, security and feature use has its own lessons. Fixing the problems that alerts reveal belongs to Performance Antipatterns, and controlling overload inside the system belongs to Back Pressure.

Quick check

Your team's alert fires every night at 2 a.m., when CPU briefly reaches 90% during a batch job. A batch job is a scheduled task that processes many records at once. Users notice nothing, and engineers have started ignoring the pages. What should you change, and why?

Tip

Use a simple test for any alert. Name the user harm it prevents, the person who acts on it, and the first step that person takes. If you cannot name all three, do not page anyone with it.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.