Visualization & Alerts: Turning Monitoring Data Into Dashboards and Pages
Learn how to build dashboards people can read in seconds, and alerts that reach the right person only when users are harmed.
In short
Dashboards and alerts turn monitoring data into something people act on: small readable dashboards, and alerts that reach the right person only when users are harmed.
14 min readFree
What You Will Learn
A running system produces a steady stream of numbers: how many requests arrived, how long each one took, and how many failed. Raw numbers are hard to act on. Visualization turns them into charts a reader can scan in seconds. Alerts watch the same numbers and notify a person when something needs action. Together, they turn monitoring data into decisions.
By the end of this lesson you will know how to pick a small set of signals for a dashboard, how to write alerts that point at harm to users, and how to keep alerts from becoming noise. A worked example follows a checkout service from start to finish.
Key words
- Metric
- A number a system records over time, such as requests per second or the share of failed requests.
- Dashboard
- A page of charts that shows a service's key numbers so a person can check its health quickly.
- Panel
- One chart or number box inside a dashboard.
- Threshold
- The level at which a number stops being normal. Crossing it can change a chart's colour or start an alert.
- Percentile
- A value that a set share of requests stay under. The p95 value is the time that 95 out of 100 requests beat.
- Error rate
- The share of requests that fail, usually shown as a percentage of all requests.
- HTTP and 5xx
- HTTP is the web protocol that browsers and servers use. A 5xx status is an HTTP answer with a code from 500 to 599, which means the server failed.
- Saturation
- How full a resource is, such as processor time, memory, or a queue (a line of waiting work).
- Symptom
- Something a user feels, such as a slow page or a failed payment.
- Cause
- An event inside the system that may or may not hurt users, such as one server restarting.
- Alert
- An automatic notification, sent to a person or a queue, when a rule's condition is true.
- Page
- An alert urgent enough to interrupt a person right away, usually by phone.
- On-call
- The person whose turn it is to respond to alerts during a given shift.
- Runbook
- A written guide that tells a responder what to check and do for one alert.
- Alert fatigue
- The state where people ignore alerts because there are too many of them, or many are false.
- Deduplication
- Merging repeated copies of the same alert into one, so one problem does not create many notices.
How Does Raw Data Become Decisions?
Systems produce three common kinds of telemetry, which means data about how they behave. Metrics are numbers recorded over time. Logs are written records of single events. Traces follow one request as it moves through several parts. This lesson works mainly with metrics, because most dashboards and alert rules are built from them. How to collect this data from your code belongs to another lesson.
A dashboard is for looking
It gives a person one place to ask how a service is doing and get a fast answer. Its job is to answer a question, not to display every number the system has.
An alert is for acting
An alert pushes a notice to a person, a ticket queue or an email list when a rule's condition holds. Its purpose is to start a response, so an alert that starts nothing is a weak alert.
Both should agree
A dashboard panel and the alert built on it should describe the same fact about users. When they disagree, people stop trusting both.
How Do You Design a Dashboard?
A useful dashboard is small. Dozens of panels with no question behind them make a reader hunt for the answer. Start with one question, such as whether shoppers can complete a purchase right now, and then choose only the charts that answer it.
Pick a Few Signals
Site Reliability Engineering, or SRE, is the discipline of keeping services running reliably. Google's SRE book gives a short starting list. If you can measure only four things about a user-facing system, measure latency, traffic, errors and saturation. People call these the four golden signals.
Latency
How long a request takes to be served. Keep successful and failed requests apart, because a fast error can pull an average down and make things look better than they are.
Traffic
How much demand the system receives, such as requests per second. Traffic gives the other charts context: a latency rise during a sale means something different from one on a quiet Sunday.
Errors
The rate of requests that fail, usually shown as a percentage of all requests.
Saturation
How full a resource is.
Charts, Time Ranges and Thresholds
Each signal needs the right chart. Latency and error rate suit a line chart over time, because a climb or a drop is easy to see. A single big number suits a current status, but it hides the trend, so pair it with a line.
Chart type
Use lines for anything that changes over time. Reserve big single numbers for the current value.
Time range
Show the last few hours by default so a spike is visible, and offer a longer range for trends. A range that is too short hides slow damage, while one that is too long hides a five-minute outage.
Threshold line
Draw a flat line at the level where the team agrees the service is unhealthy. A reader can then tell healthy from unhealthy in seconds, without doing any arithmetic.
Tip
Give each service its own row, and order the rows the way a request flows through them. A reader then follows the data in the same order it travels.
Quick check
Your checkout dashboard shows an average latency of 120 ms all day. Support says some shoppers wait more than five seconds at peak times. What change would show the problem best?
How Do You Design Alerts?
An alert should exist only when a person can do something useful about it. Good alert design answers three questions: what the alert watches, when it fires, and who receives it.
Why Alert on Symptoms, Not Causes?
A symptom is something a user feels. A cause is an internal event that may or may not hurt users. Grafana's guidance says alerts should mainly detect failures that users can see, because a restarted container matters only when the app slows down or fails. Alert on symptoms, and keep causes on dashboards where engineers can investigate them.
CPU, the share of processor time in use, is the classic trap. A CPU spike can be harmless, and a service can break while its CPU looks calm. A sustained CPU rise may help explain a failure, but it rarely deserves a page on its own. Cause-based alerts tend to produce noise, while a rule such as checkout latency staying high for minutes points at real user harm.
Thresholds, Durations and Severity
A threshold is the value that separates trouble from normal, such as an error rate above 2%. A duration is how long the value must stay past that threshold before the alert fires. Without a duration, one noisy minute can wake someone for nothing.
Severity sorts alerts by urgency. Teams that make every alert critical soon find that nothing stands out, because when everything is urgent, nothing is.
Tip
Before you turn paging on for a new rule, run it quietly and check how often it would have fired. Confirm how it recovers and how repeats are grouped before the first real page.
Routing to an On-Call Owner
Every alert needs a named owner, the team that maintains the rule and answers it. The on-call person is whoever holds the pager for the current shift. Routing decides where an alert goes. Escalation decides who gets it next if the first person does not respond. To acknowledge an alert means the responder tells the system they have taken it.
Owner
Route alerts by service ownership, not to one catch-all rotation that nobody knows well.
Escalation
Escalate as confidence in user impact grows, notifying more responders as sustained or correlated signals build.
Quiet channel
Send low-priority information to a dashboard, an alert list or a team chat channel, where people can read it without being woken.
Before you add any page, ask whether something specific must happen right now if it fires at three in the morning. If the honest answer is no, do not page anyone. Move the alert to a quieter route or remove it.
Worked Example: A Checkout Service
Imagine a checkout service that takes payments for an online shop. The team wants a Grafana dashboard that shows whether shoppers can pay, and an alert that pages the on-call engineer through PagerDuty, a paging service, only when real trouble lasts. Grafana is an open-source tool for charts and alerts. The steps below use ideas that most dashboard tools share.
- 1
Add the two user panels
Create a latency panel that plots p95 latency over time. Create an error panel that plots the 5xx error rate over time. A 5xx response means the server failed the request.
- 2
Draw the threshold lines
On the latency panel, draw a line at the response time the team promised shoppers, such as 2 seconds. On the error panel, draw a line at 2%.
- 3
Write the alert condition
Set the rule to fire when the checkout error rate stays above 2%. Set the pending period to five minutes, so one bad minute cannot page anyone.
- 4
Add the context
Write an annotation, which is a note attached to the alert. It should say what the alert means and link to the checkout dashboard and to the runbook.
- 5
Route it to the on-call engineer
Send the rule to PagerDuty with critical severity, so the on-call engineer is paged. Latency stays on the dashboard for context, and the page depends on the error rule alone.
Alert: checkout-5xx-high
Condition: 5xx error rate above 2 percent of checkout requests
Pending period: 5 minutes
Severity: critical
Route: PagerDuty, checkout on-call
Annotations: what it means, link to checkout dashboard, link to runbookThe next two Python samples copy the logic behind the latency panel and the alert rule. They use only the standard library and print fixed results, so you can run them with python3 -I. They do not connect to Grafana. The first computes a p95 value with the nearest-rank method: sort the values, then take the value at the 95% mark.
latencies_ms = [120, 180, 95, 210, 150, 130, 900, 110, 105, 140,
125, 160, 190, 99, 101, 115, 133, 147, 158, 170]
def p95(values):
ordered = sorted(values)
rank = (95 * len(ordered) + 99) // 100
return ordered[rank - 1]
print("p95 latency (ms):", p95(latencies_ms))Output
p95 latency (ms): 210
Only the 900 ms request is slower than the p95 value. An average of the same numbers would be pulled around by that one request, but the p95 line states plainly how slow the slowest tail is.
The second sample checks the duration rule. It walks through one error rate per minute and fires only after five minutes in a row above 2%.
error_rate_pct = [0.4, 2.5, 3.1, 2.2, 2.8, 2.6, 0.9, 2.4, 2.3, 2.1, 2.7]
THRESHOLD = 2.0
NEEDED_MINUTES = 5
run = 0
fired_at = None
for minute, rate in enumerate(error_rate_pct, start=1):
if rate > THRESHOLD:
run += 1
else:
run = 0
if run == NEEDED_MINUTES and fired_at is None:
fired_at = minute
print("alert fired at minute:", fired_at)Output
alert fired at minute: 6
The first spike, at minute 2, fires nothing on its own. The rule waits until five minutes in a row are above the threshold, which is why the alert fires at minute 6. The duration protects the on-call engineer from short spikes that heal themselves.
How Do You Avoid Alert Fatigue?
Alert fatigue sets in when a team gets so many alerts that people stop reacting to them. A missed alert is bad, but an alert everyone ignores can be worse, because the real page gets skipped along with the noise. When an alert keeps firing with no action, people learn to tune it out, and the important page gets tuned out too.
Tune, Group and Deduplicate
Three techniques cut noise without hiding real problems. Each one works on a different part of the problem.
Tune thresholds
Lengthen a duration or move a threshold when a rule fires on harmless blips. A good threshold catches real harm early and still avoids false alarms.
Group related alerts
When one failure breaks many services, send one grouped incident instead of a separate page for each service.
Deduplicate repeats
A condition that keeps firing should produce one open alert, not a new page every minute. Attach a fixed key to each alert so repeats of the same problem merge.
Tip
Count how many pages each incident creates. If one outage creates ten pages, grouping or deduplication is missing.
Attach Runbooks
A runbook is a written guide for one alert. It explains what the alert means, how to confirm that users are affected, which safe first steps exist, and who to call next. A page that links to its runbook gives the on-call engineer a place to start instead of a blank screen.
Put the essentials in every alert: what happened, which service is affected, the severity, the impact, and links to the dashboard and the runbook.
Tip
If you cannot write a runbook for a noisy alert, pause the rule until you can document what it should do.
Review Alerts That Fired With No Action
Alert quality decays over time, so review it on a schedule. Look at the alerts that paged someone in the last few weeks and ask whether each one led to real work, such as a rollback or a fix. Each noisy alert then ends in one of four ways: it is removed, moved to a lower level, merged into a group, or adjusted.
- 1
Pull the pages
List every page sent during the review period, with the owner of each alert.
- 2
Check for action
For each page, ask whether the responder did something real. If nothing was done, mark the alert as noise.
- 3
Choose a fate
Remove the alert, move it to a dashboard or a ticket, merge it into a group, or change its threshold or duration.
- 4
Watch the rules too
A rule that fails to run sends nothing, which looks exactly like a healthy system. Check rule health regularly, and consider alerting when a rule stops evaluating.
Putting It Together
Visualization and alerts work as a pair. The dashboard gives people a fast picture of user health, and the alert pulls a person in only when that picture crosses a line that matters. Keep the signals few, the thresholds honest, the routes clear, and the review regular.
This lesson stops short of several neighbouring topics. Collecting the data is covered in Instrumentation. Choosing what to measure for speed, uptime, security and feature use has its own lessons. Fixing the problems that alerts reveal belongs to Performance Antipatterns, and controlling overload inside the system belongs to Back Pressure.
Quick check
Your team's alert fires every night at 2 a.m., when CPU briefly reaches 90% during a batch job. A batch job is a scheduled task that processes many records at once. Users notice nothing, and engineers have started ignoring the pages. What should you change, and why?
Tip
Use a simple test for any alert. Name the user harm it prevents, the person who acts on it, and the first step that person takes. If you cannot name all three, do not page anyone with it.
Sources
- Google SRE book, Chapter 6: Monitoring Distributed Systems
- Grafana documentation: Alerting best practices
- Grafana documentation: Grafana dashboard best practices
- Grafana documentation: Best practices for alerting rules (Prometheus alerting app)
- Grafana dashboard best practices (visualizations guide, older copy)
- Grafana tutorial: dashboards, logs and alerts from scratch (Pasquale Pillitteri)
- Observability best practices with Grafana Cloud (Nextlink Labs)
- Alerting overview (Compilenrun Grafana guide)
- Grafana dashboard datasource and alerting operational guide (Meteora Web)
- Monitoring and alerting best practices to reduce alert fatigue (OneUptime)
- How to reduce alert fatigue: a practical playbook (Struct)
- Alert fatigue: how to reduce noise and protect on-call engineers (Rootly)
- Observability settings to reduce alert fatigue (iTechGuides)
- How to reduce alert fatigue: a framework for on-call teams (Datatroops)
- Alert fatigue: causes, real costs, and how to fix it (Nova AIOps)
Last updated on 18 November 2026
Related lessons
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.