Measuring the Impact: Check Whether a Design Change Really Worked
Learn how to set a goal, record a baseline, compare fairly, and decide with evidence instead of opinion.
In short
Measuring impact means setting a goal, recording a baseline and comparing fairly, so you decide whether a design change worked using evidence instead of opinion.
14 min readFree
What you will learn
Every design change is a guess until you check it. Measuring the impact means checking whether a change actually moved the behavior it was meant to move. You compare real user data from before and after the change, or against a group that did not get the change. This lesson shows you how to set a goal, record a starting point, choose the right numbers, compare fairly, and decide what to do next. By the end you should be able to tell a real improvement from a number that only looks good.
Key words
- Measuring the impact
- Checking whether a design change moved the behavior or result it was meant to change, using data instead of opinions.
- Baseline
- The measured starting point, taken before you change anything. It tells you what normal looks like.
- Primary metric
- The one main number that tells you whether the change worked.
- Behavioral data
- Records of what people actually do, such as clicks, finished tasks, and return visits.
- Attitudinal data
- What people say about how they feel, usually gathered with surveys or interviews.
- Vanity metric
- A number that looks impressive but cannot show whether the design got better.
- Control group
- A set of users who do not get the change, so you can compare them with users who do.
- Retention
- How many users come back after a set time, such as a week or a month.
- Correlation
- Two things that move together. It does not prove one caused the other.
- Causation
- One thing actually makes another thing happen.
Why start with a goal and a baseline?
Before you change anything, write one plain sentence that says what the change should do for the user. For example: more people finish checkout without giving up. Then choose one primary metric that tracks that behavior directly. A common choice is checkout completion rate.
Next, measure where that number stands today. This is your baseline. Without it, a later number means little, because you cannot tell whether it is better, worse, or just normal for your product. A clear baseline makes later results meaningful, and the baseline should be taken before the design work starts.
- 1
Write the goal
Describe the change in one plain sentence, such as: users should reach the end of setup without help.
- 2
Pick one primary metric
Choose a number that reflects the user behavior you want to change. Ask whether a better number would clearly mean a better experience.
- 3
Record the baseline
Measure the current value of that metric before you launch anything. Write down the date range you used.
- 4
Set a target and a time frame
Decide how much change would count as a win, and over what period you will look.
- 5
List guardrail metrics
Name a few numbers that must not get worse, so you notice side effects early.
Tip
Write the goal, the primary metric and the baseline down before launch. If you choose the metric after you see the results, you are more likely to pick the number that happened to rise.
Choosing one primary metric matters because it keeps you honest. Tracking many numbers invites you to report the one that moved in the right direction. Pick one, commit to it, and treat the others as supporting checks.
Without baseline numbers, nobody can tell whether a new layout reduces drop-off or simply looks nicer to the team that built it. The baseline is the honest yardstick you compare against.
How do you pick the right kind of metric?
Metrics come in two broad kinds. Behavioral metrics show what people do. Attitudinal metrics show what people think and feel. Both are useful, but they answer different questions.
Behavioral data
What users physically do, such as completion rates, error rates, drop-off rates and return visits. These usually come from analytics tools.
Attitudinal data
What users say, such as how easy a task felt, how satisfied they were, or how confident they were. These usually come from surveys.
Outcome data
Business results that a design can influence, such as the share of visitors who buy, or the share of new users who are still active later.
For proving that a change worked, behavioral data is usually the stronger evidence. Behavioral numbers tell you what happened. Attitudinal numbers tell you how it felt. The second is harder to read as proof, because feelings can shift for many reasons.
Attitudinal data still has a job. When a behavior number moves, a few interviews or survey answers can help you understand why. Pair the numbers with a handful of user conversations so you know the reason, not just the result.
Tip
Ask one question about any candidate metric: if this number improves, does that really mean users have a better experience with what I changed? If the answer needs several leaps of logic, the metric is probably too indirect.
A checklist of categories can help you think wider. Google researchers published a model called HEART in 2010, and it is used to organize user experience metrics into five groups.
Happiness
How satisfied users say they are.
Engagement
How often and how deeply people use the product in a set time.
Adoption
How many new users start using a product or feature.
Retention
How many users keep coming back over time.
Task success
Whether users finish the job they came to do.
Not every group fits every design. Use the list to check your thinking, then pick the one or two that match your goal. Treat a click as a step, not as success. A user who clicks a button has not necessarily finished the task.
Vanity metrics are the numbers that look good on a slide but cannot tell you whether the design improved. They are the most common trap in this kind of work.
A vanity metric often only goes up. Total users, app downloads, page views and shares are typical examples. They feel like progress, but they do not explain how users experience the product.
A better habit is to turn a raw count into a rate. Instead of counting downloads, measure the share of downloads that become active users within a set time. A rate can show a real change in the experience, because it is not just growing with the audience.
A metric is actionable when a change in it points you to the next thing to fix. If a number drops, you should know roughly where to look.
Watch for side effects. A redesign can raise one number while hurting another. For example, a change might bring in more sign-ups but fewer of those new people are still using the app a month later. If you only count sign-ups, you would call it a win and miss the problem.
Teams guard against this by naming guardrail metrics. These are numbers that must not get worse while you chase the primary metric. Check them every time you read the results.
How do you compare fairly and read the result carefully?
A number only means something when you compare it with another number. The three most common comparisons are below. Each is useful, but each has a weakness you need to watch.
Before and after
You compare the baseline with the period after launch. It is simple, but it fails if something else changed at the same time.
Control group
Some users keep the old design while others get the new one, and both groups are measured over the same days. This is the fairer test.
Comparison period
You compare the change period with a similar stretch of time that had no change, such as a comparable week. This helps, but seasonal events can still skew it.
Each comparison answers a slightly different question. Before and after asks whether things moved. A control group asks whether the change caused the movement. Be clear about which one you are answering.
Both groups then face the same news, prices, weather and sales. Any gap between them is much more likely to come from the design. Controlled comparisons like this are the basis of A/B testing, which has its own lesson.
Collect enough users or enough time before you trust the result. A small difference can appear by luck. Statistical significance is a check for this. It asks whether a result is unlikely to be due to chance, and it is often tested against a threshold of 0.05.
Statistical significance is not the same as importance. A very large sample can make a tiny change look reliable even when nobody would notice it. Ask two questions: is the result reliable, and is it big enough to matter to users and the business?
The amount of data you need depends on your baseline and the size of change you hope to detect.
Tip
When you choose a time frame, pick the shortest one that still gives a fairly stable rate. Very short periods bring random ups and downs, and very long periods slow down feedback.
How do you separate correlation from causation?
Correlation means two numbers move together. Causation means one thing actually produces the other. After a redesign, sales may rise, but that does not prove the redesign did it. The sales rise and the redesign might both have a different cause.
The outside factor that could explain a result is sometimes called a confounder. Common confounders include a holiday, a price cut, a seasonal sale, a marketing email, a bug fix, or a new competitor. Any of these can move the numbers during the same weeks as your change.
- 1
List everything else that changed
Write down every event in the same window: promotions, emails, releases, outages, holidays and news.
- 2
Compare with a control group
If the users who did not get the change also rose, the outside factor is a likely cause, not your design.
- 3
Check a second period
Look at a time without the event, or a repeat of the test. A real design effect should still appear.
- 4
Check the guardrails
Make sure the side effects did not grow while the main number rose.
Tip
Be suspicious of any result that arrives in the same week as a big event. Say the change was linked to the result until you have ruled out the event.
Worked example: a fitness app's weekly goal reminder
A fitness app adds a weekly goal reminder to its home screen. The reminder shows how many workouts a person has left to reach their weekly goal. The designer wants to know whether the reminder helps people log a workout within their first seven days. The steps below show how the designer checks it.
- 1
Set the goal
The goal is for more new users to log a workout early, so they build the habit while they are still new.
- 2
Choose the primary metric
The primary metric is the share of users who log a workout within seven days. This is a behavioral measure, and it is close to the behavior the reminder is meant to change.
- 3
Record the baseline
Before the reminder goes live, the designer measures the same share for users who joined before the change. That is the starting point.
- 4
Compare the groups
The designer compares the baseline with the group who saw the reminder. A stronger version would also split new users so some see the reminder and some do not, at the same time.
- 5
Check the guardrail
The designer also checks whether notification opt-outs went up. If more people turned notifications off, the reminder may be annoying some users, even if the workout share rose.
- 6
Look for confounders
The designer checks whether a New Year campaign, a new free trial, or a seasonal challenge started in the same weeks.
- 7
Decide
If the share rose clearly and opt-outs stayed flat, keep the reminder. If opt-outs jumped, change the timing or wording. If the share did not move, drop it or test a different idea.
Notice what the designer did not do. They did not decide from one screenshot or from the number of total logins. They chose one primary metric, recorded a baseline first, watched a guardrail, and checked for outside events before acting.
Tip
The reminder itself is one tactic. This lesson only shows how to measure whether it worked. Other lessons cover specific tactics such as goal trackers and reminders.
How do you decide to keep, change or drop?
Once you have a fair comparison, you can make one of three decisions. Each depends on the primary metric, the guardrails, and the size of the effect.
Keep
The primary metric improved, the change is reliable, and the guardrails held. Keep the design and keep watching the numbers.
Change
The goal was right, but a side effect appeared or the effect was weak. Adjust the design and measure again from a new baseline.
Drop
The primary metric did not move, or it moved while a serious guardrail got worse. Remove the change and record what you learned.
Be careful with effects that are real but tiny. A result can be reliable and still be too small to justify extra engineering work or new complexity. Ask whether the gain is worth the cost.
A small test can also show a big drop that is not yet proven. In that case, the drop is a reason to investigate further, not a reason to celebrate or to ignore it.
Check your thinking
Quick check
After a checkout redesign, your team sees purchases rise by 10% in the week after launch. That same week included a holiday sale. Can you say the redesign caused the rise, and what should you do to find out?
Quick check
A news app changes its article page to show a save-for-later button. More people reach the sign-up page afterwards, but fewer of the new sign-ups come back a month later. What is the best next step?
Quick check
In a survey, users say a new settings layout feels faster. But the share of users who finish their settings task has not changed. Which result should decide whether the change worked?
Summary
Measuring the impact turns a design idea into evidence. Start with a goal and one primary metric. Record a baseline before the change. Prefer behavioral data, watch vanity metrics and guardrails, and compare against a control group or a fair period. Collect enough users or time, and separate correlation from causation before you decide to keep, change or drop the design.
Tip
Keep a simple log of each change: the goal, the primary metric, the baseline, the comparison, the guardrails, and the decision. Over time, this log tells you more than any single report.
Sources
- Nielsen Norman Group: Your New UX Habit, Establishing Baselines for Impact
- Nielsen Norman Group: Vanity Metrics, Add Context to Add Meaning
- Nielsen Norman Group: Statistical Significance Isn't the Same as Practical Significance
- Kerry Rodden: The HEART framework for UX metrics
- Equinet Academy: UX Metrics, How to Measure the Success of a Design
- Nectar Labs: Data-Driven UX, Measuring Design Success
- Lyssna: UX Success Metrics
- Interaction Design Foundation: What is the HEART Framework?
- Koji: HEART Framework, Google's 5-Metric Model for Measuring User Experience
- Fuselab Creative: UX Metrics, How to Measure UX in Enterprise Products
- Medium (Design Bootcamp): Mastering UX metrics, a comprehensive guide
Last updated on 17 November 2026
Related lessons
- Design systemsService and Health Metrics: Measuring Whether a Design System Is Working
- UI/UX basicsMultivariate Testing: Try Many Design Changes at Once
- UI/UX basicsIncremental A/B Testing: Improve a Design One Small Change at a Time
- UI/UX basicsPersuasive Technology: Design That Changes What People Do
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.