# Measuring the Impact: Check Whether a Design Change Really Worked

> Learn how to set a goal, record a baseline, compare fairly, and decide with evidence instead of opinion.

**In short:** Measuring impact means setting a goal, recording a baseline and comparing fairly, so you decide whether a design change worked using evidence instead of opinion.

From "UI/UX basics", a free course by Venkat R. Murty: https://designervenkat.online/learn/ui-ux/measuring-the-impact

## What you will learn

Every design change is a guess until you check it. Measuring the impact means checking whether a change actually moved the behavior it was meant to move. You compare real user data from before and after the change, or against a group that did not get the change. This lesson shows you how to set a goal, record a starting point, choose the right numbers, compare fairly, and decide what to do next. By the end you should be able to tell a real improvement from a number that only looks good.

- **Measuring the impact:** Checking whether a design change moved the behavior or result it was meant to change, using data instead of opinions.
- **Baseline:** The measured starting point, taken before you change anything. It tells you what normal looks like.
- **Primary metric:** The one main number that tells you whether the change worked.
- **Behavioral data:** Records of what people actually do, such as clicks, finished tasks, and return visits.
- **Attitudinal data:** What people say about how they feel, usually gathered with surveys or interviews.
- **Vanity metric:** A number that looks impressive but cannot show whether the design got better.
- **Control group:** A set of users who do not get the change, so you can compare them with users who do.
- **Retention:** How many users come back after a set time, such as a week or a month.
- **Correlation:** Two things that move together. It does not prove one caused the other.
- **Causation:** One thing actually makes another thing happen.

## Why start with a goal and a baseline?

Before you change anything, write one plain sentence that says what the change should do for the user. For example: more people finish checkout without giving up. Then choose one primary metric that tracks that behavior directly. A common choice is checkout completion rate.

Next, measure where that number stands today. This is your baseline. Without it, a later number means little, because you cannot tell whether it is better, worse, or just normal for your product. A clear baseline makes later results meaningful, and the baseline should be taken before the design work starts.

1. **Write the goal:** Describe the change in one plain sentence, such as: users should reach the end of setup without help.
2. **Pick one primary metric:** Choose a number that reflects the user behavior you want to change. Ask whether a better number would clearly mean a better experience.
3. **Record the baseline:** Measure the current value of that metric before you launch anything. Write down the date range you used.
4. **Set a target and a time frame:** Decide how much change would count as a win, and over what period you will look.
5. **List guardrail metrics:** Name a few numbers that must not get worse, so you notice side effects early.

> **Tip:** Write the goal, the primary metric and the baseline down before launch. If you choose the metric after you see the results, you are more likely to pick the number that happened to rise.

Choosing one primary metric matters because it keeps you honest. Tracking many numbers invites you to report the one that moved in the right direction. Pick one, commit to it, and treat the others as supporting checks.

Without baseline numbers, nobody can tell whether a new layout reduces drop-off or simply looks nicer to the team that built it. The baseline is the honest yardstick you compare against.

## How do you pick the right kind of metric?

Metrics come in two broad kinds. Behavioral metrics show what people do. Attitudinal metrics show what people think and feel. Both are useful, but they answer different questions.

- **Behavioral data:** What users physically do, such as completion rates, error rates, drop-off rates and return visits. These usually come from analytics tools.
- **Attitudinal data:** What users say, such as how easy a task felt, how satisfied they were, or how confident they were. These usually come from surveys.
- **Outcome data:** Business results that a design can influence, such as the share of visitors who buy, or the share of new users who are still active later.

For proving that a change worked, behavioral data is usually the stronger evidence. Behavioral numbers tell you what happened. Attitudinal numbers tell you how it felt. The second is harder to read as proof, because feelings can shift for many reasons.

Attitudinal data still has a job. When a behavior number moves, a few interviews or survey answers can help you understand why. Pair the numbers with a handful of user conversations so you know the reason, not just the result.

> **Tip:** Ask one question about any candidate metric: if this number improves, does that really mean users have a better experience with what I changed? If the answer needs several leaps of logic, the metric is probably too indirect.

A checklist of categories can help you think wider. Google researchers published a model called HEART in 2010, and it is used to organize user experience metrics into five groups.

- **Happiness:** How satisfied users say they are.
- **Engagement:** How often and how deeply people use the product in a set time.
- **Adoption:** How many new users start using a product or feature.
- **Retention:** How many users keep coming back over time.
- **Task success:** Whether users finish the job they came to do.

Not every group fits every design. Use the list to check your thinking, then pick the one or two that match your goal. Treat a click as a step, not as success. A user who clicks a button has not necessarily finished the task.

Vanity metrics are the numbers that look good on a slide but cannot tell you whether the design improved. They are the most common trap in this kind of work.

A vanity metric often only goes up. Total users, app downloads, page views and shares are typical examples. They feel like progress, but they do not explain how users experience the product.

A better habit is to turn a raw count into a rate. Instead of counting downloads, measure the share of downloads that become active users within a set time. A rate can show a real change in the experience, because it is not just growing with the audience.

A metric is actionable when a change in it points you to the next thing to fix. If a number drops, you should know roughly where to look.

Watch for side effects. A redesign can raise one number while hurting another. For example, a change might bring in more sign-ups but fewer of those new people are still using the app a month later. If you only count sign-ups, you would call it a win and miss the problem.

Teams guard against this by naming guardrail metrics. These are numbers that must not get worse while you chase the primary metric. Check them every time you read the results.

## How do you compare fairly and read the result carefully?

A number only means something when you compare it with another number. The three most common comparisons are below. Each is useful, but each has a weakness you need to watch.

- **Before and after:** You compare the baseline with the period after launch. It is simple, but it fails if something else changed at the same time.
- **Control group:** Some users keep the old design while others get the new one, and both groups are measured over the same days. This is the fairer test.
- **Comparison period:** You compare the change period with a similar stretch of time that had no change, such as a comparable week. This helps, but seasonal events can still skew it.

Each comparison answers a slightly different question. Before and after asks whether things moved. A control group asks whether the change caused the movement. Be clear about which one you are answering.

Both groups then face the same news, prices, weather and sales. Any gap between them is much more likely to come from the design. Controlled comparisons like this are the basis of A/B testing, which has its own lesson.

Collect enough users or enough time before you trust the result. A small difference can appear by luck. Statistical significance is a check for this. It asks whether a result is unlikely to be due to chance, and it is often tested against a threshold of 0.05.

Statistical significance is not the same as importance. A very large sample can make a tiny change look reliable even when nobody would notice it. Ask two questions: is the result reliable, and is it big enough to matter to users and the business?

The amount of data you need depends on your baseline and the size of change you hope to detect.

> **Tip:** When you choose a time frame, pick the shortest one that still gives a fairly stable rate. Very short periods bring random ups and downs, and very long periods slow down feedback.

## How do you separate correlation from causation?

Correlation means two numbers move together. Causation means one thing actually produces the other. After a redesign, sales may rise, but that does not prove the redesign did it. The sales rise and the redesign might both have a different cause.

The outside factor that could explain a result is sometimes called a confounder. Common confounders include a holiday, a price cut, a seasonal sale, a marketing email, a bug fix, or a new competitor. Any of these can move the numbers during the same weeks as your change.

1. **List everything else that changed:** Write down every event in the same window: promotions, emails, releases, outages, holidays and news.
2. **Compare with a control group:** If the users who did not get the change also rose, the outside factor is a likely cause, not your design.
3. **Check a second period:** Look at a time without the event, or a repeat of the test. A real design effect should still appear.
4. **Check the guardrails:** Make sure the side effects did not grow while the main number rose.

> **Tip:** Be suspicious of any result that arrives in the same week as a big event. Say the change was linked to the result until you have ruled out the event.

## Worked example: a fitness app's weekly goal reminder

A fitness app adds a weekly goal reminder to its home screen. The reminder shows how many workouts a person has left to reach their weekly goal. The designer wants to know whether the reminder helps people log a workout within their first seven days. The steps below show how the designer checks it.

1. **Set the goal:** The goal is for more new users to log a workout early, so they build the habit while they are still new.
2. **Choose the primary metric:** The primary metric is the share of users who log a workout within seven days. This is a behavioral measure, and it is close to the behavior the reminder is meant to change.
3. **Record the baseline:** Before the reminder goes live, the designer measures the same share for users who joined before the change. That is the starting point.
4. **Compare the groups:** The designer compares the baseline with the group who saw the reminder. A stronger version would also split new users so some see the reminder and some do not, at the same time.
5. **Check the guardrail:** The designer also checks whether notification opt-outs went up. If more people turned notifications off, the reminder may be annoying some users, even if the workout share rose.
6. **Look for confounders:** The designer checks whether a New Year campaign, a new free trial, or a seasonal challenge started in the same weeks.
7. **Decide:** If the share rose clearly and opt-outs stayed flat, keep the reminder. If opt-outs jumped, change the timing or wording. If the share did not move, drop it or test a different idea.

Notice what the designer did not do. They did not decide from one screenshot or from the number of total logins. They chose one primary metric, recorded a baseline first, watched a guardrail, and checked for outside events before acting.

> **Tip:** The reminder itself is one tactic. This lesson only shows how to measure whether it worked. Other lessons cover specific tactics such as goal trackers and reminders.

## How do you decide to keep, change or drop?

Once you have a fair comparison, you can make one of three decisions. Each depends on the primary metric, the guardrails, and the size of the effect.

- **Keep:** The primary metric improved, the change is reliable, and the guardrails held. Keep the design and keep watching the numbers.
- **Change:** The goal was right, but a side effect appeared or the effect was weak. Adjust the design and measure again from a new baseline.
- **Drop:** The primary metric did not move, or it moved while a serious guardrail got worse. Remove the change and record what you learned.

Be careful with effects that are real but tiny. A result can be reliable and still be too small to justify extra engineering work or new complexity. Ask whether the gain is worth the cost.

A small test can also show a big drop that is not yet proven. In that case, the drop is a reason to investigate further, not a reason to celebrate or to ignore it.

## Check your thinking

**Quick check:** After a checkout redesign, your team sees purchases rise by 10% in the week after launch. That same week included a holiday sale. Can you say the redesign caused the rise, and what should you do to find out?

A. Yes. The rise came right after launch, so the redesign must have caused it.
B. Not yet. The sale is a rival explanation, so compare with a control group or a similar period without the redesign.
C. Yes, as long as the sale was small, because small sales do not change buying.
D. No, purchases can never show whether a design works, so stop measuring them.

**Answer:** B. Timing alone does not prove cause. The holiday sale is a confounder that could explain the rise. A control group or a comparable period without the redesign would show how much of the rise the design can claim.

**Quick check:** A news app changes its article page to show a save-for-later button. More people reach the sign-up page afterwards, but fewer of the new sign-ups come back a month later. What is the best next step?

A. Announce a win, because more people reached the sign-up page.
B. Treat the lower return rate as a side effect, and compare it with a control group before deciding.
C. Keep the change, because sign-up counts are vanity metrics anyway.
D. Remove all tracking so the team stops seeing the drop.

**Answer:** B. The sign-up count is a rising number that may not show a better experience. The fall in later return is a guardrail problem. Comparing it with a control group shows whether the design caused the drop before you decide to keep, change or drop it.

**Quick check:** In a survey, users say a new settings layout feels faster. But the share of users who finish their settings task has not changed. Which result should decide whether the change worked?

A. The survey, because people know best how a screen feels to them.
B. The task completion share, because it records what users actually do.
C. Neither, because both kinds of data are only opinions.
D. The number of settings pages viewed, because more views mean more interest.

**Answer:** B. Behavioral data shows what happened, so it is harder to argue with. The survey is still useful for explaining the feeling, but it should not overrule a flat task result when you judge the impact.

## Summary

Measuring the impact turns a design idea into evidence. Start with a goal and one primary metric. Record a baseline before the change. Prefer behavioral data, watch vanity metrics and guardrails, and compare against a control group or a fair period. Collect enough users or time, and separate correlation from causation before you decide to keep, change or drop the design.

> **Tip:** Keep a simple log of each change: the goal, the primary metric, the baseline, the comparison, the guardrails, and the decision. Over time, this log tells you more than any single report.

## Sources

- [Nielsen Norman Group: Your New UX Habit, Establishing Baselines for Impact](https://www.nngroup.com/articles/establishing-baselines/)
- [Nielsen Norman Group: Vanity Metrics, Add Context to Add Meaning](https://www.nngroup.com/articles/vanity-metrics/)
- [Nielsen Norman Group: Statistical Significance Isn't the Same as Practical Significance](https://www.nngroup.com/articles/practical-significance/)
- [Kerry Rodden: The HEART framework for UX metrics](https://kerryrodden.com/heart/)
- [Equinet Academy: UX Metrics, How to Measure the Success of a Design](https://www.equinetacademy.com/blog/ux-metrics-how-to-measure-the-success-of-a-design/)
- [Nectar Labs: Data-Driven UX, Measuring Design Success](https://www.createnectar.com/insights/data-driven-ux-measuring-design-success)
- [Lyssna: UX Success Metrics](https://www.lyssna.com/blog/ux-success-metrics/)
- [Interaction Design Foundation: What is the HEART Framework?](https://ixdf.org/literature/topics/heart-framework)
- [Koji: HEART Framework, Google's 5-Metric Model for Measuring User Experience](https://www.koji.so/docs/heart-framework-ux-metrics)
- [Fuselab Creative: UX Metrics, How to Measure UX in Enterprise Products](https://fuselabcreative.com/how-to-measure-ux-metrics/)
- [Medium (Design Bootcamp): Mastering UX metrics, a comprehensive guide](https://medium.com/design-bootcamp/mastering-ux-metrics-a-comprehensive-guide-to-understanding-user-behavior-and-design-impact-4f7a9063ad6e)
