esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Measuring the Impact: Check Whether a Design Change Really Worked

Learn how to set a goal, record a baseline, compare fairly, and decide with evidence instead of opinion.

In short

Measuring impact means setting a goal, recording a baseline and comparing fairly, so you decide whether a design change worked using evidence instead of opinion.

14 min readFree

What you will learn

Every design change is a guess until you check it. Measuring the impact means checking whether a change actually moved the behavior it was meant to move. You compare real user data from before and after the change, or against a group that did not get the change. This lesson shows you how to set a goal, record a starting point, choose the right numbers, compare fairly, and decide what to do next. By the end you should be able to tell a real improvement from a number that only looks good.

Key words

Measuring the impact
Checking whether a design change moved the behavior or result it was meant to change, using data instead of opinions.
Baseline
The measured starting point, taken before you change anything. It tells you what normal looks like.
Primary metric
The one main number that tells you whether the change worked.
Behavioral data
Records of what people actually do, such as clicks, finished tasks, and return visits.
Attitudinal data
What people say about how they feel, usually gathered with surveys or interviews.
Vanity metric
A number that looks impressive but cannot show whether the design got better.
Control group
A set of users who do not get the change, so you can compare them with users who do.
Retention
How many users come back after a set time, such as a week or a month.
Correlation
Two things that move together. It does not prove one caused the other.
Causation
One thing actually makes another thing happen.

Why start with a goal and a baseline?

Before you change anything, write one plain sentence that says what the change should do for the user. For example: more people finish checkout without giving up. Then choose one primary metric that tracks that behavior directly. A common choice is checkout completion rate.

Next, measure where that number stands today. This is your baseline. Without it, a later number means little, because you cannot tell whether it is better, worse, or just normal for your product. A clear baseline makes later results meaningful, and the baseline should be taken before the design work starts.

  1. 1

    Write the goal

    Describe the change in one plain sentence, such as: users should reach the end of setup without help.

  2. 2

    Pick one primary metric

    Choose a number that reflects the user behavior you want to change. Ask whether a better number would clearly mean a better experience.

  3. 3

    Record the baseline

    Measure the current value of that metric before you launch anything. Write down the date range you used.

  4. 4

    Set a target and a time frame

    Decide how much change would count as a win, and over what period you will look.

  5. 5

    List guardrail metrics

    Name a few numbers that must not get worse, so you notice side effects early.

Tip

Write the goal, the primary metric and the baseline down before launch. If you choose the metric after you see the results, you are more likely to pick the number that happened to rise.

Choosing one primary metric matters because it keeps you honest. Tracking many numbers invites you to report the one that moved in the right direction. Pick one, commit to it, and treat the others as supporting checks.

Without baseline numbers, nobody can tell whether a new layout reduces drop-off or simply looks nicer to the team that built it. The baseline is the honest yardstick you compare against.

How do you pick the right kind of metric?

Metrics come in two broad kinds. Behavioral metrics show what people do. Attitudinal metrics show what people think and feel. Both are useful, but they answer different questions.

  • Behavioral data

    What users physically do, such as completion rates, error rates, drop-off rates and return visits. These usually come from analytics tools.

  • Attitudinal data

    What users say, such as how easy a task felt, how satisfied they were, or how confident they were. These usually come from surveys.

  • Outcome data

    Business results that a design can influence, such as the share of visitors who buy, or the share of new users who are still active later.

For proving that a change worked, behavioral data is usually the stronger evidence. Behavioral numbers tell you what happened. Attitudinal numbers tell you how it felt. The second is harder to read as proof, because feelings can shift for many reasons.

Attitudinal data still has a job. When a behavior number moves, a few interviews or survey answers can help you understand why. Pair the numbers with a handful of user conversations so you know the reason, not just the result.

Tip

Ask one question about any candidate metric: if this number improves, does that really mean users have a better experience with what I changed? If the answer needs several leaps of logic, the metric is probably too indirect.

A checklist of categories can help you think wider. Google researchers published a model called HEART in 2010, and it is used to organize user experience metrics into five groups.

  • Happiness

    How satisfied users say they are.

  • Engagement

    How often and how deeply people use the product in a set time.

  • Adoption

    How many new users start using a product or feature.

  • Retention

    How many users keep coming back over time.

  • Task success

    Whether users finish the job they came to do.

Not every group fits every design. Use the list to check your thinking, then pick the one or two that match your goal. Treat a click as a step, not as success. A user who clicks a button has not necessarily finished the task.

Vanity metrics are the numbers that look good on a slide but cannot tell you whether the design improved. They are the most common trap in this kind of work.

A vanity metric often only goes up. Total users, app downloads, page views and shares are typical examples. They feel like progress, but they do not explain how users experience the product.

A better habit is to turn a raw count into a rate. Instead of counting downloads, measure the share of downloads that become active users within a set time. A rate can show a real change in the experience, because it is not just growing with the audience.

A metric is actionable when a change in it points you to the next thing to fix. If a number drops, you should know roughly where to look.

Watch for side effects. A redesign can raise one number while hurting another. For example, a change might bring in more sign-ups but fewer of those new people are still using the app a month later. If you only count sign-ups, you would call it a win and miss the problem.

Teams guard against this by naming guardrail metrics. These are numbers that must not get worse while you chase the primary metric. Check them every time you read the results.

How do you compare fairly and read the result carefully?

A number only means something when you compare it with another number. The three most common comparisons are below. Each is useful, but each has a weakness you need to watch.

  • Before and after

    You compare the baseline with the period after launch. It is simple, but it fails if something else changed at the same time.

  • Control group

    Some users keep the old design while others get the new one, and both groups are measured over the same days. This is the fairer test.

  • Comparison period

    You compare the change period with a similar stretch of time that had no change, such as a comparable week. This helps, but seasonal events can still skew it.

Each comparison answers a slightly different question. Before and after asks whether things moved. A control group asks whether the change caused the movement. Be clear about which one you are answering.

Both groups then face the same news, prices, weather and sales. Any gap between them is much more likely to come from the design. Controlled comparisons like this are the basis of A/B testing, which has its own lesson.

Collect enough users or enough time before you trust the result. A small difference can appear by luck. Statistical significance is a check for this. It asks whether a result is unlikely to be due to chance, and it is often tested against a threshold of 0.05.

Statistical significance is not the same as importance. A very large sample can make a tiny change look reliable even when nobody would notice it. Ask two questions: is the result reliable, and is it big enough to matter to users and the business?

The amount of data you need depends on your baseline and the size of change you hope to detect.

Tip

When you choose a time frame, pick the shortest one that still gives a fairly stable rate. Very short periods bring random ups and downs, and very long periods slow down feedback.

How do you separate correlation from causation?

Correlation means two numbers move together. Causation means one thing actually produces the other. After a redesign, sales may rise, but that does not prove the redesign did it. The sales rise and the redesign might both have a different cause.

The outside factor that could explain a result is sometimes called a confounder. Common confounders include a holiday, a price cut, a seasonal sale, a marketing email, a bug fix, or a new competitor. Any of these can move the numbers during the same weeks as your change.

  1. 1

    List everything else that changed

    Write down every event in the same window: promotions, emails, releases, outages, holidays and news.

  2. 2

    Compare with a control group

    If the users who did not get the change also rose, the outside factor is a likely cause, not your design.

  3. 3

    Check a second period

    Look at a time without the event, or a repeat of the test. A real design effect should still appear.

  4. 4

    Check the guardrails

    Make sure the side effects did not grow while the main number rose.

Tip

Be suspicious of any result that arrives in the same week as a big event. Say the change was linked to the result until you have ruled out the event.

Worked example: a fitness app's weekly goal reminder

A fitness app adds a weekly goal reminder to its home screen. The reminder shows how many workouts a person has left to reach their weekly goal. The designer wants to know whether the reminder helps people log a workout within their first seven days. The steps below show how the designer checks it.

  1. 1

    Set the goal

    The goal is for more new users to log a workout early, so they build the habit while they are still new.

  2. 2

    Choose the primary metric

    The primary metric is the share of users who log a workout within seven days. This is a behavioral measure, and it is close to the behavior the reminder is meant to change.

  3. 3

    Record the baseline

    Before the reminder goes live, the designer measures the same share for users who joined before the change. That is the starting point.

  4. 4

    Compare the groups

    The designer compares the baseline with the group who saw the reminder. A stronger version would also split new users so some see the reminder and some do not, at the same time.

  5. 5

    Check the guardrail

    The designer also checks whether notification opt-outs went up. If more people turned notifications off, the reminder may be annoying some users, even if the workout share rose.

  6. 6

    Look for confounders

    The designer checks whether a New Year campaign, a new free trial, or a seasonal challenge started in the same weeks.

  7. 7

    Decide

    If the share rose clearly and opt-outs stayed flat, keep the reminder. If opt-outs jumped, change the timing or wording. If the share did not move, drop it or test a different idea.

Notice what the designer did not do. They did not decide from one screenshot or from the number of total logins. They chose one primary metric, recorded a baseline first, watched a guardrail, and checked for outside events before acting.

Tip

The reminder itself is one tactic. This lesson only shows how to measure whether it worked. Other lessons cover specific tactics such as goal trackers and reminders.

How do you decide to keep, change or drop?

Once you have a fair comparison, you can make one of three decisions. Each depends on the primary metric, the guardrails, and the size of the effect.

  • Keep

    The primary metric improved, the change is reliable, and the guardrails held. Keep the design and keep watching the numbers.

  • Change

    The goal was right, but a side effect appeared or the effect was weak. Adjust the design and measure again from a new baseline.

  • Drop

    The primary metric did not move, or it moved while a serious guardrail got worse. Remove the change and record what you learned.

Be careful with effects that are real but tiny. A result can be reliable and still be too small to justify extra engineering work or new complexity. Ask whether the gain is worth the cost.

A small test can also show a big drop that is not yet proven. In that case, the drop is a reason to investigate further, not a reason to celebrate or to ignore it.

Check your thinking

Quick check

After a checkout redesign, your team sees purchases rise by 10% in the week after launch. That same week included a holiday sale. Can you say the redesign caused the rise, and what should you do to find out?

Quick check

A news app changes its article page to show a save-for-later button. More people reach the sign-up page afterwards, but fewer of the new sign-ups come back a month later. What is the best next step?

Quick check

In a survey, users say a new settings layout feels faster. But the share of users who finish their settings task has not changed. Which result should decide whether the change worked?

Summary

Measuring the impact turns a design idea into evidence. Start with a goal and one primary metric. Record a baseline before the change. Prefer behavioral data, watch vanity metrics and guardrails, and compare against a control group or a fair period. Collect enough users or time, and separate correlation from causation before you decide to keep, change or drop the design.

Tip

Keep a simple log of each change: the goal, the primary metric, the baseline, the comparison, the guardrails, and the decision. Over time, this log tells you more than any single report.

Sources

How is this lesson?

Last updated on 17 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.