esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Gather Lessons, Prioritize & Integrate: Turn Test Results Into Product Change

Learn a three-step habit: record every test, rank the lessons by impact and effort, then build the best ones into your design system, defaults and flows.

In short

Turning test results into product change takes three steps: record every test, rank the lessons by impact and effort, then build the best ones into your design.

13 min readFree

What you will learn

Running a test is only half of the work. A team can test often and still waste its effort if the results sit in a folder and nobody uses them. This lesson teaches a three-step habit. First you gather what you learned from each test. Then you prioritize, which means ranking those lessons. Last you integrate, which means building the best lessons into the product. By the end, you should be able to look at a pile of past tests and decide which ones should change your design.

Key words

Experiment
A planned change to a product that you compare against the current version to see what happens.
A/B test
A test where some users see version A and others see version B, so you can compare the two.
Hypothesis
A clear guess written before the test, such as: if we change this, people will do that, because of this reason.
Metric
The number you use to judge the test, such as how many people finish a step.
Outcome
What actually happened: a win, a loss, or an inconclusive result.
Inconclusive
The test did not show a clear difference either way.
Impact
How much a change is likely to help users or the business.
Effort
How much time, work and money a change takes to build.
Backlog
The list of ideas and tasks that are waiting to be done.
Design system
A shared set of parts, such as buttons, forms, colours and rules, that a product team reuses.
Default
The choice a product uses when the user does nothing to change it.
Learning repository
One shared, searchable place where every test and its lesson is written down.
ICE score
A score made from three parts: Impact, Confidence and Ease.
RICE score
A score made from four parts: Reach, Impact, Confidence and Effort.

Why do good lessons get lost?

Teams often stop at the winner. A test wins, the change goes live, and everyone moves on to the next idea. The reason behind the win is never written down. Figr, a testing guide, compares this to a scientist who finds something new but never publishes it. The gain is there for a while, but the team cannot build on it.

Losing and flat tests are just as useful. A test that lost tells you what users do not respond to. A test that was inconclusive may mean the change was too small, the test ran too briefly, or the idea simply does not matter much to users. Without a record, a team may run the same test again months later and get the same unclear answer.

Tip

Write up a failed or flat test with the same care as a winning one. You will need it more than you expect, usually when someone new asks why the team never tried a certain idea.

Step 1: Gather

Gather means putting every experiment, test result and user observation into one shared place. Include the failures and the inconclusive results. For each item, record the hypothesis, the metric and the outcome. A learning repository is not a dumping ground, so each entry should be short, clear and easy to search later.

What does each entry need?

  • The hypothesis

    The guess you tested and the reason you believed it. A result means little if nobody remembers why the test existed.

  • The metric

    The number used to judge the change. Write it down before the test starts, so nobody can pick a friendlier number after seeing the result.

  • The outcome

    Win, loss or inconclusive, with a short note on what users actually did.

  • The observations

    Notes, quotes or clips from users who took part in research or watched the test. These explain the numbers.

One shared place

The place can be a shared drive, a wiki or a research tool with a search bar. Some research tools keep test results, interview notes and reports together, and can filter them by tags or themes. What matters most is that a teammate can find an old test in a few minutes. Name one person who keeps the place tidy. A shared space with no owner slowly fills with messy notes that nobody trusts.

  1. 1

    Collect

    Pull in every test from the last period, including tests you stopped early.

  2. 2

    Fill in the four fields

    For each test, write the hypothesis, the metric, the outcome and any observations.

  3. 3

    Tag it

    Add a label for the part of the product it touched, such as sign-up, search or checkout.

  4. 4

    Write one lesson

    Add one sentence that says what this tells you about users, even when the test was flat.

  5. 5

    Link the decision

    If the test changed a screen or a rule, link the entry to that design decision.

Tip

A one-sentence lesson is worth more than a long report. If you cannot say what you learned in one sentence, the test may have had an unclear goal.

Step 2: Prioritize

Prioritize means ranking your lessons so the team works on the most useful ones first. Most teams have more lessons than time. Ranking with clear rules gives an honest order instead of one based on whoever speaks loudest. The Nielsen Norman Group describes several prioritization methods that base decisions on stated criteria rather than opinion. The goal is to act on the lessons that matter most for the product goal you chose.

What are impact and effort?

The simplest method is the impact-effort grid. You place each lesson on a grid with impact on one side and effort on the other. This creates four groups that help you decide what to do first.

  • Quick wins

    High impact and low effort. These usually go first.

  • Big bets

    High impact and high effort. Plan them carefully and try a prototype before you commit.

  • Fill-ins

    Low impact and low effort. Do them when there is spare time.

  • Money pits

    Low impact and high effort. Drop them or leave them for later.

Teams often place lessons together. People can vote with coloured dots, with one colour for impact and one for effort. A designer may judge the effect on users, while an engineer judges the work needed. Keep the finished grid after the meeting, so everyone can see why each lesson landed where it did.

How do you score with ICE or RICE?

A grid is quick, but a score gives you a number to sort by. Two common scores are ICE and RICE. ICE uses Impact, Confidence and Ease, where Ease is the opposite of effort. RICE multiplies Reach, Impact and Confidence, then divides the result by Effort, where Reach means how many users the change touches. Teams with a small list often start with ICE. Teams with better user numbers often move to RICE. Some teams use the grid for a quick talk and a score for the final order.

RICE=R×I×CE\mathrm{RICE} = \frac{R \times I \times C}{E}

In words: multiply Reach, Impact and Confidence, then divide the result by Effort. A lesson that reaches many users, helps them a lot, is backed by good evidence and is cheap to build gets a high score. Confidence is how sure you are. A lesson backed by several tests and user interviews deserves more confidence than a hunch. Without research, confidence is only an opinion, so the score is only as good as the evidence behind it.

  1. 1

    List the lessons

    Put every lesson from the repository on one list, including the losses and the flat results.

  2. 2

    Pick one method

    Use the impact-effort grid for a quick talk, or ICE or RICE when you need a sorted list.

  3. 3

    Score with the same rules

    Use the same scale for every lesson, so the scores can be compared.

  4. 4

    Sort the list

    Put the highest scores at the top and read the top ten out loud.

  5. 5

    Check the goal

    Ask whether the top lessons move the one goal your team chose. If not, change the order on purpose and say why.

  6. 6

    Write the decision

    Record the final order and the reason for each top lesson.

Tip

Score the lessons together in one room, and write the reason next to each score. A score with no reason is hard to defend a month later, when someone asks why a lesson was skipped.

Step 3: Integrate

Integrate means turning the top lessons into design decisions. A lesson that stays in a report changes nothing. A lesson that becomes a shared part, a default or a changed flow keeps working after the people who ran the test have moved on. This is how improvements build on each other instead of staying as separate experiments.

How do you update the design system?

A design system holds the parts a team reuses. When a lesson wins, decide where it belongs. Often it belongs in a pattern, which is a shared solution to a common problem that links to the parts it uses. One design-system guide notes that the teams who solve real problems are often the best people to add new patterns back into the shared system. GOV.UK, the UK government's design system, labels parts that still need research as experimental, so designers know which ones are not yet proven.

How do you change defaults and flows?

A default is the choice a product makes for the user. If a winning test showed a better starting choice, the default should change too. Otherwise, people who never touch settings keep seeing the old version. A flow is the set of steps a user takes to finish a task. If a test showed that a shorter flow works better, rebuild the flow with the shorter version as the standard. This lesson only shows where these changes go. How to choose the right default is taught in other lessons.

How do you re-check the results?

After you ship a change, watch the same metric you chose before the test. Check that the gain is still there a few weeks later, and that nothing else got worse. Then update the repository entry with what happened after launch. Booking.com is described in one experimentation guide as a team where each experiment's learning feeds back into the framework for the next one. That loop is what makes the learning stick.

  1. 1

    Pick the top lessons

    Start from the ranked list, not from the newest test.

  2. 2

    Write one decision per lesson

    Describe the change in plain words, such as: the sign-up form now asks only for the essential fields.

  3. 3

    Update the shared parts

    Change the pattern, component, default or flow that the lesson affects.

  4. 4

    Tell the team

    Let designers, writers and builders know what changed and why.

  5. 5

    Set a re-check date

    Choose a date to look at the metric again, and write it in the calendar.

  6. 6

    Record the result

    Add the after-launch outcome to the repository entry.

Worked example: a language app's onboarding

Imagine a language-learning app team with three past tests. The first tested a reminder that helps people keep a daily streak. The second tested a progress bar on the lesson screen. The third tested a shorter sign-up form. The team wants to decide what goes into the standard onboarding flow, which is the set of screens a new learner sees first.

  1. 1

    Gather

    The team opens its repository and finds all three tests. Each entry has a hypothesis, the metric the team chose before the test, and an outcome. The progress bar entry also has a short note from user sessions that people looked at it but did not change how they studied.

  2. 2

    Score with ICE

    The team rates the shorter sign-up as high impact, high confidence and easy to build, because the work is mostly removing fields. The reminder gets high impact and high confidence, since it showed a clear gain, and a medium ease score because it needs notification setup. The progress bar gets lower impact, decent confidence and a harder build, because it needed new data on the lesson screen.

  3. 3

    Read the grid

    On the impact-effort grid, the shorter sign-up and the reminder both land in the quick-win group. The progress bar lands in the lower group, so it is not a priority.

  4. 4

    Integrate the winners

    The team places the shorter sign-up and the reminder into the standard onboarding flow. It also updates the shared form pattern in the design system, so every new form starts with the shorter version.

  5. 5

    Shelve the loser for now

    The progress bar stays in the repository with a note: flat result, shelved, and worth a new test if the lesson screen changes. Nothing is thrown away.

  6. 6

    Re-check

    The team sets a date to look at the same metric after launch. It writes the after-launch result into each of the three entries, so the next team sees the full story.

Common mistakes to avoid

  • Saving only the winners

    Losses and flat results show what users ignore. Without them, the team repeats old tests.

  • Ranking by volume of voices

    The loudest person in the room is not a measure. Use the same criteria for every lesson.

  • Shipping and forgetting

    A change with no owner and no re-check can quietly drift back to the old version.

  • Changing the metric afterwards

    If you pick a new number after seeing the result, the lesson is not trustworthy.

  • Treating one test as a law

    A result applies to the users, screen and context it was tested in. Check before you copy it to a new place.

Check your understanding

Quick check

Your team ran six A/B tests this quarter. Two won, one lost, and three were inconclusive. A manager wants to ship the two winners and forget the other four. What should you do instead, and why?

Quick check

A teammate proposes a change that would reach most of your users and looks like a big win. The only evidence is one conversation with a single user, and the build will take several months. Which score is the most honest?

Key takeaways

  • Gather

    Record every test, including failures and flat results, in one shared, searchable place with the hypothesis, the metric and the outcome.

  • Prioritize

    Rank the lessons with a simple method such as an impact-effort grid, ICE or RICE, and check the order against your product goal.

  • Integrate

    Turn the top lessons into changes in the design system, defaults and flows, then re-check the metric and record what happened.

Good teams treat each test as a step in a chain. When the learning is gathered, ranked and built in, the next test starts from a stronger place than the last one.

Sources

How is this lesson?

Last updated on 17 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.