# A/B Tests & Experiments: Let Real User Data Pick Your Design System Standards

> Learn how teams compare two versions of a shared part with real users, so the design system adopts what works instead of what someone argues for.

**In short:** An A/B test compares two versions of a shared part with real users, so a design system adopts what works instead of what someone argues for.

From "Design systems", a free course by Venkat R. Murty: https://designervenkat.online/learn/design-systems/a-b-tests-experiments

## What you will learn

A design system is a shared set of parts, rules, and guidance that a team uses to build its products. When someone changes a shared part, like a button, the change touches many screens at once. That makes it risky to guess. In this lesson you will learn how an A/B test lets a team check a change with real users before the change becomes a standard in the system. You will also see how the results should be used, and why a single test is never the end of the story.

- **Design system:** The shared parts, rules, and guidance that teams use to build products in a consistent way.
- **Component:** One reusable piece of an interface, such as a button, a form field, or a card.
- **A/B test:** An experiment that shows two versions of one thing to different people, chosen at random, and then compares what they do.
- **Control (version A):** The current version. It is the baseline you compare against.
- **Variant (version B):** The new version you want to test against the control. In this lesson, it means the test version, not a style option of a component.
- **Hypothesis:** A clear guess, written so it can turn out to be wrong, about what a change will do and why.
- **Metric:** A number you measure, such as how many people click a button.
- **Statistical significance:** A sign that a difference between two versions is unlikely to be random chance.

## How does an A/B test work?

An A/B test is a simple idea. You prepare two versions of one element. You split your users into two groups at random. Each group sees one version. Then you measure which version makes more people do the thing you care about. The version that does better is the winner. Testing tools can handle the random split for you, so a designer can often set up a test without writing code.

1. **Write one clear hypothesis:** Say what you expect to happen and why. For example: “If the button is more visible, more shoppers will add the item to their cart.” A good hypothesis can be proven wrong.
2. **Set up the control and the variant:** The control (A) is the version people see today. The variant (B) is the one change you want to test. Change only one thing, so you know what caused any difference.
3. **Choose one main metric:** Pick the one number that will decide the winner. Write it down before the test starts.
4. **Split users at random and run the test:** Each visitor is shown A or B by chance. Let the test run until you have enough users and enough time, as described below.
5. **Read the result and decide:** Check whether the winner is clearly better, and whether the difference is large enough to matter. Then decide what to do next.

Keep in mind that a test only compares two versions. It does not explain why people behaved the way they did. Designers often pair a test with other research, such as watching session recordings or reading support messages, to understand the reason behind the numbers.

> **Tip:** If you test two things at once, such as a new color and a new label, you will not know which change caused the result. Test one change at a time. Some teams use a method called multivariate testing to study several changes together, but that is a more advanced method.

Keep the hypothesis and the metric on paper. Many tests fail not because the idea was bad, but because the team could not agree later on what success meant.

### Why pick one main metric?

A metric is the number that tells you whether one version works better. Teams often choose a main metric, sometimes called the primary metric, and a few other numbers to watch. The main metric should match the goal of the change. It should describe a task being finished, not just a glance at the page.

- **Click-through rate:** The share of people who click a specific button or link after they see it. It is common for button tests.
- **Conversion rate:** The share of visitors who complete a goal, such as buying something or signing up.
- **Completion rate:** The share of people who finish a multi-step task, like filling in a form.
- **Guardrail metrics:** Numbers you watch to make sure the winner does not harm something else, such as people leaving a checkout halfway through.

Guardrail metrics matter more in design than many new designers expect. Users are not a fresh group every time. If a change wins on clicks but makes people lose trust or give up later, the win is not a real win.

### How many users and how much time does a test need?

A result can look exciting after a few hours and still be false. Small tests often show big differences just by chance. Statistical significance is the check that helps you tell real differences from random noise. It asks how likely it is that the gap happened by luck.

Being significant does not mean the change is important. A difference can be real but still too small to be worth the work. Teams should ask both questions: is the difference real, and is it big enough to matter?

Plan the sample size before the test starts. Sample size is the number of users each version needs before the result can be trusted. Sites with a lot of traffic reach that number faster. Sites with little traffic need longer tests, or they must test bigger changes.

> **Tip:** Do not stop a test the moment the winner looks clear. Stopping early is a common way for teams to ship a false winner. Decide the sample size and the run time in advance, then stick to the plan.

## Why does a design system benefit from testing?

A design system is used by many product teams and many screens. Choices about its shared parts are often debated. People have strong opinions about button shapes, form layouts, and card designs. A test gives the group something better than opinions to look at.

- **It settles debates:** When two people disagree about a button style, a test can show which version users pick, instead of a long meeting deciding it.
- **It shows the real effect:** A component may look good in a design file but behave poorly in a real product. A test uses real user behavior, so it shows what actually happens.
- **It protects many screens:** A system change can touch every screen that uses the part. A wrong choice is therefore costly and hard to undo across a whole product.
- **It helps adoption:** Teams are more willing to use a system when they trust its choices. Evidence builds that trust.

A senior designer on Netflix’s design system team has said that she would like more design systems to test new design decisions before adopting them. Her point was that untested decisions can lower business numbers, and that can make teams hesitant to use the system.

Testing also keeps the system honest. Without it, a part can become standard just because the most senior person liked it. With it, the team can point to evidence, and the decision can be explained to others later.

## Worked example: the green Add to cart button

Here is a simple example from an online shop. The team wants to know whether a green Add to cart button on the product page works better than the current gray one. The steps below show how the idea is used from start to finish. The exact color values belong to another lesson, so we only talk about the idea here.

1. **Start with the problem:** The team sees that many shoppers look at a product but do not add it to their cart. The gray button may blend in with the rest of the page. This is a guess, so the team will test it.
2. **Write the hypothesis:** The team writes: if the Add to cart button is green instead of gray, more shoppers will add the product to their cart.
3. **Build the control and the variant:** Version A (control) keeps the current gray button. Version B (variant) uses the green button. Size, wording, and position stay the same, so only the color changes.
4. **Pick the metric and guardrails:** The main metric is the add-to-cart rate, the share of product page visitors who add an item. The team also watches guardrails, such as whether people still reach the checkout.
5. **Split the traffic and run the test:** Visitors are shown A or B at random. The team runs the test until it reaches the planned sample size, and does not stop when the early numbers look nice.
6. **Read the result:** The green button wins on the add-to-cart rate, and the guardrails look healthy. The team now has evidence for the change.
7. **Update the primary button component:** The team changes the primary button component in the design system. Every product that uses that component inherits the new default. The team does not repaint each screen by hand.
8. **Document the reason:** The team writes a short note in the system documentation. It records the hypothesis, the control and variant, the main metric, the result, the product page context, and the date of the test.

> **Tip:** The note matters as much as the win. A year later, someone may ask why the button is green. A clear record tells them the decision came from a test on a product page, and it shows what was measured.

## How do you use the results?

A test ends with a decision. There are three common outcomes: a clear winner, a clear loser, or an inconclusive result. Each one leads to a different action. Read the result as a product decision, not as a score on a scoreboard.

1. **Roll out the winner as the standard:** If the variant clearly wins on the main metric and the guardrails hold, update the shared component so the winning version becomes the default for new work.
2. **Record the reason in the guidelines:** Write the decision where designers and developers will read it. Explain the test, the context, and why the team chose the new version. Detailed rules are covered in other lessons, but the test reason should always be recorded.
3. **Keep testing:** Plan the next test. Look for the next pain point, and test the next change. A design system improves over time through small, checked steps.

Remember that a test shows what worked for one context and one audience. A button that wins with shoppers on phones in one country may not win on desktop screens for a different group. That is why teams keep checking results, and why they write down where the evidence came from.

If a test is inconclusive, do not force a winner. Report that the difference was too small to be sure. You can run a new test with a bigger change, more users, or a clearer hypothesis.

## Common mistakes to avoid

- **Testing too many things at once:** When several parts change together, the team cannot tell which change caused the result.
- **Stopping early:** Calling a winner before the planned sample size is reached often leads to a false winner.
- **Testing without a problem:** Some teams run tests just to test. A better start is a real problem, found in user research, analytics, or support feedback.
- **Ignoring the audience:** Testing should be done with the right users. Results from one group may not hold for another.
- **Using a test to avoid thinking:** A test answers a question you already asked. It does not replace good design judgment. Some questions are better answered by interviews or usability tests.

> **Tip:** Ask a simple question before you start: what decision will this test change? If the answer is nothing, the test is probably not worth running.

## Check your understanding

**Quick check:** Your design team cannot agree whether the primary button in the system should have rounded or square corners. Which plan is the best way to decide?

A. Ask the most senior designer to choose, then document the choice.
B. Run an A/B test where the control is the current corner style and the variant changes only the corners, with one main metric such as click-through rate on the button, and check guardrails such as form completion.
C. Show both versions to the team in a meeting and vote on the one that looks better.

**Answer:** B. The test uses one clear change, a control and a variant, and a main metric tied to the button’s job. Opinion and votes do not use real user behavior, so they cannot settle the question.

**Quick check:** After four days, a test of a new form button shows the variant with more clicks. The planned sample size has not been reached. What should the team do?

A. Roll out the variant to the system now, since the early numbers look clear.
B. Keep the test running until the planned sample size and time are reached, and watch the guardrails.
C. Stop the test and keep the control, because any early difference is wrong.

**Answer:** B. Early results can be misleading, so the team should finish the plan. Stopping early and rolling out at once risks a false winner, while dropping the test entirely throws away a test that is still running correctly.

**Quick check:** A test on a checkout page for shoppers on phones shows that a rounded button wins. A product manager wants the rounded button as the rule for every screen in the system. What is the best response?

A. Accept the rule, because one winning test proves the style works everywhere.
B. Treat the result as strong evidence for that context, record it in the guidelines, and test again with other audiences or screen sizes before making it a universal rule.
C. Reject the result, because a test can never guide a design system decision.

**Answer:** B. A test shows what worked for one context and audience. The team should keep the evidence, state its limits, and check other contexts before applying it everywhere.

## Key takeaways

An A/B test compares a control with one variant, using a clear hypothesis, one main metric, a random split, and enough users and time. In a design system, it helps the team choose shared parts based on evidence. The winner can become the standard, but the reason should be recorded, and testing should continue, because one test only shows what worked in one context.

> **Tip:** Before you propose any change to a shared component, write three things on one page: the hypothesis, the main metric, and the decision the result will change.

## Sources

- [UXPin: A/B Testing in UX Design: When and Why It's Worth It](https://www.uxpin.com/studio/blog/ab-testing-in-ux-design-when-and-why/)
- [LogRocket: Understanding A/B testing in UX research](https://blog.logrocket.com/ux-design/understanding-ab-testing-ux-research/)
- [Ranktracker: A/B Testing in UI/UX Design: The Complete Guide](https://www.ranktracker.com/blog/ab-testing-in-ui-ux-design-the-complete-guide/)
- [ANODA: A/B Testing in UX Design: A Practical Guide](https://www.anoda.mobi/ux-blog/ab-testing-in-ux-design-how-to-run-split-tests-and-improve-user-experience-and-conversion)
- [Back Market engineering team: How we A/B Test our Design System Components](https://medium.com/@stevedodierlazaro/how-we-a-b-test-our-design-system-components-af7a75ed4cae)
- [Kissmetrics: A/B Testing Statistical Significance: When to Call a Winner](https://kissmetrics.io/blog/ab-testing-statistical-significance)
- [Convert Hooks: Statistical Significance in A/B Testing, Explained](https://converthooks.com/articles/conversion-optimization/statistical-significance-in-a-b-testing-explained)
- [Humblytics: A/B Testing Statistical Significance Explained](https://humblytics.com/blog/a-b-testing-statistical-significance-explained)
- [Articos: A/B Testing Sample Size guide](https://www.articos.com/blog/ab-testing-sample-size)
- [Kirro: A/B testing UX: a guide for designers and teams](https://kirro.io/ab-testing-ux)
- [Winsavvy: How to Use A/B Testing to Refine UI Design Decisions](https://www.winsavvy.com/how-to-use-a-b-testing-to-refine-ui-design-decisions/)
- [Looppanel: How A/B Testing Helps in UX Design](https://www.looppanel.com/blog/ab-testing-ux-design)
- [Let's Update Skills: How to Conduct A/B Testing to Improve UX](https://www.letsupdateskills.com/article/how-to-conduct-ab-testing-to-improve-ux)
- [PM Toolkit: A/B Testing Guide: Statistical Significance Explained Simply](https://pmtoolkit.ai/learn/experimentation/ab-testing-guide)
