A/B Tests & Experiments: Let Real User Data Pick Your Design System Standards
Learn how teams compare two versions of a shared part with real users, so the design system adopts what works instead of what someone argues for.
In short
An A/B test compares two versions of a shared part with real users, so a design system adopts what works instead of what someone argues for.
12 min readFree
What you will learn
A design system is a shared set of parts, rules, and guidance that a team uses to build its products. When someone changes a shared part, like a button, the change touches many screens at once. That makes it risky to guess. In this lesson you will learn how an A/B test lets a team check a change with real users before the change becomes a standard in the system. You will also see how the results should be used, and why a single test is never the end of the story.
Key words
- Design system
- The shared parts, rules, and guidance that teams use to build products in a consistent way.
- Component
- One reusable piece of an interface, such as a button, a form field, or a card.
- A/B test
- An experiment that shows two versions of one thing to different people, chosen at random, and then compares what they do.
- Control (version A)
- The current version. It is the baseline you compare against.
- Variant (version B)
- The new version you want to test against the control. In this lesson, it means the test version, not a style option of a component.
- Hypothesis
- A clear guess, written so it can turn out to be wrong, about what a change will do and why.
- Metric
- A number you measure, such as how many people click a button.
- Statistical significance
- A sign that a difference between two versions is unlikely to be random chance.
How does an A/B test work?
An A/B test is a simple idea. You prepare two versions of one element. You split your users into two groups at random. Each group sees one version. Then you measure which version makes more people do the thing you care about. The version that does better is the winner. Testing tools can handle the random split for you, so a designer can often set up a test without writing code.
- 1
Write one clear hypothesis
Say what you expect to happen and why. For example: “If the button is more visible, more shoppers will add the item to their cart.” A good hypothesis can be proven wrong.
- 2
Set up the control and the variant
The control (A) is the version people see today. The variant (B) is the one change you want to test. Change only one thing, so you know what caused any difference.
- 3
Choose one main metric
Pick the one number that will decide the winner. Write it down before the test starts.
- 4
Split users at random and run the test
Each visitor is shown A or B by chance. Let the test run until you have enough users and enough time, as described below.
- 5
Read the result and decide
Check whether the winner is clearly better, and whether the difference is large enough to matter. Then decide what to do next.
Keep in mind that a test only compares two versions. It does not explain why people behaved the way they did. Designers often pair a test with other research, such as watching session recordings or reading support messages, to understand the reason behind the numbers.
Tip
If you test two things at once, such as a new color and a new label, you will not know which change caused the result. Test one change at a time. Some teams use a method called multivariate testing to study several changes together, but that is a more advanced method.
Keep the hypothesis and the metric on paper. Many tests fail not because the idea was bad, but because the team could not agree later on what success meant.
Why pick one main metric?
A metric is the number that tells you whether one version works better. Teams often choose a main metric, sometimes called the primary metric, and a few other numbers to watch. The main metric should match the goal of the change. It should describe a task being finished, not just a glance at the page.
Click-through rate
The share of people who click a specific button or link after they see it. It is common for button tests.
Conversion rate
The share of visitors who complete a goal, such as buying something or signing up.
Completion rate
The share of people who finish a multi-step task, like filling in a form.
Guardrail metrics
Numbers you watch to make sure the winner does not harm something else, such as people leaving a checkout halfway through.
Guardrail metrics matter more in design than many new designers expect. Users are not a fresh group every time. If a change wins on clicks but makes people lose trust or give up later, the win is not a real win.
How many users and how much time does a test need?
A result can look exciting after a few hours and still be false. Small tests often show big differences just by chance. Statistical significance is the check that helps you tell real differences from random noise. It asks how likely it is that the gap happened by luck.
Being significant does not mean the change is important. A difference can be real but still too small to be worth the work. Teams should ask both questions: is the difference real, and is it big enough to matter?
Plan the sample size before the test starts. Sample size is the number of users each version needs before the result can be trusted. Sites with a lot of traffic reach that number faster. Sites with little traffic need longer tests, or they must test bigger changes.
Tip
Do not stop a test the moment the winner looks clear. Stopping early is a common way for teams to ship a false winner. Decide the sample size and the run time in advance, then stick to the plan.
Why does a design system benefit from testing?
A design system is used by many product teams and many screens. Choices about its shared parts are often debated. People have strong opinions about button shapes, form layouts, and card designs. A test gives the group something better than opinions to look at.
It settles debates
When two people disagree about a button style, a test can show which version users pick, instead of a long meeting deciding it.
It shows the real effect
A component may look good in a design file but behave poorly in a real product. A test uses real user behavior, so it shows what actually happens.
It protects many screens
A system change can touch every screen that uses the part. A wrong choice is therefore costly and hard to undo across a whole product.
It helps adoption
Teams are more willing to use a system when they trust its choices. Evidence builds that trust.
A senior designer on Netflix’s design system team has said that she would like more design systems to test new design decisions before adopting them. Her point was that untested decisions can lower business numbers, and that can make teams hesitant to use the system.
Testing also keeps the system honest. Without it, a part can become standard just because the most senior person liked it. With it, the team can point to evidence, and the decision can be explained to others later.
Worked example: the green Add to cart button
Here is a simple example from an online shop. The team wants to know whether a green Add to cart button on the product page works better than the current gray one. The steps below show how the idea is used from start to finish. The exact color values belong to another lesson, so we only talk about the idea here.
- 1
Start with the problem
The team sees that many shoppers look at a product but do not add it to their cart. The gray button may blend in with the rest of the page. This is a guess, so the team will test it.
- 2
Write the hypothesis
The team writes: if the Add to cart button is green instead of gray, more shoppers will add the product to their cart.
- 3
Build the control and the variant
Version A (control) keeps the current gray button. Version B (variant) uses the green button. Size, wording, and position stay the same, so only the color changes.
- 4
Pick the metric and guardrails
The main metric is the add-to-cart rate, the share of product page visitors who add an item. The team also watches guardrails, such as whether people still reach the checkout.
- 5
Split the traffic and run the test
Visitors are shown A or B at random. The team runs the test until it reaches the planned sample size, and does not stop when the early numbers look nice.
- 6
Read the result
The green button wins on the add-to-cart rate, and the guardrails look healthy. The team now has evidence for the change.
- 7
Update the primary button component
The team changes the primary button component in the design system. Every product that uses that component inherits the new default. The team does not repaint each screen by hand.
- 8
Document the reason
The team writes a short note in the system documentation. It records the hypothesis, the control and variant, the main metric, the result, the product page context, and the date of the test.
Tip
The note matters as much as the win. A year later, someone may ask why the button is green. A clear record tells them the decision came from a test on a product page, and it shows what was measured.
How do you use the results?
A test ends with a decision. There are three common outcomes: a clear winner, a clear loser, or an inconclusive result. Each one leads to a different action. Read the result as a product decision, not as a score on a scoreboard.
- 1
Roll out the winner as the standard
If the variant clearly wins on the main metric and the guardrails hold, update the shared component so the winning version becomes the default for new work.
- 2
Record the reason in the guidelines
Write the decision where designers and developers will read it. Explain the test, the context, and why the team chose the new version. Detailed rules are covered in other lessons, but the test reason should always be recorded.
- 3
Keep testing
Plan the next test. Look for the next pain point, and test the next change. A design system improves over time through small, checked steps.
Remember that a test shows what worked for one context and one audience. A button that wins with shoppers on phones in one country may not win on desktop screens for a different group. That is why teams keep checking results, and why they write down where the evidence came from.
If a test is inconclusive, do not force a winner. Report that the difference was too small to be sure. You can run a new test with a bigger change, more users, or a clearer hypothesis.
Common mistakes to avoid
Testing too many things at once
When several parts change together, the team cannot tell which change caused the result.
Stopping early
Calling a winner before the planned sample size is reached often leads to a false winner.
Testing without a problem
Some teams run tests just to test. A better start is a real problem, found in user research, analytics, or support feedback.
Ignoring the audience
Testing should be done with the right users. Results from one group may not hold for another.
Using a test to avoid thinking
A test answers a question you already asked. It does not replace good design judgment. Some questions are better answered by interviews or usability tests.
Tip
Ask a simple question before you start: what decision will this test change? If the answer is nothing, the test is probably not worth running.
Check your understanding
Quick check
Your design team cannot agree whether the primary button in the system should have rounded or square corners. Which plan is the best way to decide?
Quick check
After four days, a test of a new form button shows the variant with more clicks. The planned sample size has not been reached. What should the team do?
Quick check
A test on a checkout page for shoppers on phones shows that a rounded button wins. A product manager wants the rounded button as the rule for every screen in the system. What is the best response?
Key takeaways
An A/B test compares a control with one variant, using a clear hypothesis, one main metric, a random split, and enough users and time. In a design system, it helps the team choose shared parts based on evidence. The winner can become the standard, but the reason should be recorded, and testing should continue, because one test only shows what worked in one context.
Tip
Before you propose any change to a shared component, write three things on one page: the hypothesis, the main metric, and the decision the result will change.
Sources
- UXPin: A/B Testing in UX Design: When and Why It's Worth It
- LogRocket: Understanding A/B testing in UX research
- Ranktracker: A/B Testing in UI/UX Design: The Complete Guide
- ANODA: A/B Testing in UX Design: A Practical Guide
- Back Market engineering team: How we A/B Test our Design System Components
- Kissmetrics: A/B Testing Statistical Significance: When to Call a Winner
- Convert Hooks: Statistical Significance in A/B Testing, Explained
- Humblytics: A/B Testing Statistical Significance Explained
- Articos: A/B Testing Sample Size guide
- Kirro: A/B testing UX: a guide for designers and teams
- Winsavvy: How to Use A/B Testing to Refine UI Design Decisions
- Looppanel: How A/B Testing Helps in UX Design
- Let's Update Skills: How to Conduct A/B Testing to Improve UX
- PM Toolkit: A/B Testing Guide: Statistical Significance Explained Simply
Last updated on 17 November 2026
Related lessons
- Machine Learning FoundationsTrain - Test Data: splitting your data
- Design systemsDesign System Examples: How to Read Real Systems as Models
- Design systemsPilot: Test a Design System on One Real Team Before Full Rollout
- System designData Management in System Design: Where Data Lives and How It Stays Correct
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.