A/B Testing for Small Businesses: How to Run Tests That Mean Something
A/B testing sounds simple: show two versions, keep the winner. In practice, most small business tests end too early, test too little traffic or measure the wrong thing, and the "winner" is just noise. Here is how to run tests you can actually trust.
Key takeaways
- An A/B test compares two versions by randomly splitting your audience and measuring one primary result.
- Calculate the sample size before you start, and do not stop the test early because one version is ahead.
- With low traffic, test bold changes on high-traffic pages, or test in email and ads where volume is easier to reach.
- Most ideas do not win. A test that shows no difference still saves you from a change that would not have helped.
Every marketing team has opinions about headlines, button colors, prices and images. A/B testing replaces opinions with evidence. Done well, it helps you improve conversion step by step. Done badly, it gives false confidence and leads to changes that make no difference, or even hurt.
Experimentation experts are blunt about how often ideas fail. In their book Trustworthy Online Controlled Experiments, Ron Kohavi, Diane Tang and Ya Xu describe how only about a third of ideas tested at Microsoft improved the metrics they were designed to improve. If large companies with expert teams see that, small businesses should expect plenty of tests to show no clear winner too.
What an A/B test is
An A/B test, also called a split test, shows version A (the control) to one randomly chosen group and version B (the variant) to another, at the same time. You then compare a single primary metric, such as conversion rate, between the groups. Because the groups are random and the timing is the same, a large enough difference can be credited to the change you made.
What you can test
- Website pages: headlines, offers, page layout, form length, calls to action, images, pricing display.
- Emails: subject lines, send times, content length, offers. Most email platforms include built-in split testing.
- Ads: creative, copy and audiences. Google Ads and Meta both offer experiment features, though their names and options change.
- Offline: two versions of a flyer with different promo codes, or two scripts for outbound calls.
Common mistake
Stopping the test as soon as one version pulls ahead. Early results swing wildly. If you check every day and stop the first time the difference looks significant, you dramatically increase the chance of declaring a false winner. This is often called the "peeking" problem.
The four ingredients of a trustworthy test
1. A clear hypothesis
Write it before you build anything, in this form: "Because we believe [insight], changing [element] to [new version] will increase [metric]." For example: "Because customers ask about delivery times in chat, adding the delivery date next to the buy button will increase checkout starts." A hypothesis forces you to test something with a reason behind it, and helps you learn even when it loses.
2. One primary metric
Pick the single number that decides the winner, usually a conversion rate. You can watch secondary metrics, like average order value or bounce rate, to catch side effects, but choose the primary one in advance. Picking the best-looking metric after the test is a recipe for false wins.
3. Enough sample size
The number of visitors you need depends on your current conversion rate and the smallest improvement you care about detecting. Smaller baseline rates and smaller improvements need far more traffic. Free calculators, such as Evan Miller's widely used online sample size calculator, do the math for you.
4. A fixed duration
Decide how long the test will run before you start, and run it for whole weeks so weekday and weekend behavior are both included. Many practitioners suggest at least one to two full business cycles. Avoid running tests across unusual periods, such as a major holiday sale, unless that is exactly what you want to learn about.
How to run an A/B test step by step
- Find the opportunity. Use analytics to spot a page or step with lots of traffic and a big drop-off. Customer questions, session recordings and reviews help explain why people leave.
- Write the hypothesis. Use the "Because... changing... will increase..." format and pick one primary metric.
- Calculate sample size and duration. Enter your baseline conversion rate and the minimum lift you care about into a calculator, then divide by your weekly traffic to estimate how many weeks you need.
- Build the variant. Change only what your hypothesis describes. If you change five things, you will not know which one mattered.
- Check tracking. Preview both versions, complete a test conversion in each and confirm it is recorded correctly.
- Launch and leave it alone. Split traffic 50/50 and do not change the pages, budgets or targeting mid-test.
- Read the result at the planned end. Look at the primary metric, the confidence level and the secondary metrics.
- Record and act. Write down the hypothesis, result and what you learned in a simple test log, then roll out, iterate or move on.
Testing with low traffic
Many small business websites get a few thousand visitors a month. That is too few to detect small improvements in reasonable time. You still have good options:
- Test bigger changes. A completely different offer or page structure might produce a large difference that is detectable with less traffic. Button color tweaks will not.
- Test higher up the funnel. Measure clicks to the next step instead of final purchases. There are more clicks than sales, so you reach significance faster.
- Use email and ads. An email list of 10,000 lets you test subject lines in a single send. Ad platforms can split thousands of impressions quickly.
- Run sequential tests carefully. Comparing "before" and "after" periods is weaker than a true split test because seasonality and other changes interfere, but it can still show large effects. Keep everything else stable and compare equal periods.
- Use qualitative research. Five-second tests, customer interviews and user testing with five people can reveal big problems without any statistics.
| Your monthly traffic | What you can realistically test | Better alternatives |
|---|---|---|
| Under 1,000 visitors | Very little on the website | Customer interviews, user testing, email tests |
| 1,000 to 10,000 | Bold changes on top pages, click-through goals | Email subject lines, ad creative tests |
| 10,000 to 100,000 | Headlines, offers, form changes on key pages | Combine with session recordings to find ideas |
| Over 100,000 | Smaller refinements and multiple tests a month | A structured testing roadmap |
These bands are rough guides. Your real limits depend on your conversion rate and the size of change you want to detect.
Hypothetical example
Suppose a local accounting firm's website gets about 3,000 visitors a month and its contact form converts at 2%. Testing a new button label would take many months to detect a small lift. Instead, the owner tests a bold change: replacing the long contact form with a short "book a 15-minute call" calendar. The calculator shows the test can detect a large lift in about six weeks. The new version wins clearly, and the owner logs the result before planning the next test.
Reading results without fooling yourself
Most testing tools report a confidence level or "probability to beat" figure. A common convention is to accept a result at 95% statistical significance. That does not mean you are 95% sure B is better in a simple sense; it means a difference this large would be unlikely if the versions truly performed the same. Treat results near the line with caution.
- Check the size, not just the significance. A significant but tiny lift may not be worth the cost of the change.
- Look at segments carefully. Mobile versus desktop splits can be useful, but slicing into many segments will produce random "wins." Treat segment findings as ideas for new tests.
- Watch for side effects. A pop-up that raises email signups might reduce purchases. Secondary metrics catch this.
- Re-test big wins. If a result seems too good to be true, run it again before rebuilding your site around it.
Pro tip
Keep a simple test log with the date, hypothesis, metric, sample size, result and lesson. After a year you will have a record of what your customers respond to, which is far more valuable than any single winning test.
Tools and practical considerations
Google Optimize, once a popular free option, was shut down by Google in September 2023. Many website builders, e-commerce platforms, landing page tools and email platforms now include built-in testing, and dedicated testing tools are available at a range of prices. Features and pricing change often, so compare current options against your traffic and budget.
Testing tools usually rely on cookies or similar technologies to keep each visitor in the same group. In the EU, UK and other places with strict consent rules, check whether your testing setup needs consent and how visitors who decline are handled. Also be careful with price testing: showing different prices to different customers can raise fairness and consumer protection concerns in some countries, so get advice before testing prices directly.
When A/B testing is worth it
- You have enough traffic or list size for the change you want to detect
- The decision matters and is hard to call by judgment alone
- You can keep conditions stable for several weeks
When to skip it
- Fixing obvious problems like broken forms or slow pages
- Traffic is so low the test would take many months
- Legal or brand requirements decide the answer anyway
Your A/B testing checklist
- Pick one high-traffic page or email with a clear drop-off
- Write a hypothesis and choose one primary metric
- Calculate sample size and a fixed test duration in full weeks
- Change only the element your hypothesis describes
- Confirm tracking works in both versions before launch
- Do not stop early or change settings mid-test
- Record the result and lesson in a test log
Frequently asked questions
How long should an A/B test run?
Long enough to reach your calculated sample size, and in full weeks so all days are represented. For many small business tests, that means at least two weeks, and often longer.
What is the difference between A/B and multivariate testing?
An A/B test compares two versions. Multivariate testing changes several elements at once and tests combinations. It needs far more traffic, so most small businesses should stick to A/B tests.
Can I test more than two versions?
Yes, this is often called A/B/n testing. Each extra version splits your traffic further and increases the time needed, so add variants only if you have plenty of traffic.
What should I do if a test shows no difference?
Record it as a valid result. It tells you that element probably is not a major driver of behavior. Keep the simpler or cheaper version and move on to a bolder idea.
Related reading
The bottom line
A/B testing works when you start with a clear hypothesis, pick one metric, calculate the sample size and let the test run its full course. With limited traffic, test bold ideas, measure earlier steps and use email and ads for faster learning. Expect many tests to show no winner, log every result, and let the evidence, not the loudest opinion, decide what changes.
