Email A/B Testing: What to Test and How to Read the Results

Marios ChaidousisBy 8 min read
Email A/B Testing: What to Test and How to Read the Results

Email A/B testing compares two versions of an email to learn which one better achieves a specific goal. Test one meaningful change at a time, split a suitable audience fairly, and judge the result using the metric tied to your goal—not opens alone.

What email A/B testing can tell you

In an A/B test, one group receives version A and another receives version B. You change a single element, such as the subject line or call to action, while keeping the rest of the campaign as similar as possible. If the audience and sending conditions are comparable, the result helps you decide which version to use next time.

A test does not prove that one phrase will always win. It tells you how those versions performed with that audience, for that offer, at that time. Treat the result as evidence to guide your next campaign, not a universal rule about all subscribers.

Before writing either version, decide what action you want people to take. A retailer might want a product purchase; a local service business might want a quote request; a nonprofit might want a donation. The right success metric depends on that action.

What to test in an email

Choose an element that could plausibly affect the action you want. Changing several things at once may produce a different result, but you will not know which change caused it.

  • Subject line: Compare a direct benefit with a curiosity-led approach, while keeping the offer and email body the same.
  • Preheader: Test whether adding useful context beside the subject line encourages the right subscribers to open.
  • Call to action: Compare button wording, such as “Book a consultation” versus “See available times.” Keep its destination consistent.
  • Offer presentation: Test how you explain the same offer, rather than quietly changing its value or terms.
  • Email structure: Compare a short message with a more detailed version when the audience may need additional explanation.
  • Personalization: Test a relevant detail, such as a product category or service area, only if your data is accurate and useful.
  • Send time: Compare timing in a separate test. Avoid changing the send time and subject line together if you want to learn which mattered.

For example, a home-cleaning business could keep the same seasonal offer and email body while testing two subject lines. A dentist could compare two booking prompts, but should avoid wording that pressures people about sensitive health concerns. Consider your industry’s privacy, advertising, and consent requirements before personalizing messages.

Subject line A: “A cleaner home before your next gathering”
Subject line B: “See our home-cleaning availability”

Call to action A: “Book a consultation”
Call to action B: “Choose a time to talk”

How to run an email A/B test step by step

  1. Set one goal. Choose the business outcome first—for example, completed bookings, orders, qualified replies, or donations. Pick one primary metric that reflects it.
  2. Write a clear hypothesis. State what you are changing and why you think it may help. For example: “A service-specific button may lead to more appointment requests than a general ‘Learn more’ button.”
  3. Choose a comparable audience. Use the same eligibility rules for both versions. If you are testing a campaign for recent customers, do not send version A to recent customers and version B to older leads.
  4. Change one variable. Keep the offer, audience, landing page, and other email elements consistent unless one of them is the variable under test.
  5. Divide recipients fairly. Use your email platform’s randomized split where available. For a small list, consider whether each group is large and representative enough to support a useful decision.
  6. Set the measurement window and decision rule. Decide how long you will wait for the action to happen and what evidence would be enough to choose a version. Do this before reviewing the results to reduce the temptation to cherry-pick.
  7. Send and let the test run. Avoid changing the campaign or stopping it early because one version has a temporary lead. Check that both groups received the intended versions.
  8. Review the result and record the lesson. Compare the primary metric, check that the difference is meaningful for the business, and note the audience and conditions. Use the learning to shape a future test.

Make the test large enough to be informative, but do not assume there is one suitable subscriber count for every business. The number you need depends on your audience size, usual response, and how large a difference you are trying to detect. If a list is small, use the result as a directional clue, repeat the test in a comparable campaign, or test a higher-impact change.

How to read email A/B test results

Start with the primary metric you chose, then check the supporting metrics to understand what happened. A subject line might attract more opens but fewer qualified visits. A shorter message could receive fewer clicks overall while producing more completed bookings. The business outcome should guide your decision.

MetricUseful whenWhat to watch for
Click rateYou want subscribers to visit a product, booking page, or article.Check that the destination and tracking work, and distinguish meaningful clicks from automated activity where possible.
Conversion rateThe goal is an order, booking, sign-up, or other completed action.Make sure the landing page and attribution window are the same for both groups.
Revenue per recipientYou are comparing promotional campaigns for an online store.Consider order value, refunds, and whether the offer changed between versions.
Reply or lead qualityYou want inquiries for a service or business-to-business offer.Review whether replies are relevant, not just whether there are more of them.
Unsubscribe and complaint signalsYou need to monitor whether a version creates an unwanted experience.A short-term lift in clicks may not be worth harming trust or list health.

Use open rates cautiously. Privacy protections and automated security systems can register opens or clicks without a person reading or engaging with an email. That can make a small difference look more convincing than it is. Prefer conversions, revenue, qualified replies, or human-verified clicks when those measures are available.

Also look beyond the winner label. Ask whether the difference is large enough to matter, whether it is consistent across comparable audiences, and whether the test was fair. If the outcome is close or the groups are too small, call it inconclusive rather than declaring a winner. A repeat test may be more useful than acting on noisy results.

Common email A/B testing mistakes

Testing too many changes: If the subject line, offer, layout, and button all differ, you cannot isolate what influenced the outcome. Keep a test focused.

Picking a winner by opens alone: Opens are easy to see, but they may not represent human interest or business value. Connect the metric to the action you actually want.

Sending each version to a different type of person: A group of loyal customers may respond differently from a group of new subscribers, regardless of the email. Split comparable recipients randomly.

Stopping as soon as one version leads: Early results can change as more people respond. Use a predetermined measurement window and decision rule.

Repeating a test without a question: Testing random wording changes can consume time without producing a useful lesson. Keep a simple record of the hypothesis, audience, variable, and result.

Test note: “For the spring appointment email, we compared a specific booking prompt with a general information prompt. The booking prompt led to more completed requests in this send. We will check the same approach with a comparable audience before treating it as a consistent preference.”

A practical checklist before you send

  • Have I chosen one primary business goal and a metric that reflects it?
  • Does my hypothesis explain why this change could help?
  • Are the two groups eligible in the same way and split fairly?
  • Is only one main element different between the versions?
  • Do both versions use the same offer, destination, and measurement window?
  • Have I checked the email on mobile and tested every link?
  • Will I account for automated opens or clicks where the reporting tool allows it?
  • Have I decided what I will do if the result is close or inconclusive?
  • Does the copy respect subscriber consent and the expectations set when they signed up?

Tools for running and learning from tests

Many email platforms provide native A/B testing, scheduling, and basic campaign reports. You can also keep a simple spreadsheet to record each test and its outcome. For more reliable decisions, make sure your measurement distinguishes human engagement from security scanners and link-preview bots where possible.

MailFlow includes A/B testing for subject lines, scheduling, and bot-filtered analytics, including scanner detection and human-verified clicks. You can review its AI email marketing features if you want those capabilities in one platform. These tools can help with campaign execution and measurement, but they cannot replace a clear hypothesis or a suitable audience.

If you are choosing software, compare the capabilities and limits that matter to your list and sending volume. MailFlow has a plans and pricing page; free or native tools may be enough when you send occasionally and need only a basic comparison.

Conclusion: make your next test answer one question

Email A/B testing is most useful when it answers a focused question about a real business goal. Choose one change, compare similar recipients, measure clicks or downstream actions carefully, and be willing to call a close result inconclusive. Before your next campaign, write down the hypothesis and success metric, then use what you learn to plan one better-informed test.

Frequently asked questions

How many subscribers do I need for an email A/B test?

There is no single subscriber count that makes every email A/B test reliable. The useful sample depends on your usual response rate, list size, and the size of the difference you want to detect. If your groups are small, treat the result as directional, repeat the test with a comparable audience, or choose a change with clearer practical value.

How long should I run an email A/B test?

Set a measurement window before sending, based on how quickly your subscribers typically act and how long the offer remains relevant. Do not stop just because one version leads early. For purchases or bookings that may happen after a visit, allow time for those actions and use the same attribution window for both groups.

Should I use open rate or click rate to choose the winner?

Choose the metric that matches your goal. Open rates can be affected by privacy features and automated systems, so they are a weak basis for many decisions. Clicks can be more useful for traffic goals, while completed orders, bookings, qualified replies, or revenue are better when those are the outcomes you want.

Can I test more than one thing in the same email campaign?

You can, but changing multiple elements between versions makes it difficult to tell which change affected the result. For a straightforward A/B test, keep everything the same except one main variable. More complex experiments require enough audience and a design that can separate the effects of each change.

What should I do if my email A/B test has no clear winner?

Record the result as inconclusive rather than choosing whichever version has a tiny lead. Check whether the audience was large and comparable enough, whether tracking worked, and whether the metric reflects your goal. You can repeat the test with a similar campaign or try a more substantial change that has a clearer reason to matter.

Author

Founder of MailFlow, software engineer working across the full stack and every major programming language, specialising in AI automation, email deliverability and SEO.

Related articles

Start free with MailFlow

Free plan forever, no card needed. Upgrade when you grow.

See pricing