Email A/B testing: a practical guide

Torn between two subject lines, two buttons or two send times? Instead of guessing, email A/B testing lets your recipients settle it for you, with numbers to back it up. It is the simplest way to stop steering on instinct and improve campaign after campaign. Here is what you can test, how to run it without tripping yourself up, and above all how to read the results without kidding yourself.

What is A/B testing in email marketing?

A/B testing, also called a split test, means creating two versions of the same email that differ on a single element (variant A and variant B). You send each version to a small sample of your list, measure which one performs better, then roll out the winner to all the remaining contacts. In practice, the process comes down to three steps.

Most email marketing platforms build this feature in and automate the whole flow, including picking the winner and sending the final version. A/B testing is not reserved for experts: it is a reflex you can adopt from your very first campaign.

What can you test in an email?

Almost everything can be tested, but some elements carry far more weight than others. Here are the five classic levers, from the most rewarding to the most refined:

• The subject line: this is the most common test and often the most profitable, because it decides the open. Compare short against long wording, a question against a statement, with or without an emoji, with or without the first name.
• The sender name: “Marie from MyBrand” against “MyBrand”, or a human address against a noreply@. The impact on trust and opens can be clear.
• The content: text length, images or no images, block order, tone of voice. These tests mainly move engagement and clicks.
• The call-to-action (CTA) button: its wording (“Grab the deal” against “See the offer”), its color, size or position. This is a direct lever on the click rate.
• The send time and day: Tuesday 10 a.m. against Thursday 2 p.m., for example. Careful though, this test follows special rules we detail further down.

A word on priority: tests on the subject line and the CTA generally deliver more value than tests on the sender name or the footer. Start there. Note too that the subject line mainly influences the open rate, while the content and CTA act on the click rate.

What method should you follow for a reliable test?

A badly framed A/B test produces false conclusions, which is worse than no test at all. Three rules keep the approach solid.

One variable at a time. This is the non-negotiable rule. If you change both the subject line and the CTA, and version B wins, you will never know which of the two made the difference. To test several elements, run several successive tests, or use a dedicated multivariate test if your list is very large.

A clear hypothesis before you launch. Do not test “just to see”. Frame an intention: “I believe a subject line with the recipient’s first name improves opens”. That way you know what to measure and what to do with the result.

The right success metric. Choose the metric before the test, based on what you are testing. A subject line test is judged on the open rate, a CTA test on the click rate, an offer test on conversions. Judging a subject line test on clicks makes no sense.

What sample size and duration do you need?

This is the point that separates a real test from a lucky guess. A gap measured on 40 recipients proves nothing; on several thousand, it becomes credible.

Sample size. Common benchmarks point to a minimum of around 1,000 recipients per variant to hope for a reliable result, so at least 2,000 contacts involved in the test. It is not a magic threshold: the smaller the performance gap you expect, the more volume you need to detect it. Below a few hundred contacts per version, an A/B test has little chance of settling anything clearly, and sometimes you are better off simply sending your best hunch.

Indicative benchmarks, July 2026, aggregated from provider guides and email marketing platforms. Exact thresholds depend on your baseline open rate and the size of the difference you are chasing.

Duration. Let the test run long enough to smooth out swings from one moment of the day to another. A window of 3 to 7 days is often recommended for standard campaigns; some tools pick a winner after just a few hours, which can be enough for a simple subject line test but stays more fragile. In B2B, stretch the window: your contacts check their email mostly during office hours.

How do you interpret A/B test results?

One version wins by two-tenths of a point? That is not necessarily a winner. The real question is whether the gap is statistically significant, meaning too pronounced to be down to chance. The convention in email marketing is to aim for a 95% confidence level: below that, the result is treated as inconclusive.

In practice, most platforms display a confidence indicator directly, or only declare a winner once the threshold is reached. If your tool does not, hold on to this simple principle: a small gap on a small sample proves nothing, a clear gap over several thousand sends is credible. Three reflexes to conclude well:

1. Check significance before acting. As long as the 95% threshold is not met, treat the test as unsettled and do not turn it into a general rule.
2. Tie the result to a real goal. A better open rate that produces neither clicks nor conversions brings nothing. Trace the chain up to the metric that actually matters to you.
3. Document and repeat. Log the hypothesis, the winner and the gap. A/B testing is a cycle: each test feeds the next, campaign after campaign.

A test that leads to no decision, or whose gap is not significant, is not a failure: it is a hypothesis ruled out, and time saved.The MiisterSoftware team, the principle of continuous optimization.