Creating and Scheduling an A/B Test Campaign

A/B test campaigns let you test different subject line and preview text combinations on a portion of your contact list before sending to everyone. SendX sends your variations to a test group, measures which one gets the highest open rate engagement, then automatically sends the winning version to the rest of your contacts.

A/B testing is valuable when you're unsure which subject line will resonate best with your audience. Rather than guessing, you let actual engagement data decide. This is particularly useful for promotional campaigns, product announcements, or any email where maximizing opens matters. Over time, A/B testing helps you learn what language, tone, and formatting your subscribers respond to.

Here's the basic flow of an A/B test campaign:

Contacts who received a test variation do not receive the final winner send. They're excluded since they already got the email.

If you already have A/B test campaigns, you'll see them listed on this page.

On the first screen, you'll add your subject line and preview text combinations.

Adding Variations

You start with two variation fields. Click Add Variation to add more. There's no maximum limit, but keep in mind that more variations require a larger test group to get meaningful results.

For each variation, enter:

Using the AI Subject Line Assistant

If you need help coming up with ideas, click the AI assistant icon next to any subject line field. Enter a rough idea of what you want to say, click Generate, and SendX will suggest alternatives. You can use any suggestion directly or use it as inspiration.

Once you've added all your variations, click Next to proceed.

Choose which editor you want to use for creating your email:

Before moving forward, preview your email to see how it looks:

Click Send Test Email to send a preview to yourself or your team. This lets you see exactly how the email appears in a real inbox before scheduling.

When you're satisfied with your design, click Continue.

Choose who will receive your A/B test campaign.

Include Recipients

Select your audience using:

Exclude Recipients

You can also exclude specific contacts from this campaign. Use the exclude option to select lists, tags, or segments that should not receive this email.

This is useful for scenarios like excluding contacts who received an email recently. To do this, create a segment with that condition beforehand, then select it in the exclude section.

This is where you define how your A/B test will run.

Test Percentage

Choose what percentage of your selected contacts will be in the test group:

The test percentage is divided evenly among your variations. If you have 3 variations and select 30%, each variation gets sent to 10% of your contacts.

Wait Time Before Sending Winner

Configure how long SendX should wait after sending the test before determining a winner and sending to the remaining contacts. This gives recipients time to open the test emails so the results are meaningful.

A longer wait time gives more accurate results but delays your full campaign. Consider your audience's typical email checking habits when setting this.

How the Winner Is Determined

SendX automatically selects the variation with the highest open rate as the winner. This is not configurable. Once the wait time passes, the winning variation is sent to all remaining contacts who weren't part of the test.

Click Continue to proceed to review.

The review screen shows a summary of your campaign settings. Check that everything looks correct:

Fixing Issues

If there are any problems with your campaign (like missing content or invalid settings), they'll be listed at the top of the review screen. Fix any issues before proceeding to scheduling.

Choose when to send your campaign:

Send Right Away

Select this option to start the A/B test immediately after you click Submit.

Send Later

Schedule the campaign for a future date and time:

The A/B test will begin at your scheduled time. The winner will be sent after your configured wait period.

Google Analytics Tracking

Enable Track campaign click report on Google Analytics to add UTM parameters to all links in your email. This lets you track campaign traffic in Google Analytics.

Click Send to confirm all settings, then Submit to schedule your campaign.

You can edit an A/B test campaign after scheduling but before it starts sending. Navigate to Campaigns > A/B Split Test, find your campaign, and click to edit. You can modify subject lines, content, recipients, or scheduling.

To cancel a scheduled campaign, open it and change its status or delete it before the scheduled send time.

A/B test campaigns have specialized reporting that shows how each variation performed and which one won.

The Overview tab shows a comparison table of all your variations with these columns:

This table lets you quickly see which variation won and how each performed relative to the others. The winner shows a green checkmark, while non-winning variations show a red X.

The Performance tab provides detailed metrics for each variation. Use the dropdown in the top right to select which variation to view.

Overall Performance Metrics

Engagement Over Time

A line chart showing opens and clicks over time. This helps you understand when recipients engaged with each variation.

Geographic and Device Stats

Pie charts showing where your recipients are located and what devices/email clients they used to open your email.

Use the variation dropdown to switch between different subject lines and compare their detailed performance. This helps you understand not just which one won, but why it might have performed better.

If you're happy with your email design, click Save as template to reuse it in future campaigns.

Winner Determination

The winning variation is determined solely by open rate. Click rate and other metrics are tracked but don't affect which variation wins.

Test Group Size Considerations

While SendX doesn't enforce a minimum contact count, A/B testing works best with larger lists. With small lists, the test group may be too small to produce statistically meaningful results. As a general guideline, aim for at least 100 contacts per variation in your test group for more reliable winner selection.

Excluded Contacts

Contacts who receive a test variation are automatically excluded from the final winner send. They won't receive duplicate emails.

What happens if I have a very small list?

The campaign will still run, but your results may not be reliable. With only a few contacts per variation, a single open can dramatically swing the percentages. Consider using a higher test percentage (30-40%) for smaller lists to get more data points.

How long should I wait before sending the winner?

It depends on your audience. Most email opens happen within the first 24-48 hours. A 2-day wait is common, but you might go shorter for time-sensitive promotions or longer if your audience tends to check email less frequently.

Can I use personalization in subject lines?

Yes. You can use contact personalization tags like first name in your subject lines, just as you would in regular campaigns.

Do I need different preview text for each variation?

No. You can test subject lines while keeping the same preview text across variations, or test different preview text with the same subject line, or vary both. It's up to you.