Definition & context
A/B testing (or multivariate testing) can be used to evaluate the performance of your campaign on a sample of your target. The testing possibilities are up to you, whether you want to test the subject of the design, the design itself, texts, or even your target’s behaviour, be it the number of clicks or total revenue from the campaign.
You can choose to send only one message to a small percentage of your database, or test many designs at once, simultaneously.
The best performing message can be selected manually, or Splio can automatically select and send your best message, based on the criteria you chose.
Openings: If you want to run an A/B test on best open rates, please refer to the dedicated section in our article on Apple Mail Privacy Protection.
How to - Steps
Creating an A/B Test campaign
To create an A/B Test, create a regular email campaign and choose your initial population.
The A/B Test will take only a sample of the population, and will keep the remaining recipients as target for the final action.
Once your population is computed, add the first sequence, select A/B Test, and input the date and time you wish to launch the test.
Your scenario will contain 2 actions by default, but you can add more with the configuration panel. Click on the A/B Test Configuration box to open the modal window.
A/B Test Configuration
The volumes tab is where you select the number of branches for your test, one action (message) per branch.
If you wish to test 3 messages, click on the 3 branch icon. If you wish to test more than 6 messages, you can set a custom number in the input.
You can then select the number of recipients that will be selected for each action.
The samples' recipients are selected at random from the initial population.
A/B Test Settings
The settings tab contains the validation mode. You can either keep the final message selection manual or set a criterion, date and time for the automatic selection and sending of the final message.
Automatic selection criteria
Splio can select the best performing message for you. The available criteria are:
- Best open rate (unique openers)
- Best click rate (unique clickers)
- Most buyers
- Highest total sales
To make its selection, Splio will analyze, at the date and time set, all the tested message's statistics, and select the best according to the criteria.
If there is a tie on the criteria selected, Splio will send you an email and ask for your manual input.
What happens if you want to stop the sending of the automatic winner?
If you have selected the option of the winner design to be sent out automatically at the end of the A/B test but for some reason you wish to stop this action and not send it. It is possible to do so by editing the campaign and switching back to a manual selection of the winner as shown below:
Select "manual selection of final design":
The other tabs contain the same settings as for the regular campaigns, such as the seed list and the submission speed limit.
These settings will be applied to all actions of the A/B Test.
Configure the messages (actions)
Once you have validated the A/B Test settings, including the number of branches, select all the messages you wish to test.
You can then save the scenario, review the campaign summary, and launch the A/B Test.
Comparing the results
You can compare the results of your A/B Test and evaluate each message's performance.
Go to the campaign's list, find your A/B Test and click on "Compare A/B Tests".
The campaign overview presents all actions within the test, the targeting summary, as well as the global statistics, which are presented as a comparative table of all the tested messages' statistics.
Minimum and maximum values are highlighted, for each indicator. You can have the highlight switch from absolute values to rates.
Choosing the message
If you chose the automatic A/B Test validation, your message is selected and sent automatically at the time and date you selected. When the final message is sent, it is available in the campaigns list within your A/B Test campaign.
If you have to choose the final message manually, go to the campaigns list and click on the edit button at the campaign level.
Open the scenario editor, and click on "Choose final action". From there, you will be able to compare the statistics of tested messages, select the best performing message, and select the date and time to send the message to the remaining recipients.
After the complete sending of the A/B Test campaign, both of the designs will be automatically duplicated and available for use into the Designer module.
When - Our tips to time your campaign well
Test one thing at a time
In order to run a successful A/B test, you should only test one thing at a time so you can analyse your results: either the subject line, the pre-header, or both or the body of your design. It is important to isolate the tested area to be able to identify what has delivered results afterwards.
Make sure that you have something to base your analysis on. If you have existing statistics, you will be able to measure this against your test.
Testing simultaneously and continuously
Challenge yourself with new tests: although you should focus on one thing at a time, you can still run several A/B test campaigns in parallel. For example, you can test the length, urgency, mention of a promotion or use of the recipient’s name.
Timing is everything. Make sure you test different days and time of the day: emails don’t perform the same depending on when you send them.
Have a clear goal in mind
You need to increase the open rate: focus on subject line and pre-headers
You need to increase your click-through-rate downloads or subscriptions: focus on your content.
- word order (try reversing the order)
- tone (neutral, friendly, provocative)
- length (try shorter, try longer)
- personalization (try including their first name)
Try and introduce pre-headers to trigger the customer’s curiosity and open your emails, the more they are linked to the subject line, the better. See example:
Subject line: We’re so glad you signed up. Here’s what’s next.
Pre-header text: You’re in! Welcome to the easiest way to shop online. Let’s get acquainted.
It's all about the visual. Re-arrange the positioning of the product catalogue to achieve a different impact and boost click rate.