A/B testing (or multivariate testing) can be used to evaluate the performance of your campaign on a sample of your target. The testing possibilities are up to you, whether you want to test the subject of the design, the design itself, texts, or even your target’s behaviour, be it the number of clicks or total revenue from the campaign.
You can choose to send only one message to a small percentage of your database, or test many designs at once, simultaneously.
The best performing message can be selected manually, or Splio can automatically select and send your best message, based on the criteria you chose.
Creating an A/B Test campaign
To create an A/B Test, create a regular email campaign and choose your initial population.
The A/B Test will take only a sample of the population, and will keep the remaining recipients as target for the final action.
Once your population is computed, add the first sequence, select A/B Test, and input the date and time you wish to launch the test.
Your scenario will contain 2 actions by default, but you can add more with the configuration panel. Click on the A/B Test Configuration box to open the modal window.
A/B Test Configuration
The volumes tab is where you select the number of branches for your test, one action (message) per branch.
If you wish to test 3 messages, click on the 3 branch icon. If you wish to test more than 6 messages, you can set a custom number in the input.
You can then select the number of recipients that will be selected for each action.
The samples' recipients are selected at random from the initial population.
A/B Test Settings
The settings tab contains the validation mode. You can either keep the final message selection manual or set a criterion, date and time for the automatic selection and sending of the final message.
Automatic selection criteria
Splio can select the best performing message for you. The available criteria are:
- Best open rate
- Best click rate
- Most buyers
- Highest total sales
To make its selection, Splio will analyze, at the date and time set, all the tested message's statistics, and select the best according to the criteria.
If there is a tie on the criteria selected, Splio will send you an email and ask for your manual input.
What happens if you want to stop the sending of the automatic winner?
If you have selected the option of the winner design to be sent out automatically at the end of the A/B test but for some reason you wish to stop this action and not send it. It is possible to do so by editing the campaign and switching back to a manual selection of the winner as shown below:
Select "manual selection of final design":
The other tabs contain the same settings as for the regular campaigns, such as the seed list and the submission speed limit.
These settings will be applied to all actions of the A/B Test.
Configure the messages (actions)
Once you have validated the A/B Test settings, including the number of branches, select all the messages you wish to test.
You can then save the scenario, review the campaign summary, and launch the A/B Test.
Comparing the results
You can compare the results of your A/B Test and evaluate each message's performance.
Go to the campaign's list, find your A/B Test and click on "Compare A/B Tests".
The campaign overview presents all actions within the test, the targeting summary, as well as the global statistics, which are presented as a comparative table of all the tested messages' statistics.
Minimum and maximum values are highlighted, for each indicator. You can have the highlight switch from absolute values to rates.
Choosing the message
If you chose the automatic A/B Test validation, your message is selected and sent automatically at the time and date you selected. When the final message is sent, it is available in the campaigns list within your A/B Test campaign.
If you have to choose the final message manually, go to the campaigns list and click on the edit button at the campaign level.
Open the scenario editor, and click on "Choose final action". From there, you will be able to compare the statistics of tested messages, select the best performing message, and select the date and time to send the message to the remaining recipients.
N. B. : After the complete sending of the A/B Test campaign, both of the designs will be automatically duplicated and available for use into the Designer module.