NOTE: This article is for customers with the "classic" Admin interface (horizontal top-navigation bar).



If your interface is different, view the New Admin Experience version of this article.

A/B Testing campaigns allow you to create two or more sample groups of recipients. You then choose to send each group either completely different messages or different versions of the same message (using different subject lines, from fields, etc.) to see which is more effective. After you configure and deploy the campaign, Higher Logic Thrive Marketing Enterprise (Thrive Marketing Enterprise) will manage sending each version and tracking the results. At a date and time you set, it will determine which version is the "winner" of the test according to the criteria you entered during campaign setup.

The information provided by A/B testing can be important to fine-tune your emails. For example, is one subject line more effective than another? Does sending at a specific time of day lead to better success?

Campaign Info

Campaign Details

• Basic A/B campaigns can only use groups for enrollment.
• You can have between 2 and 100 samples. During campaign setup, you’ll specify whether each sample is sent to a percentage of the enrolled recipients or a set number. (e.g., 10% of recipients receive sample A and 10% receive sample B, or 500 recipients receive sample A and 500 receive sample B.)
• Samples can have different subject lines, From addresses, time of send, or even completely different messages.
• You’ll set a time after the samples are sent to determine which is the "winner." The winner can be determined by either open rate or click rate.
• When the winner is determined, you can choose to receive a notification. You can also choose to automatically send the winner to the rest of the enrolled recipients.

Campaign Ideas

• Test recipient open rate based on different subject lines.
• Design two different messages for the same product, and determine which is better in driving web traffic to your sales page.
• Send the same message in the morning and in the afternoon to see if time of send affects open rate.

Create & configure an A/B Testing campaign

Step 1 - Create the campaign

To create an A/B Testing campaign:

1. Navigate to the Automation tab.
2. Give your A/B test campaign a descriptive name, and then click GO.

3. Select your industry from the first menu. Then, select the A/B Testing Campaign Type from the second.
4. Click Select.

Step 2 - Configure the campaign

For this step, you'll select the group(s) of recipients, your test variable, and select the test's "winner" criteria.

1 - Select campaign recipients

The first choice to make is which recipients will be enrolled.

1. Enrollment is restricted to groups for A/B Testing campaigns. Click the Select Group(s) link to choose one or more groups.

NOTE: When the campaign is deployed, all recipients in these groups will be added to the campaign. Any new members added to the groups after the campaign is deployed will not be added to the campaign.

2. If you'd like to filter the recipients in your selected group(s), click + condition to add a condition. You can select conditions like Registered for an Event, Submitted a Landing Page, and Has a Specific Number of Points. Simply click + condition again to continue adding additional conditions, if desired.
3. Select how many samples you'd like to use in your campaign. You can have anywhere from 2 to 26. Click Submit to create the sample.
4. Then, choose how large each sample should be. You can choose to set the sample size based on the number of recipients or a percentage of the total (select the desired option from the menu).

1. If you choose % of total, the number you enter will be the percentage of recipients added to each sample when the campaign is deployed. The value must be between 1 and 100. You can also drag the available slider to determine the percentages.
2. If you choose Recipients, the number you enter will be the number of recipients added to each sample when the campaign is deployed. NOTE: Because recipients are not enrolled until the campaign is deployed, it’s possible to have a larger number of recipients assigned to the samples than are in the campaign. If you want to use large sample sizes, you may want to first conduct a Recipient Search for the groups you're using for enrollment. This will give you an estimate of how many recipients you can expect.

When the campaign is deployed, random campaign members will be added to fill each sample until it meets the threshold you specified. These campaign members will be sent the appropriate message for their sample. The remaining recipients will not be sent a test message.

An important thing to remember when setting up your samples is that they are large enough to give a usable result. Choosing a very small percentage from a very small sample will likely leave you with a result due more to random chance than any meaningful differences.

2 - Select test variable

Next, select the variable you would like to test.

• Select message if you would like to send a completely different message to each sample. You will be prompted to select the message to send to each group. Then, select a date and time to send your messages. Lastly, click the send options link to enter the other information (subject line, from address, etc.) to apply to all of the messages sent.
• Select subject line to enter the subject line for each sample. Then, select a date and time to send your messages. Lastly, click the send options link to select the message you want to send and enter the other information (subject line, from address, etc.) to apply to all of the messages sent.
• Select pre-header to change the pre-header text for each message. This text will appear when the message is viewed as part of a list in certain email clients (for example, Gmail.). Then, select a date and time to send your messages. Lastly, click the send options link to select the message you want to send and enter the other information to apply to all of the messages sent.
• Select from field to enter a from address for each sample. If your account has a default from address option, it will appear. Then, select a date and time to send your messages. Lastly, click the send options link to select the message you want to send and enter the other information to apply to all of the messages sent.
• Select time of send to have each sample message sent at a different date and time. Then, click the send options link to select the message you want to send and enter the other information to apply to all of the messages sent.

3 - Determine winner

At this point, you'll specify how Thrive Marketing Enterprise should determine which message “wins” the A/B test: By open rate or click rate.

1. From the first menu, select either open rate or click rate: Open rate is the percentage of messages opened by the recipients in each sample; click rate is the number of clicks per message in each sample.

2. You can choose either to have no minimum rate or a minimum rate for the winner:

1. If you choose no minimum rate, then whichever sample has the better percentage will be the winner.
2. If you choose a minimum rate, you will be prompted to enter a percentage. In order to be considered the winner, a sample has to have an open/click rate equal to or greater than the percentage you enter. If no sample meets that threshold, then no sample will be considered the winner.

3. Select a date and time to determine the winner. This is the time that Thrive Marketing Enterprise calculates which message “won.” It must be after the last send time for all messages sent. It is a best practice to give your samples plenty of time to determine a winner. Typically, this is one or two business days (less time will prevent your recipients from all having a chance to see the message).

4. Lastly, you need to decide what to do when the winner is determined:

1. The first option is to send a notification to the email address you enter with the results of the A/B test.
2. The second option is to automatically send the winner to the remaining campaign members. This means that, after the winner is determined, the winning message will be sent to all recipients in the campaign who were not sent any test message.

EXAMPLE: You run a test with three messages – A, B, and C – which are each sent to 10% of the enrolled recipients. When the test is finished, message A has won. Any recipient who was not sent any test message will be sent message A. Any recipient who was sent messages B or C will not be sent message A.

Step 3 - Review the campaign

Click Next to review your campaign's configuration on the resulting page.

• If you're satisfied with your campaign, click Next to continue.
• If you want to make some changes, click Setup Campaign in the top bar.

Step 4 - Deploy the campaign

Click Next again to go to the Deploy Campaign page. From here, you can schedule the campaign to either start immediately upon deployment or on a specific date. Similarly, choose whether to end the campaign manually or on a specific date and time.

Click Deploy Campaign to enroll recipients and begin your A/B Test per your selected time-frame.

What's next?

Once your is deployed, you're going to want to track it's progress. See Track A/B Testing Campaigns to learn more.