The A/B Testing feature enables you to test variations of a message and then review the results in order to better understand your subscribers based on their interaction (or lack of) with the message.

This works by you configuring two slightly different versions of a message -- and then sending one version of the message to a subset of the test group, and the different version to another subset. It's this difference that you're "testing." A third subset of the test group will the automatically receive the "winning" version at the conclusion of the testing period.

NOTE: A/B Testing can be set up by designing a new message or editing an existing one.

By examining which version performed better (e.g., had more opens or more clicks), you can better understand what your subscribers prefer.

Watch the video

A/B Testing types

You can perform A/B Testing on the following aspects of a message; these are the "types." The table offers guidance on how each type can be used.

Type	Description
Friendly From address	Use this to test whether different from addresses impact recipient actions and responsiveness. Choose this on the Properties tab > Envelope sub-tab.
Subject Line	Use this to test whether the language of the subject line impacts recipient actions and responsiveness. Choose this on the Properties tab > Envelope sub-tab.
One Story	Use this to test how message content, that is basically similar, is perceived by and responded to. In the Message Designer, select one story as the target of the testing. Refer to A/B Testing message stories, below.

When you decide on what type of testing you want to run, there are a few basic steps to setting up A/B Testing. You will:

Design your new message or existing message
Configure a few basic options, such as its properties, recipients and exclusions, and a reply email address
Review the message and then set schedule and delivery options

1 - Choose or create a message

A/B Testing can be done when creating a message or by editing an existing message. Both methods start by accessing the Messages List page.

To access the Messages > List page:

Access the Admin interface.
Navigate to the Messages > List.

Continue with A/B Testing for an existing message (immediately below) or jump to A/B Testing for a new message.

A/B Testing an existing message

NOTE: These instructions are for Message Designer-based messages only; they are not applicable to other message types.

If the Type column is not displayed, click the column choose below the list and select Type.

Make sure that the message you want to use is a Message Designer type message.
Click the message to select it and then click Edit in the upper right to open the message in the Message Designer.
Continue with 2 - Configure the properties, below.

A/B Testing a new message

Click the plus symbol to the right of the page title.

Leave the default of Message Designer and click Create.
On the template-selection page:

Click a template to choose it or use the dropdown to access the templates in other folders. OR...
Click or click here to create a new Template! to create a new template and then select one from the Template Gallery or Your (account's) Templates. (Again, use the dropdown to access templates in other folders.)

Continue with 2 - Configure the properties, below.

NOTE: Selecting a template automatically moves you to the Message Designer page with your template loaded.

NOTE: See the Design tab section of the Create a Message (MD2) article for instructions on how to build or edit your message. Then return here and resume with the next section.

IMPORTANT: If you are A/B Testing the One Story type, refer to A/B Testing message stories, below, for instructions on how to configure that in Message Designer. When you've finished, click the link to return here to continue.

2 - Configure the properties

Now that you have the message designed, work through the fields of the Properties tab to configure the message's A/B Testing properties.

The Properties tab has four several sub-tabs:

Name
Targeting
Envelope
Analytics

Name sub-tab

In the upper section of this tab:

Specify a name for the message (required).
Optionally, choose a folder other than the default, Main Folder.
Optionally, specify a send date and time other than the default of the current day and the soonest hour.

TIP: Review your settings and then proceed to the Targeting tab. You can return here to make changes.

Targeting sub-tab

In the upper section of this tab:

Navigate the folder structure to select (click) groups that you want to include and exclude. You can include and exclude multiple groups.

In the above image, in the Interests group, the 14 members of the Space Station group have been included and the six members of the Moon Colony group have been excluded; note the changes to their icons on the left. In this case, if any of those six Moon Colony members are also in the Space Station group, they will be excluded from the A/B Testing group.

In the lower section of this tab:

Specify up to 15 Email Addresses to also receive a copy of the test message.
Use the dropdown to select one account-specific Test Group to receive a copy of the test message.
Click show advanced targeting to set any or all of the three exclusions.

TIP: Review your settings and then proceed to the Envelope tab. You can return here to make changes.

Envelope sub-tab

On this tab:

If you're A/B Testing type is not One Story, click the a | b button to choose either Friendly From or Subject Line.
Complete the fields by specifying email addresses in the From, Friendly From, and Reply To fields, and typing your subject content into the Subject Line field.

If your A/B Testing type is Friendly From, there will be a Friendly From A field and a Friendly From B field; be sure to complete both.
If your A/B Testing type is Subject Line, there will be a Subject Line A field and a Subject Line B field; be sure to complete both.
If your A/B Testing type is neither Friendly From nor Subject Line, there will be one each of these fields.

NOTE: The Use Targeted Subjects option is not available with A/B Testing.

TIP: Review your settings and then proceed to the Analytics tab. You can return here to make changes.

Analytics sub-tab

If you want to use Google Analytics when A/B Testing this message, see Google Analytics (MD2).

TIP: Review your settings and then proceed to the Review & Send tab. You can return here to make changes.

3 - Review and send the test messages

The final part of the process has a few components. On the Review & Send tab, you can:

review and test the message,
solicit feedback on it, and
set up scheduling and delivery options.

NOTE: See the Review & Send tab section of the Create a Message (MD2) article for comprehensive instructions on the options. Then return here and resume with the next section if you want to have the "winning" message automatically sent to your non-participant contacts.

IMPORTANT: When using A/B Testing, you should send a "test" message to a few recipients to ensure it displays as expected. Test recipients will be sent both versions of the message.

See Create a Message (MD2) for information on testing a message.

Send to Winner feature

With A/B Testing, your target group has been split in order to see which testing version performed better. The Send to Winner feature let's you automatically send the better-performing version to the remainder of the target group (i.e., those members who were not sent a test message).

NOTE: This feature is not compatible with the Send Message Now and Triggered scheduling options. If you've selected either of those, make a different selection to enable/display this feature.

Notes

There must be at least 20 contacts in the target group to use this feature.
If there is a tie between the A and B versions, the A version will be sent to the remaining contacts.
In the New Admin Interface, you can cancel sending the "winner" message by navigating to the Messages > Sent list page, choosing the "winner" message, and clicking More > Stop Sending.

To set up Send to Winner:

Navigate to the Delivery Methods section and select Send to Winner.
Click the dropdowns to set:

What percentage (5% to 40%, in 5% increments) of contacts to send each version of the message to.
NOTE: The "...remaining contacts %" value in the next option will auto-adjust based on your selection.
How long to wait, after the test messages were sent, to send to the non-participants.
NOTE: This must be at least 4 hours after the initial send time.
What constitutes the "winner:" Highest Delivery Rate, Highest Open Rate, Highest Click Rate, or Lowest Unsubscribe Rate.

A/B Testing message stories

You can set up A/B Testing for one story in a message to determine whether the layout and/or content of your stories could be improved.

Note

If you want to set up targeting while working with a story, navigate to the Options tab > Envelope sub-tab and choose a group beforehand; refer to Options: Targeting. Otherwise, targeting will have to be set up afterwards.

A/B Testing a single story

To test a story:

In Message Designer, click a story to select it. This is your A version.

In the editor panel on the left, click A/B Test this story.

Choose a content type from the pop up or copy the A version. This will be the B version for the test.
Update the B version in the editor panel on the left and click Close to save your changes.

Both versions display so you can easily compare their differences.

You can edit or delete either of these stories. If you delete either story, A/B Testing will be automatically disabled.

Return to the 2 - Configure the properties section to continue.

Set up targeting

If you have chosen a target group on the Options tab > Envelope sub-tab, you can isolate a subset of that group to receive the A version of the story.

To target a subset:

Click the A version of the story to select it.
Click the Targeting tab and then check the Send this story... box to see the list of target groups.
Navigate the folder structure to select (click) a group that you want to include. The group name will appear in the shaded field under Included.

NOTE: You can include only one group. If you add a group and then click a different group, the prior group will be removed.

Click Close to save your changes.

A/B Testing an entire message

You can test an entire message, but it has to be done manually.

Select a target group or compile a group of select recipients.
Manually send the A version to a segment (50%) of the group, and then send the B version to the other segment (50%).

While this is not automated like standard A/B Testing, it will enable you to get two sets of results for comparison.

Use random segmenting

You can also use Higher Logic Informz's random segmenting feature to quickly create subsets a target group. Learn how to do this in Splitting and Random Segmenting.

A/B Testing (MD2)