A/B Testing enables you to test variations of your mailing. The basic approach is this - take your target group and identify a reasonable percentage as a test group. Send one version of the mailing to half of the test group, and a different version of the mailing to the other half. By examining which version did better (i.e., had more opens and clicks), you can better understand what your subscribers prefer.

You can perform an A/B Test on:

• Friendly From address
• Subject Line
• One Story
• All Stories

A/B Testing helps you to determine what changes can be implemented to improve delivery and open results for your mailings.

How to set up A/B Testing

A/B Testing is available on the Set-Up tab > Properties area of a mailing. Click Edit to select the type of test you want to perform.

On the next page, select the type of testing you want to perform (the default is None).

If you select Friendly From or Subject Line, the Envelope area displays to allow you to create both versions of either the friendly from or subject line. These are shown below:

To edit, click Edit to open a new pop-up window. Make your changes and then click Update to save.

Friendly From

Subject Line

Testing a subject line is a very worthwhile exercise. The subject line remains a very important factor in whether a mailing is opened.

One Story

Testing one story at a time is a good approach if you're wondering whether the layout of the stories in your mailing could be improved.

When the One Story option is selected, you can toggle between the A Version and the B Version.

Click on the A Version and the content for that mailing version will appear. Pick a story within the A Version of the mailing and select "edit." This One Story option only allows a single story within the mailing to be tested.

Click the check box to Enable A/B Testing for this Story In the editor window that appears.

Next, click the Copy A content to Version B tab. This will automatically copy and paste all of the HTML from Version A of your mailing and paste it into Version B of your mailing. Go into Version B and make the changes you want - compared to the A version - and then save that story.

You can toggle between the A and B versions using the available tabs shown below.

All Stories

This testing option enables testing on every story in your mailing. The process to create and edit a story is identical to the One Story option.

Sending

Once you've created your mailing and are ready to send, the A/B Test option you've selected is displayed on the Send tab.

Reporting

To check the status of the mailing after it has been sent, navigate to the Sent tab on the mailing list page.

Hover over the icon to the left of the mailing name and select Report. A comparison of both mailing versions is displayed.

Send to Winner feature

With A/B tests, you typically split your target group to see which responds better to what you're testing. This could be different subject lines, friendly froms, stories, graphic density, font size, etc. The Send to Winner feature let's you automatically send the better-performing version to the remainder of the target group outside your test segment.

The only thing you have to do is decide:

• How long to wait before sending after the test
• and how you want to determine who the "winner" is.

1. After specifying the type of A/B test you want to do on the mailing's Set-Up tab, navigate to the Send tab.
2. Click Schedule Mailing.
3. Set your Schedule Option to Once, and choose an initial send time.
4. Set your Delivery Method to Send to Winner.
5. Configure your Send to Winner criteria:

1. Percentage - Each A/B version of the mailing can be set to a percentage of subscribers - anywhere from 5% to 40%, in increments of 5%. After the winner is determined, the rest of the targeted subscribers will receive the winning version.

EXAMPLE: If the you select 20%, then 20% of the target group receive Version A and 20% receive Version B. After the winning version is determined, the remaining 60% of the target group receives the winning version.

2. Timing - Select the desired date and time. At this date and time, the following will happen: The final tallies are made for the two A/B versions; the winner is determined; the mailing is sent to the rest of the target group. The time and date selected must be at least 4 hours after the initial publication of the A/B test. This ensures that enough time has passed to get feedback for a winner.
3. Determining winning version - Because each organization's metric for determining success may vary, you can choose one of four options: (1) Highest Delivery Rate, (2) Highest Open Rate, (3) Highest Click Rate, or (4) Lowest Unsubscribe Rate.

Notes

A mailing has to have at least 20 subscribers in order to use A/B Send to Winner. If it doesn’t, the mailing just uses A/B Testing as normal.

If there is a tie between the A version and B version, the A version is sent to the remaining subscribers.

A mailing that uses A/B Test is still a regular mailing. It is not a triggered mailing.

A/B Testing & reporting for Recurring Mailings

A recurring mailing is one that is queued up to be sent on a regular, scheduled basis (e.g., sending a meeting reminder on the 15th of every month).

If you're using the Send to Winner feature with A/B testing, this presents a new wrinkle in reporting because there is an inherent time lag between the test groups being sent the mailing and the remaining subscribers receiving the "winning" version from the test group.

Mailing Activity Report statuses

If the initial A/B mailings have been sent but the final mailing has not yet been sent to the remaining subscribers in the target group, all reporting activities display this fact in the yellow bar at the top of the report:

When the original A/B Test is completed, the reports display a message in the yellow bar giving the results of the test:

The Options report now shows information about any A/B Send to Winner data used in the publishing of the mailing: