Email Variants: Content Testing Emails in EXM

A/B Testing of Emails

 25 Days of Sitecore EXM reaches the midway point with Day 13! Today, we’re taking a bigger look at A/B Variant Testing within Sitecore Email Experience Manager.  Sitecore Experience Platform comes complete with an entire suite of content testing tools, and email was not left out!

Setting Up Our Marketing Strategy

On our fictitious site, we have two landing pages that we want to market, but unsure which one will drive the most value, or be the most successful. So, in a small scale test, I have created two landing pages and selected goals on each item that have a low or a high value. I have called these pages Low Value and High Value. to simulate the value of a visit after clicking through on an email.

I’m using the Tracking field to record the goal to trigger when someone lands on this page. And for clarity, I have adjusted the Points on these goals to derive at a value.

Specifying Our Target Segment

Using List Manager, I have created a list made from a CSV file of email addresses and contacts that I have imported from a 3rd party system. However, you could also use a Segmented List as well.

Creating our EXM Message

Flipping to EXM, I have created a new Regular Message and have begun to edit my content. Below you see that I have created Two Variants to this message: A and B.

Variant A links to the Low-Value landing page that is simulating a not-so-successful marketing message.

Variant B is linking to a High Value landing page that is simulating a very high performing, well-received page.

Preparing the Test

On the Delivery tab, we now have options for adjusting the content test and specifying whether we want EXM to automatically determine a test winner after a specified length of time (Hours, Weeks, Months).  Or to manually determine the winner. 

For this test, we are going to manually select a winner. This will pause the dispatch indefinitely until the content author comes back to EXM to Select a Winner.

For this example, we are going to have EXM send out to 10% of our Contact List and then pause to allow us to select a winner.

Starting The Test

Once you feel the AB Test rules are set, click the Start A/B Test button on the Delivery tab of our message.

This is going to kick off the Dispatch process and send to 10% of the list, which equals 5 contacts on a 50 person list, and then pause.

Using Mailtrap.io for my own development environment, so that I”m not actually sending email messages, you can see that I have sent/received two emails showing both Variant A and Variant B.  I open each of the messages, click on both links in order to generate some activity.

Wasn’t 10% sending 5 messages?

Very observant! Yes, the test sent 5 messages, but because of Mailtrap.io’s Free tier, I can only send 2 emails/second. EXM works very fast, and 5 messages were sent within the first second, and Mailtrap.io only allows me to see 2 of those messages.

Going back to the EXM Dashboard and clicking on In Progress campaigns, you can see our EXM Message is shown, and that the status is Select Winner. This is notification to the message is ready for testing.

Picking a Winner

Because we selected choosing a Manual A/B Variant test, we need to go back to the message’s Delivery tab. The table will show how both variants performed. In this example, we have one landing page that generated way more value than another landing page.

We’re going to select that winner. Which will prompt a confirmation box and then continue dispatching to the rest of the contacts in the message list.

EXM provides a great progress bar for seeing the email progression as it sends out emails. For extremely large lists, it’s actually fun watching it fill across.

After a few seconds, on our short 50 contact list, we completed the dispatch and sending process. Going back to Mailtrap.io you can see that we have received a number of new email messages, but all of our chosen Variant, which in this case is Variant B.

Looking at the Stats

Last, but not least, pull up the Message Statistic report to see how well your message has performed. We have successfully sent an A/B Test using EXM Message Variants! Huzzah!

EXM Live! Webinar Today!

Don’t forget to register for my Sitecore EXM webinar happening TODAY!

Register for EXM Live! Webinar

Tagged as: 25days, EXM

Categorised in: EXM, Sitecore