Subject Line Testing in 3 Easy Steps

Subject line testing is one of those things that's always on the “to do” list, but somehow seems hard to make happen. There are a thousand reasons why: lack of creativity to come up with variations, last-minute issues to resolve before the campaign goes out, uncertainty about how to split up the list …

Subject line testing isn’t as difficult or as time consuming as it may seem. Like any new habit, the hardest part is simply getting started. So, I’d like to share with you three easy steps to follow and apply to your own email program. This organized framework helps breakdown the process in a manageable way, which may improve the likelihood the habit will stick!

Step 1: Decide what to test.

It may be a small part of the email, but there are dozens of ways you can test a subject line. To get started, make a master list of all the possible attributes that may be tested in your program, such as these common elements:

Word choice

• Action verbs: get, save, go, buy
• Benefit statements: exclusive, just for you, best
• Offer strategy: $ savings, % off, free shipping

Tone

• Format: question, statement, exclamation, cliffhanger (…)
• Sentiment: phrases that exude an emotional response, such as “glow in the prettiest spring arrivals” (love/joy), “don’t be seen in last year’s look” (disgust)
• Urgency: limited time, ending today

Other elements

• Personalization: first name, geography, loyalty status
• Capitalization: lower case, Mixed Case, ALL CAPS
• Emojis: with and without

Once you've completed your list of test elements, choose one to implement first. It’s important that you choose only one testing element at a time to prevent muddling the response metrics. Too many testing variables will make it difficult to know which element drove results.

Step 2: Determine your audience.

Choosing your test segments doesn’t need to include in-depth analytics. It could be as straightforward as asking your ESP about its technical abilities, and using the method it offers as part of its platform. If there are concerns about impacting customer engagement, choose a low-risk campaign (e.g., your welcome message) or subscriber group (e.g., loyalty members) to start. Two common ways to split a test campaign are as follows:

• A/B testing is a random division of your list in half, an “A” group and a “B” group. Most ESPs offer this as part of their user interface, making it easy to select and allow the ESP to do the heavy lifting. You only need to write and provide two versions of the subject line as you're scheduling the campaign to send. One subject line version is sent to the “A” group and the second subject line is sent to the “B” group.
• Split testing is a method that includes a control as well as test groups. This approach holds out a control group, and sends the test subject lines to a smaller percentage of your list. With this method, you can test more than two variations of the same element. (Tip: be cautious about how many versions you create, as the smaller sample sizes risk less statistically reliable results.) Some ESP platforms also allow you to send just the test segments first, evaluate the responses, and subsequently send the winning subject line to the remainder of the list.

Step 3: Measure results.

After you send the test campaign, it’s important to measure the impact of the tested variable to determine whether the new tactic had a positive or negative effect on subscriber engagement.

• Choose metrics to measure in advance. Open or read rate is an obvious choice to measure with a subject line test. However, there are other metrics such as complaint rate and deleted without reading that are also very useful in determining the impact of a subject line. It may be relevant to look at account activity as well, especially when testing with a loyalty segment or call to action. Whatever you measure, choose metrics in advance of the campaign so success is clearly defined.
• Allow ample time to gather results. Within 24 hours you should glean enough responses to determine a winner, as most consumers are avidly checking email throughout the day. However, before pulling final metrics and declaring a winner, consider waiting three days to seven days post-send to be sure you’ve captured all responses.

While conducting a subject line test can be done by following these three easy steps, to have the greatest success, there's a fourth step you should add to the process: apply the learnings to optimize future campaigns. After you’ve completed the first test successfully, invest a bit of planning time to evaluate how the test results can be implemented in your email strategy. Then review your list of possible test elements and choose a different one to test in each of the coming weeks. Using this approach, it doesn’t take long to draft a plan for the next few months. This simple plan provides an easy-to-follow map that’s not overwhelming to execute, and helps embed the new testing habit into your operating system.

The Greek philosopher Heraclitus said, “The only thing that is constant is change.” As consumer preferences continue to change at a rapid pace, embracing this kind of subject line testing practice is mutually beneficial. It keeps you informed of ever-changing consumer tastes, and it keeps your program fresh and interesting to subscribers.

Bonnie Malone leads the consulting, client training, and knowledge and editorial organization of Return Path, an email data solutions provider.