10 Tactics for Online Testing

Almost any question can be answered cheaply, quickly and finally by a test campaign. And that’s the way to answer them — not by arguments around a table. Go to the court of last resort — the buyers of your product. —Claude Hopkins, Scientific Advertising, 1923

Savvy catalogers have long used testing to improve their mail businesses. And as the Web matures, catalogers are bringing the same discipline to their online marketing efforts.

This article offers 10 tips for running direct marketing tests in the online world. The first seven are common to online and offline. The last three are unique to online marketing, for they exploit the Web’s high speed and low cost.

Tests Common to Both Channels

1. Move bigger levers first. In descending order of importance, the three essential elements of a direct marketing campaign include:

- list — who, and how many people receive the offer;

- offer — what merchandise you offer, at what price, and with what service; and

- creative — how is the merchandise presented, described and displayed.

If your goal is to double online sales, your best bet is to double qualified traffic to your site. This generally is easier than doubling average order value or doubling site conversion (worthy goals, too, but harder to achieve).

If you’ve not yet tested paid search, paid inclusion, local search, affiliate marketing, eBay, Amazon and other online marketing programs, do so. Such “list” tests can give you the greatest chance of bumping sales.

After list, focus on your offer. Is your site presenting the right merchandise at the right prices? Would your conversion lift from a free shipping offer offset the cost? Tip: When setting a minimum order size for an offer, place it above your average order size.

Finally, focus on creative — that is, how your site looks and works. Does your homepage highlight the breadth of your merchandise? Are your product detail pages clear, with relevant information above the fold (i.e., visible without scrolling)? Is your checkout process smooth, fast and intuitive?

2. Test shouts, not whispers. Testing takes effort, attention and sometimes money. Don’t test tiny tweaks. Favor bold tests that have the potential to change your business. Tip: A sure sign of a bold test is that it may make some people inside your company slightly uncomfortable.

Subtle tests will, at best, yield subtle results, often too small to detect.

3. Maintain test notebooks. Document each test you do, why you did it and what happened. Short pre-test and post-test summaries keep you from repeating mistakes or wasting time on questions already answered.

Before the test, write a clear hypothesis of what you’re trying to prove or disprove. Here’s an example:

Test No. 6, October 2005: Our hypothesis is that bringing visitors into our site from paid search to the new simplified product-page template will increase conversions relative to the current grid-style, productpage template.

Before the test, also record your decision metrics and the roll-out plans. For example:

If the new pages increase closing by a significant amount (i.e., 50 more orders than the control for the week), we’ll discard the grid template in favor of the simpler template.

After the test, record numeric results, your interpretation and suggestions for next steps. For instance:

Results: Despite one large order, the simple treatment actually reduced conversion a bit.

Next steps: Keep the grid; test another challenger later this month.

Given the value of test notebooks — indeed, they become your marketing department’s shared institutional memory — it’s worth maintaining two copies.

4. Test one thing at a time. To isolate the effect of any variables, traditional testing mandates changing a single factor at a time.

Testing online usually is cheaper and faster than a traditional in-the-mail test. As such, there’s less need to cram everything into one massive test. Start with a series of simple, one-factor tests. You’ll quickly learn what matters.

Tip for more advanced marketing teams: Look into multivariant testing (MVT), which entails scientific testing design of experiments. MVT offers marketers the chance to vary many factors at once in a statistically valid way. MVT design and analysis require more skill. A little training for your team from a seminar or statistical consultant goes a long way.

5. Separate signal from noise. All tests have some element of random statistical “noise.” For instance, say you took a mailing list of 10,000 people, randomly split it into two cells of 5,000 each, and on the same day mailed each cell the same catalog. Even with exactly the same treatment to both groups, one cell — by chance alone — will have a few more orders, and thus a higher response rate.

Be sure you can distinguish a marketing signal from marketplace noise. Familiarize yourself with basic statistical-significance calculations. As a general rule of thumb, if you plot conversion rates over time, a test must increase conversion by more than 1.5 times the normal range of variability to be significant.

If your team isn’t using these statistical-significance formulas yet, you can get training from The Direct Marketing Association (www.the-dma.org/seminars/statistics/), or check out the Excel spreadsheet we posted with the basic formulas at www.rimmkaufman.com/statistics.

6. Assign unique tracking codes. Online testing without proper tracking is like flying an airplane blindfolded. To be able to read your tests, each cell needs a unique tracking code. Make sure your tracking application can report key metrics (e.g., visits, sales, ad expense) for each code. Also be sure your tracking application allows you as many keycodes as you need.

Some marketers embed meaning in the various digits of the code, where a code such as 0510E02 could mean the second (02) e-mail test (E) sent in October 2005 (0510). Other marketers use sequential numbers for their cells, and log each code’s meaning in a spreadsheet or database.

In either case, your team must maintain scrupulous electronic records, matching keycodes to their corresponding cells and campaigns, with accompanying information regarding the creative and offer. Many a test has been lost to keycode mismanagement.

7. Manage expectations. Make sure your team understands that most tests produce no results. Because your current marketing approach represents years or decades of thoughtful improvement, many alternatives won’t test out better. And the more successes your testing program obtains, the harder it becomes to move the needle.

Should the difficulty of hitting homeruns stop you from stepping up to the plate? Not at all. While testing wins may be scarce, the big wins that can follow more efficient marketing are well worth the effort.

Online Testing

Compared to offline testing, online testing can be so cheap and fast that it becomes a difference in kind, not just degree. The following tips exploit this difference in cost and speed.

8. When firing shotguns, you don’t need great aim. In the offline world, versioning is expensive and often limited due to printing and bindery constraints. In the online world, versioning usually is inexpensive, often free.

Suppose you’re testing a holiday sales e-mail. You may have six decent ideas for the subject line, two possibilities for the large seasonal image and three reasonable candidates for the hero product. Considering all combinations of subject, image and product, you could send 36 different e-mail versions (six times two times three).

Which version will prove most effective? If your testing framework can support it, and if the creative can be built in a cookie-cutter format, then send them all. This brute-force, or shotgun, approach can be surprisingly effective at finding winners.

9. Test within, not between. In the offline world, tests typically are slow events, requiring time to prepare, launch and analyze. With online testing, results come immediately, often within hours. Smart marketers can exploit this early information to test within a campaign, rather than just between campaigns.

Here’s how: Suppose you mail a monthly e-mail to your housefile. Getting the subject line right can have a large impact on open and conversion rates. Many catalogers test multiple subject lines on each mailing. Then after the campaign, the marketing team reviews results by subject line, learning what worked best so as to improve future e-mails.

Instead, pull a random 10 percent of recipients from your e-mail file and mail to them first. Randomly nth these e-mails across myriad subject line ideas, and mail them. Wait six hours. Evaluate the subject line ideas using their six-hour open rates for initial blast. (You’d prefer to use conversion rates, but for many catalogers six hours is likely too short to see meaningful sales.)

Then — and here’s the trick — mail the remaining 90 percent of your file using the winning subject line from the initial blast.

This allows you to reap the benefits of sending the better e-mail immediately, not later in a future campaign.

10. Automate testing. In the offline world, a test is a discrete marketing effort that takes careful planning and execution. Indeed, testing is anything but automatic.

Meanwhile, the online world is beginning to offer an exciting new opportunity: automated test robots. Automated testing may not yet be supported by your e-commerce platform or any of your advertising partners. But within 12 months, they will.

Here’s how automated testing works: You load up a set of different marketing messages (e.g., ads on an advertising network, featured products on your homepage) and instruct the platform as to which metric you want to optimize (e.g., clickthrough rate, clicks, sales). The platform then rotates through the set of messages, tracking which performs best. As the platform learns what message performs best, it preferentially serves the winner, improving your results.

Early examples of automated test robots include Google’s “automatically optimize ad serving for my ads” adwords option (www.adwords.google.com), and Offermatica’s “mbox” platform (www.offermatica.com).

Conclusion

Claude Hopkins’ praise for testing rings as true today as when written 82 years ago. May your online tests surprise you, bringing you fresh insights to grow your business.

Alan Rimm-Kaufman, Ph.D., leads the Rimm-Kaufman Group, a direct marketing service and consulting firm helping catalogers with online marketing and paid search. He may be reached via his Web site at www.rimmkaufman.com.