How and What to Test Effectively

Testing is the key to direct marketing success. The key, however, is knowing what, how and when to test.

While testing is important, it’s not always cost-effective. This month, I’ll discuss how to structure tests, how to read the results and when it makes economic sense to do so.

When creating test panels, you’ll need to weigh a number of factors. The method that’s mathematically most accurate may not be cost-effective. Conversely, the most cost-effective method may produce skewed results. And in some cases, it may not even make economic sense to test.

The point is to determine the economic benefit from the test compared with its cost. You also want to be sure that in the end you’ll be able to read the results accurately and come to a conclusion.

It’s important to think through the test and ask yourself what you’re going to learn in the end. The goal of testing is to increase revenue and/or reduce costs. You need to know you can measure against these objectives.

Basic Rules

When setting up your test, follow these basic rules:

- Only test one variable at a time.

- Always test against a control group.

- Test panels must be selected from the same list universe.

- Mail dates must be the same for all test panels.

Only test one thing at a time. You must be able to read the results accurately. Every variable you introduce can influence the outcome. If you’re testing two different offers against each other and against a control group, ensure that all variables and conditions are the same for each test group.

For example, the same catalog should be mailed to all test panels on the same date. Moreover, the offer should be promoted in the exact same way: If you’re promoting the offer on the front cover for one test panel, do the same for the other test panel.

Promoting your offer on the cover for one group and on the inside order form for another doesn’t constitute a valid test. The offer’s design also must be the same for each group. Thus, the only variable in your testing should be the words that describe the offer.

How to Select Your Sample Size

The most accurate testing method is to nth select the panel from the entire mailing across the board. This means the test panel has the same makeup as the main mailing. Using this method will increase the postage cost, because separate mail ZIP streams will be required.

A second method is to select the test panel geographically to minimize the postage cost. But this method isn’t recommended as it’s not representative of the main mailing, thus skewing the response reading.

Some mailers will create test panels from the highest level of the postage group. This is less skewed than the geographical method, and it has less negative impact on postage costs. Mailers also will use particular lists or segments of the house list.

The previously discussed pros and cons still apply here. It’s imperative to select the test panels after the merge/purge process to maintain statistical accuracy.

For example, say you want to test a free-shipping offer against an offer for $10 off an order of $99 or more. This test will require not two but three separate test panels. Let’s also assume you want to test these offers against your entire housefile of 120,000 buyers.

When selecting the test panels from the housefile, one approach might be to nth every third name, so you end up with 40,000 names per group. Again, this method of selecting your test panels must be taken across all of the recency, frequency and monetary (RFM) value segments of your customer file.

If you’d prefer to test to a smaller group of the housefile until actual results are known, you easily can select three test panels of 10,000 customers each. In this case, you would want to take a random nth of 30,000 names from the universe of 120,000 total buyers.

Next, you would nth the test group of 30,000 into three equal groups of 10,000 records each.

The number of records (house and prospects) you have per test cell will depend on your overall response rate and/or average order size.

The lower the response rate, the higher the number of names you should test per panel. That’s because you need to yield “enough” orders (i.e., the response rate) to have a valid test and read. In a typical consumer catalog, the minimum number of names I like to test is 10,000 per panel.

Most likely, you’ll want to test this same offer against outside prospect lists. You may find a particular offer works better when presented to prospects than to your own customers and/or vice versa.

When testing to outside rented names, it’s better to test those lists you know will work. Select a large enough sample to split into three equal test panels. A minimum of 30,000 names is needed if you’re going to split them into three groups of 10,000 each.

In other words, you wouldn’t want to take a test list of 10,000, run it through the merge/purge netting around 8,000 records, then split this group into three equal groups of 2,666 each. What’s more, you’ll want to test more than one outside list.

Results can vary by list, therefore, test about five outside lists. Each sample size would be too small for testing purposes.

A typical test matrix might look like the example in the chart above. In this example, the control is no offer at all. However, through this testing process, the control will be whatever offer (if you continue to use an offer every time) that you decide to use on an ongoing basis. Some mailers have used an offer so often that it becomes the control. In this case, you may end up testing no offer against the control (e.g., free shipping or $10 off).

Whatever offer you make to your customers and prospects on an ongoing basis should be your control.

Re-testing, Testing and Testing

Many of the things you test must be tested more than once to validate your findings. For a variety of reasons, results aren’t always consistent from one test to another. That’s why it’s important to re-test if the impact or magnitude of what you’re testing justifies doing so.

Let’s look at the types of things to test more than once:

Catalog trim size. A slight change in the trim size of your book can reduce your costs without having a negative impact on the sales. This isn’t an easy thing to test cost effectively, since it requires producing two completely different catalogs. Therefore, testing a new trim size (and re-testing) will depend upon the degree of the physical size of the change you’re planning to make.

Customer contact strategy. This involves determining how many times you should be mailing to your housefile. It can be a difficult test to set up since the test panels will need to be frozen and mailed, spanning at least a six-month period.

Timing test. Here you may want to determine the best time to mail to both customers and prospects. If, for example, you’d normally mail in the second or third week of September, what happens when you try to lengthen the fall buying season by mailing in the last week of August?

Major repositioning. You may decide to reposition the catalog entirely to give the book a new look. This is risky business with a huge upside. Perhaps you need to change the name of your catalog. Doing this is extremely risky, and it should be tested more than once before an actual name change occurs. Generally, those on your housefile will react differently to a name change than prospects.

Reading Results (and Economics)

Do the results appear to be logical? Are they believable? You’ll need at least a 5-percent variance, or difference, in the revenue per catalog mailed for the results to be real. A variance of less than 5 percent may not be statistically valid.

So what do you do? Re-test. If the difference is slight the second time around, perhaps you should stick with your control. When reading the results, factor in any cost differences. Exclude the costs associated with the test itself. Include only the costs associated with the rollout itself, should you decide to do so.

Let’s assume you wanted to test a new paper stock that has the potential to reduce your in-the-mail costs by $5/M. If you get $1.50 per catalog in revenue, and your test achieves $1.50, then your contribution to profit and overhead will increase by 33 percent. Postage costs will increase during the test phase, but you don’t factor it into the analysis; it won’t be there when you roll out.

In conclusion, do the results tell you what you already know? Do you believe and/or accept the results even if they differ from your thinking?

As long as your test is properly structured, you should be able to read the results and proceed accordingly. Always weigh the potential benefit against the time and cost to set up the test.

Remember, testing is the key to success. Don’t assume anything!

Stephen R. Lett is president of Lett Direct, a catalog consulting firm specializing in marketing, circulation planning, forecasting and analysis. He can be reached at (317) 844-8228 or by e-mail at slett@lettdirect.com.