How to Expertly Evaluate Your Campaign Results

Expert evaluation of catalog mailings was problematic in the past, but mostly just for those few catalogers who had retail stores. That’s not the case anymore.

Now virtually every catalog company is multichannel as customers increasingly use the Internet to place orders. The result often is a haphazard online collection of key codes.

In this article, we’ll examine the problems with traditional campaign analysis and how you can use a matchback between orders and mail files to substantially overcome these issues.

The Problems With Traditional Factored Allocations

While unattributable orders usually have been in the range of 10 percent to 20 percent, now unattributable orders are frequently more than 50 percent due to customers’ failure to provide a key code — even when asked — while placing orders on catalog Web sites. With 57 percent of shoppers presently crossing channels and Web sales trending upwards each year, traditional means of evaluating results through key-code analysis has become increasingly problematic.

Traditional methods to determine factored allocation of unknown orders are flawed for several reasons. First, as a general rule, statistical validity of results per segment evaluated requires a minimum response of 100 orders. So if you expect a 1 percent response to a mailing, a list segment size of 10,000 names is required. That’s why test orders of lists generally are 5,000 (2 percent expected response) to 10,000 names (1 percent expected response).

But those 100 orders must be real orders, not factored for unknowns or projected response. So if half of your incoming orders are of unknown origin, the segment size for statistical significance at a 1 percent response level actually is 20,000 names. Such segment or test list quantities for small mailers generally are risky or impossible.

Second, unallocated orders normally are factored against known response. For example, if 20 percent of response is unknown, the known response to each segment is increased by 20 percent across all segments. Unfortunately, unknown response never is random or proportional across all lists or segments.

Traditionally, most unknown response tends to come disproportionately from one’s own buyer file, unless during the order-entry process the system automatically adds a key code to known customers. If key codes are applied, then the opposite is the case: Unknown orders come disproportionately from prospects.

However, with the advent of the Internet, a new bias has developed. Those customers who traditionally order on the Net end up disproportionately in the unattributable order pool. After factored allocation, lists with many Net shoppers get an inadequate allotment of buyers and look as if they’re under-performing relative to other segments and lists.

In short, as unattributable orders grow (as a percentage of total orders), factored allocation becomes problematic. Results are more likely to be misread due to statistical insignificance of segment size and biased attribution. There’s a risk of over-mailing lists that really are under-performing by not contributing to Web orders. There’s also the risk of under-mailing lists that drive heavy Web performance.

For those mailers with the third channel of retail stores, the problem is further compounded. For most, the retail channel simply represents an immeasurable black hole, with sales that are difficult to attribute back to the catalog mailing through any means.

The Matchback

A matchback compares the names and addresses on a mail tape with those of responders from all channels within the mailing’s season. A report is produced from the match that’s much like a key-code report showing number of orders and sales by key code. It’s important to retain this information by channel and separately for those orders for which a key code actually was collected and those which had no key code.

A matchback generally is provided by the service bureau that processed your merge/purge and produced the mail tape. Since there’s a moderate charge for this, it’s usually done one time after, say, 70 percent to 90 percent of the mailing’s response is received and before you need to make key decisions about an upcoming campaign. Recently, these one-off matchbacks also have been provided, sometimes at no charge, by cooperative databases.

The advantages of a single matchback include the ability to establish customized complex rules and reports that reflect the uniqueness of your business. The primary disadvantage is that it’s a one-time snapshot run at the end of the campaign and therefore precludes one from seeing changing patterns of response during the campaign.

DoubleClick developed a product called ChannelView, an application service provider-based, campaign-analysis tool that allows a continuous matchback process with multiple open campaigns and live analysis. Users log on to a secure Web site to view results in real time. Once set up, orders are transmitted to Abacus daily or weekly and can be matched to multiple open campaigns including e-mail blasts.

Besides salving one’s curiosity, continual matchback allows you to view changing response patterns by channel and file source. The program can be customized to some extent for your mail program and creates a centralized results database.

While ChannelView can be highly cost-effective for large mailers — especially those with multiple catalog titles — it can be very costly for smaller mailers. You need at least six annual drops and more than 2 million annual circulation for ChannelView to be cost-effective. Also, the set-up process can be tedious, and we’ve seen fairly sophisticated mailers experience long delays in getting it right. Depending on the complexity of your circulation plan and segmentation scheme, you still must download raw data for secondary analysis.

Finally, many larger, more sophisticated mailers maintain an internal matchback program. While such programs can be highly customized and integrated with existing databases and modeling programs, they can be extremely costly to build and maintain. You often can match only housefile data unless a mail file can be retained on-site, usually necessitating list owner approval.

Mechanics of the Matchback

Orders already ascribed key codes and unattributable orders should be matched back to the mail tape(s). By matching known orders, you validate the matchback process or sometimes identify major problems with phone rep/Web site collection of key codes. Understand that usually 15 percent to 20 percent of known orders aren’t matched to the mail tape due to errors and/or pass-along catalogs.

The best practice generally is to add unattributable orders that are matched to the mail tape to known orders. For example, a specific key code may have 135 orders assigned to it prior to the matchback based on traditional collection techniques. When unattributable orders are matched to the mail tape, 51 orders may be additionally identified as belonging to that key code. These would be added to the 135 orders, resulting in 186 orders for the key code.

Following the matchback, a pool of names will remain from the unattributable orders that didn’t match the mail tape. Use your judgment as to what to do with them, since they may have resulted from other marketing programs, word of mouth, old catalogs or errors in matching.

If you can tell they’ve arrived at your Web site from e-mail campaigns, natural and paid key-word searches or affiliates, then you should remove them from the “unknown” pool and give credit where it’s due. You may want to then allocate the remaining balance back to each key code in proportion to the allocation of all orders in the matchback to the full mail file.

Case Study: The Flax Companies

San Francisco-based the Flax Companies produces the Flax Art & Design, T.Shipley and Reliable Home Office catalogs. In one of its recent mailings, about 37 percent of total sales within a particular mailing’s season came from the Web and had no key codes.

Of those, 53 percent were allocated to the catalog via matchback. The remaining unallocated were matched to e-mail campaigns or affiliate marketing messages, leaving only 22 percent of the season’s Web sales as truly unattributable to a marketing campaign. Additionally, post-matchback, only 8 percent of all phone and mail orders that season remained unallocated.

In that same mailing, the matchback identified the following:

— 46,984 names (less than 9 percent of the total mailing) that would’ve been considered profitable using factored allocation were in fact below breakeven; and

— 3 25,993 names (less than 5 percent of the mailing) that were considered unprofitable using factored allocation were profitable.

Rules for Matchbacks

Most companies have multiple active campaigns from which orders could result. So there must be some logic applied to how orders are allocated. In general, it’s accepted practice to allocate an order to the most recent active campaign from which that order could’ve resulted. So although a person may have been included in the last three monthly mailings, assuming that his or her order carries an order date, the person would be matched to the campaign that’s active on that order date. If the order carries no date (not a desirable scenario), the order should be matched to the earliest campaign included in the matching process.

Other mailers may allocate a customer’s first order to a prospect key code if an individual appears as a prospect in one campaign; the second order would be allocated to a buyer code in the subsequent open campaign. Still others want to allocate in proportion to overlapping response curves.

Your unique circumstances should be discussed with your service bureau or ChannelView to determine appropriate rules to maximize the process’s utility.

Conclusion

As you can see from the Flax Companies’ case study above, much better mailing decisions can result from a matchback. Sometimes, the very survival of a catalog program can be at stake if the catalog doesn’t get full credit for having driven orders placed through either the Web site or retail stores. Matchbacks must become a routine part of evaluating your results.

John Lenser is the president and founder of LENSER, a San Rafael, Calif.-based catalog consultancy and list management and brokerage firm. Lenser founded the San Francisco Music Box Co. and previously was president of Hearthsong and David Kay. He can be reached at (415) 446-2500.

Fatemeh Khatibloo-McClure is the director of circulation for the Flax Companies, where she manages three catalog titles. She also serves on the board of directors for the Catalog & eCommerce Club of Northern California.

They wrote this article at the request of the Catalog Success editors.