Contributions to Profit: The Art and Science of Testing
Not long ago, a client of mine was all but certain that by upgrading the paper used for its catalog, sales would increase. Unfortunately, the client scoffed at the notion of running a pro forma break-even analysis to determine how much revenue was needed to offset the additional paper and postal costs. In fact, it took the client’s paper merchant, printer, service bureau reps and myself to convince the cataloger to set up a test before making a global change to this new paper grade.
Simple A/B Split Test
Once convinced, the client set up a pretty straightforward, scientific A/B split test, sending half of its customers and prospects a catalog on its regular paper, and the other half the book with the higher grade paper. To keep the test scientific, the service bureau chose every other nth name from each list segment. In the world of science, the “A” portion of names mailed were the “control” group — the goal was to beat the test “B” portion.
When we analyzed the results after the test, the client and I came to different conclusions. The cataloger concluded the test was a success. After all, there was a marginal increase in response rates, driving in a few new customers in some, but not all, of its list segments. I saw something different. Although results were up slightly in some prospecting segments, housefile response decreased — especially its single buyer segments. This meant that the cataloger was converting fewer new customers (who were just trying out its products) to multibuyers — a key metric and a clear indicator of future success.
Just as important, the incremental cost of the better paper grade wasn’t covered by the slight bump in sales. Prospects were costing more to acquire, and the company was losing slightly more money up front that previously would have to be made up by future mailings and orders. The company was making less profit on a per-order basis.
The client at that point was willing to lose a bit more up front to increase sales and put out a sharper-looking product. Eventually, the increased costs would catch up with it, however, as the business suffered and the paper grade had to be scaled back. Therein lies the moral of the story: Many times even scientific tests succumb to the subjective.
There are infinite varieties of tests that can be configured, measured and ultimately interpreted. Some of these tests may be profitable — and some may not.
Some suggestions for effective testing:
• Test against something. Your current catalog, e-mail, Web site, merchandise, etc., is your control.
• Create a test hypothesis. Use this example: If we do “X,” we expect “Y” to occur.
• Always set up logical tests based on scientific principles. Test one variable at a time. If you test more than one, you’ll never know which one made the difference.
• Always do your math up front. Get all expenses in advance of the test and set up a pro forma profit and loss and break-even analysis.
• Always do the math at the end. Before the artistic, interpretive side of testing begins, have the math in front of you. Many times the numbers will drive the decision.
List variables:
• Recency. Test the most recent names for a new list test. If your list seems to be working, test older segments of the list. Test average order size test selects. Test new lists in your category and outside names.
• Creative and printing variables. Paper, trim size, pages, formats, photography, copy, ink-jet messages, cover
versions, dot whacks, inserts.
• Offer variables. Gift with purchase, dollar or percentage off, free shipping, etc.
Jim Gilbert is president of Gilbert Direct Marketing Inc., a catalog and direct marketing consultancy. He can be reached at (561) 302-1719 or jimdirect@aol.com.
Jim Gilbert has had a storied career in direct and digital marketing resulting in a burning desire to tell stories that educate, inform, and inspire marketers to new heights of success.
After years of marketing consulting, Jim decided it was time to “put his money where his mouth was" and build his own e-commerce company, Premo Natural Products, with its flagship product, Premo Guard Bed Bug & Mite Sprays. Premo in its second year is poised to eclipse 100 percent growth.
Jim has been writing for Target Marketing Group since 2006, first on the pages of Catalog Success Magazine, then as the first blogger for its online division. Jim continues to write for Total Retail.
Along the way, Jim has led the Florida Direct Marketing Association as their Marketing Chair and then three-term President, been an Adjunct Professor of Direct and Digital marketing for Miami International University, and created a lecture series, “The 9 Immutable Laws of Social Media Marketing,” which he has presented across the country at conferences and universities.
