Tesing 1, 2, 3...

So what had it actually tested that produced this clear result? Absolutely nothing — the test and control groups were treated the same at each point in the test. They got identical catalogs at identical times.

I once performed the above test myself in cooperation with one of my catalog clients. We called it the “no-test test,” because we went through all the motions of a test, only we didn’t do a single thing differently between the control and test segments. The results from both segments basically should have been the same — but they weren’t.