Mysterious & Bizarre Results

In this cataloger’s case, order counts for various test groups had been between 10 and 30 orders — enough to persuade the cataloger the trends were real, but not enough to satisfy a statistician.

In the end, the statisticians were right: The results weren’t repeatable. When the cataloger rolled out to what he thought were his winners, he actually was rolling out to a purely random group of lists, which ended up producing poor results — like any random group of lists would.