10 Tactics for Online Testing

5. Separate signal from noise. All tests have some element of random statistical “noise.” For instance, say you took a mailing list of 10,000 people, randomly split it into two cells of 5,000 each, and on the same day mailed each cell the same catalog. Even with exactly the same treatment to both groups, one cell — by chance alone — will have a few more orders, and thus a higher response rate.