How Crutchfield Does In-House Testing

Testing is at the core of every direct marketer/cross-channel retailer's DNA. Without it, you're left wondering what works and what doesn't. A well-executed test — be it an A/B split, multivariate, in-house focus group, etc. — provides insight into what the consumer is going to do next. In a session at last week's eTail East conference in Baltimore, Archie Miller, senior manager of web design and development at consumer electronics retailer Crutchfield, detailed how his company uses in-house testing to learn about consumers.

Running at least one test per month at its corporate offices in Charlottesville, Va., Crutchfield tests its website, in-store kiosks, emails and even catalogs. The tests are compromised of three to four participants, who are generally recruited from Craigslist and are compensated with either $60 in cash or an $80 gift card (people almost always take the gift card, Miller said). The tests require two conference rooms — one in which the testing takes place, another where Crutchfield employees watch the tests via video — and generally take 45 minutes to an hour to conduct, then an hour to two hour debriefing period after the testing.

The equipment Crutchfield uses for its testing is minimal and rather inexpensive (Miller estimated that the total cost is around $2,500). It includes a web cam, microphone, basic computer, TechSmith morae usability software and an A/V cart. One Crutchfield employee serves as the greeter and facilitator of the test. The paperwork Crutchfield puts together for any given test includes recruiting forms/letters, consent forms, intro scripts for test facilitators and thank-you cards sent to test subjects.

With everything in place, testing can begin. The stakeholders in the test (i.e., Crutchfield executives watching from another room) all see the same thing at the same time, Miller said. The goal is to find those “a-ha moments,” where Crutchfield can learn from the actions of the test subjects — the homepage doesn't provide for easy navigation to a particular product, for example.

The secret recipe Crutchfield bakes into every in-house test it conducts is one-part participation, one-part discussion, two-parts consensus and four-parts action. Participants can't do anything wrong; they're there to provide insight into consumer behavior. Just one person mentioning a problem is enough for Crutchfield to consider a change or action, Miller said. Here are some of his tips to running an effective in-house testing program:

• specify your details up front — know what you're testing for;
• plan your test out, including running a dry test first;
• take advantage of outside testing tools, such as UserTesting.com, which provides results within 24 hours of a test, and can segment results by gender, age, income, among a host of other categories;
• be cautious of participants being too complimentary to your business — you want to know what you need to improve upon;
• test regularly (Miller would like to see Crutchfield be able to test twice a month, rather than it's current one test per month);
• provide a quiet, comfortable testing environment;
• don't just test when you're launching a new project;
• test at the beginning of campaigns so that your learnings can be implemented right away;
• weed out professional testers — Miller notes that professional testers tend to ask about compensation first, so be on the lookout;
• run niche tests that you can extract specific takeaways from (e.g., a women's apparel retailer testing men who are going to buy clothes for their wives online);
• involve stakeholders in the post-testing discussion; and
• hand off test results for action.