CASE STUDY: MEC Optimizes Web Sales 
Via A/B Testing

PROBLEM: Mountain Equipment Co-Op (MEC), a cross-channel retailer of outdoor gear and apparel, wanted to increase its web sales.

SOLUTION: Hired a third-party provider of testing and personalization products for online retailers.

RESULTS: MEC's conversion rates and web revenues have increased since it began conducting optimization testing on its site nearly three years ago.

For MEC (think the REI of Canada), the inability to simultaneously test pages or functionalities on its website led to a lot of head-scratching. Testing was a part of the cross-channel retailer's culture, but tests were run at different points in time. This often left MEC wondering whether the results of its tests were affected by outside variables (e.g., a holiday, free shipping promotion) unrelated to what it was trying to measure in the first place and, more importantly, if it was leaving money on the table.

In August 2008, MEC partnered with Amadesa, a provider of testing solutions for online retailers, to help it optimize website conversions via simultaneous A/B and multivariate testing. Specifically, MEC is using a tool called InPlace, which allows it to see real-time test results from a dashboard right on the web page where a test is being run. Amadesa's tool has enabled MEC to leverage insights gleaned from tests to implement revenue-generating changes to its website.

"To be able to measure different versions of tests side-by-side at the exact same moment in time to see what everybody on the site thinks is the best took a lot of opinion out of the mix," says Bob Hermanutz, e-commerce development manager at MEC. "A/B testing allows us to take individual ideas, put them into a pot and see how our website visitors react by reviewing the data."

Testing Yields 
Conversion Gains
MEC's average website test runs nearly three weeks, with individual tests dependent on the web page in question. Pages that naturally see more traffic (e.g., homepage, product page) require a shorter period of time to reach statistical significance compared to other pages (e.g., shopping cart page). Deciding what to test is a shared responsibility. MEC's web team suggests tests to Amadesa, and Amadesa's web optimization consultants propose tests to MEC. Tagging of the actual web pages to be tested is done by MEC's in-house IT team.

A recent test that proved fruitful for MEC occurred on one of its product pages.

The retailer tested showing descriptions of products with images on the product page vs. just showing images without descriptions. To MEC's surprise, the image-only version of the page converted much better.

This change was turned live by MEC's IT team as soon as the winner of the test was determined. Within nine days of the change to the product page, revenue per visit had increased 12 percent. Changes that are hard-coded on MEC's site, on the other hand, take four weeks to eight weeks for implementation, Hermanutz notes.

No matter the size or scope of the test, MEC's end goal is always to increase conversions. "More subtle changes to our site take more time before we get a significant result, and they're going to be a smaller fraction of a percent improvement, but even a fraction of a percent improvement to our conversion rate has a good ROI," says Hermanutz.

Future tests that have been discussed by the MEC team include a more thorough examination of site navigation; product detail page tests in preparation for a redesign of those pages; and a test of the value of the large atmosphere image displayed on MEC's homepage.