Building a Loyal and Profitable Customer Base

By Richard Vermillion

During the first big wave of data warehousing in the 1990s, companies were sometimes described as being "data rich and information poor," a reference to the lack of business insights that still prevailed even after large amount of data had been collected and centralized. With the explosion of communications channels in recent years, that phrase has new currency, as many retailers have been struggling to integrate all these new data sources — and to make use of them to drive business results.

Big data technologies such as Hadoop were developed to help large internet companies (Facebook, Google and Yahoo) sift through staggering amounts of data, but many of the applications at first seemed remote from the challenges faced by retailers. Web companies may have huge data sets, but these tend to be "skinny" — billions of rows about a few things. Retailers, on the other hand, may not have the same scale of data, but their data is richer and more complex. Fortunately, as these tools mature, they're proving to be equally as valuable in solving retail marketing problems.

One midsize retailer with a strong online presence recently found that just the first step in their planning process was taking 30 hours to 50 hours per month to run. This process required rolling up hundreds of millions of line-item purchases by stores, product categories, planograms and customer segments. Because the job was so intensive, it had to be scheduled around other processes, which meant that it could take a long time to complete. Migrating this process to a Hadoop cluster reduced total run time to four hours. Now the planning group has far more time to spend on subsequent steps in merchandising planning, making the retailer more nimble in its merchandising.

In this case, the significant gains in throughput come from Hadoop's ability to split a big problem into manageable chunks, spread the workload across multiple servers and, at the end, reassemble all of the pieces to provide business insights. Retailers have many such complex, intensive processes — e.g., offer management, cross-channel analysis, demand forecasting, to name a few — that present good candidates for the technology. For those retailers with a large online presence, integrating traffic data with shopping and customer data is well-suited to these technologies.

Of course, the old adage of "if it ain't broke, don't fix it" applies here. If your existing infrastructure supports current and near-term analytical needs, then an investment in big data solutions doesn't make sense. On the other hand, if there are performance bottlenecks or business needs that can't be met without costly hardware or software upgrades, big data tools deserve serious consideration. Retailers considering trying out big data technologies have a number of issues to weigh, including the following:

Buy or lease: One of the attractive aspects of big data solutions is their use of open-source software and commodity hardware, which results in system costs that tend to be lower than comparable commercial products from big IT vendors. Moreover, companies have the option to use cloud-based offerings, either on a pilot or permanent basis.
Finding qualified staff: While the available software and hardware options are plentiful, there's a shortage of experienced professionals in this area because it's relatively new and in high demand. Here, the most expedient answer may be to hire contractors or consulting firms that can guide internal staff through initial pilot projects.
Support: Because a substantial number of big data software is open source, technical support can be less robust than commercial offerings. However, there's a growing number of companies that offer support services, as well as a healthy community of user forums. And, as with most open-source initiatives, a larger user base results in both better support as well as more robust installation and management tools.

Before embarking on a big data initiative, retailers should identify one or two business processes that can benefit from deeper or faster analytics. Piloting one or two cases that create measurable business value is a way to gain experience and prove the business case for larger investment. The good news is that retailers are finding success by adding these tools to their overall information strategy and that the technology has many features that make getting started affordable.

Richard Vermillion is the CEO of Fulcrum, an analytics and marketing technology firm.