Big Data Without Big Headaches

While "big data" is certainly a popular topic, retailers of all sizes struggle to understand exactly what big data is and how to use it to improve their bottom line. Many assume big data is a black box that churns out valuable insights that are then massaged into charts and graphs. Not so. Big data is a game changer that allows retailers to harness all accessible information on customer shopping behaviors and turn it into insight. However, in discussions with retailers, I've learned that many don't know how to get to these insights. This article won't solve your big data problems, but it will provide insight into how exactly merchants can look to big data for help.

The commonly held belief is that merchants need experts in order to analyze big data. As this McKinsey & Co. study shows, however, there's a shortage of qualified data scientists to do that. Contrary to popular belief, retailers don't need to hire $200,000 per year data scientists or invest in complicated data architectures and tools to manage their data. Companies can steer around the data scientist bottleneck by using readily available software tools, such as Wise.io and BigML.

Tools like these make big data accessible to non-IT functions, including merchandisers and marketers. Data scientists will play an important role in developing and ensuring the integrity of these abstractions, but for most retailers the use of purpose-built data refinery solutions takes away much of the technical complexity so merchandisers can easily wield the power of big data insight.

However, many retailers don't have machine learning and data science at the core of their company's competence. Think about the three specific areas served by the data scientist and you'll see what I mean. Data scientists are experts in data architecture, machine learning and analytics, and most retailers don't need a highly specialized data team of the sort you'd find at Amazon.com or eBay. For many, the solution to the data scientist bottleneck will be found by the commercial design and deployment of data refineries that abstract away as much of the technical complexity as possible.

In case you're still in doubt, let's look more closely at these three areas:

Data Architecture
Nearly every retailer is interested in capturing user behavior on engagements, purchases, offline transactions and social data, as well as catalog and customer profiles. Limiting the scope of data analysis to this basic functionality allows for the creation of templates with standard data inputs. This makes it easier to capture customer data and analyze it. To simplify data architecture, you can use the 80/20 rule: 80 percent of big data use cases (which is all most retailers need) can be achieved with 20 percent of the technology and effort.

Machine Learning
With machine learning, computers "learn" to discern patterns from data and infer insights without being explicitly programmed to do so. For example, machine learning can be used to detect trends in millions of shopping transactions, which can then make predictions about future customer behavior. A large part of a data scientist's job is crafting "features," which are meaningful combinations of data that the computer looks for and that make machine learning effective.

Luckily, features can be developed into templates. Every commerce site has a notion of buy flow and user segmentation, for example. What if merchandisers could define the general features that they were looking for and apply them to their own template, bypassing the data scientist as middleman and translator? This would eliminate the bottleneck and put machine learning technology in the hands of merchandisers.

Analytics
It's never easy to automatically surface the most valuable insights from large amounts of data. That said, there are ways that allow business experts to experiment like a data scientist. Consider how the machine learns from the results or "outputs" that it discovers. This is the critical feedback loop, and business experts need to be able to modify that loop. It's another opportunity to provide a "consumerized" interface that makes insight and analytics easily accessible to a non-data scientist.

The three areas that I outlined are fairly complex and require a lot of technical skill and training, making it hard to scale across the entire spectrum of retailers. Applying a purpose-built data refinery approach to the broader set of big data issues in e-commerce will eliminate the data scientist bottleneck and still allow retailers, big and small, to benefit from the insight that data capture and analysis provide. When domain experts can work directly with machine learning systems we'll have entered a new age of big data where we learn from each other. Hopefully then big data will actually solve more problems than it creates.

Dan Darnell is the vice president of product marketing at Baynote, a provider of personalized customer experience solutions for cross-channel retailers.