The Benefits of Managing Bots for Retailers

In this day and age, all retailers have websites. In fact, websites are integral to a retailer’s online business strategy, as the content that you put on your website can better help you sell to consumers. That said, anytime you have a website, you have content. And anytime you have content, you have bots. This is because your content has value for entities beyond your intended audience (i.e., customers). Then, what's a bot?

By definition, a bot (short for "web robot") is a software program that operates as an agent for an individual, group of individuals, organization or even legitimate businesses. Many organizations default to simply blocking bots, categorizing them all as malicious security threats. This strategy, however, eliminates opportunities to leverage different types of bots that can help improve customer experiences. Bots can represent anywhere from 40 percent to 60 percent of a company’s overall web traffic, so it’s important to think about why they’re there, what they’re doing, and how they’re affecting your online strategy — for better or worse.

Good Bots vs Bad Bots
Bots can impact your business in many different ways. Bad bots can cost you sales, damage your brand, get between you and your customers, and degrade your competitive advantage. The following examples illustrate threats posed by malicious bots across a variety of situations:

• Grey marketing/inventory grabbing: Many retail sites employ a strategy of time- or inventory-limited sales to generate heightened buzz and awareness, as well as increase add-on sales once shoppers come to their site. They spend time and resources marketing a product or service that has extremely high demand and low inventory (e.g., a limited edition sneaker, limited run designer purse) only to discover that bots have snapped them up within seconds of being released. This can leave your customers feeling duped and frustrated, and they may turn to a competitor’s site to find the product elsewhere.
• Price scraping: Many retail sites sell commodity products or services that are highly price sensitive, which presents an opportunity for competitors to scrape your website for price matching to win customers at your expense, all thanks to bots.
• Content scraping: If providing your customers with proprietary information and research related to the goods you sell (e.g., a guide for matching printers and toners with compatible printers, an auto parts diagram with instructions for installing purchased parts) is a differentiator for your business, bots could threaten this advantage by scraping the information to reuse on competitor sites.

Good bots, on the other hand, do exist, and they can provide benefits to retailers, such as creating better customer experiences highlighted in the following examples:

• Search engines: Even beyond Google, Bing and Baidu, there are hundreds of smaller search engines specializing in different types of consumer goods that help customers find products on your site.
• Online advertising: These third-party services can help retailers attract customers by evaluating the interests of visitors to various sites based on the content viewed and then presenting them with targeted content on pages where the advertiser operates.
• SEO, analytics or marketing: These services help consumers find you in product or brand searches and price comparisons, as well as evaluate the content of your site for attracting customers. In some cases, the crawlers that run these services can account for as much as 20 percent of bot traffic.
• Prioritizing partners: Reseller partners often rely on scraping to keep up with the changes on your website. While you don’t want to block them from accessing this important information, their heavy load has a significant impact on your site’s performance and, ultimately, the user’s experience. What do you do? What compromises are possible?
• Web analytics: Regardless of good or bad bots, one common complaint of many retailers is the difficulty in analyzing website traffic data to improve the customer experience. To optimize the transaction rate of a site, businesses must monitor and manage a range of considerations, from page layout to content to ease of use. This requires data analysis based on how client systems are interacting with your site. With 40 percent to 60 percent of traffic generated by automated bots, however, any strategies designed based on this data risk misfiring.

How to Balance the Good With the Bad
To maximize the opportunities presented by good bots without exposing your business to the risks of the bad, retailers must shift their mind-set from one of bot mitigation to one of bot management. It’s also important to better understand the types of bots coming to your site as well as their goals. What are their impacts on your business? On your IT infrastructure?

In addition, talk with your IT team to learn about your bot policy. In addition to compromising the possibilities of good bots, simply blocking all bot traffic doesn't always address the root cause of bad bots, and can increase bot evasion. Finally, think about the kind of information you want to share as part of your online strategy. How can bots help you deliver better experiences? Are they hindering your ability to provide those experiences? Bot management, as opposed to mitigation, allows you to build a strategy that revolves around your business rather than the bots.

Renny Shen is senior product marketing manager at Akamai Technologies, a content delivery network (CDN) services provider.