E-commerce Insights: The (Free) Benefits of Google’s Webmaster Central

• Robots.txt tab is next. This lets you see which areas of your site you’ve asked spiders to avoid. Compliance with robots.txt is voluntary; spam scrapers and other bad guys can (and do) ignore it. But Googlebot is well-mannered and obeys your robots.txt. A syntax error or logic error in your robots.txt can destroy rankings, so it makes sense to glance at it every few months. On my site, I see we’re giving Google freedom to roam, but we’ve opted to exclude ourselves from the Internet archives (www.archive.org) via the following two lines: