How sites that rank for SEO queries treat AI crawlers in robots.txt

The 749 domains are every site that ranked in Google's US top ten for the 170 queries behind this site's content plan. They include SEO software vendors, agencies, publishers, documentation sites, communities, and platforms. 684 returned a valid robots.txt as plain text. This is not a random sample of the web; it represents sites that compete for SEO and AI search attention.

AI companies separate crawlers that fetch pages for answers and citations from those that collect training data. We grouped OAI-SearchBot, Claude-SearchBot, and PerplexityBot as search agents, and GPTBot, ClaudeBot, Google-Extended, and CCBot as training agents.

Among sites competing in our niche, blocking AI crawlers is uncommon, and blocking the search agents that produce citations is rarer still. The sites that do block tend to block training agents while allowing search agents, which keeps them eligible for citations in ChatGPT search and Perplexity while opting out of model training. If visibility in AI answers matters to you, that split is the policy most consistent with your goal.

607 files (89 percent) included a Sitemap line. Explicitly naming Googlebot or Bingbot was uncommon, at 63 and 56 files respectively, and almost never to block them. Newer agents were named less often than older ones: Claude-SearchBot appeared in 50 files compared with 115 for ClaudeBot, suggesting many files have not been updated since newer agents were announced.

robots.txt shows stated policy, not enforcement; some sites block AI agents at the CDN or firewall instead, which this method cannot see. We counted only site-wide blocks of the root path, so partial blocks of sections are not included. Sites on subdomains were treated separately. Our free AI crawler access checker runs the same parsing logic on any single site; SerionFlow does not manage robots.txt for you.