robots.txt for AI crawlers: how to allow or block GPTBot, ClaudeBot, and others

The user agents below are the ones that affect visibility in AI assistants and answer engines. The names are the tokens to use in User-agent lines; vendors document them publicly and change them occasionally, so re-check annually.

Frame the decision per crawler as a trade between two outcomes: being retrievable and citable in that assistant, versus having your content used for model training. OAI-SearchBot, ClaudeBot, and PerplexityBot are primarily about retrieval and citation, which is the outcome GEO wants. GPTBot and CCBot are more about training corpora. Google-Extended affects Gemini grounding and training but not Googlebot, so blocking it does not remove you from Google search or AI Overviews.

A common 2026 configuration for a site that wants AI citations is to allow OAI-SearchBot, ClaudeBot, PerplexityBot, and Bingbot, and to make a deliberate choice on GPTBot, CCBot, and Google-Extended based on how the business feels about training use.

Write one group per crawler rather than relying on the wildcard group, because a crawler that finds a group addressed to it uses only that group. To allow a crawler everywhere except private paths, give it a group with your standard Disallow lines. To block a crawler entirely, give it a group with Disallow: /. Keep the Sitemap line in the file regardless.

If your file has only User-agent: * rules, every AI crawler follows them, which is usually fine. The problem arises when a template adds a specific group for one AI crawler and forgets the others, or when a specific group exists with a Disallow: / copied from a "block AI" snippet without the site owner realising it also blocks retrieval. Explicit groups make the intent legible and testable.

robots.txt is only one layer. CDNs including Cloudflare offer one-click AI crawler blocking, and some accounts have it on by default; web application firewalls and bot-management rules can return 403 to AI user agents regardless of what the file says. Verify by fetching a page with each user agent string and checking for a 200 and the full HTML. SerionFlow's free AI crawler access checker runs this test on a URL for the common AI agents, and the free robots.txt checker evaluates the file itself.

Also confirm the page content is present in the initial HTML. AI crawlers do not execute JavaScript, so a page that renders its content client-side is effectively invisible to them even when allowed.

Pages SerionFlow serves under your domain are ordinary first-party HTML at a configured path such as /explore, so they inherit your robots.txt and CDN settings. If you want those pages cited, the AI crawler groups must allow that path and the CDN must not block the agents. The site-wide robots.txt remains yours to control; SerionFlow does not modify it.