The AI crawlers and what each one controls
The user agents below are the ones that affect visibility in AI assistants and answer engines. The names are the tokens to use in User-agent lines; vendors document them publicly and change them occasionally, so re-check annually.
| User agent | Operator | Allowing it enables | Notes |
|---|---|---|---|
| GPTBot | OpenAI | Training use and retrieval for OpenAI products | Widely blocked by templates; check before assuming |
| OAI-SearchBot | OpenAI | Appearing in ChatGPT search results | Separate from GPTBot; allow this for search visibility |
| ClaudeBot | Anthropic | Retrieval and citation in Claude | Respects robots.txt groups addressed to it |
| PerplexityBot | Perplexity | Indexing for Perplexity answers | Perplexity also documents a user-triggered fetch agent |
| Google-Extended | Use of content for Gemini training and grounding | Does not affect Googlebot crawling or AI Overviews | |
| CCBot | Common Crawl | Inclusion in the Common Crawl dataset used by many models | Blocking reduces presence in future training sets |
| Bingbot | Microsoft | Bing and Microsoft Copilot | Standard search crawler; also feeds Copilot |
Deciding what to allow
Frame the decision per crawler as a trade between two outcomes: being retrievable and citable in that assistant, versus having your content used for model training. OAI-SearchBot, ClaudeBot, and PerplexityBot are primarily about retrieval and citation, which is the outcome GEO wants. GPTBot and CCBot are more about training corpora. Google-Extended affects Gemini grounding and training but not Googlebot, so blocking it does not remove you from Google search or AI Overviews.
A common 2026 configuration for a site that wants AI citations is to allow OAI-SearchBot, ClaudeBot, PerplexityBot, and Bingbot, and to make a deliberate choice on GPTBot, CCBot, and Google-Extended based on how the business feels about training use.
Copy-ready robots.txt groups
Write one group per crawler rather than relying on the wildcard group, because a crawler that finds a group addressed to it uses only that group. To allow a crawler everywhere except private paths, give it a group with your standard Disallow lines. To block a crawler entirely, give it a group with Disallow: /. Keep the Sitemap line in the file regardless.
- Allow example: "User-agent: ClaudeBot" then "Disallow: /api/" and "Disallow: /admin/".
- Block example: "User-agent: GPTBot" then "Disallow: /".
- Keep the wildcard group for everything else and put your Sitemap line at the end.
Why a wildcard is not enough
If your file has only User-agent: * rules, every AI crawler follows them, which is usually fine. The problem arises when a template adds a specific group for one AI crawler and forgets the others, or when a specific group exists with a Disallow: / copied from a "block AI" snippet without the site owner realising it also blocks retrieval. Explicit groups make the intent legible and testable.
Verifying the rules are actually in effect
robots.txt is only one layer. CDNs including Cloudflare offer one-click AI crawler blocking, and some accounts have it on by default; web application firewalls and bot-management rules can return 403 to AI user agents regardless of what the file says. Verify by fetching a page with each user agent string and checking for a 200 and the full HTML. SerionFlow's free AI crawler access checker runs this test on a URL for the common AI agents, and the free robots.txt checker evaluates the file itself.
Also confirm the page content is present in the initial HTML. AI crawlers do not execute JavaScript, so a page that renders its content client-side is effectively invisible to them even when allowed.
What this means for SerionFlow-published pages
Pages SerionFlow serves under your domain are ordinary first-party HTML at a configured path such as /explore, so they inherit your robots.txt and CDN settings. If you want those pages cited, the AI crawler groups must allow that path and the CDN must not block the agents. The site-wide robots.txt remains yours to control; SerionFlow does not modify it.
Clear answers
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
+
Blocking GPTBot stops OpenAI's general crawler. ChatGPT search visibility is governed separately by OAI-SearchBot, so a site can block GPTBot for training while allowing OAI-SearchBot for search results. Check OpenAI's current documentation, as the split has changed over time.
Does Google-Extended affect my Google rankings?
+
No. Google-Extended controls whether your content is used for Gemini training and grounding. It does not change Googlebot crawling, indexing, ranking, or appearance in AI Overviews, which use the standard Google index.
Do AI crawlers actually obey robots.txt?
+
The major operators, OpenAI, Anthropic, Perplexity, Google, and Common Crawl, document that their named crawlers honour robots.txt. Some smaller scrapers do not. For enforcement rather than politeness, use CDN bot rules, but remember those also block the crawlers you want.
How do I check whether my CDN is blocking AI crawlers?
+
Fetch a page with each AI user agent string and look for a 403 or a challenge page instead of a 200 with full HTML. In Cloudflare, review the AI crawler and bot management settings in the dashboard. SerionFlow's free AI crawler access checker automates the fetch test.
Should I allow AI crawlers on every page?
+
Allow them on the pages you want cited and block the paths you would block for any crawler: APIs, admin areas, internal search, and duplicate parameter URLs. There is no benefit to letting AI crawlers into paths that are not meant to be read.
Continue exploring
Related SerionFlow resources
More in Guides
- robots.txt monitoring: how to detect changes and outages
- Title tag length and best practices
- How to connect Webflow to Google Search Console and use the data
- What is AI SEO and why does it matter?
- What is IndexNow and how do you verify it is working?
- What is llms.txt and how do you create one?
- Writesonic vs SE Ranking: which fits your workflow?
- AEO link building: how to earn links and mentions that answer engines trust
Make the next move obvious
Let SerionFlow turn your market evidence into momentum.
Confirm the market, rank the opportunities that matter, create brand matched pages on your domain, and keep moving with controlled weekly intelligence.