Improve AI visibility · Free · No signup
AI Crawler Access Checker
Review robots.txt rules for major search and AI crawlers.
Public information only. Do not enter passwords, private URLs, or confidential data.
What this AI crawler access checker checks and how to act on it
This AI crawler access checker reads your live robots.txt and evaluates the URL you enter for 19 search and AI crawlers, including OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Googlebot, and Bingbot, reporting for each whether it is allowed or blocked and the rule that decided it. It also flags a noindex on the page and checks for an llms.txt file. It evaluates robots.txt rules only: a CDN or firewall that returns 403 to a crawler is not visible here. Blocking at the CDN is common and silent, because Cloudflare can block AI crawlers with one setting, so also check your CDN bot settings or server logs.
Allowing a crawler enables retrieval and citation in the corresponding assistant; blocking it ends that channel. Google-Extended controls Gemini training and grounding, not Googlebot crawling or AI Overviews. Because AI crawlers do not execute JavaScript, also confirm the page content is in the initial HTML; a page that renders client-side is invisible to them even when allowed.
Read the guide
Step-by-step explanations of what this tool checks and how to fix what it finds.