Robots.txt Checker

This robots.txt checker fetches the live robots.txt from the host of the URL you enter, parses every User-agent group, and tests that exact URL against Googlebot and against a generic crawler using longest-match rules. It reports the winning rule and its line number, so an unexpected block can be traced to the line that caused it. It also flags a site-wide Disallow for all crawlers, blocked CSS, JavaScript, or image paths, Crawl-delay values, lines crawlers cannot parse, a missing Sitemap directive, and whether AI crawlers have groups of their own.

Three failures matter most. A server error on robots.txt makes Google treat the whole site as disallowed until the file recovers. A Disallow rule left over from staging can block the entire site or its CSS and JavaScript paths, starving the renderer. And AI crawler groups for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended can be blocked by a copied template or a CDN default, which ends any chance of citation in AI answers.

Run the checker after every deploy and whenever a CDN or CMS setting changes, one URL per run: the homepage, a key landing page, a representative asset path, and one URL you expect to be blocked. For a per-bot verdict across GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers, use the AI crawler access checker. robots.txt controls crawling, not indexing: a page blocked here can still appear in results if it is linked elsewhere, so use a noindex tag to keep a page out of the index.

What this checker cannot tell you: whether a CDN or firewall returns 403 to a crawler before robots.txt is consulted, whether an allowed page actually contains its content in the static HTML, or whether a blocked page is still indexed from links. Pair it with the AI crawler access checker, a fetch with JavaScript disabled, and Search Console's Pages report.