How to test and validate a robots.txt file

robots.txt tells crawlers which paths they may request. It does not remove pages from the index; a page blocked by robots.txt can still be indexed from links and shown with no snippet. To keep a page out of results, use a noindex meta tag or X-Robots-Tag header and let it be crawled. To stop crawlers hitting expensive endpoints such as /api/ or faceted search, robots.txt is the right tool.

It is also advisory. Well-behaved crawlers from Google, Bing, and the main AI companies obey it; scrapers do not. Treat it as crawl management, not security.

Request https://yourdomain.com/robots.txt directly. It must return HTTP 200 with a text/plain content type and be under 500 KiB, which is Google's parsing limit. A 404 is treated as "allow everything"; a 5xx or timeout makes Google treat the site as fully disallowed until the file recovers, which is the single most damaging robots.txt failure. Check that the www and non-www hosts and the http and https schemes all serve the same file, since each host-scheme pair is a separate robots.txt in the protocol.

The file is groups of lines. Each group starts with one or more User-agent lines followed by Allow and Disallow rules, and Sitemap lines can appear anywhere. Common syntax faults are a rule before any User-agent line, a Disallow with no path (which allows everything), Windows line endings mixed with Unix, and a byte-order mark at the start of the file that makes the first line unparseable. Comments start with #. Paths are case-sensitive and must start with /.

Syntax can be valid and still wrong. Pick ten URLs that must be crawlable, including your homepage, a product or article page, your sitemap, and your CSS and JS asset paths, plus five URLs that must be blocked. Test each against Googlebot, Bingbot, and the AI agents using a tester that implements the longest-match rule. SerionFlow's free robots.txt checker fetches the live file, parses it, and reports allow or block per URL and user agent; Google Search Console's robots.txt report shows the version Google last fetched and any parse errors.

Blocking CSS or JavaScript paths is a frequent silent failure: the page is crawlable but Google cannot render it, and rendering-dependent content is lost.

GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Google's AI training control), and CCBot (Common Crawl) each honour their own User-agent group. A blanket Disallow for these ends any chance of citation in the corresponding assistants, which may or may not be what you intend. Decide deliberately, write explicit groups, and test them. Note that Google-Extended controls training use of your content, not Googlebot crawling, so blocking it does not remove you from AI Overviews. Some CDNs, including Cloudflare, can block AI crawlers at the edge regardless of robots.txt, so check that layer too.

Disallow: / left over from staging. A Sitemap line pointing at a relative path or a 404. Blocking /wp-content/ or /assets/ and starving the renderer. Using Disallow to hide pages that are already indexed, which freezes their stale snippet in place. Expecting robots.txt to apply across subdomains; each host needs its own file. And deploying a new file without re-testing, because the previous one was fine.