robots.txt monitoring: how to detect changes and outages

robots.txt is a single file with site-wide consequences and no safety net. Google fetches it roughly once a day per host, caches it for up to 24 hours, and treats a 5xx response as "disallow everything" until it recovers. A Disallow: / that ships from a staging template, a CDN rule that starts blocking Googlebot from the file, or a well-meaning edit that blocks /assets/ can all cut crawling without any error in your analytics until rankings slip.

Monitoring turns a weeks-long mystery into a same-day alert with a diff.

Availability: the file must return HTTP 200 with a text content type on every host and scheme. Content change: the bytes of the file must not change without a corresponding deploy. Rule regression: a fixed list of must-crawl URLs must still evaluate to allow for Googlebot, Bingbot, and any AI crawlers you permit, and a list of must-block URLs must still evaluate to block.

Any uptime monitor that can request a URL and assert on status code and response body works. Add a check for https://yourdomain.com/robots.txt that expects 200, expects the body to contain a known string such as "User-agent:", and alerts on failure within one cycle. Add the same for the www host if you serve both. Because the failure mode is severe, route this alert to the on-call channel, not to a marketing inbox.

A short script run daily, and as a post-deploy step, fetches the file, computes a hash, compares it with the last approved hash, and on a difference stores the new version with a timestamp and posts the diff to your team channel. Keeping every version in a repository or bucket means a regression can be traced to a specific deploy and rolled back with confidence. If your robots.txt is generated from code, as it is on many frameworks, also add a test in CI that renders the file and compares it with a golden copy.

Maintain a fixed list of URLs in two groups: must be crawlable (homepage, key landing pages, sitemap, CSS and JS paths) and must be blocked (/api/, admin, faceted parameters). Weekly, and whenever content changes, evaluate every URL against Googlebot, Bingbot, and the AI agents you allow using a parser that implements longest-match rules. SerionFlow's free robots.txt checker can be used for ad-hoc runs; a scripted parser is better for scheduled runs. Fail the check if any URL changes group.

For an availability failure, fix the server or CDN first and verify the file returns 200, then request a recrawl in Search Console's robots.txt report. For a content change, read the diff; if it was unintended, redeploy the approved version and re-run the rule test. For a rule regression, correct the rule, redeploy, and check Search Console over the next few days for the affected URLs. Because Google caches the file for up to a day, expect a lag between the fix and normal crawling resuming.