robots.txt disallow all: how it works and when to use it

Each pattern is a complete file; groups are separated by a blank line.

A crawler follows the most specific user-agent group that matches it and ignores the others, so in the 'all except Googlebot' pattern, Googlebot reads only its own group. Within a group, Google applies the most specific matching rule by path length, and when an Allow and a Disallow are equally specific, the less restrictive Allow wins. Paths are case-sensitive. The file must be at the root of each host and protocol: a robots.txt on www does not cover a subdomain.

It does not keep pages out of search results. If other sites link to a blocked URL, Google can index the URL without crawling it and show it with no description. It does not remove pages already indexed; they may stay until Google decides to drop them. It does not stop non-compliant bots, scrapers, or people. To remove a live site from search, allow crawling, serve noindex on every page, and use Search Console's Removals tool for urgent cases.

Teams often put Disallow: / on staging environments, then copy the file to production during a launch, blocking the entire live site. Password protection or IP allowlisting on staging is safer: crawlers cannot access the content at all, and there is no robots.txt to leak into production. If you must use a staging robots.txt, generate it from environment configuration so production can never receive it.

Fetch /robots.txt on the live domain after each launch or migration and confirm it does not contain Disallow: / for user-agent *. Search Console's robots.txt report shows the version Google last fetched and any errors. Our free robots.txt checker tests whether specific URLs are allowed for Googlebot, Bingbot, and AI crawlers, and the robots.txt monitoring guide explains how to get alerted when the file changes. SerionFlow is the wrong tool for managing robots.txt on your site; it does not edit your server files, so changes happen in your host or CMS.