Noindex in robots.txt: what to use instead

In July 2019 Google proposed formalising the Robots Exclusion Protocol as an internet standard and announced it would retire support for unpublished rules, including noindex, from 1 September 2019. The protocol was later published as RFC 9309 in 2022. robots.txt controls crawling, not indexing; the correct place for indexing directives is the page or its HTTP headers.

Pick the method that fits the content type.

If a URL is disallowed in robots.txt, Google does not fetch it and never sees a noindex tag or header on it. The URL can still be indexed from links, appearing without a description. To remove a page from search, allow crawling and serve noindex. Once it has dropped out, you can leave it noindexed, or block crawling afterwards if you also want to save crawl capacity, accepting that Google may keep the URL in its records.

For a staging or private site, password protection is the safest choice: nothing can be crawled or indexed. If the site must be public but unindexed, set an X-Robots-Tag: noindex header for every response at the server, CDN, or host level, which avoids editing every template. Most hosts support custom headers; on Vercel, a headers entry in vercel.json applies it. serionflow.com uses this approach to noindex its authenticated app routes.

Run URL Inspection's live test; the indexing section reports whether indexing is allowed and whether noindex was detected in the meta tag or header. Check response headers in the browser's network panel or with a command-line request. View the raw HTML for the meta tag, since JavaScript can add or remove it after load. Our robots.txt checker shows the rules in your live file, so leftover Noindex lines are easy to spot and remove. SerionFlow is the wrong tool for applying noindex across your existing site; set that in your HTML or server headers.