Blocked by robots.txt: what it means and how to fix it

They look similar but mean different things.

Open the live robots.txt at the root of the exact host and protocol shown in the report. Look for Disallow lines in the group that applies to Googlebot, which is the group naming Googlebot if one exists, otherwise User-agent: *. Rules match URL prefixes and can use * and $ wildcards, so Disallow: /*? blocks every URL with a query string. Our free robots.txt checker tests a URL against the live file and names the matching rule.

A Disallow: / left over from staging blocks the whole site. Rules meant for parameters, such as Disallow: /*?, also block legitimate paginated or filtered pages. Blocking /wp-content/ or JavaScript and CSS folders stops Google rendering pages properly. Platform defaults can block paths like /search or /collections/*+* on stores. And a CDN or host may serve a different robots.txt than the one in your repository, so always check the live file.

Edit the rule where the file is generated: a physical file, a CMS or SEO plugin setting, framework configuration, or host settings. Deploy, then confirm the live file changed. Google caches robots.txt for up to 24 hours in most cases; Search Console's robots.txt report lets you request a recrawl of the file. Then use URL Inspection to request indexing for important URLs.

Blocks are appropriate for crawl traps such as infinite calendars and faceted filters, internal search results, and admin areas. If those appear in the report, no action is needed. If they appear under 'Indexed, though blocked' and you want them gone from search, temporarily allow crawling and serve noindex so Google can see it, because a block prevents Google from reading the noindex. SerionFlow is the wrong tool for editing robots.txt rules on your site; find and change the rule where your file is generated.