WordPress robots.txt: Default File, Editing, and Rules

What we measured · 2026-10-22

What Google shows for "wp robots.txt" right now

Method: Live Google results (United States, desktop, English) for "wp robots.txt" retrieved through DataForSEO's SERP API on 2026-10-22. Page types are our own classification of each ranking URL.

The top 9 organic results for "wp robots.txt" were 4 guides or explainers, 3 product or service pages, 2 forum or community threads. The first three were wordpress.org, docs.wpvip.com, www.reddit.com. Community threads (www.reddit.com, stackoverflow.com) ranked on page one, a sign searchers want first-hand answers, not only vendor pages.

Google showed an AI Overview for this query, citing 5 source domains including wpengine.com, yoast.com, www.reddit.com, inspectwp.com. Pages that state the answer directly and early are the ones an Overview can quote.

People Also Ask listed: "Is robots.txt legal?"; "Is robots.txt still used?"; "How do I change the robots.txt file in WordPress?"; "How to fix blocked by robots.txt error?". The sections below answer the ones within this page's scope.

Top organic results for "wp robots.txt", 2026-10-22
Position	Domain	Page type
1	wordpress.org	Product or service page
2	docs.wpvip.com	Product or service page
3	www.reddit.com	Forum or community thread
4	wpengine.com	Guide or explainer
5	perishablepress.com	Product or service page
6	www.seopress.org	Guide or explainer

Where WordPress robots.txt comes from

Visit yourdomain.com/robots.txt and you will see a file, but you will not find it in your WordPress directory. WordPress builds it on request through the do_robots function. On a standard install the output is a user-agent line for all crawlers, a disallow rule for /wp-admin/, an allow rule for /wp-admin/admin-ajax.php so front-end features keep working, and a Sitemap line pointing to the core sitemap.

If the 'Discourage search engines from indexing this site' setting is ticked, current WordPress versions add a noindex robots meta tag to pages rather than relying only on robots.txt. A surprising number of live sites still have that box ticked after launch, so check Settings, Reading first if your pages are not appearing in search.

Three ways to edit it

Choose one method and stick to it, because they can conflict.

Ways to change robots.txt in WordPress
Method	How	Best for
SEO plugin editor	Yoast: Tools, File editor. Rank Math: General Settings, Edit robots.txt	Most site owners
Physical file	Upload robots.txt to the web root by SFTP or the host's file manager	Full control; overrides the virtual file and plugin editors
robots_txt filter	Add a filter in a small plugin or child theme to append rules	Developers who want rules in version control
Managed host settings	Some hosts manage robots.txt for staging and production	Check your host before editing

A safe starting file

For most WordPress sites, this is enough: User-agent: * on the first line, then Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, and a Sitemap line with the full URL of your sitemap index, for example the one your SEO plugin generates. Add rules only for a specific reason, such as internal search result pages (Disallow: /?s=) that create endless crawlable URLs.

Avoid the older advice to block /wp-includes/, /wp-content/plugins/, or theme folders. Google renders pages with their CSS and JavaScript, and blocking those resources can stop it from seeing the page as visitors do.

Handling AI crawlers

Decide your policy per crawler rather than with one blanket rule. Search-oriented agents such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot fetch pages so they can be cited in answers; training-oriented agents such as GPTBot, ClaudeBot, and Google-Extended govern use in model training. Blocking a training agent does not have to block citation. Whatever you choose, never block Googlebot or Bingbot, since AI Overviews and Copilot depend on those indexes. Our AI crawler access checker reads your live file and shows which agents are allowed.

Common mistakes and how to fix them

The most damaging mistake is a leftover Disallow: / from a staging site, which blocks everything. The second is using robots.txt to hide pages from search: a blocked URL can still be indexed from links, and Google cannot see a noindex tag on a page it is not allowed to crawl. Use a noindex meta tag for pages that should stay out of results and leave them crawlable. Third, two sources of truth: a physical file plus a plugin editor means the plugin edits do nothing. After any change, test the live file with our robots.txt checker and the robots.txt report in Search Console. SerionFlow is the wrong tool for editing a WordPress robots.txt; it does not change files or settings on your WordPress site, so use your SEO plugin or host.

Clear answers

Frequently asked questions

Does WordPress create a robots.txt by default?

Yes. It serves a virtual robots.txt generated on each request, so the file exists at /robots.txt even though there is nothing on disk.

How do I fix 'Blocked by robots.txt' on WordPress?

Open the live robots.txt, find the Disallow rule that matches the reported URL, and remove or narrow it in whichever method controls your file. Then use URL Inspection to request a recrawl.

Is robots.txt still used?

Yes. All major search engines and most AI crawlers from large providers read it, though it is a request rather than an enforcement mechanism.

About the author

Shadrack Kumi · Founder, SerionFlow

Shadrack builds SerionFlow's site research, page generation, and Search Console tooling, and runs the free tools these guides pair with. Search Console figures cited on these pages come from serionflow.com's own property.

Continue exploring

Related SerionFlow resources

Let SerionFlow turn your market evidence into momentum.

Confirm the market, rank the opportunities that matter, create brand matched pages on your domain, and keep moving with controlled weekly intelligence.

Start free Explore plans