Check technical SEO · Free · No signup
Robots.txt Generator
Draft a clean robots.txt from your live file, with common private paths, a Sitemap line, and an explicit policy for AI search and training crawlers.
Public information only. Do not enter passwords, private URLs, or confidential data.
What this robots.txt generator checks and how to act on it
A robots.txt generator drafts the file that tells crawlers which paths they may fetch. This generator reads your live robots.txt and sitemap, then drafts a clean file with common private paths disallowed, an explicit group allowing AI search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, commented rules for training crawlers you can opt out of, and a Sitemap line.
In our study of 684 robots.txt files from sites ranking for SEO queries, only 8 percent blocked any AI crawler site-wide, and blocking AI search crawlers was rarer than blocking training crawlers. Whatever you choose, never block Googlebot or Bingbot, and remember that robots.txt controls crawling, not indexing: use noindex to keep a page out of results.
How to use the robots.txt generator
- Enter your homepage URL.
- The tool reads your live robots.txt and sitemap, then drafts a clean file with an AI crawler policy.
- Delete paths that do not exist on your site, decide on training crawlers, and publish at /robots.txt.
- Test the new file with the robots.txt checker.
What it does not check
- CMS-specific needs beyond common WordPress and store paths.
- CDN or firewall bot rules, which robots.txt cannot change.
Read the guide
Step-by-step explanations of what this tool checks and how to fix what it finds.