robots.txt crawl-delay: how it works and when to use it

Place the directive in the group for the crawler it applies to.

A crawl-delay of 10 means at most one request every 10 seconds from that crawler, or about 8,640 requests a day. For a site with 50,000 pages, that is not enough to recrawl the site in a week. Values of 1 to 5 are a mild slowdown; values above 10 can significantly delay discovery of new and updated pages for crawlers that honour them.

Google adjusts its crawl rate based on how quickly and reliably your server responds. If Googlebot is causing load problems, Google's documentation recommends temporarily returning 500, 503, or 429 status codes, which makes Googlebot slow down, and then removing them once load is under control; returning them for more than a day or two can cause pages to drop. Google retired the crawl rate limiter setting in Search Console in early 2024.

Some AI crawlers request pages aggressively. Rather than slowing every bot, decide per crawler: allow the search and citation crawlers you want, such as OAI-SearchBot and PerplexityBot, and disallow training crawlers if you choose. CDN-level bot management and rate limiting can control abusive traffic that ignores robots.txt entirely. Check server logs to see which agents actually cause load before changing rules.

Most sites should not. Modern hosting handles normal crawler load, and the directive mainly slows search engines that would otherwise help you. Use it only for a specific crawler causing measurable load, with a small value, and revisit it. Our robots.txt checker shows any crawl-delay values in your file and which crawlers they apply to. SerionFlow is the wrong tool for managing crawl rate; that is controlled by your server, CDN, and robots.txt.