clock
AB-111
Crawl-delay directive in robots.txt
What it means
robots.txt includes a Crawl-delay directive to control crawl rate.
Why it matters
Crawl-delay prevents aggressive crawling that can overload your server and ensures agents respect your rate limits.
✗
What's wrong
robots.txt with no Crawl-delay directive.
✓
What's right
Crawl-delay: 1 in robots.txt.
Check this rule on your site
Enter your URL to check just this one rule (AB-001).