Robots.txt File

🔍 Discovery Pillar: Discovery Quick fix Severity: low Est. cost: $0 Gap: documentation Fix: deterministic

What it means

Your site has a robots.txt file that tells crawlers and AI agents what they can access.

Without this file, AI agents don't know which parts of your site they're allowed to read, and may skip your content entirely.

This rule is verified by:

http html Source: Website content

✗

No robots.txt file found. Agents assume nothing is allowed.

✓

A clear robots.txt welcomes AI agents to explore your content.

Enter your URL to check just this one rule (AB-001).

This rule is also available as JSON: /rules/AB-001.json