seo · Standard
Robots Exclusion Protocol (RFC 9309)
Published by IETF · Free
The actual internet standard defining robots.txt — what directives mean and how crawlers must treat them.
Why it matters
Reading the specification clears up most robots.txt misconceptions faster than any second-hand summary.
What it helps with
- Writing correct crawl directives for search and AI bots
- Understanding precedence, grouping and matching rules
Hosted by IETF. Rankly describes this resource but does not reproduce its content.
Related resources
See how your website scores on these foundations with a free Rankly audit.
Analyze website →