Per-bot AI rules correct variant
Correctly-spelled, non-contradictory groups for named AI bots (GPTBot, ClaudeBot, …) in robots.txt.
Site-wide file — one per host.
This signal isn't a page; the container fetches it at a fixed path, and the
correct/broken/authbroken variant is chosen by the request host. On this host the canonical path(s)
below serve the correct variant. Add
?mode=… to override locally.
Canonical path(s)
Broken variant
A misspelled token (GPT-Bot), a group that both Allows and Disallows /, an empty group, and a duplicated group.
Defect metrics
| Defect boolean |
|---|
perBotRuleMisspelledToken |
perBotRuleContradictoryAllowDisallow |
perBotRuleEmptyGroup |
perBotRuleDuplicateGroup |
Fetch the live file: https://agentic-correct.crawler-test.com/robots.txt