This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung →
AI crawler in robots.txt: who locks out is not quoted
Answer machines only cite what their crawlers are allowed to read. If the robots.txt of a website blocks GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended with “Disallow: /”, the page is not present for these systems – regardless of how good content and structure are. The check area reads the robots.txt of the domain and reports every complete block.
- The block can be a conscious decision, for example for copyright reasons. Then the finding is not a mistake, but the confirmation that the decision works – who wants AI visibility, cancels it.
- Many locks come from copied templates or plugin presets that blanketly lock out all bots. Check if the lock is desired.
- Lock partial paths is allowed and does not count as a block: “Disallow: /internal/” leaves the public content readable.
