This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung
GEO / AI
Does my robots.txt block the AI crawlers of ChatGPT, Claude and Perplexity?
Whether a website can appear in the answers of ChatGPT, Claude or Perplexity is not decided in the text, but in a small file at the root directory: the robots.txt. She tells every crawler what to read. The AI providers operate their own crawlers with their own names — GPTBot and OAI SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google Extended for AI use on Google. If one of these names or all bots is completely banned in the file, the page for this system does not exist. The GEO check area reads the robots.txt of the domain and reports every complete block.
How the check reads what the file says
A robots.txt consists of groups: each one starts with one or more user-agent lines and names what is allowed and what is prohibited. For each known AI crawler, the test area first searches for its own group; if there are none, the group applies to all bots, marked with an asterisk. Only a ban on the entire performance counts as a block, i.e. the line Disallow with a single slash. Bans on individual paths — such as an internal area or a search result — are normal and cost no points. If all tested crawlers are locked, the rule is zero of twelve points; Individuals are locked, with six. If the file is completely missing or unreachable, nothing is considered locked and could not be read due to a network error, the rule remains without deduction, The unknown never becomes a lack.
A lock can be wanted – then the result is the confirmation
Some operators deliberately block AI crawlers because they do not want their texts to flow into language models. This is a legitimate decision, and the finding does not contradict it: he states that the block works, and that the page is therefore not cited. If you want both protection and visibility, you can differentiate: OAI SearchBot and PerplexityBot are used for search and citation, GPTBot and Google-Extended are used for training. More common than the conscious decision, however, is the unnoticed: security plugins and copied templates block all unknown bots, and no one has ever looked. That's what the finding is for. Sharing is a line in the file; It works on the next crawl.
- Open the robots.txt of your domain in the browser and search for the names GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, as well as a group for all bots with disallow and slash.
- Decide consciously: Visibility in AI responses or protection against training usage — and distinguish search crawlers from training crawlers.
- Check security plugins and templates for blanket bot locks; They are the most common cause.
- Have the page checked again after a change; the robots.txt is freshly read for each test.
