This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung →
robots.txt and sitemap: the two files that control access
The check looks at whether robots.txt names a sitemap, and whether it's reachable. robots.txt controls what may be crawled; the sitemap says what exists at all — two different jobs that often get confused.
- The most common mistake is a reference to a sitemap that no longer exists, or that sits on a different host. The check reports both, because a dead reference has the same effect as no reference at all.
- The second most common is an exclusion that blocks too much — a directory with stylesheets and scripts, say. Search engines then can't render the page properly, and rate it accordingly.
- The audit service itself abides by this file: it doesn't crawl addresses it blocks for its product token — and explicitly states the exclusion in the result, instead of presenting it as missing content.
FAQ
Does the sitemap have to be listed in robots.txt?
It's not required, but it's the most reliable way to announce it — regardless of whether it's also submitted through a separate tool.
What happens if robots.txt is missing?
Then everything counts as allowed. That's not an error; the sitemap reference is just missing then, and targeted exclusions aren't possible.
Does robots.txt block a page from the index?
No — it only forbids crawling. A blocked page can still end up in the index anyway, then with no content. noindex is what excludes a page from the index, but the search engine only reads it if it's allowed to crawl the page.