This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung
Indexing
Why do robots.txt and the sitemap need each other?
robots.txt is the first file a search engine reads on your website — and that's where the link to the sitemap, the machine-readable table of contents for every page, should be. The "Indexing" audit area checks the entire chain: is there a robots.txt, does it link to a sitemap, is that sitemap reachable, and does it point to the right domain.
robots.txt as your website's house rules
robots.txt is a simple text file that sits at a fixed address: your-domain.at/robots.txt. Search engines read it before every visit. It states which areas of the website crawlers may visit — and, ideally, a line pointing the way to the sitemap. The check requests the file and, along the way, checks whether any blocking rule accidentally affects the page being audited itself. A block at this spot would mean search engines aren't even allowed to visit the page — a silent exclusion nobody notices in everyday use.
The sitemap as a table of contents
The sitemap is a machine-readable list of every page on your website. Search engines use it to quickly find new and changed pages — pages with weak internal linking especially would be hard to discover without it. The check follows the link from robots.txt and checks whether the sitemap is actually reachable. A common, silent error gets specifically checked here: after a relaunch or domain change, the sitemap line sometimes still points to the old domain, or a mistyped one. The link then leads nowhere, with nobody noticing — the finding points exactly to that.
Sampling into the sitemap
The check also draws a sample from the addresses listed in the sitemap and requests them individually. Three outcomes are distinguished: addresses that respond directly with status 200 — the target state. Addresses that only arrive via a redirect — they work, but waste crawl effort, since the sitemap should name the final address. And addresses that end in an error — they don't belong in the sitemap at all, and appear in the audit report along with their status code. That way, the table of contents stays a reliable source instead of a collection of dead links.
- Open your-domain.at/robots.txt in the browser — the file should appear and contain a "Sitemap:" line.
- Open the sitemap address named there — it must load and list your current pages.
- Check after every relaunch or domain change whether the Sitemap line still points to the right domain.
- Have the sitemap maintained automatically, so new pages get added and deleted ones disappear.
- Check that no blocking rule in robots.txt accidentally blocks important pages.
