This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung →
Indexing
Why do robots.txt and the sitemap need each other?
robots.txt is the first file a search engine reads on your website — and that's where the link to the sitemap, the machine-readable table of contents for every page, should be. The "Indexing" audit area checks the entire chain: is there a robots.txt, does it link to a sitemap, is that sitemap reachable, and does it point to the right domain.
robots.txt as your website's house rules
robots.txt is a simple text file that sits at a fixed address: your-domain.at/robots.txt. Search engines read it before every visit. It states which areas of the website crawlers may visit — and, ideally, a line pointing the way to the sitemap. The check requests the file and, along the way, checks whether any blocking rule accidentally affects the page being audited itself. A block at this spot would mean search engines aren't even allowed to visit the page — a silent exclusion nobody notices in everyday use.
The sitemap as a table of contents
The sitemap is a machine-readable list of every page on your website. Search engines use it to quickly find new and changed pages — pages with weak internal linking especially would be hard to discover without it. The check follows the link from robots.txt and checks whether the sitemap is actually reachable. A common, silent error gets specifically checked here: after a relaunch or domain change, the sitemap line sometimes still points to the old domain, or a mistyped one. The link then leads nowhere, with nobody noticing — the finding points exactly to that.
Sampling into the sitemap
The check also draws a sample from the addresses listed in the sitemap and requests them individually. Three outcomes are distinguished: addresses that respond directly with status 200 — the target state. Addresses that only arrive via a redirect — they work, but waste crawl effort, since the sitemap should name the final address. And addresses that end in an error — they don't belong in the sitemap at all, and appear in the audit report along with their status code. That way, the table of contents stays a reliable source instead of a collection of dead links.
- Open your-domain.at/robots.txt in the browser — the file should appear and contain a "Sitemap:" line.
- Open the sitemap address named there — it must load and list your current pages.
- Check after every relaunch or domain change whether the Sitemap line still points to the right domain.
- Have the sitemap maintained automatically, so new pages get added and deleted ones disappear.
- Check that no blocking rule in robots.txt accidentally blocks important pages.
FAQ
What is robots.txt?
robots.txt is a text file in a website's root directory that tells search engines which areas they may visit. It always sits at the same address: the domain, followed by /robots.txt. It should also link to the website's table of contents with a Sitemap line.
What is a sitemap?
A sitemap is a machine-readable file that lists every page on a website. Search engines use it to quickly find new and changed content. It should contain only addresses that respond directly and with no errors — redirects and error pages have no place in it.
Does the sitemap have to be listed in robots.txt?
It's the most reliable route, since every search engine looks for robots.txt in the same spot. The entry is a single line with the full sitemap address. Our check only awards this rule's points if the entry is present and the named sitemap actually responds.
Why does my Sitemap line point to the wrong domain?
It's usually a leftover: when moving from a test environment or an old domain, the robots.txt got copied along, and the Sitemap line still names the earlier address. The check specifically detects this case and flags it in the finding when the sitemap's domain differs from the domain being audited.