This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung
Indexing
Why does a non-existent address have to provide status 404?
Each response from a web server has a status code. 200 means: This page exists, pick it up. 404 means: This address does not exist. Some websites answer any address with 200 and then show a pretty error page — unobtrusive for visitors, a problem for search engines: they can no longer distinguish error pages from real pages, add them to the index and waste crawl budget on addresses that never existed. The check area indexing therefore calls up an invented address of the domain for each check and reports if it is answered with 200.
What exactly is checked
The probe creates an address that cannot exist on any website — with a timestamp in the path so that it never exists by chance — and calls it with the same headers as a search engine crawler. If 404 or 410 comes back, everything is fine; That is the rule. If 200 comes, a result with the called address as a receipt arises. The result does not change the point value because the rulebook idx-v1 remains unchanged and issued seals retain their meaning; it appears in the report and in the task list. Forwarding an unknown address to the homepage also counts as a problem if the homepage then responds with 200 – this also blurs the line between existing and missing for search engines.
Where the behavior comes from and how to stop it
Common causes are single-page applications where the server delivers each address to the same start file and only the browser decides if something is missing; blanket forwarding rules that send everything unknown to the homepage; and editorial systems whose error page was set up as a normal page. The correction is in each case at the same point: The error page may be nice, but must be delivered with status 404. For single-page applications, this does a server-side rendering or rule that separates known paths from unknown ones; in the case of transmission rules, it is sufficient to remove the flat-rate rule. After the conversion, the behavior can be checked with any tool that displays status codes.
- Go to an invented address of your domain and check the status code in the developer tools: 404 is correct, 200 is the finding.
- Set up the error page in the editing system or server so that it comes with status 404 - you can still design it.
- Remove blanket rules that redirect anything unknown to the homepage.
- Check again after a relaunch; New server configurations often bring the behavior back.
