This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung
SEO
How do I recognize an invalid line in the robots.txt?
A robots.txt tells search engines which areas of your website they are allowed to retrieve. Google skips lines that are invalid. Lighthouse warns that an invalid robots.txt can keep search engines away from public pages or share pages that should not appear in the search. Expert mode checks each line and names invalid ones with their number.
A rule before the user agent does not apply to a crawler
If an instruction is before the first user-agent line, no crawler follows it after Lighthouse. Anyone who wanted to block such an area left it open. The robots.txt is also readable for everyone: who calls it, sees all paths that you block.
How the expert mode checks the robots.txt
Expert mode loads the site’s robots.txt and checks each line for errors that Lighthouse also reports: unknown statement, a pattern that doesn’t start with “/” or “*”, a “$” in the middle of the pattern, a sitemap without a valid address and rules without a user agent. To do this, he reports lines without colon and empty user-agent lines. The file names each line with number and reason.
Google Reads These Rules
Google supports the user-agent, allow, disallow and sitemap fields. Others like crawl-delay override it. A path starts with "/", and in the path, uppercase and lowercase do not count, in the name of the field. The character "*" stands for any number of characters, "$" for the end of the address. Name the sitemap with the full address including protocol and domain. What is behind 500 KiB, Google no longer reads.
Examples
Lock for an area
Disallow: /intern
User-agent: *
Disalow: /lager-alt
Sitemap: /sitemap.xmlUser-agent: *
Disallow: /intern
Disallow: /lager-alt
Sitemap: https://www.beispiel.at/sitemap.xmlFirst the user-agent line, then the rules; the sitemap with a complete address.
- Open your robots.txt at the address of your website with /robots.txt at the end and correct the lines that the file mentions.
- Start each group with a user-agent line, including Allow and Disallow with paths starting from "/".
- Check the file in the robots.txt report of Google Search Console.
- Write the instructions exactly as in the description of Google: User-agent, Allow, Disallow, Sitemap. A typo is hardly noticeable when reading, an examination finds it immediately.




