Skip to content
EUID Audit service & seal, Home
Get your website audited

This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung

SEO

How do I recognize an invalid line in the robots.txt?

A robots.txt tells search engines which areas of your website they are allowed to retrieve. Google skips lines that are invalid. Lighthouse warns that an invalid robots.txt can keep search engines away from public pages or share pages that should not appear in the search. Expert mode checks each line and names invalid ones with their number.

A rule before the user agent does not apply to a crawler

If an instruction is before the first user-agent line, no crawler follows it after Lighthouse. Anyone who wanted to block such an area left it open. The robots.txt is also readable for everyone: who calls it, sees all paths that you block.

How the expert mode checks the robots.txt

Expert mode loads the site’s robots.txt and checks each line for errors that Lighthouse also reports: unknown statement, a pattern that doesn’t start with “/” or “*”, a “$” in the middle of the pattern, a sitemap without a valid address and rules without a user agent. To do this, he reports lines without colon and empty user-agent lines. The file names each line with number and reason.

Google Reads These Rules

Google supports the user-agent, allow, disallow and sitemap fields. Others like crawl-delay override it. A path starts with "/", and in the path, uppercase and lowercase do not count, in the name of the field. The character "*" stands for any number of characters, "$" for the end of the address. Name the sitemap with the full address including protocol and domain. What is behind 500 KiB, Google no longer reads.

Examples

Lock for an area

Not met
Disallow: /intern
User-agent: *
Disalow: /lager-alt
Sitemap: /sitemap.xml
Passes
User-agent: *
Disallow: /intern
Disallow: /lager-alt
Sitemap: https://www.beispiel.at/sitemap.xml

First the user-agent line, then the rules; the sitemap with a complete address.

  • Open your robots.txt at the address of your website with /robots.txt at the end and correct the lines that the file mentions.
  • Start each group with a user-agent line, including Allow and Disallow with paths starting from "/".
  • Check the file in the robots.txt report of Google Search Console.
  • Write the instructions exactly as in the description of Google: User-agent, Allow, Disallow, Sitemap. A typo is hardly noticeable when reading, an examination finds it immediately.

Deutsche Fassung dieser Seite

  • Seal Digitale Barrierefreiheit
  • Seal Daten & Recht
  • Seal Technische Qualität
  • Seal Digital Excellence
  • Seal Cyberversicherung

Audited. For a better digital future.euid.com

How do I recognize an invalid line in the robots.txt? · euid.com