This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung
Indexing
What belongs in a robots.txt and what does not?
A robots.txt includes a few lines: for whom the rules apply, which areas crawlers should not retrieve and where the sitemap is located. A small company’s website comes with four lines. Not included are locks on style and script files, instructions such as Noindex, which Google does not read there, and paths that should remain secret, because the file can open anyone. If the file is missing entirely, Google assumes that there are no restrictions.
The lines Google reads
Google reads these fields: User-agent says for which crawler the following rules apply; The star stands for everyone. Disallow calls a path that the crawler should not retrieve. Allow takes part of it out again. Sitemap indicates the full address of the sitemap and stands anywhere, regardless of the groups. Other fields Google does not read, invalid lines it skips. What does not hit a disallow line may be retrieved. Comments start with a rhombus. The file must be called robots.txt, stored as a text file in UTF-8 and in the main directory.
How a Crawler Applies the Rules
The file only applies to the host, protocol and port under which it is located: The robots.txt of www.ihre-domain.at does not regulate anything for shop.ihre-domain.at. If several rules apply to an address, the one with the longest path applies; if two are the same length, the permission applies according to the standard RFC 9309. Paths differ in upper and lower case and apply from the beginning of the address: Disallow: /search also meets /search-and-finde.html. If a crawler has its own group, it no longer reads the general group under the star on Google. Google reads at most 500 KiB of the file and usually keeps it in cache storage for up to 24 hours.
What is meaningfully blocked
Google cites as the purpose of the file mainly to save the server from too many retrievals. Therefore, blocks are useful for areas that generate many addresses without their own content for the search: the results pages of the internal search, the shopping cart, the checkout. For a website with a few dozen pages, often no block is necessary. Then the file consists of the group for all crawlers and the sitemap line.
What does not belong in it
Lock folders with style and script files: Google needs these files to display the page. Lines like Noindex or Crawl-delay: Google does not read them, and a noindex belongs on the page itself. Paths to Confidential Areas: RFC 9309 states that listed paths are public and thus findable and that the rules are not access control. And block pages that should disappear from search: Google can still record a blocked address if other websites link to it.
Examples
robots.txt of a company website
User-agent: *
Disallow: /wp-content/
Disallow: /intern-preisliste/
Noindex: /danke/
Crawl-delay: 10User-agent: *
Disallow: /suche/
Disallow: /warenkorb/
Sitemap: https://www.ihre-domain.at/sitemap.xmlThe first version locks the style and script file folder, publicly names a confidential path, and contains two lines that Google does not read. The second blocks two areas without their own content for search and calls the sitemap.
- Go to her-domain.at/robots.txt and go through the file line by line: For each block, someone should be able to say what it is for.
- After each change, have it checked whether homepage, style files and scripts can be retrieved further.




