Skip to content
EUID Audit service & seal, Home
Get your website audited

This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung

Indexing

Which traps in robots.txt harm indexing?

The robots.txt tells search engines which areas they should not retrieve. Two types of entries cause more damage than they prevent. First, blocking for stylesheets, scripts, and asset folders: Google renders pages like a browser, and without those files, it sees a disintegrated display, wrongly evaluates mobile-friendliness, and fails to find reloaded content. Second, directives that Google does not read: Noindex, Nofollow or crawl-delay in the robots.txt; Google has support for this on the 1st. September 2019 ended, and the lines pretend a security that does not exist. The indexing test area reads robots.txt and reports both.

The standard sets Allow and Disallow, more lines each crawler reads according to its own rule

RFC 9309, the Internet standard for robots.txt from September 2022, defines the lines User-agent, Allow and Disallow: for which crawler a rule applies, what it is allowed to retrieve and what not. A crawler can read more lines, he does not have to. Therefore, the same line does not work for one service and for the other. Google lists as supported fields User-agent, Allow, Disallow and Sitemap and writes that other fields such as crawl-delay are not supported; Invalid lines are passed. Crawl-delay is intended to slow down the calls of a crawler. Bing described in 2012 that his crawler pays attention to this line, and advises against high values, because then he retrieves fewer pages. A line of noindex does not appear in the standard or in Google’s list.

A blocked page can be in the search result, only without a description

A lock in the robots.txt prohibits retrieval, not display. Google writes: A blocked page can still enter the index if other websites link to it; in the search result is the address, but no description. The Search Console lists such cases as "Indexed, although blocked by robots.txt file". This creates a trap of two parts: Whoever locks one side and at the same time gives it a noindex, achieves the opposite of the intended. Google reads the noindex only when the page is retrieved, and the retrieval bans the block. For locked script files, according to Google: it does not execute them, and what they reload does not reach Google.

Without the locks, Google is allowed to load the same files as the browser of your visitors

Google is on 27. October 2014 in its technical guidelines: Those who prohibit the retrieval of script and style files immediately deteriorate how well Google displays and records the content, and this can lead to worse rankings. Today’s introduction to robots.txt puts it this way: Files without which the crawler understands the page harder should not be blocked. If the lines are removed, Google may load the same files as a browser on the next retrieval. The result can be viewed: The URL check of the Search Console shows after "Test live URL" under "Show tested page" a screen photo of the page as Google displays it.

Each disallow line is publicly readable

The robots.txt can be opened by anyone in the browser. RFC 9309 states in its safety instructions: Whoever lists paths in the robots.txt makes them public and thus findable, and the file does not replace effective protective measures. So a line like Disallow: /internal/ or Disallow: /backup/ is exactly the folder that nobody should find. Crawlers who adhere to the rules do not call them; Anyone else can type in the address. Google gives two ways for pages that are not to be searched: a noindex on the page or a password. What is confidential belongs behind a registration.

A blanket ban also applies to the search crawlers of the AI services

The crawlers of AI providers read the same file, and this adds two traps. The first: A Disallow: / under User-agent: *, about a remainder from the test operation, also applies to the search crawlers of the AI services. OpenAI writes that a website that excludes OAI SearchBot is not shown in ChatGPT’s search responses; Anthropic writes that its crawlers pay attention to the robots.txt. The second: a crawler follows only the group that most accurately matches its name; This is how Google describes it for its crawlers and Bing for Bingbot. Anyone who creates their own group for an AI crawler should therefore repeat all the rules that should apply to him, because the general group under the star is then eliminated for him. Google-Extended only affects Google’s AI models and, according to Google, does not change the inclusion in the search.

What is a trap

Rendering traps include locks whose path ends on style files or scripts, as well as locks of typical asset folders: wp-content and wp-includes on WordPress, assets, static, css, js, images, media, fileadmin and typo3conf on other systems. A lock in the internal area, search or shopping cart is not a trap and is not mentioned. Lines that Google does not read are Noindex, Nofollow and Crawl-delay. Crawl-delay notices Bing, where the line brakes the calls. Google has discontinued unofficial support for Noindex in robots.txt 2019; If you want to keep a page from the index, you need a meta tag or a header on the page itself. The result indicates the affected lines. It does not change the point value because the set of rules idx-v1 remains unchanged; it appears in the report and in the task list.

Why the barriers are mostly from ancient times

Before 2015, it was considered good style to lock everything except the content to save crawl budget. Since Google renders pages completely, the opposite is true: everything that belongs to the presentation must be retrievable. Many robots.txt files still contain the old entries because they were copied from templates and were never verified. The correction is the removal of the rows; who really needs to protect individual directories does so via access rights, not via the robots.txt. For noindex intentions, the statement belongs on the page: as a meta tag in the header or as the server’s X-Robots tag header.

  • Remove locks on folders and files that belong to the display: CSS, JS, images, asset folders.
  • Replacement You noindex lines in the robots.txt through a meta tag robots with noindex on the relevant pages.
  • Use the Google Search Console URL verification tool to check if the rendered view is complete.
  • Keep the robots.txt short: sitemap reference, real exclusions, nothing else.
  • Capital and lower case counts in the path: Disallow: /Internal/ does not block /internal/. Check each lock against the spelling in the address bar.
  • A lock without slash at the end reaches further than it looks: Disallow: /image also locks /image/ and /formation.html. With slash, i.e. Disallow: /picture/, she only hits the folder.
  • If your robots.txt has its own groups for individual crawlers, read each group as if the others didn’t exist. If a block that is under the star is missing in a group, it does not apply to this crawler.
  • With modular systems such as Wix, the robots.txt can often not be processed directly, according to Google. Then search the system’s settings for visibility to search engines.
  • Do not take robots.txt from a template or from another website. For each line, your agency should be able to say what it is there for.

More on this topic

Deutsche Fassung dieser Seite

  • Seal Digitale Barrierefreiheit
  • Seal Daten & Recht
  • Seal Technische Qualität
  • Seal Digital Excellence
  • Seal Cyberversicherung

Audited. For a better digital future.euid.com