This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung →
Indexing
Which traps in robots.txt harm indexing?
The robots.txt tells search engines which areas they should not retrieve. Two types of entries cause more damage than they prevent. First, blocking for stylesheets, scripts, and asset folders: Google renders pages like a browser, and without those files, it sees a disintegrated display, wrongly evaluates mobile-friendliness, and fails to find reloaded content. Second, directives that no crawler understands — Noindex, nofollow or crawl-delay in robots.txt — which have been ineffective for years and pretend a security that does not exist. The indexing test area reads robots.txt and reports both.
What is a trap
Rendering traps include locks whose path ends on style files or scripts, as well as locks of typical asset folders: wp-content and wp-includes on WordPress, assets, static, css, js, images, media, fileadmin and typo3conf on other systems. A lock in the internal area, search or shopping cart is not a trap and is not mentioned. Noindex, Nofollow and Crawl-delay are considered ineffective directives: Google has discontinued unofficial support for Noindex in robots.txt 2019; If you want to keep a page from the index, you need a meta tag or a header on the page itself. The result indicates the affected lines. It does not change the point value because the set of rules idx-v1 remains unchanged; it appears in the report and in the task list.
Why the barriers are mostly from ancient times
Before 2015, it was considered good style to lock everything except the content to save crawl budget. Since Google renders pages completely, the opposite is true: everything that belongs to the presentation must be retrievable. Many robots.txt files still carry the old entries because they were copied from templates and were never verified. The correction is the removal of the rows; who really needs to protect individual directories does so via access rights, not via the robots.txt. For noindex intentions, the statement belongs on the page: as a meta tag in the header or as the server’s X-Robots tag header.
- Remove locks on folders and files that belong to the display: CSS, JS, images, asset folders.
- Replacement You noindex lines in the robots.txt through a meta tag robots with noindex on the relevant pages.
- Use the Google Search Console URL verification tool to check if the rendered view is complete.
- Keep the robots.txt short: sitemap reference, real exclusions, nothing else.
