Skip to content
EUID Audit service & seal, Home
Get your website audited

This translation has not been editorially reviewed yet. The German version is authoritative. Deutsche Fassung

Indexing

How do I recognize the real Googlebot?

At the IP address, never at the name. The identifier "Googlebot" in the access protocol is an indication of the sender, and any program can send it; Google itself writes that it is often faked by other crawlers. Real is access if its IP address belongs to Google. On a case-by-case basis, Google mentions a double query in the Internet naming system (DNS), lists of its IP areas published for firewall and bot protection.

The identifier proves nothing

Every retrieval of a website indicates in the header user-agent who is requesting. Nobody checks this information. Google calls spammers and other troublemakers who access websites and pretend to be Google. Also, checking tools send the identifier to see what a website search engine is delivering. For your firewall, this has consequences in both directions. Whoever trusts the name, lets imitators through. And whoever brakes accesses with this name in general, because there are conspicuous many, may meet the real crawler.

The individual case: back and forth queries in the DNS

For a single IP address from the access protocol, Google describes this path. The reverse query asks which name the IP address belongs to. The name must end on googlebot.com, google.com or googleusercontent.com. The forward query asks which IP address belongs to this name. It must be the same as in the Minutes. The cross-check is necessary because the backward entry is determined by the owner of an IP address. Your hoster handles both queries with the host command; The example below shows Google’s own sample address.

The permanent solution: Google's IP lists

For firewall and bot protection, Google publishes the IP areas of its crawlers as files, separately by type of access. The common-crawlers.json file indicates the addresses of general crawlers such as Googlebot; they always adhere to the robots.txt when automatically retrieved. The special-crawlers.json file applies to special cases such as the AdsBot. Other files apply to requests made by a user; Google cites as an example the confirmation of the ownership of a website. Such calls do not depend on the robots.txt. The addresses are in the files as areas in CIDR spelling. A protection that reads these lists lets through real traffic and does not have to trust the name.

What happens if protection rejects the real Googlebot

If protection responds to the real Googlebot with an error of the 4xx series, about 403, Google treats this like a page that does not exist: all 4xx responses except 429 are treated the same, and the address is removed from the index. At 429 and errors of the 5xx series, Google slows down the retrieval temporarily. Visitors do not notice it, the website loads as always. It becomes visible in the Search Console: in the page indexing report and in the URL checking tool, whose live test shows what Google is getting right now.

Same sample for AI crawlers

OpenAI also publishes the IP addresses of its crawlers, a list for GPTBot, OAI SearchBot and ChatGPT users. Anyone who wants to appear in the search of ChatGPT should, according to OpenAI, allow the OAI SearchBot in the robots.txt and let access from the published IP areas through. A bot protection can thus also decide here by origin. Which of these crawlers you allow is your own question; She answers the article "Can I block AI crawlers, and what do I give up?".

Examples

Two accesses with the identifier Googlebot

Not met
host 203.0.113.7
Host 7.113.0.203.in-addr.arpa. not found: 3(NXDOMAIN)
Passes
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For the first address there is no name on Google: the identifier is faked. The second is back and forth.

  • Let the hoster output the traffic of the last week with the identifier Googlebot, with IP address and status code.
  • Have some of these IP addresses checked with the duplicate DNS query.
  • Ask the provider of your firewall or bot protection if it recognizes search engine crawlers on Google’s IP lists and what the setting is called.
  • In Search Console, use the URL verification tool and "Test Live URL" to see if Google can retrieve your homepage and a subpage.

More on this topic

Deutsche Fassung dieser Seite

  • Seal Digitale Barrierefreiheit
  • Seal Daten & Recht
  • Seal Technische Qualität
  • Seal Digital Excellence
  • Seal Cyberversicherung

Audited. For a better digital future.euid.com