List of Google's special-case crawlers

The special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product about the crawl process. For example, AdsBot ignores the global robots.txt user agent (*) with the ad publisher's permission. The general technical properties of Google's crawlers also apply to the special-case crawlers.

The special-case crawlers may ignore robots.txt rules and so they operate from a different IP range than the common crawlers. The IP ranges are published in the special-crawlers.json object. The special-case crawlers' reverse DNS mask matches rate-limited-proxy-***-***-***-***.google.com.

The following list shows the special-case crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the User-agent: line in robots.txt, and the products that are affected by crawl preferences for the crawler. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about.

APIs-Google

User-Agent in HTTP requests

APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)

robots.txt

User-agent token in robots.txt

APIs-Google

---

The global user agent (*) is ignored.

Example robots.txt group

user-agent: APIs-Google
allow: /archive/1Q84
disallow: /archive/

Affected products Crawling preferences addressed to the APIs-Google user agent affect the delivery of push notification messages by Google APIs.

AdsBot Mobile Web

User-Agent in HTTP requests

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)

robots.txt

User-agent token in robots.txt

AdsBot-Google-Mobile

---

The global user agent (*) is ignored.

Example robots.txt group

user-agent: AdsBot-Google-Mobile
allow: /archive/1Q84
disallow: /archive/

Affected products Crawling preferences addressed to the AdsBot-Google-Mobile user agent affect Google Ads' ability to check web page ad quality.

AdsBot

User-Agent in HTTP requests

AdsBot-Google (+http://www.google.com/adsbot.html)

robots.txt

User-agent token in robots.txt

AdsBot-Google

---

The global user agent (*) is ignored.

Example robots.txt group

user-agent: AdsBot-Google
allow: /archive/1Q84
disallow: /archive/

Affected products Crawling preferences addressed to the AdsBot-Google user agent affect Google Ads' ability to check web page ad quality.

AdSense

User-Agent in HTTP requests

Desktop agent

Mediapartners-Google

Mobile agent

(Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)

robots.txt

User-agent token in robots.txt Mediapartners-Google

---

The global user agent (*) is ignored.

Example robots.txt group

user-agent: Mediapartners-Google
allow: /archive/1Q84
disallow: /archive/

Affected products Crawling preferences addressed to the Mediapartners-Google user agent affect Google AdSense. The AdSense crawler visits participating sites in order to provide them with relevant ads.

Google-Safety

User-Agent in HTTP requests

Google-Safety

robots.txt The Google-Safety user agent ignores robots.txt rules.

Affected products The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. As such it's unaffected by crawling preferences.