How HTTP status codes affect Google's crawlers
This page describes how different
HTTP status codes,
impact Google's ability to crawl your web content. We cover the top 20 status codes that Google
encounters on the web. More exotic status
codes, such as
418 (I'm a teapot),
aren't covered.
HTTP status codes
HTTP status codes are generated by the server that's hosting the site when it responds to a request made by a client, for example a browser or a crawler. Every HTTP status code has a different meaning, but often the outcome of the request is the same. For example, there are multiple status codes that signal redirection, but their outcome is the same.
Search Console generates error messages for status codes in the 4xx—5xx range,
and for failed redirections (3xx). If the server responded with a
2xx status code, the content received in the response may be considered for
indexing.
The following table contains the most encountered HTTP status codes by Google and an explanation how Google handles each status code.
| HTTP status codes | |
|---|---|
|
Google considers the content for processing (for example, in the case of Google Search, for
indexing). If the content suggests an error for Google Search,
an empty page or an error message, Search Console will show a
200 (success)
Google passes on whatever it received to the next processing step (which is product specific). For Google Search, the next system is the indexing pipeline. The indexing systems may index the content, but that's not guaranteed. 201 (created)
202 (accepted)
Google waits for the content for a limited time, then passes on whatever it received to the next processing step (which is product specific). The timeout is user agent dependent, for example Googlebot Smartphone may have a different timeout than Googlebot Image. 204 (no content)
Google wasn't able to receive any content and therefore can't process it. |
|
By default, Google's crawlers follow up to 10 redirect hops. However, specific products' crawlers may have different limits. For example, Googlebot generally follows 10 redirect hops when crawling for general web content, but Google Inspection Tools doesn't follow redirects.
Any content Google receives from the redirecting URL is ignored, and the final target
URL's content is processed instead. For robots.txt files, learn how
Google handles a robots.txt that returns a 301 (moved permanently)
Google follows the redirect, and Google systems use the redirect as a strong signal that the redirect target should be processed. 302 (found)
By default, Google's crawlers follow the redirect, and Google systems use the redirect as a weak signal that the redirect target should be processed. Other products may handle the redirect differently. 303 (see other)
304 (not modified)
Google crawlers signal the next processing system that the content is the same as last time it was crawled. In the case of Google Search, the indexing pipeline may recalculate signals for the URL, but otherwise the status code has no effect on indexing. 307 (temporary redirect)
Equivalent to 302.
308 (moved permanently)
Equivalent to 301.
|
|
Google doesn't use the content from URLs that return
Any content Google receives from URLs that return a 400 (bad request)
All
In the case of Google Search, the indexing pipeline removes the URL from the index if
it was previously indexed. Newly encountered 401 (unauthorized)
403 (forbidden)
404 (not found)
410 (gone)
411 (length required)
429 (too many requests)
Google's crawlers treat the |
|
Any content Google receives from URLs that return a
Once the server starts responding with a 500 (internal server error)
Google decreases the crawl rate for the site. The decrease in crawl rate is proportionate to the number of individual URLs that are returning a server error. For Google Search, Google's indexing pipeline removes from the index URLs that persistently return a server error. 502 (bad gateway)
503 (service unavailable)
|