Magento 2.3.3 Varnish problems. Magento's VCL file is not working

Question

I have managed to set up Varnish for magento 2 using nginx to handle SSL. However if I try to use the vcl file provided by magento varnish stops working with the usual

"Error 503 Backend fetch failed Backend fetch failed"

error. There is nothing in the logs nothing shows up anywhere. There is something dodgy in the configuration file. Anyone else had the same issue?

Update log file:

- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 6 fetch
- Timestamp Fetch: 1575437126.769898 0.000150 0.000150
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: 2019年12月04日 05:25:26 GMT
- RespHeader Server: Varnish
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 5
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/5.2)
- VCL_call DELIVER
- RespUnset Age: 0
- RespHeader Pragma: no-cache
- RespHeader Expires: -1
- RespHeader Cache-Control: no-store, no-cache, must-revalidate, max-age=0
- RespUnset Server: Varnish
- RespUnset X-Varnish: 5
- RespUnset Via: 1.1 varnish (Varnish/5.2)
- VCL_return deliver
- Timestamp Process: 1575437126.769912 0.000163 0.000014
- RespHeader Content-Length: 278
- RespHeader Connection: close
- Timestamp Resp: 1575437126.769948 0.000200 0.000036
- ReqAcct 1158 0 1158 264 278 542
- End

Other part of the log file:

- BereqHeader grace: none
- BereqHeader Accept-Encoding: gzip
- BereqProtocol HTTP/1.1
- BereqHeader X-Varnish: 9
- VCL_call BACKEND_FETCH
- VCL_return fetch
- FetchError no backend connection
- Timestamp Beresp: 1575437127.586058 0.000018 0.000018
- Timestamp Error: 1575437127.586061 0.000021 0.000003
- BerespProtocol HTTP/1.1
- BerespStatus 503
- BerespReason Service Unavailable
- BerespReason Backend fetch failed
- BerespHeader Date: 2019年12月04日 05:25:27 GMT
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
- BerespHeader Content-Type: text/html; charset=utf-8
- BerespHeader Retry-After: 5
- VCL_return deliver
- Storage malloc Transient
- ObjProtocol HTTP/1.1
- ObjStatus 503
- ObjReason Backend fetch failed
- ObjHeader Date: 2019年12月04日 05:25:27 GMT
- ObjHeader Server: Varnish
- ObjHeader Content-Type: text/html; charset=utf-8
- ObjHeader Retry-After: 5
- Length 278
- BereqAcct 0 0 0 0 0 0
- End

I managed to narrow it down to a 404 error in the apache access.log. This seems to be the issue:

127.0.0.1 - - [04/Dec/2019:06:28:59 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:29:09 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:29:19 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:29:29 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:29:39 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:31:15 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:31:25 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"

varnishlog -g raw -i Backend_health shows

 0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000516 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000516 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000798 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000393 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000796 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 3 5 10 0.000514 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 2 5 10 0.000511 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 1 5 10 0.000609 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000646 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000650 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000580 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000661 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000842 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000931 0.000000 HTTP/1.1 404 Not Found
 0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000669 0.000000 HTTP/1.1 404 Not Found

the requested file is there and it is accessible. Unsure what causes the problem.

• Danila Vershinin: The error typically means that the VCL points to invalid backend (check port or host name), or there's actually a problem with the backend. To know for sure, try varnishlog while accessing the site, and then post the output here.
• Zoltan veress: Well if I remove the magento provided vcl file it works like charm so I do not expect the setup to be faulty there is something dodgy in the file which I am unable to find. Also I pasted the log output to the original post above.

Answer 1

Based on the logs provided, a backend health check fails in Varnish for /pub/health_check.php.

If you have configured NGINX with root /path/to/magento/pub (which you should, as it's more secure), then you need to edit the health check VCL code in order to remove pub, e.g. here:

Change your generated VCL code (earlier link is just to point location in VCL code):

.url = "/pub/health_check.php";

to

.url = "/health_check.php";

Then restart Varnish.

Why: since NGINX's web root is pointed to pub directory, all the web-accessible files in that directory are accessible from webroot, e.g. http://example.com/foo and not http://example.com/pub/bar.

This is rather a bug with Magento 2 default VCL generation. Ideally, it should detect pub vs no-pub setup and generate the proper VCL.

If anything, they should have defaulted to pub as webroot setup as it is most correct. (makes most of the Magento source files non-accessible from the web browser, thus secure).