NGINX caching upstream server when it shouldn't be

Question

I want to set up an NGINX server which provides the following functionality:

1. When a request is made NGINX to get the page at /path/to/page, it fetches the page at /path/to/page.
2. If the upstream server is down or NGINX can't connect to it for some reason, NGINX returns a cached version of the page if it has one.
3. If the cached file is over 6 hours old, don't use it, just return a 502.
4. If the upstream server is available, never use the cache.

I have an NGINX config here which I think should work based on my understanding of the docs, but it doesn't and I can't see why. The problem is with point (4), this NGINX server returns the cached version of the file even if the upstream server is online.

daemon off;
error_log /dev/stdout info;
events {
}
http {
 proxy_cache_path
 "/home/jack/Code/NGINX Caching/Codebase/cache" # Cache path
 keys_zone=cache:10m # Name of cacahe, max size for keys 10 megabytes
 levels=1:2 # Don't store all cached files in a single directory
 max_size=500m # Max size of cache
 inactive=6h; # Cached file deleted if not used within six hours
 proxy_cache_valid 6h;
 proxy_cache_key "$request_method$request_uri";
 access_log /dev/stdout;
 server {
 listen 8080;
 location ~ ^/(.+)$ {
 proxy_pass http://0.0.0.0:8000/1ドル;
 proxy_cache cache;
 proxy_cache_valid 6h;
 proxy_buffering on;
 proxy_cache_use_stale error timeout;
 }
 }
}

Replace the proxy_cache_path with a path to a directory on your machine, and run another webserver on your machine on port 8000. When I modify a file served by the server on port 8000, NGINX doesn't see the change until I erase the cache. The issue is with NGINX and not my client (Firefox), even if I turn off caching in the browser, NGINX returns a 200 with the old file contents.

Answer 1

Can you please check if these two directives might help you:

proxy_cache_revalidate: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_revalidate

and

proxy_cache_use_stale: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_use_stale

There is a video from nginx.conf '17 online describing all the cool things you could achive with caching: https://www.youtube.com/watch?v=xZrOjmAkFC8. maybe this is also of interest for you.

• Jack M: No, not really.
• J J: Hey Jack, I'm testing your scenario in my lab environemnt and let you know the result.
• Jack M: No need, I figured it out. I had misunderstood how the proxy_cache_use_stale and proxy_cache_valid directives worked.

Answer 2

So, it seems I misunderstood the NGINX proxy cache directives. The docs are quite confusing on this subject, so I'll lay it out point by point.

This official help page gives a decent overview of the various directives, however it makes no mention of something which as it turns out is a very important conceptual building block in understanding how NGINX caching works: the notion of a cached file being stale.

NGINX's default behavior is to always use the cache if it's there, rather than querying the upstream server. With this config, a minimal config to do caching, NGINX will query the upstream server the first time a page is accessed, and then use the cached version forever after that:

events {
}
http {
 proxy_cache_path
 /path/to/cache
 keys_zone=my_cache:10m;
 proxy_cache_key "$request_method$request_uri";
 server {
 listen 8080;
 location ~ ^/(.+)$ {
 proxy_pass http://0.0.0.0:8000/1ドル;
 proxy_cache cache;
 }
 }
}

You can use the proxy_cache_valid directive to tell NGINX when a cached file should be considered "stale". For example, if we set proxy_cache_valid 5m, then 5 minutes after a cache file is created NGINX will stop serving it and querying the upstream server again on the next request. If the upstream is down, NGINX will return a 502. However, during those five minutes, NGINX will still use the cache even if the upstream server is available, so this is still not what we want.

NGINX has another directive, proxy_cache_use_stale, gives NGINX conditions under which it may use cached files even if they're stale. We can combine these together to get a server which caches pages, makes them stale immediately (or almost immediately), and then only uses them if the upstream is down:

events {
}
http {
 proxy_cache_path
 /path/to/cache
 keys_zone=my_cache:10m;
 proxy_cache_key "$request_method$request_uri";
 server {
 listen 8080;
 location ~ ^/(.+)$ {
 proxy_pass http://0.0.0.0:8000/1ドル;
 proxy_cache cache;
 proxy_cache_valid 1s;
 proxy_cache_use_stale error timeout;
 }
 }
}

This config has almost the behavior we want, except that if the upstream server goes down for an extended period of time, NGINX will continue to use the cache indefinitely. As far as I know there is no way to tell NGINX to totally invalidate/clear a cached file after a given amount of time. Normally that's what proxy_cache_valid is for, but we're already using that for a different purpose, to make files stale after 1 second so they're only used when the upstream is down. We would need some next level after "stale" that means the file is completely invalidated, but I don't think that exists in NGINX.

So the simplest solution is to just clear the cache manually. It's sufficient to just delete all files in the cache directory (or its subdirectories) which were last modified more than 6 hours ago, or whatever you want the expiry time to be. On a Linux system, you can run this script every 5 minutes, for example:

find /path/to/cache -type f -mmin +360 -delete

• J J: FYI: there is a proxy_cache_purge directive. But I think this is only available in nginx plus. You can see an example of that in the video I mentioned in my answer. I guess thats not a much better solution than the manual / cron job find and delete way you mentioned.