Client should retry when there's just one 404 and a bunch of errors

Code Issues Proposed changes

During a rebalance, it's expected that we may get a 404 for data that
does exist elsewhere in the cluster. Normally this isn't a problem; the
proxy sees the 404, keeps digging, and one of the other primaries will
serve the response.
Previously, if the other replicas were heavily loaded, the proxy would
see a bunch of timeouts and the fresh (empty) primary, treat the 404 as
good, and send that on to the client.
Now, have the proxy throw out that first 404 (provided it doesn't have a
timestamp); it will then return a 503 to the client, indicating that it
should try again.
Add a new (per-policy) proxy-server config option,
rebalance_missing_suppression_count; operators may use this to increase
the number of 404-no-timestamp responses to discard if their rebalances
are going faster than replication can keep up, or set it to zero to
return to the previous behavior.
Change-Id: If4bd39788642c00d66579b26144af8f116735b4d

This commit is contained in:

Clay Gerrard

2020年08月05日 11:19:48 -05:00

committed by Tim Burke

parent 06091172c2

commit 754defc39c

9 changed files with 149 additions and 29 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

7
etc/proxy-server.conf-sample

View File

@@ -200,6 +200,12 @@ use = egg:swift#proxy

# the number of seconds configured by timing_expiry.

# timing_expiry = 300

#

# Normally, you should only be moving one replica's worth of data at a time

# when rebalancing. If you're rebalancing more aggressively, increase this

# to avoid erroneously returning a 404 when the primary assignments that

# *didn't* change get overloaded.

# rebalance_missing_suppression_count = 1

#

# By default on a GET/HEAD swift will connect to a minimum number storage nodes

# in a minimum number of threads - for replicated data just a single request to

# a single node one at a time. When enabled concurrent_gets allows the proxy

@@ -307,6 +313,7 @@ use = egg:swift#proxy

# write_affinity =

# write_affinity_node_count =

# write_affinity_handoff_delete_count =

# rebalance_missing_suppression_count = 1

# concurrent_gets = off

# concurrency_timeout = 0.5

# concurrent_ec_extra_requests = 0

Client should retry when there's just one 404 and a bunch of errors

7 etc/proxy-server.conf-sample Unescape Escape View File

7
etc/proxy-server.conf-sample

View File