Allow concurrent bulk deletes

Code Issues Proposed changes

Before, server-side deletes of static large objects could take a long
time to complete since the proxy would wait for a response to each
segment DELETE before starting the next DELETE request.
Now, operators can configure a concurrency factor for the slo and bulk
middlewares to allow up to N concurrent DELETE requests. By default, two
DELETE requests will be allowed at a time.
Note that objects and containers are now deleted in separate passes, to
reduce the likelihood of 409 Conflict responses when deleting
containers.
Upgrade Consideration
=====================
If operators have enabled the bulk or slo middlewares and would like to
preserve the prior (single-threaded) DELETE behavior, they must add the
following line to their [filter:slo] and [filter:bulk] proxy config
sections:
 delete_concurrency = 1
This may be done prior to upgrading Swift.
UpgradeImpact
Closes-Bug: 1524454
Change-Id: I128374d74a4cef7a479b221fd15eec785cc4694a

This commit is contained in:

Tim Burke

2016年04月29日 12:12:00 -05:00

parent 9482ef89a2

commit e09c4ee780

6 changed files with 247 additions and 69 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

16
etc/proxy-server.conf-sample

View File

@@ -618,19 +618,23 @@ use = egg:swift#bulk

# max_failed_extractions = 1000

# max_deletes_per_request = 10000

# max_failed_deletes = 1000

#

# In order to keep a connection active during a potentially long bulk request,

# Swift may return whitespace prepended to the actual response body. This

# whitespace will be yielded no more than every yield_frequency seconds.

# yield_frequency = 10

#

# Note: The following parameter is used during a bulk delete of objects and

# their container. This would frequently fail because it is very likely

# that all replicated objects have not been deleted by the time the middleware got a

# successful response. It can be configured the number of retries. And the

# number of seconds to wait between each retry will be 1.5**retry

# delete_container_retry_count = 0

#

# To speed up the bulk delete process, multiple deletes may be executed in

# parallel. Avoid setting this too high, as it gives clients a force multiplier

# which may be used in DoS attacks. The suggested range is between 2 and 10.

# delete_concurrency = 2

# Note: Put after auth and staticweb in the pipeline.

[filter:slo]

@@ -651,6 +655,12 @@ use = egg:swift#slo

#

# Time limit on GET requests (seconds)

# max_get_time = 86400

#

# When deleting with ?multipart-manifest=delete, multiple deletes may be

# executed in parallel. Avoid setting this too high, as it gives clients a

# force multiplier which may be used in DoS attacks. The suggested range is

# between 2 and 10.

# delete_concurrency = 2

# Note: Put after auth and staticweb in the pipeline.

# If you don't put it in the pipeline, it will be inserted for you.

Allow concurrent bulk deletes

16 etc/proxy-server.conf-sample Unescape Escape View File

16
etc/proxy-server.conf-sample

View File