openstack/swift - swift - OpenDev: Free Software Needs Free Tools

This patch specifies a set of configuration options required to build
a TLS context, which is used to wrap the client connection socket.
Closes-Bug: #1906846
Change-Id: I03a92168b90508956f367fbb60b7712f95b97f60

With DLO and SLO, we validate that we can read the first segment before
sending data to the client; this helps catch auth errors where the user
has access to read the manifest but not the segments.
Sometimes, though, that validation fails for transient reasons; if the
proxy couldn't get enough responses from primaries to determine whether
the object exists (for example), we should send back a 503 to indicate
to the client that it should retry the request.
Change-Id: Ice5358ff85ee2d5fe60785b73b67dea493044a2c

Change-Id: Icdbebb55e85763a7bd2a269753ced61a99e557be

No version of eventlet that I'm aware of hasany sort of support for
eventlet.wsgi.WRITE_TIMEOUT; I don't know why we've been setting that.
On the other hand, the socket_timeout argument for eventlet.wsgi.Server
has been supported for a while -- since 0.14 in 2013.
Drive-by: Fix up handling of sub-second client_timeouts.
Change-Id: I1dca3c3a51a83c9d5212ee5a0ad2ba1343c68cf9
Related-Change: I1d4d028ac5e864084a9b7537b140229cb235c7a3
Related-Change: I433c97df99193ec31c863038b9b6fd20bb3705b8

Tox trying to install latest versions for building docs which may
not be supported by stable and lower branches, so should be
restricted by respective version's upper-constraints.txt
Change-Id: I599d4ea430cc1e65bb50d0481617b005febf6de2

Add a new config option to SLO, allow_async_delete, to allow operators
to opt-in to this new behavior. If their expirer queues get out of hand,
they can always turn it back off.
If the option is disabled, handle the delete inline; this matches the
behavior of old Swift.
Only allow an async delete if all segments are in the same container and
none are nested SLOs, that way we only have two auth checks to make.
Have s3api try to use this new mode if the data seems to have been
uploaded via S3 (since it should be safe to assume that the above
criteria are met).
Drive-by: Allow the expirer queue and swift-container-deleter to use
high-precision timestamps.
Change-Id: I0bbe1ccd06776ef3e23438b40d8fb9a7c2de8921

Change-Id: I6ca407b86658191b2db1e806d284cdd6348c86e1

This gets us retries "for free" and should reduce gate flakiness.
Change-Id: Ia2e4c94f246230a3e25e4557b4b2c1a3a67df756

This should help make tests that rely on up-to-date listings more reliable.
Change-Id: Ib888c84fa629cc78f48a8251eda98c0fa51242c0

This should make us less likely to fail in the gate because of listing
consistency issues.
Drive-by: Remove the allowed_headers override; that hasn't been necessary
since we updated the default value a while back.
Change-Id: Id280bc93ed6e899a62b0115fdf144a564ef0cd8d
Related-Change: Ib82e175096716e42aecdab48f01f079e09da6a1d

Previously these were all hardcoded; let operators tweak them as needed.
Significantly, this also allows operators to disable error-limiting
entirely, which may be a useful protection in case proxies are
configured with a single memcached server.
Use error_suppression_limit and error_suppression_interval to mirror the
option names used by the proxy-server to ratelimit backend Swift
servers.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Ife005cb8545dd966d7b0e34e5496a0354c003881

...even after the operator has configured interval. This was raised
during the Oct 2020 ops feedback session.
Change-Id: Ie44de8f15df63813367b76e1b96be456fac9b2b0

This patch moves the tables describing configuration options for each
server type from the deployment_guide.rst doc to separate per-server
documents. The new per-server documents are grouped under a config
directory with a config index doc. The config index doc is listed in
the top level index and provides a single starting point to navigate
to the individual server docs.
Change-Id: I6cedd98586febb5dc949c088ee44e160385ed324

Add the librsvg2-* package to bindep.txt. This package has been
required to build docs since the Related-Change. The package is added
to a new 'doc' profile in bindep.txt; if missing, the package will be
reported by:
 bindep doc
The 'doc' profile is added to the tox 'bindep' env command so that, if
missing, the package will always be reported by:
 tox -e bindep
Change-Id: I6c60c31ca8002133ab77f05f59359b25315b299c
Related-Change: I26cefda80d3234df68d7152b404e0a71da74ab90
Closes-Bug: 1903038

We sometimes see probe test failures in the gate where `resetswift`
fails like
 umount: /mnt/sdb1: target is busy.
It seems to be because `swift-init all kill` is just a stop with a
status check after the fact. Add an orphan sweep afterwards to make
sure worker subprocesses got cleaned up, too.
Change-Id: I533b436a35e5c3deb7488e9961b55f2274c14751

UPPER_CONSTRAINTS_FILE is old name and deprecated
-https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
This allows to use lower-constraints file as more
readable way instead of UPPER_CONSTRAINTS_FILE=<lower-constraints file>.
[1] https://review.opendev.org/#/c/722814/
Change-Id: I7e3cdef5b1bcb28019055d29dbd95073e92882fc

Change-Id: I4ecc98bbe6d9817fa40fbb54f743859550e84994

Since fb08d477eb was merged, storlets
invocation on proxy fails because the close method is called even if
iterator has list_iterator type which doesn't have the close method.
This change replaces the direct call of the close method by
close_if_possible, to avoid Exception if the response iterator is
list or tuple, which don't support the close method.
Change-Id: I851cbb17465b5693dda5206aeae0a3191c9fe4f4
Closes-Bug: #1901453 

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: Ic8a087c600ef0624ee815a330119ccc12c287f0f

Change-Id: Ica2b637c9ee015c2cf129715e33d1ed1968b0a94

Python modules related to coding style checks (listed in blacklist.txt in
openstack/requirements repo) are dropped from lower-constraints.txt
they are not needed during installation.
Change-Id: Ic777b435723f663b408c778f661811439b486071

This allows operators to throttle expirers as needed.
Partial-Bug: #1784753
Change-Id: If75dabb431bddd4ad6100e41395bb6c31a4ce569

They finally got rid of Thread.isAlive, having added Thread.is_alive as
an alias back in 2.6.
array.tostring is also gone; use array.tofile instead, but only on py3.
Change-Id: I0b7e426c60118a5df879a8908729d7feb12dfc1b

In a situation where two nodes does not have the same version of a ring
and they both think the other node is the primary node of a partition,
a race condition can lead to the loss of some of the objects of the
partition.
The following sequence leads to the loss of some of the objects:
 1. A gets and reloads the new ring
 2. A starts to replicate/revert the partition P to node B
 3. B (with the old ring) starts to replicate/revert the (partial)
 partition P to node A
 => replication should be fast as all objects are already on node A
 4. B finished replication of (partial) partition P to node A
 5. B remove the (partial) partition P after replication succeeded
 6. A finishes replication of partition P to node B
 7. A removes the partition P
 8. B gets and reloads the new ring
All data transfered between steps 2 and 5 will be lost as they are not
anymore on node B and they are also removed from node A.
This commit make the replicator/reconstructor to hold a replication_lock
on partition P so that remote node cannot start an opposite replication.
Change-Id: I29acc1302a75ed52c935f42485f775cd41648e4d
Closes-Bug: #1897177 

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: Iaff6cdd9e8de2589a56113867550e35d52cc068f

Capture the on the wire status code for logging because we change the
logged status code sometimes.
Closes-Bug: #1896518
Change-Id: I27feabe923a6520e983637a9c68a19ec7174a0df