openstack/swift - swift - OpenDev: Free Software Needs Free Tools

md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() is added to
swift/common/utils.py. This encapsulation is identical to the one being
added to oslo.utils, but is recreated here to avoid adding a dependency.
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
While this patch seems large, it is really just the same change over and
again. Reviewers need to pay particular attention as to whether the
keyword parameter (usedforsecurity) is set correctly. Right now, all
of them appear to be not used in a security context.
Now that all the instances have been converted, we can update the bandit
run to look for these instances and ensure that new invocations do not
creep in.
With this latest patch, the functional and unit tests all pass
on a FIPS enabled system.
Co-Authored-By: Pete Zaitcev
Change-Id: Ibb4917da4c083e1e094156d748708b87387f2d87

Change-Id: I0a77025237549b192b6b695b6f18512dff28af6c

...and bump up their timeout, since that seems more likely to happen if
we have to retry.
Change-Id: Ie05521f6cd146234dc5615c96ad19681b43e9110

...and, since the previous tag didn't have the Bandit pin, make the
rolling upgrade job non-voting. We should plan on backporting this so we
can check that upgrades from stable branches are still OK.
See also: https://github.com/PyCQA/bandit/issues/654
Change-Id: If7f3ad8b275271d748426133232ed06c2a1cd1de

Change-Id: I495fb1ec2394130c7274368662b58212ca375854

* Drop osc from test-requirements (and lower-constraints)
 I'm not clear on where/how we use it; I think it was a hold-over from
 swift3?
* Pin python-keystoneclient in our py2-constraints
 Something sees to have changed with the pip resolver that means it
 keeps trying to install a newer, py3-only version for our py2 jobs.
Change-Id: Ie37ac077517e1ece5fa6bf163d1ab5e316ced509

Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I6e62f7cc0474087edfd7f0ca133c75dce71cc795
Closes-Bug: #1889951 

Previously a traceback was logged when the ssync receiver experienced
a problem reading the wsgi input. Now this is logged as an error but
without a traceback.
Change-Id: Ifadbe603d5fa5473397a32e14ae1b8023cb53eb2
Closes-Bug: #1889951 

This is to fix the invalid argument formatting in exception messages.
Change-Id: Ibce0db9fc75a07bbd6eb43ab1a412e6eb73fd27f

This used to be necessary when we were explicitly calling next() on the
app iter, but we don't need it any more.
Change-Id: I0ae13905443aa8c8a1941c3212d01c9a5ae674e6
Related-Change: I27feabe923a6520e983637a9c68a19ec7174a0df

With DLO and SLO, we validate that we can read the first segment before
sending data to the client; this helps catch auth errors where the user
has access to read the manifest but not the segments.
Sometimes, though, that validation fails for transient reasons; if the
proxy couldn't get enough responses from primaries to determine whether
the object exists (for example), we should send back a 503 to indicate
to the client that it should retry the request.
Change-Id: Ice5358ff85ee2d5fe60785b73b67dea493044a2c

We only need the invalidation post-rsync, since rsync was changing data
on disk behind Swift's back. Move the REPLICATE call down into the
rsync() helper function and drop it from the reconstructor entirely.
Change-Id: I576901344f1f3abb33b52b36fde0b25b43e54c8a
Closes-Bug: #1818709 

This only applies to post-sync REPLICATE calls, none of which actually
look at the response anyway.
Change-Id: I1de62140e7eb9a23152bb9fdb1fa0934e827bfda

Change-Id: Icdbebb55e85763a7bd2a269753ced61a99e557be

No version of eventlet that I'm aware of hasany sort of support for
eventlet.wsgi.WRITE_TIMEOUT; I don't know why we've been setting that.
On the other hand, the socket_timeout argument for eventlet.wsgi.Server
has been supported for a while -- since 0.14 in 2013.
Drive-by: Fix up handling of sub-second client_timeouts.
Change-Id: I1dca3c3a51a83c9d5212ee5a0ad2ba1343c68cf9
Related-Change: I1d4d028ac5e864084a9b7537b140229cb235c7a3
Related-Change: I433c97df99193ec31c863038b9b6fd20bb3705b8

Tox trying to install latest versions for building docs which may
not be supported by stable and lower branches, so should be
restricted by respective version's upper-constraints.txt
Change-Id: I599d4ea430cc1e65bb50d0481617b005febf6de2

Add a new config option to SLO, allow_async_delete, to allow operators
to opt-in to this new behavior. If their expirer queues get out of hand,
they can always turn it back off.
If the option is disabled, handle the delete inline; this matches the
behavior of old Swift.
Only allow an async delete if all segments are in the same container and
none are nested SLOs, that way we only have two auth checks to make.
Have s3api try to use this new mode if the data seems to have been
uploaded via S3 (since it should be safe to assume that the above
criteria are met).
Drive-by: Allow the expirer queue and swift-container-deleter to use
high-precision timestamps.
Change-Id: I0bbe1ccd06776ef3e23438b40d8fb9a7c2de8921

Change-Id: I6ca407b86658191b2db1e806d284cdd6348c86e1

This gets us retries "for free" and should reduce gate flakiness.
Change-Id: Ia2e4c94f246230a3e25e4557b4b2c1a3a67df756

This should help make tests that rely on up-to-date listings more reliable.
Change-Id: Ib888c84fa629cc78f48a8251eda98c0fa51242c0

This should make us less likely to fail in the gate because of listing
consistency issues.
Drive-by: Remove the allowed_headers override; that hasn't been necessary
since we updated the default value a while back.
Change-Id: Id280bc93ed6e899a62b0115fdf144a564ef0cd8d
Related-Change: Ib82e175096716e42aecdab48f01f079e09da6a1d

Previously these were all hardcoded; let operators tweak them as needed.
Significantly, this also allows operators to disable error-limiting
entirely, which may be a useful protection in case proxies are
configured with a single memcached server.
Use error_suppression_limit and error_suppression_interval to mirror the
option names used by the proxy-server to ratelimit backend Swift
servers.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Ife005cb8545dd966d7b0e34e5496a0354c003881

...even after the operator has configured interval. This was raised
during the Oct 2020 ops feedback session.
Change-Id: Ie44de8f15df63813367b76e1b96be456fac9b2b0

This patch moves the tables describing configuration options for each
server type from the deployment_guide.rst doc to separate per-server
documents. The new per-server documents are grouped under a config
directory with a config index doc. The config index doc is listed in
the top level index and provides a single starting point to navigate
to the individual server docs.
Change-Id: I6cedd98586febb5dc949c088ee44e160385ed324

Add the librsvg2-* package to bindep.txt. This package has been
required to build docs since the Related-Change. The package is added
to a new 'doc' profile in bindep.txt; if missing, the package will be
reported by:
 bindep doc
The 'doc' profile is added to the tox 'bindep' env command so that, if
missing, the package will always be reported by:
 tox -e bindep
Change-Id: I6c60c31ca8002133ab77f05f59359b25315b299c
Related-Change: I26cefda80d3234df68d7152b404e0a71da74ab90
Closes-Bug: 1903038