openstack/swift - swift - OpenDev: Free Software Needs Free Tools

This has been causing most/all of our tox-based jobs to run with
unconstrained dependencies (though test requirements were properly
pinned).
Change-Id: Icf1b63e39ec321887d9fe6afe7b212c813ad1d1e

Currently, when object-server serves PUT request and DiskFile
writer write file chunks to disk, there is no explicit
eventlet sleep called. When network outpace the slow disk IO,
it's possible one large and slow PUT request could cause
eventlet hub not to schedule any other green threads for a
long period of time. To improve this, this patch enable the
configurable yield parameter 'cooperative_period' into object
server controller write path.
Related-Change: I80b04bad0601b6cd6caef35498f89d4ba70a4fd4
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I1c0aba9830433f093d024b4c39cd3a3b2f0d69f1

We would like to remove any Statsd related code from logs.py. That
requires that SwiftLogAdapter no longer provides a StatsdClient
interface by default. However, for backwards compatibility the main
utils.get_logger/get_prefixed_logger entrypoints must provide a
SwiftLogAdapter that does have a StatdsClient interface.
The new utils._patch_statsd_methods helper function is therefore used
to retrospectively patch a SwiftLogAdapter instance with the
StatsdClient interface when necessary. The _patch_statsd_helper is
used in get_logger, and again when we clone a logger in
get_prefixed_logger.
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: I44694b92264066ca427bb96456d6f944e09b31c0

Verify that the reconciler is able to move objects to a container that
has, or will have, versioning enabled, and versions of the moved object
can subsequently be created.
Change-Id: I019447614ebadbb9e2cc8a18c0369bc16a89c0d9

Change-Id: I5c9af73cc66e17dc662d8c4d97cbf71ded1fed6f

A failing CORS test in the gate discovered that, when running with
eventlet==0.38.0, container and account HEAD requests returned
Content-Type of application/json to clients regardless of the requested
format. This was due to the backend HEAD response no longer having a
Content-Length header, causing the listing_formats middleware to not
modify the returned Content-Type (see Related-Change).
The Related-Change fixed the client facing issue by making
listing_formats middleware ensure the correct Content-Type is returned
to clients even when Content-Length is absent in the backend
response. The Related-Change also ensured that the 204 response to
clients always has a Content-Length header.
This patch directly fixes the problem of backend account and container
server HEADs no longer having 'Content-Length: 0' by adding it
explicitly. This violates the RFC prohibition of a 204 response having
a Content-Length header [1], but preserves Swift's historic behavior
and is consistent with the proxy-server's 204 response to clients.
[1] https://httpwg.org/specs/rfc7230.html#header.content-length
Related-Change: If724485e1425d1481d10b9255436301e346f07e8
Change-Id: Idacc59c5f43367926eff5221ee7fc417a9bc2d50

Update the 2023.1 release notes configuration to build from
unmaintained/2023.1.
Change-Id: I3e0962570145adb672bdba71f6de8f3da034b859

A failing CORS test in the gate discovered that we were responding
application/json to ?format=txt requests (which is maybe not even a
valid value for that qs param?), but only when running with
eventlet==0.38.0
This avoids the problem of backend container server HEADs no longer
having 'Content-Length: 0' by fixing the client HEAD resp headers before
we check for chunked-transfer resp.
Drive-By: refactor listing_formats to use HeaderKeyDict and always set
Content-Length explicitly
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: If724485e1425d1481d10b9255436301e346f07e8

This patch fixes a bug in the Related-Change that causes a prefix to
be added twice at the start of an 'exception' log message.
Previously, PrefixLoggerAdapter.exception would forward to the wrapped
LogAdapter.exception, and not call back to
PrefixLoggerAdapter.process. The prefix therefore needed to be added
in both PrefixLoggerAdapter.exception and PrefixLoggerAdapter.process
(for other log level methods).
SwiftLogAdapter.exception *does* call back to SwiftLogAdapter.process,
and so since the Related-Change it is not necessary to add the prefix
in both the SwiftLogAdapter.exception and SwiftLogAdapter.process
methods.
Related-Change: I8988c0add6bb4a65cc8be38f0bf527f141aac48a
Change-Id: Ia6e1f007989b0ef455b8dca8155b386a3fd9e8e1

Give this test file a less specific name in anticipation of further
mpu-related probe tests being added.
Change-Id: Iea01928e1daad25f3425f486b9dda4c6fb58510c

There's an upstream eventlet bug that seems to cause process hangs
during an atexit hook; unfortunately, that means that every time we
call "once" in probe tests, we can hang indefinitely waiting for a
process that won't terminate.
See https://github.com/eventlet/eventlet/issues/989
Now, wait with a timeout; if it pops, kill the offending process and
hope for the best. Do this by patching out subprocess.Popen.wait, but
only in probe tests -- this ensures that we won't impact any real
systems, while also ensuring a broad coverage of probe tests (as
opposed to, say, plumbing some new wait_timeout kwarg into all the
Manager call sites).
Closes-Bug: #2088027
Change-Id: I8983eafbb575d73d1654c354815a7de7ae141873

In order to modernize swift's statsd configuration we're working to
separate it from logging. This change is a pre-requisite for the
Related-Change in order to simplfy the stdlib base logger instance
wrapping in a single extended SwiftLogAdapter (previously LogAdapter)
which supports all the features swift's servers/daemons need
from our logger instance interface.
Related-Change-Id: I44694b92264066ca427bb96456d6f944e09b31c0
Change-Id: I8988c0add6bb4a65cc8be38f0bf527f141aac48a

Otherwise, test-run state (eventlet.tpool initialization, in particular)
can bleed over and cause hangs.
Closes-Bug: #2088026
Change-Id: I4f7dd0755b8dc8806d9b9046ac192d94ca705383

Previously, queue-clean-up would use the same timestamp/offset as the
tombstone written down in the old policy to clean up now-moved data.
This could lead to bad behaviors, as the reconciler-written tombstone
could itself be enqueued to be reconciled, at the same timestamp. If
that happened, replication would never bring the DB replicas to
consistency, causing the reconciler to get different answers for
whether there was work to do.
Now, add an extra offset bump between the tombstone in the old policy
and queue-clean-up. Also add an extra offset to the moved data in the
new policy, to keep it one ahead of queue-clean-up. New ordering looks
like:
- data/ts in old policy at t0 (as well as queue entry time to move it)
- tombstone in old policy at t0_1
- queue clean-up time at t0_2
- moved data/ts in new policy at t0_3
Closes-Bug: #2028175
Change-Id: Ib0dda0d338f48336d18d3d817a0c5994e201042e

Their binary wheels may crash python on older processors.
Related-Change: https://review.opendev.org/c/openstack/pyeclib/+/933338
Depends-On: https://review.opendev.org/c/openstack/requirements/+/934074
Change-Id: Ibc5b96421721e8e0a257acd06fdbfa5d82907bea

Add functional tests for copying from an SLO using PUT requests with
an x-copy-from header.
Add functional tests for copying from an SLO using part-number parameter.
Change-Id: Ic40c4b104b11de9ab09485e82543521b37ea1daa

In the past we have had some async pendings that repeatedly fail for months at a time. This patch adds an OldestAsyncPendingTracker class which manages the tracking of the oldest async pending updates for each account-container pair. This class maintains timestamps for pending updates associated with account-container pairs. It evicts the newest pairs when the max_entries is reached. It supports retrieving the N oldest pending updates or calculating the age of the oldest pending update.
Change-Id: I6d9667d555836cfceda52708a57a1d29ebd1a80b

This commit differentiates async pendings that are unlinked due to a successful object update (unlinks)from those async pendings that are unlinked due to a newer async_pending existing for the same object (outdated_unlinks).
Change-Id: I66e16207f3e368248617fc7ed3c6b5c80c54b1b5

Drive-Bys:
 * DRY out redundent configuration examples in expiring objects overview
 documentation.
 * Add missing delay_reaping man page docs.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I8879dbd13527233c878dff764ec411ce9619ee39

This change enhances the FakeInteralClient to make it easier to use the
existing stubs to test error handling behaviors and make assertions
about the behaviors.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Ib652a752ba91dbc791eef2a1d6940bcb0d16e36a

Move py311 to experimental, as py312 is now the highest required
version. Also swap py39 and py38 jobs between experimental and
check/gate; we probably should have done that for 2024.1 -- oh well.
See also:
- https://opendev.org/openstack/openstack-zuul-jobs/commit/3d6a033e
- https://opendev.org/openstack/openstack-zuul-jobs/commit/05389d5c
Change-Id: I418acfaeca20c93fa19b0f32a95bc53953efe183

Previously, we'd install the lowest (mostly common) denominator of deps
for our func-py3 jobs. This left us not testing up-to-date deps for
many python versions, though, and could even prevent us from testing on
new python versions entirely. For example, py312 cannot install
greenlet 2.0.2, the last version to support py36; this caused gate
failures when OpenStack as a whole tried to run Swift's func tests on
Ubuntu 24.04.
Change-Id: I0e55d8b4c5ed520c0aad8cc952a9fa1683905356

Change-Id: I8c65c7829fbe167e0c79239d577c38e90b93b5cf

Some submodules have previously been broken out of the utils
module. This patch adds automodule directives for the new modules to
the source documentation.
Change-Id: I985205fda95f01d226e81dcbfe0d6dbbb5b69c96
Related-Change: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a

In our updater we define `obj = '/%s/%s/%s' % (acct, cont, update['obj'])` which is a path, not an object. This commit renames the variable appropriately
Change-Id: I218fa6759df9af84002420a66a721740c0039003

The AWS behaviour is weird: if I have my client signer always use
the casing that will be sent on the wire, I reliably get back 405s,
but if my client signer always converts to upper-case despite
sending lower-case on the wire, it's roughly a coin toss for whether
I get back a 405 or a 403 (with the string-to-sign indicating that
the server was expecting lower-case). This is true for both v2 and
v4 signatures, both in headers and in query params.
Logically, though, it seems like using what's on the wire *ought* to
be the way to go, and AWS's consistency in that case lends some
credibility.
Unfortunately, that breaks an existing functional test unless awscrt
is installed, as boto3 (or botocore? somebody) will upper-case the
request method by default, but trusts awscrt to handle things if
available. So skip the part of the test that uses a lower-cased
request method if we can't import awscrt.
Change-Id: Ia6cacc51b2744986e703469f9e5215209194de8a

Now that openstack/constraints has dropped support for py38, we have
to manage constraints ourselves.
Change-Id: I9046c6ea91c98f15c4c491ce3de4a5aaf682f6f5

Change-Id: I224e8706f1ee72b718e35a25ea417a1cf3a35938

As per 2025.1 testing runtime[1], we need to test on Ubuntu
Noble (which will be taken care by depends-on tempest and devstack
patches to move base jobs to Noble) and at least single job to run on
Ubuntu Jammy (for smooth upgrade from previous releases).
This commit adds a new job to run on Jammy which can be removed
in future cycle when testing runtime test next version of Ubuntu
as default.
Depends-On: https://review.opendev.org/c/openstack/tempest/+/932156
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html
Change-Id: Ibab68b459c9510148b67ec1aecab73db11192afe

Fixes a bug with the x-open-expired feature where our magic header
does not get copied when refetching all manifests that causes
404 on HEAD requests with part-number=N query parameter since
the object-server returns an empty response body and the
proxy needs to refetch. The fix also applies to segment GET
requests if the segments have expired.
Change-Id: If0382d433f73cc0333bb4d0319fe1487b7783e4c