openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Add functional tests for copying from an SLO using PUT requests with
an x-copy-from header.
Add functional tests for copying from an SLO using part-number parameter.
Change-Id: Ic40c4b104b11de9ab09485e82543521b37ea1daa

In the past we have had some async pendings that repeatedly fail for months at a time. This patch adds an OldestAsyncPendingTracker class which manages the tracking of the oldest async pending updates for each account-container pair. This class maintains timestamps for pending updates associated with account-container pairs. It evicts the newest pairs when the max_entries is reached. It supports retrieving the N oldest pending updates or calculating the age of the oldest pending update.
Change-Id: I6d9667d555836cfceda52708a57a1d29ebd1a80b

This commit differentiates async pendings that are unlinked due to a successful object update (unlinks)from those async pendings that are unlinked due to a newer async_pending existing for the same object (outdated_unlinks).
Change-Id: I66e16207f3e368248617fc7ed3c6b5c80c54b1b5

Drive-Bys:
 * DRY out redundent configuration examples in expiring objects overview
 documentation.
 * Add missing delay_reaping man page docs.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I8879dbd13527233c878dff764ec411ce9619ee39

This change enhances the FakeInteralClient to make it easier to use the
existing stubs to test error handling behaviors and make assertions
about the behaviors.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Ib652a752ba91dbc791eef2a1d6940bcb0d16e36a

Move py311 to experimental, as py312 is now the highest required
version. Also swap py39 and py38 jobs between experimental and
check/gate; we probably should have done that for 2024.1 -- oh well.
See also:
- https://opendev.org/openstack/openstack-zuul-jobs/commit/3d6a033e
- https://opendev.org/openstack/openstack-zuul-jobs/commit/05389d5c
Change-Id: I418acfaeca20c93fa19b0f32a95bc53953efe183

Previously, we'd install the lowest (mostly common) denominator of deps
for our func-py3 jobs. This left us not testing up-to-date deps for
many python versions, though, and could even prevent us from testing on
new python versions entirely. For example, py312 cannot install
greenlet 2.0.2, the last version to support py36; this caused gate
failures when OpenStack as a whole tried to run Swift's func tests on
Ubuntu 24.04.
Change-Id: I0e55d8b4c5ed520c0aad8cc952a9fa1683905356

Change-Id: I8c65c7829fbe167e0c79239d577c38e90b93b5cf

Some submodules have previously been broken out of the utils
module. This patch adds automodule directives for the new modules to
the source documentation.
Change-Id: I985205fda95f01d226e81dcbfe0d6dbbb5b69c96
Related-Change: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a

In our updater we define `obj = '/%s/%s/%s' % (acct, cont, update['obj'])` which is a path, not an object. This commit renames the variable appropriately
Change-Id: I218fa6759df9af84002420a66a721740c0039003

The AWS behaviour is weird: if I have my client signer always use
the casing that will be sent on the wire, I reliably get back 405s,
but if my client signer always converts to upper-case despite
sending lower-case on the wire, it's roughly a coin toss for whether
I get back a 405 or a 403 (with the string-to-sign indicating that
the server was expecting lower-case). This is true for both v2 and
v4 signatures, both in headers and in query params.
Logically, though, it seems like using what's on the wire *ought* to
be the way to go, and AWS's consistency in that case lends some
credibility.
Unfortunately, that breaks an existing functional test unless awscrt
is installed, as boto3 (or botocore? somebody) will upper-case the
request method by default, but trusts awscrt to handle things if
available. So skip the part of the test that uses a lower-cased
request method if we can't import awscrt.
Change-Id: Ia6cacc51b2744986e703469f9e5215209194de8a

Now that openstack/constraints has dropped support for py38, we have
to manage constraints ourselves.
Change-Id: I9046c6ea91c98f15c4c491ce3de4a5aaf682f6f5

Change-Id: I224e8706f1ee72b718e35a25ea417a1cf3a35938

As per 2025.1 testing runtime[1], we need to test on Ubuntu
Noble (which will be taken care by depends-on tempest and devstack
patches to move base jobs to Noble) and at least single job to run on
Ubuntu Jammy (for smooth upgrade from previous releases).
This commit adds a new job to run on Jammy which can be removed
in future cycle when testing runtime test next version of Ubuntu
as default.
Depends-On: https://review.opendev.org/c/openstack/tempest/+/932156
[1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html
Change-Id: Ibab68b459c9510148b67ec1aecab73db11192afe

Fixes a bug with the x-open-expired feature where our magic header
does not get copied when refetching all manifests that causes
404 on HEAD requests with part-number=N query parameter since
the object-server returns an empty response body and the
proxy needs to refetch. The fix also applies to segment GET
requests if the segments have expired.
Change-Id: If0382d433f73cc0333bb4d0319fe1487b7783e4c

Previously FakeSwift would raise a KeyError for a call that had no
matching registered response *before* capturing the call. This
prevents tests reliably asserting that only expected calls have been
made. In particular, if the code being tested handles the KeyError
gracefully then it was possible to write quite reasonable test
assertions that passed despite unexpected calls being made.
Tests in test/unit/container/test_reconciler.py seem to rely on this
behaviour, so this patch adds a capture_unexpected_calls option for
FakeSwift which defauts to True. This allows the reconciler tests to
opt out of the new stricter call capturing.
Change-Id: Idc6b6b5a2b665538e861700f5d0996fc39368f5b

Apparently errno.EUCLEAN is not available on OS X, which can complicate
running unit tests.
Change-Id: Iaa3d7756949b4a67d4afe8a53b242ed9f41e9374

Of the last 100 successful runs, the average runtime has been ~4800s -- we
still get too many timeouts.
Change-Id: Ic5ba02198d51f938605787281488efccb25c0242

We've seen significant delays in the arm pipeline lately, with some changes
waiting nine or more hours for jobs to run. We should do what we can to
alleviate demand for arm64 nodes; I'm not sure any other projects have four
arm jobs on every change.
Change-Id: I7929ec3b856d90201c4cc4bf201e634446d3ad88

If py38 and py312 pass, it is reasonable to assume py39, py310, and py311
would, too.
Change-Id: Iefbfafe7c1005c9843a97d54c05398d4f0a3d2b0

We introduced this a while back, but forgot to add it then.
Related-Change: Ia13ee5da3d1b5c536eccaadc7a6fdcd997374443
Change-Id: Ib65ddf50d7f5c3e27475626000943eb18e65c73a

This is a path, not a request. See the parent class, Putter, and
all the callers of MIMEPutter.connect().
Related-Change: I12a6e41150f90de746ce03623032b83ed1987ee1
Change-Id: I42c7dcf2b4a3bf8bd023fdc1ce7d454ce01885a0

When Expirer is iterating task containers and doing listings, it's
possible one of the container nodes hosting the task container may
become overloaded (or have a really low backend ratelimit set).
Object-expirer should expect UnexpectedResponse and continue to try
and list the task objects in the next container.
And if the task container doesn't exist, expirer should not try to
delete the non-existent containers, before continue to work on
the next container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Id1966fa22725a02471e2d7c5a42fb243b1cfcf6a

Previously the object-expirer would not respect an internal_client_conf_path option if its config was loaded from a "legacy" config filepath (e.g. **/object-expirer.conf). Legacy object-expirer config expected config sections for an internal-client to be included in the same file. However, "modern" object-expirer config files (e.g. **/object-server.conf) expect internal-client config to be loaded from a separate file specified by internal_client_conf_path and defaulting to 'internal_client_conf_path'.
"Legacy" expirer config is still used in production clusters but it is useful to use a shared internal-client config. This patch therefore makes the internal_client_conf_path option always be respected regardless of the path of the object-expirer conf file. If 'internal_client_conf_path' is not specified, "modern" config will continue to use the default '/etc/swift/internal-client.conf', but "legacy" config will default to the path of the expirer conf file (i.e. the same as the previous behavior).
Related-Change: Ib21568f9b9d8547da87a99d65ae73a550e9c3230
Change-Id: I24ec702cd2ed074ca9df084cefc896418cece394

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I0e73e5548efbf93e433507b63fb41e34148b81a2

The datetime.utcfromtimestamp() is deprecated in Python 3.12.
Replace datetime.utcfromtimestamp() with datetime.fromtimestamp().
Change-Id: I01d6b94de394413aa13a045ab2c36504e65a6f5a
Signed-off-by: Takashi Natsume <takanattie@gmail.com>

py2 http responses don't have a 'headers' attribute, so don't try to
use it.
Unfortunately the unit test infrastructure's FakeConn class does
have a 'headers' attribute (whose value doesn't match the result of
calling its 'getheaders()' method!), so this bug was not caught by
tests in the Related-Change.
Related-Change: I96f28ab0b2b5f9374c399e8905ee240e7b093f8b
Change-Id: I2cd820280b8c69cafc5730183903c9d379d8dde5

Previously the in-process functional test setup attempted to insert
etag-quoter in the proxy pipeline by replacing the substring 'cache
listing_formats' with 'cache etag-quoter listing_formats'. However,
the pipeline had already been modified to have 'cache domain-remap
listing_formats', so the etag-quoter replacement failed silently.
This patch modifies the etag-quoter replacement to match just
'cache'. Some helper functions are also introduced to make the
pipeline modification more uniform and to consistently verify the
intended modifications have been achieved, or otherwise raise an
exception.
Change-Id: I640837a4da985bcbe48f9ae5eeb56385b11034eb

A print statement, presumably for debugging, crept in with the
Related-Change.
Change-Id: I0cccb960b51446a3aa58be95c8c9d06cc6b6eada
Related-Change: I69974cc8a39731c980b54137b799a36b2e63a44a

Some s3api functional tests were setting boto logging level to DEBUG,
which results in a very large amount of logging output from the test
runner, including large request bodies. This makes it extremely hard
to inspect test output.
This patch makes the quiet_boto_logging context manager revert the
logger level to its original value rather than indiscriminately
leaving it set to DEBUG.
Change-Id: I1dd9603adf9a19e89da5a461d3c6810a3432ae46

Maintaining testing requirements for our py36 gate jobs
since we have been remiss in updating them regularly. This
will help us test swift which operators might be running.
Change-Id: I58fa780460f7ac84a02d508fd95770bada991326

Related-Bug: #2080600
Change-Id: Id5b0bf6efcb2c16e10334d2edb268194fa1ec008

Add file to the reno documentation build to show release notes for
stable/2024.2.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2024.2.
Sem-Ver: feature
Change-Id: I9a384cc42eecc5ffd6f24fa58653a672190017df

We used to have a fairly bimodal distribution of successful ARM test
times: roughly 2/3 of the time, tests would take 800-1800s (~15-30min),
and 1/3 of the time they would take more like 2000-5400s (~30-90min).
Starting around August, the bimodal distribution is gone, with only the
longer expected test times remaining. Timeouts have increased, leaving
the ARM jobs frequently failing spuriously. Increase the timeouts to
get better signal from the tests.
Change-Id: I56165dc6d877bd9e56d7160dd28c6c60f339d730

Change-Id: I374abcdd93664322cb440cd083f1ea4edf04d730