openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I32fec7540bee70e77140964c5983d133a572fa7b

It used to be that when a logger encountered an error trying to handle
/ emit a message, it would attempt to log about the error via the same
logger. Long ago, this could lead to infinite recursion in Swift's
LoggerFileObject, so we fixed it and added a test that included
showing the bad stdlib behavior.
Recently, stdlib did a similar fix so the recursion doesn't happen. See
 - https://github.com/python/cpython/issues/91555 and
 - https://github.com/python/cpython/pull/131812
Of course, now our test breaks (at least, on 3.13.4). Relax the
assertion a little.
Related-Change: Ia9ecffc88ce43616977e141498e5ee404f2c29c4
Change-Id: Id2f490d4204b8eaf07857cb84ed783bec19b8511

Change-Id: Iea1adfb93891e4bde62a11bfcef478d9b9696fd4

Swift does not return all the parameters of objects in a listing
(e.g. ChecksumType and ChecksumAlgorithm) so pop these from listings
before making assertions.
Change-Id: Ieb7a9783731c11f1c08db398eae07ffafa127460

The labeled timing metrics previously have sample_rate similar to
those added for non-labeled metrics. However the down sampling has
not been helpful when using labeled metrics to investigate customer
issues, for example those related to object server REPLICATE requests.
This patch changes labeled timing metrics to not have sample_rate. The
non-labeled metrics are unrelated to this effort thus not changed.
Related-Change: I05336b700120ab5fcf922590d6a12f73112edb50
Change-Id: Ia6e856ffaf8fd1b4a905e6976ebdc62ed5ddf32f

We're increasingly trying to move toward having S3 tests that pass
against both AWS and Swift. Especially as we move to replace our old
boto(2) tests, it would be nice to have the replacements be cross-
compatible from the get-go.
However, there was previously a risk in leaning too-heavily on that
test suite: only the functional test suite would be exercised in the
gate with Keystone. Now, run both test suites to ensure we don't
leave implementation gaps in s3token.
Change-Id: I873c32107b4174bd2968097b5fe96d97f260a256

Previously (at least on OSX) this could lead to errors like
 socket.gaierror: [Errno 8] nodename nor servname provided, or not known
when we actually went and tried to connect to send the update (!!)
Change-Id: I86f6e731d1ee273c6772974ce597ac91be3937be

Change-Id: I0a27efe897b4e8ce2c21da1a3603a2a77c02eb69

Locally, this reduced the test time from 240s to 0.16s.
Change-Id: I5a4786b6782c06f8e6bd9fab5d4dae683a970242

Change-Id: I05336b700120ab5fcf922590d6a12f73112edb50

Change-Id: Id5583e3a1e4515ec3c8a972f647aaaabfba673bc
Related-Change: I2e57dacb342b5758f16b502bb91372a2443d0182

Modified code to use native labeled metrics API introduced by the
Related-Change.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: I115ffb1dc601652a979895d7944e011b951a91c1
Change-Id: I5a96626862644fe23a0d4bc53e85989770c8ceec

When a DiskFile is opened, its on-disk files are first listed and then
xattr metadata is read. If a listed file no longer exists when its
metadata is read then a DiskFileNotExist exception was raised which
would previously cause the object server to return a 404.
It isn't appropriate to return 404 when the list of on-disk files
suggested that the object existed (i.e. included a .data file) and
there is no evidence that the object no longer exists: the missing file
may well have been *replaced* by a concurrent request.
For example, if a POST request races with another concurrent POST
request, the object's .meta file may be replaced by a new .meta file,
and reading the original .meta file will fail. Similarly, if a POST
races with a concurrent PUT request, the PUT request may replace the
.data file causing the POST request handler to fail to read the
original .data file. In neither case has the object ever been
non-existent.
This issue was observed during a period of many concurrent POST
requests to an object in a production system. A significant number of
the POSTs returned 404 despite the object never being deleted.
This patch modifies DiskFile to raise a new DiskFileStateChanged
exception when metadata cannot be read from a data, meta, or ts file
that was listed in the object's datadir. The object server may translate
this exception to a 503 response rather than a 404, depending on the
request method:
 * POSTs, GETs and HEADs that encounter this transient loss of a
 file will now return 503 rather than 404. The DiskFile may
 still exist.
 * DELETEs that encounter this transient loss of a file would
 previously have proceeded but will now return 503: a replacement
 file may contain an X-Delete-At that would have prevented the
 DELETE proceeding.
 * PUTs that encounter this transient loss of a file will continue to
 proceed as before: a replacement meta file may contain an
 X-Delete-At that requires updates to the expirer queue but those
 updates would presumably be handled by the concurrent POST that
 has replaced the meta file; a replacement .data file will remain
 on disk until the DiskFile is next opened and one of the .data
 files will be cleaned up.
Closes-Bug: #2054791
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I2f698c25ed65b236e851e5a307d48a12cef62b33

Swift already supports that much, at least. AWS used to not support any
conditional PUTs, but that's changed somewhat recently; see
- https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/
- https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/
Drive-By: Fix retry of a CompleteMultipartUpload with changed parts; it
should 404 rather than succeed in writing the new manifest.
Change-Id: I2e57dacb342b5758f16b502bb91372a2443d0182

The timing stats decorators are moved from utils to base_storage_server.py
to avoid a circular import of HTTPException.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Idc4b52de0a04ebfc0e353162bd791d4e0e20eac3

openSUSE stopped providing OpenStack packages some time ago.
Co-authored-by: Takashi Kajinami <kajinamit@oss.nttdata.com>
Change-Id: I60fcbac1cb6ec370de565903375be411e8fc3415

pip 23.1 removed the "setup.py install" fallback for projects
that do not have pyproject.toml and now uses a pyproject.toml
which is vendored in pip.
To address that, this change adds the minimal pyproject.toml
to enable pbr to be properly used to build editable wheels.
This is required to support installing devstack on
centos stream 9 and related distros with GLOBAL_VENV=True
Without this change the wsgi scripts are not generated in
editable mode. i.e. pip install -e /opt/stack/keystone
See https://pip.pypa.io/en/stable/news/#v23-1
and https://github.com/pypa/pip/issues/8368 for more
details on the removal of the fallback support.
setuptools v64.0.0 is used to support editable installs
via its PEP-660 implmentation
https://github.com/pypa/setuptools/pull/3488
Change-Id: I82e0c130c4022164bc8e4d1120aae9bdb9a43b5d

This got missed when we dropped support for py36.
Related-Change: I6f0cbde23f0a0db75bdb3e598ee862af0bad0dba
Change-Id: I3dcca36744bbdbce7e5e22e50b3a3fa263071717

We recently removed the "func-py3" tox env in favor of just "func" -- as a
result, this went back to running unit tests!
Keep "func-py3" for older releases where that tox env *is* still defined,
though, as they won't have py27 around to run "func".
Related-Change: I53153c4fde043e964e1daa7bbf2089e0471dede2
Change-Id: Ibbb8c305a8b8bac7ca1a8853208cd031957adac1

Some tests previously made assertions on statsd client methods that
were implemented by a delegated interface patched onto a logger
instance. Those tests have run against a StatsdClient since the
Related Change, so the use of the term 'delegate' in tests no longer
makes sense.
Change-Id: Ie47fa50a226da6ff7b067d6a367758ac9f12ef4c
Related-Change: I3a677bb67c5700da48f89c847f652b4610ba47c2

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I811642fccd916bd9ef71846a8108d50a462740f0

Change-Id: I6f0cbde23f0a0db75bdb3e598ee862af0bad0dba

A new CRCHasher class wraps crc functions to mimic the hashlib
hasher interface.
For crc32, use the zlib implementation.
For crc32c, use ISA-L if we can find it; on py38+ this includes the
ISA-L bundled in pyeclib's binary wheels.
If ISA-L is not available, check for kernel crypto API support for crc32c;
use it if available. Otherwise, raise a NotImplementedError.
No callers yet, but these will be used by s3api in a later patch.
Change-Id: Ic0c55e307ce10b56b569c9fee728c445a2300cbd

This update implements Sigv4-streaming (chunked upload) as described in
the Amazon S3 documentation:
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
Closes-Bug: #1810026
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: ashnair <ashnair@nvidia.com>
Change-Id: I7be1ce9eb5dba7b17bdf3e53b0d05d25ac0a05b0

The file is not used because PBR is used.
Change-Id: Iefef64f63cbd8847a6b0322edc4a8ce85a6b8580

This will prevent client timeouts during large object copies, similar to existing support in
bulk operations and SLO manifest writes. The operator can set the frequency using the
yield_frequency inside conf file.
To use heartbeat, simply add "heartbeat=on" query parameter, which will:
- Returns a 202 Accepted response
- Sends periodic 1 byte whitespace heartbeat every yield_frequency
- Returns final status and headers
- Supports both plain text and JSON
Closes-Bug: #1738306
Change-Id: Iaf4355f2e0edbd6dde226231f3d7a61a5e53ecfa

Bobcat went EOL recently; add Epoxy job.
Related-Change: https://review.opendev.org/c/openstack/releases/+/948240
Change-Id: Ibad603c4a07d49ce55dce26d1af4c506d95b193f

Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: I155572bc5f4d955c918c951d6d2af9dbbae0136d