openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Adds a test that verifies extra body content beyond the content-length
is ignored provided that the checksum value matches that of the
content-length bytes.
Add comment to explain why this is the case.
Drive-by: add clarifying comment to unit test.
Change-Id: I8f198298a817be47223e2f45fbc48a6f393b3bef
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

See https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
for some background.
This covers both "normal" objects and part-uploads for MPUs. Note that
because we don't write down any client-provided checksums during
initiate-MPU calls, we can't do any verification during complete-MPU
calls.
crc64nvme checksums are not yet supported; clients attempting to use
them will get back 501s.
Adds crt as a boto3 extra to test-requirements. The extra lib provides
crc32c and crc64nvme checksum support in boto3.
Co-Authored-By: Ashwin Nair <ashnair@nvidia.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Id39fd71bc59875a5b88d1d012542136acf880019

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I9a31eda60da71ff9f9db8b32b517338e99896667
Signed-off-by: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Generated-By: openstack/openstack-zuul-jobs:roles/prepare-zanata-client/files/common_translation_update.sh

S3 includes the expected base64 digest in a BadDigest response to a
multipart complete POST request.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ie20ccf10846854f375c29be1b0b00b8eaacc9afa

If we already know enough about the error to feel confident in
quarantining, we don't really need a whole traceback about it;
that just clutters logs.
Change-Id: Iab0e62c85b33d699f96d744faaa16420b7148b47

Change-Id: I88e2e743ccbd6292bc1570ae0efbdd45dcced8cc

Change-Id: I80142c6c0654b65b5755e7e828bcc4969a10f4f1

The existing doc is quite outdated and is based on ancient versions.
Update it according to the following points, so that it works in recent
versions.
* Use python3- packages instead of python-/python2- packages
* xinetd is no longer available in recent CentOS
* Remove old unused test dependencies such as nose and mock
* Remove netifaces which is no longer a hard dependency
Change-Id: I8bf87f858406dc1a32139a0071b53cfb90864108

Change-Id: I74381567978025e7f528e169692bbaf9bac45de2

datetime.timezone.utc[1] has been available in Python 3 and can be used
instead of datetime.UTC which is available only in Python >=3.11 .
[1] https://docs.python.org/3.13/library/datetime.html#datetime.timezone.utc
Change-Id: I92bc82a1b7e2bcb947376bc4d96fc603ad7d5b6c

Change-Id: I4eff901aaa334c8a73cebfc578cea14d23e6c365

... because Python 2.x is no longer supported.
Change-Id: I3167a539b3e26ceb35976fbd7a2356ba59d4a5e4

Change-Id: I32fec7540bee70e77140964c5983d133a572fa7b

It used to be that when a logger encountered an error trying to handle
/ emit a message, it would attempt to log about the error via the same
logger. Long ago, this could lead to infinite recursion in Swift's
LoggerFileObject, so we fixed it and added a test that included
showing the bad stdlib behavior.
Recently, stdlib did a similar fix so the recursion doesn't happen. See
 - https://github.com/python/cpython/issues/91555 and
 - https://github.com/python/cpython/pull/131812
Of course, now our test breaks (at least, on 3.13.4). Relax the
assertion a little.
Related-Change: Ia9ecffc88ce43616977e141498e5ee404f2c29c4
Change-Id: Id2f490d4204b8eaf07857cb84ed783bec19b8511

Change-Id: Iea1adfb93891e4bde62a11bfcef478d9b9696fd4

Swift does not return all the parameters of objects in a listing
(e.g. ChecksumType and ChecksumAlgorithm) so pop these from listings
before making assertions.
Change-Id: Ieb7a9783731c11f1c08db398eae07ffafa127460

The labeled timing metrics previously have sample_rate similar to
those added for non-labeled metrics. However the down sampling has
not been helpful when using labeled metrics to investigate customer
issues, for example those related to object server REPLICATE requests.
This patch changes labeled timing metrics to not have sample_rate. The
non-labeled metrics are unrelated to this effort thus not changed.
Related-Change: I05336b700120ab5fcf922590d6a12f73112edb50
Change-Id: Ia6e856ffaf8fd1b4a905e6976ebdc62ed5ddf32f

We're increasingly trying to move toward having S3 tests that pass
against both AWS and Swift. Especially as we move to replace our old
boto(2) tests, it would be nice to have the replacements be cross-
compatible from the get-go.
However, there was previously a risk in leaning too-heavily on that
test suite: only the functional test suite would be exercised in the
gate with Keystone. Now, run both test suites to ensure we don't
leave implementation gaps in s3token.
Change-Id: I873c32107b4174bd2968097b5fe96d97f260a256

Previously (at least on OSX) this could lead to errors like
 socket.gaierror: [Errno 8] nodename nor servname provided, or not known
when we actually went and tried to connect to send the update (!!)
Change-Id: I86f6e731d1ee273c6772974ce597ac91be3937be

Change-Id: I0a27efe897b4e8ce2c21da1a3603a2a77c02eb69

Locally, this reduced the test time from 240s to 0.16s.
Change-Id: I5a4786b6782c06f8e6bd9fab5d4dae683a970242

Change-Id: I05336b700120ab5fcf922590d6a12f73112edb50

Change-Id: Id5583e3a1e4515ec3c8a972f647aaaabfba673bc
Related-Change: I2e57dacb342b5758f16b502bb91372a2443d0182

Modified code to use native labeled metrics API introduced by the
Related-Change.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: I115ffb1dc601652a979895d7944e011b951a91c1
Change-Id: I5a96626862644fe23a0d4bc53e85989770c8ceec

When a DiskFile is opened, its on-disk files are first listed and then
xattr metadata is read. If a listed file no longer exists when its
metadata is read then a DiskFileNotExist exception was raised which
would previously cause the object server to return a 404.
It isn't appropriate to return 404 when the list of on-disk files
suggested that the object existed (i.e. included a .data file) and
there is no evidence that the object no longer exists: the missing file
may well have been *replaced* by a concurrent request.
For example, if a POST request races with another concurrent POST
request, the object's .meta file may be replaced by a new .meta file,
and reading the original .meta file will fail. Similarly, if a POST
races with a concurrent PUT request, the PUT request may replace the
.data file causing the POST request handler to fail to read the
original .data file. In neither case has the object ever been
non-existent.
This issue was observed during a period of many concurrent POST
requests to an object in a production system. A significant number of
the POSTs returned 404 despite the object never being deleted.
This patch modifies DiskFile to raise a new DiskFileStateChanged
exception when metadata cannot be read from a data, meta, or ts file
that was listed in the object's datadir. The object server may translate
this exception to a 503 response rather than a 404, depending on the
request method:
 * POSTs, GETs and HEADs that encounter this transient loss of a
 file will now return 503 rather than 404. The DiskFile may
 still exist.
 * DELETEs that encounter this transient loss of a file would
 previously have proceeded but will now return 503: a replacement
 file may contain an X-Delete-At that would have prevented the
 DELETE proceeding.
 * PUTs that encounter this transient loss of a file will continue to
 proceed as before: a replacement meta file may contain an
 X-Delete-At that requires updates to the expirer queue but those
 updates would presumably be handled by the concurrent POST that
 has replaced the meta file; a replacement .data file will remain
 on disk until the DiskFile is next opened and one of the .data
 files will be cleaned up.
Closes-Bug: #2054791
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I2f698c25ed65b236e851e5a307d48a12cef62b33

Swift already supports that much, at least. AWS used to not support any
conditional PUTs, but that's changed somewhat recently; see
- https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/
- https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/
Drive-By: Fix retry of a CompleteMultipartUpload with changed parts; it
should 404 rather than succeed in writing the new manifest.
Change-Id: I2e57dacb342b5758f16b502bb91372a2443d0182

The timing stats decorators are moved from utils to base_storage_server.py
to avoid a circular import of HTTPException.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Idc4b52de0a04ebfc0e353162bd791d4e0e20eac3

openSUSE stopped providing OpenStack packages some time ago.
Co-authored-by: Takashi Kajinami <kajinamit@oss.nttdata.com>
Change-Id: I60fcbac1cb6ec370de565903375be411e8fc3415

pip 23.1 removed the "setup.py install" fallback for projects
that do not have pyproject.toml and now uses a pyproject.toml
which is vendored in pip.
To address that, this change adds the minimal pyproject.toml
to enable pbr to be properly used to build editable wheels.
This is required to support installing devstack on
centos stream 9 and related distros with GLOBAL_VENV=True
Without this change the wsgi scripts are not generated in
editable mode. i.e. pip install -e /opt/stack/keystone
See https://pip.pypa.io/en/stable/news/#v23-1
and https://github.com/pypa/pip/issues/8368 for more
details on the removal of the fallback support.
setuptools v64.0.0 is used to support editable installs
via its PEP-660 implmentation
https://github.com/pypa/setuptools/pull/3488
Change-Id: I82e0c130c4022164bc8e4d1120aae9bdb9a43b5d