openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Added the new field to be able to log the access key during the
s3api calls, while reserving the field to be filled with auth relevant
information in case of other middlewares. Added respective code to
the tempauth and keystone middlewares.
Since s3api creates a copy of the environ dict for the downstream
request object when translating the s3req.to_swift_req the environ
dict that is seen/modifed in other mw module is not the same instance
seen in proxy-logging - using mutable objects get transfered into the
swift_req.environ.
Change the assert in test_proxy_logging from "the last field" to
the index 21 in the interests of maintainability.
Also added some regression tests for object, bucket and s3 v4 apis and
updated the documentation with the details about the new field.
Signed-off-by: Vitaly Bordyug <vbordug@gmail.com>
Change-Id: I0ce4e92458e2b05a4848cc7675604c1aa2b64d64

We stopped writing pickled rings more than twelve years ago. Any
cluster that was going to upgrade from then has, or can pick any of
the multitude of intermediary releases to pause at and push rings.
We can also safely assume that regions will be present for devices;
that change is nearly as old.
As a side-effect, clean up some old tests that did nonsense things
like having 7 assignments per row for a part-power-2 ring.
UpgradeImpact: remove ability to read rings written by swift <1.7.0
circa 2012
Related-Change: I799b9a4c894d54fb16592443904ac055b2638e2d
Related-Change: Ifefbb839cdcf033e6c9201fadca95224c7303a29
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: Ic8322b18d51b40f586cb217a0d1b2f345e1d8df6

Add anycrc as a soft dependency in case ISA-L isn't available.
Plus we'll want it later: when we start writing down checksums,
we'll need it to combine per-part checksums for MPUs.
Like with crc32c, we won't provide any pure-python version as the
CPU-intensiveness could present a DoS vector. Worst case, we 501
as before.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: Ia05e5677a8ca89a62b142078abfb7371b1badd3f
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Assert that BadDigest responses due to checksum mismatch do not include
the expected or computed values.
Change-Id: Iaffa02c3c02fa3bc6922f51ecf28a39f4b24ccf2
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Adds a test that verifies extra body content beyond the content-length
is ignored provided that the checksum value matches that of the
content-length bytes.
Add comment to explain why this is the case.
Drive-by: add clarifying comment to unit test.
Change-Id: I8f198298a817be47223e2f45fbc48a6f393b3bef
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

See https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
for some background.
This covers both "normal" objects and part-uploads for MPUs. Note that
because we don't write down any client-provided checksums during
initiate-MPU calls, we can't do any verification during complete-MPU
calls.
crc64nvme checksums are not yet supported; clients attempting to use
them will get back 501s.
Adds crt as a boto3 extra to test-requirements. The extra lib provides
crc32c and crc64nvme checksum support in boto3.
Co-Authored-By: Ashwin Nair <ashnair@nvidia.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Id39fd71bc59875a5b88d1d012542136acf880019

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I9a31eda60da71ff9f9db8b32b517338e99896667
Signed-off-by: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Generated-By: openstack/openstack-zuul-jobs:roles/prepare-zanata-client/files/common_translation_update.sh

S3 includes the expected base64 digest in a BadDigest response to a
multipart complete POST request.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ie20ccf10846854f375c29be1b0b00b8eaacc9afa

Change-Id: I88e2e743ccbd6292bc1570ae0efbdd45dcced8cc

Change-Id: I80142c6c0654b65b5755e7e828bcc4969a10f4f1

The existing doc is quite outdated and is based on ancient versions.
Update it according to the following points, so that it works in recent
versions.
* Use python3- packages instead of python-/python2- packages
* xinetd is no longer available in recent CentOS
* Remove old unused test dependencies such as nose and mock
* Remove netifaces which is no longer a hard dependency
Change-Id: I8bf87f858406dc1a32139a0071b53cfb90864108

Change-Id: I74381567978025e7f528e169692bbaf9bac45de2

datetime.timezone.utc[1] has been available in Python 3 and can be used
instead of datetime.UTC which is available only in Python >=3.11 .
[1] https://docs.python.org/3.13/library/datetime.html#datetime.timezone.utc
Change-Id: I92bc82a1b7e2bcb947376bc4d96fc603ad7d5b6c

... because Python 2.x is no longer supported.
Change-Id: I3167a539b3e26ceb35976fbd7a2356ba59d4a5e4

Change-Id: I32fec7540bee70e77140964c5983d133a572fa7b

It used to be that when a logger encountered an error trying to handle
/ emit a message, it would attempt to log about the error via the same
logger. Long ago, this could lead to infinite recursion in Swift's
LoggerFileObject, so we fixed it and added a test that included
showing the bad stdlib behavior.
Recently, stdlib did a similar fix so the recursion doesn't happen. See
 - https://github.com/python/cpython/issues/91555 and
 - https://github.com/python/cpython/pull/131812
Of course, now our test breaks (at least, on 3.13.4). Relax the
assertion a little.
Related-Change: Ia9ecffc88ce43616977e141498e5ee404f2c29c4
Change-Id: Id2f490d4204b8eaf07857cb84ed783bec19b8511

Change-Id: Iea1adfb93891e4bde62a11bfcef478d9b9696fd4

Swift does not return all the parameters of objects in a listing
(e.g. ChecksumType and ChecksumAlgorithm) so pop these from listings
before making assertions.
Change-Id: Ieb7a9783731c11f1c08db398eae07ffafa127460

The labeled timing metrics previously have sample_rate similar to
those added for non-labeled metrics. However the down sampling has
not been helpful when using labeled metrics to investigate customer
issues, for example those related to object server REPLICATE requests.
This patch changes labeled timing metrics to not have sample_rate. The
non-labeled metrics are unrelated to this effort thus not changed.
Related-Change: I05336b700120ab5fcf922590d6a12f73112edb50
Change-Id: Ia6e856ffaf8fd1b4a905e6976ebdc62ed5ddf32f

We're increasingly trying to move toward having S3 tests that pass
against both AWS and Swift. Especially as we move to replace our old
boto(2) tests, it would be nice to have the replacements be cross-
compatible from the get-go.
However, there was previously a risk in leaning too-heavily on that
test suite: only the functional test suite would be exercised in the
gate with Keystone. Now, run both test suites to ensure we don't
leave implementation gaps in s3token.
Change-Id: I873c32107b4174bd2968097b5fe96d97f260a256

Previously (at least on OSX) this could lead to errors like
 socket.gaierror: [Errno 8] nodename nor servname provided, or not known
when we actually went and tried to connect to send the update (!!)
Change-Id: I86f6e731d1ee273c6772974ce597ac91be3937be

Change-Id: I0a27efe897b4e8ce2c21da1a3603a2a77c02eb69

Locally, this reduced the test time from 240s to 0.16s.
Change-Id: I5a4786b6782c06f8e6bd9fab5d4dae683a970242

Change-Id: I05336b700120ab5fcf922590d6a12f73112edb50

Change-Id: Id5583e3a1e4515ec3c8a972f647aaaabfba673bc
Related-Change: I2e57dacb342b5758f16b502bb91372a2443d0182

Modified code to use native labeled metrics API introduced by the
Related-Change.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: I115ffb1dc601652a979895d7944e011b951a91c1
Change-Id: I5a96626862644fe23a0d4bc53e85989770c8ceec

When a DiskFile is opened, its on-disk files are first listed and then
xattr metadata is read. If a listed file no longer exists when its
metadata is read then a DiskFileNotExist exception was raised which
would previously cause the object server to return a 404.
It isn't appropriate to return 404 when the list of on-disk files
suggested that the object existed (i.e. included a .data file) and
there is no evidence that the object no longer exists: the missing file
may well have been *replaced* by a concurrent request.
For example, if a POST request races with another concurrent POST
request, the object's .meta file may be replaced by a new .meta file,
and reading the original .meta file will fail. Similarly, if a POST
races with a concurrent PUT request, the PUT request may replace the
.data file causing the POST request handler to fail to read the
original .data file. In neither case has the object ever been
non-existent.
This issue was observed during a period of many concurrent POST
requests to an object in a production system. A significant number of
the POSTs returned 404 despite the object never being deleted.
This patch modifies DiskFile to raise a new DiskFileStateChanged
exception when metadata cannot be read from a data, meta, or ts file
that was listed in the object's datadir. The object server may translate
this exception to a 503 response rather than a 404, depending on the
request method:
 * POSTs, GETs and HEADs that encounter this transient loss of a
 file will now return 503 rather than 404. The DiskFile may
 still exist.
 * DELETEs that encounter this transient loss of a file would
 previously have proceeded but will now return 503: a replacement
 file may contain an X-Delete-At that would have prevented the
 DELETE proceeding.
 * PUTs that encounter this transient loss of a file will continue to
 proceed as before: a replacement meta file may contain an
 X-Delete-At that requires updates to the expirer queue but those
 updates would presumably be handled by the concurrent POST that
 has replaced the meta file; a replacement .data file will remain
 on disk until the DiskFile is next opened and one of the .data
 files will be cleaned up.
Closes-Bug: #2054791
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I2f698c25ed65b236e851e5a307d48a12cef62b33