openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: Ib6bf702e38495e52e3b2f5ca95ed17c519018474
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

The disable_fallocate function provided in common.utils doesn't really
have a way to undo it - it's tested independently in test_utils.
It shouldn't be used on test_diskfile or else test_utils fallocate tests
will fail afterwards.
Change-Id: I6ffa97b39111ba25f85ba7cfde21440d975dc760
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

Change-Id: I0168ab113fdda60ed858ed0928356699399d4044
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Closes-Bug: #2122543
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I8a2a96394734899ee48e1d9264bf3908968c51a8

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I2e2a4e2c448319e6531372ae06ab81eb58edc57e

Added the new field to be able to log the access key during the
s3api calls, while reserving the field to be filled with auth relevant
information in case of other middlewares. Added respective code to
the tempauth and keystone middlewares.
Since s3api creates a copy of the environ dict for the downstream
request object when translating the s3req.to_swift_req the environ
dict that is seen/modifed in other mw module is not the same instance
seen in proxy-logging - using mutable objects get transfered into the
swift_req.environ.
Change the assert in test_proxy_logging from "the last field" to
the index 21 in the interests of maintainability.
Also added some regression tests for object, bucket and s3 v4 apis and
updated the documentation with the details about the new field.
Signed-off-by: Vitaly Bordyug <vbordug@gmail.com>
Change-Id: I0ce4e92458e2b05a4848cc7675604c1aa2b64d64

S3 stopped returning DisplayNamme in the Owner field of object
listings [1], so the tests need to stop asserting that it is present.
Further work is needed to drop DisplayName from the Swift s3api
responses [2].
[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_Owner.html
[2] https://bugs.launchpad.net/swift/+bug/2120622
Change-Id: Ia915a65313394910c74ae826c912b5549e833a7b
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

The RuntimeError gives more of a hint than the TypeError, but we haven't
really solved the issue. If there's a busted pyeclib install, it's safe
to assume it won't have ISA-L alongside it; just proceed assuming it's
not available.
Closes-Bug: #2120591
Related-Change: I64a85eb739fb72efe41f1ee829e463167246b793
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I2791566b208327b1fb536fb56a363337ab3f3941

ssync unit tests would sometimes fail when making assertions about the
ssync receiver log messages. Test runner output would show that the
messages were eventually being logged. However, the assertions could
be made before the ssync receiver request thread had completed.
A trampoline had been previously been used to workaround this, but
that is clearly insufficient. The author found that increasing the
trampoline interval would help reduce the rate of failures, but not
eliminate them.
This patch introduces a custom GreenPool for the unit test wsgi object
server so that tests can deterministically wait for the receiver
request handling thread to exit before making assertions.
Closes-Bug: #212065
Change-Id: I09ad8bb1becae46a78902d1d384a9f27a3d54b38
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

If pyeclib dist is missing files then the isal loading would blow up
with "TypeError: 'NoneType' object is not iterable". This patch
changes that to a RuntimeError with a more useful message.
Change-Id: I64a85eb739fb72efe41f1ee829e463167246b793
Closes-Bug: #2120591
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Previously it was possible for a Timestamp to have inconsistent raw
and normal values. For example:
>>> ts1 = Timestamp(1755077566.523385)
>>> (ts1.normal, ts1.raw, (~ts1).normal)
('1755077566.52339', 175507756652338, '8244922433.47661')
This results in the invert function not being reliably reversible:
(~(~ts1)).normal
'1755077566.52338'
The cause is that the normal value is based on Timestamp.timestamp
which preserves the precision of the value given to the constructor,
whereas the invert function uses the limited precision raw value.
This patch forces Timestamp.timestamp to have the limited precision
value of Timestamp.raw.
Change-Id: I4e7fd6078aae3f284628303f20ced66aa702c466
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I75bd005a4a3bc79c1bd8f8fa1153a64059970865

Related-Change-Id: I8c342c4751ba3ca682efd152e90e396e9f8eb851
Change-Id: I9df32c5aae4e681d488419f36982ffc36589d50a
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

If a PUT subrequest body iterator times out while the object server is
reading it, the object server will handle the timeout and return a 500
response to the ssync receiver.
Previously, the ssync receiver would attempt to drain the remainder of
the subrequest body iterator and then resume reading the next
subrequest from the SSYNC body. However, the attempt to drain the
subrequest iterator would fail (silently) because the timeout had
already caused the iterator to exit.
The ssync receiver would therefore treat any remaining subrequest body
as the preamble to the next subrequest. This remaining subrequest body
content was likely to cause the protocol parsing to fail, but in the
worst case could be erroneously interpreted as a valid subrequest.
(The exact failure mechanism depends on what state the
eventlet.wsgi.Input is left in when the timeout fired.)
This patch ensures that the ssync receiver will terminate processing
an SSYNC request if an exception occurs while reading a subrequest
body.
Closes-Bug: #2115991
Change-Id: I585e8a916d947c3da8d7c0e8a85cf0a8ab85f7f0
Co-authored-by: Tim Burke <tim.burke@gmail.com>
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

fetch_crypto_keys can fail like
get_keys(): from callback: 'utf-8' codec can't encode character '\udcc0' in position 1: surrogates not allowed:
Traceback (most recent call last):
 File ".../swift/common/middleware/crypto/crypto_utils.py", line 166, in get_keys
 keys = fetch_crypto_keys(key_id=key_id)
 File ".../swift/common/middleware/crypto/keymaster.py", line 148, in fetch_crypto_keys
 keys['container'] = self.keymaster.create_key(
 File ".../swift/common/middleware/crypto/keymaster.py", line 322, in create_key
 path = path.encode('utf-8')
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc0' in position 1: surrogates not allowed
This doesn't fix *all* non-utf8 paths, but
- it was easy enough to avoid the non-swift ones, which have been seen
 in prod, and
- there's ample precedent in other middlewares for checking API version.
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: I8c342c4751ba3ca682efd152e90e396e9f8eb851

There is an infinite loop if multiple quota limits are set and exceeded,
eventually resulting in a 500 response due to a RecursionError ("maximum
recursion depth exceeded").
The issue is the delayed rejection, required to support container_acls.
If any quota is exceeded the middleware needs to return directly,
without proceeding to check other quota settings.
The fix is basically to add a "return self.app". However, there is quite
some redundant code, thus moving this into its own method.
Another test with multiple exceeded quotas has been added, which is
failing without the bugfix.
Closes-Bug: #2118758
Change-Id: I49ec4c5f6c83f36ce1d38f2f1687081c71488286
Signed-off-by: Christian Schwede <cschwede@redhat.com>

Previously, when the audit process detected gaps and/or overlaps in a
DB's shard ranges, it would log a warning that included a list of all
impacted shard ranges. The log message can grow long when there are
gaps or overlaps involving many shard ranges: so long that syslog
might raise an OSError (Message too long).
This patch shortens these log warning messages to only include a count
of the number of gaps and/or overlaps. The count may still be useful
to observe how a problem has developed over time. The detailed
information is better accessed using the swift-manage-shard-ranges
repair command.
Change-Id: I055c40395807708de60882f53652d9533a495d09
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

There's a bunch of moving pieces here:
- Add a new RingWriter class.
 Stick it in a new swift.common.ring.io module. You *can* use it like
 the old gzip file, but you can also define named sections which can
 be referenced later on read. Section names may be arbitrary strings,
 but the "swift/" prefix is reserved for upstream use. Sections must
 contain a single length-value encoded BLOB. If sections are used, an
 additional BLOB is written at the end containing a JSON section-index,
 followed by an uncompressed offset for the index.
 Move RingReader to ring/io.py, too.
- Clean up some ring metadata handling:
 - Drop MD5 tracking in RingReader. It was brittle at best anyway, and
 nothing uses it. YAGNI
 - Fix size/raw_size attributes when loading only metadata.
- Add the ability to seek within RingReaders, though you need to know
 what you're doing and only seek to flush points.
- Let RingBuilder objects change how wide their replica2part2dev_id
 arrays are. Add a dev_id_bytes key to serialized ring metadata.
 dev_id_bytes may be either 2 or 4, but 4 requires v2 rings. We
 considered allowing dev_id_bytes of 1, but dropped it as unnecessary
 complexity for a niche use case.
- swift-ring-builder version subcommand added, which takes a ring. This
 lets operators see the serialization format of a ring on disk:
 $ swift-ring-builder object.ring.gz version
 object.ring.gz: Serialization version: 2 (2-byte IDs), build version: 54
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: Ia0ac4ea2006d8965d7fdb6659d355c77386adb70

We stopped writing pickled rings more than twelve years ago. Any
cluster that was going to upgrade from then has, or can pick any of
the multitude of intermediary releases to pause at and push rings.
We can also safely assume that regions will be present for devices;
that change is nearly as old.
As a side-effect, clean up some old tests that did nonsense things
like having 7 assignments per row for a part-power-2 ring.
UpgradeImpact: remove ability to read rings written by swift <1.7.0
circa 2012
Related-Change: I799b9a4c894d54fb16592443904ac055b2638e2d
Related-Change: Ifefbb839cdcf033e6c9201fadca95224c7303a29
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: Ic8322b18d51b40f586cb217a0d1b2f345e1d8df6

Add anycrc as a soft dependency in case ISA-L isn't available.
Plus we'll want it later: when we start writing down checksums,
we'll need it to combine per-part checksums for MPUs.
Like with crc32c, we won't provide any pure-python version as the
CPU-intensiveness could present a DoS vector. Worst case, we 501
as before.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Change-Id: Ia05e5677a8ca89a62b142078abfb7371b1badd3f
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Assert that BadDigest responses due to checksum mismatch do not include
the expected or computed values.
Change-Id: Iaffa02c3c02fa3bc6922f51ecf28a39f4b24ccf2
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Adds a test that verifies extra body content beyond the content-length
is ignored provided that the checksum value matches that of the
content-length bytes.
Add comment to explain why this is the case.
Drive-by: add clarifying comment to unit test.
Change-Id: I8f198298a817be47223e2f45fbc48a6f393b3bef
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

See https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
for some background.
This covers both "normal" objects and part-uploads for MPUs. Note that
because we don't write down any client-provided checksums during
initiate-MPU calls, we can't do any verification during complete-MPU
calls.
crc64nvme checksums are not yet supported; clients attempting to use
them will get back 501s.
Adds crt as a boto3 extra to test-requirements. The extra lib provides
crc32c and crc64nvme checksum support in boto3.
Co-Authored-By: Ashwin Nair <ashnair@nvidia.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com>
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Id39fd71bc59875a5b88d1d012542136acf880019

S3 includes the expected base64 digest in a BadDigest response to a
multipart complete POST request.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ie20ccf10846854f375c29be1b0b00b8eaacc9afa

Change-Id: I88e2e743ccbd6292bc1570ae0efbdd45dcced8cc

Change-Id: I80142c6c0654b65b5755e7e828bcc4969a10f4f1

datetime.timezone.utc[1] has been available in Python 3 and can be used
instead of datetime.UTC which is available only in Python >=3.11 .
[1] https://docs.python.org/3.13/library/datetime.html#datetime.timezone.utc
Change-Id: I92bc82a1b7e2bcb947376bc4d96fc603ad7d5b6c

... because Python 2.x is no longer supported.
Change-Id: I3167a539b3e26ceb35976fbd7a2356ba59d4a5e4

Change-Id: I32fec7540bee70e77140964c5983d133a572fa7b