openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I80142c6c0654b65b5755e7e828bcc4969a10f4f1

datetime.timezone.utc[1] has been available in Python 3 and can be used
instead of datetime.UTC which is available only in Python >=3.11 .
[1] https://docs.python.org/3.13/library/datetime.html#datetime.timezone.utc
Change-Id: I92bc82a1b7e2bcb947376bc4d96fc603ad7d5b6c

... because Python 2.x is no longer supported.
Change-Id: I3167a539b3e26ceb35976fbd7a2356ba59d4a5e4

Change-Id: I32fec7540bee70e77140964c5983d133a572fa7b

It used to be that when a logger encountered an error trying to handle
/ emit a message, it would attempt to log about the error via the same
logger. Long ago, this could lead to infinite recursion in Swift's
LoggerFileObject, so we fixed it and added a test that included
showing the bad stdlib behavior.
Recently, stdlib did a similar fix so the recursion doesn't happen. See
 - https://github.com/python/cpython/issues/91555 and
 - https://github.com/python/cpython/pull/131812
Of course, now our test breaks (at least, on 3.13.4). Relax the
assertion a little.
Related-Change: Ia9ecffc88ce43616977e141498e5ee404f2c29c4
Change-Id: Id2f490d4204b8eaf07857cb84ed783bec19b8511

Swift does not return all the parameters of objects in a listing
(e.g. ChecksumType and ChecksumAlgorithm) so pop these from listings
before making assertions.
Change-Id: Ieb7a9783731c11f1c08db398eae07ffafa127460

The labeled timing metrics previously have sample_rate similar to
those added for non-labeled metrics. However the down sampling has
not been helpful when using labeled metrics to investigate customer
issues, for example those related to object server REPLICATE requests.
This patch changes labeled timing metrics to not have sample_rate. The
non-labeled metrics are unrelated to this effort thus not changed.
Related-Change: I05336b700120ab5fcf922590d6a12f73112edb50
Change-Id: Ia6e856ffaf8fd1b4a905e6976ebdc62ed5ddf32f

Previously (at least on OSX) this could lead to errors like
 socket.gaierror: [Errno 8] nodename nor servname provided, or not known
when we actually went and tried to connect to send the update (!!)
Change-Id: I86f6e731d1ee273c6772974ce597ac91be3937be

Change-Id: I0a27efe897b4e8ce2c21da1a3603a2a77c02eb69

Locally, this reduced the test time from 240s to 0.16s.
Change-Id: I5a4786b6782c06f8e6bd9fab5d4dae683a970242

Change-Id: I05336b700120ab5fcf922590d6a12f73112edb50

Change-Id: Id5583e3a1e4515ec3c8a972f647aaaabfba673bc
Related-Change: I2e57dacb342b5758f16b502bb91372a2443d0182

Modified code to use native labeled metrics API introduced by the
Related-Change.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: I115ffb1dc601652a979895d7944e011b951a91c1
Change-Id: I5a96626862644fe23a0d4bc53e85989770c8ceec

When a DiskFile is opened, its on-disk files are first listed and then
xattr metadata is read. If a listed file no longer exists when its
metadata is read then a DiskFileNotExist exception was raised which
would previously cause the object server to return a 404.
It isn't appropriate to return 404 when the list of on-disk files
suggested that the object existed (i.e. included a .data file) and
there is no evidence that the object no longer exists: the missing file
may well have been *replaced* by a concurrent request.
For example, if a POST request races with another concurrent POST
request, the object's .meta file may be replaced by a new .meta file,
and reading the original .meta file will fail. Similarly, if a POST
races with a concurrent PUT request, the PUT request may replace the
.data file causing the POST request handler to fail to read the
original .data file. In neither case has the object ever been
non-existent.
This issue was observed during a period of many concurrent POST
requests to an object in a production system. A significant number of
the POSTs returned 404 despite the object never being deleted.
This patch modifies DiskFile to raise a new DiskFileStateChanged
exception when metadata cannot be read from a data, meta, or ts file
that was listed in the object's datadir. The object server may translate
this exception to a 503 response rather than a 404, depending on the
request method:
 * POSTs, GETs and HEADs that encounter this transient loss of a
 file will now return 503 rather than 404. The DiskFile may
 still exist.
 * DELETEs that encounter this transient loss of a file would
 previously have proceeded but will now return 503: a replacement
 file may contain an X-Delete-At that would have prevented the
 DELETE proceeding.
 * PUTs that encounter this transient loss of a file will continue to
 proceed as before: a replacement meta file may contain an
 X-Delete-At that requires updates to the expirer queue but those
 updates would presumably be handled by the concurrent POST that
 has replaced the meta file; a replacement .data file will remain
 on disk until the DiskFile is next opened and one of the .data
 files will be cleaned up.
Closes-Bug: #2054791
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I2f698c25ed65b236e851e5a307d48a12cef62b33

Swift already supports that much, at least. AWS used to not support any
conditional PUTs, but that's changed somewhat recently; see
- https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/
- https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/
Drive-By: Fix retry of a CompleteMultipartUpload with changed parts; it
should 404 rather than succeed in writing the new manifest.
Change-Id: I2e57dacb342b5758f16b502bb91372a2443d0182

The timing stats decorators are moved from utils to base_storage_server.py
to avoid a circular import of HTTPException.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Idc4b52de0a04ebfc0e353162bd791d4e0e20eac3

Some tests previously made assertions on statsd client methods that
were implemented by a delegated interface patched onto a logger
instance. Those tests have run against a StatsdClient since the
Related Change, so the use of the term 'delegate' in tests no longer
makes sense.
Change-Id: Ie47fa50a226da6ff7b067d6a367758ac9f12ef4c
Related-Change: I3a677bb67c5700da48f89c847f652b4610ba47c2

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I811642fccd916bd9ef71846a8108d50a462740f0

A new CRCHasher class wraps crc functions to mimic the hashlib
hasher interface.
For crc32, use the zlib implementation.
For crc32c, use ISA-L if we can find it; on py38+ this includes the
ISA-L bundled in pyeclib's binary wheels.
If ISA-L is not available, check for kernel crypto API support for crc32c;
use it if available. Otherwise, raise a NotImplementedError.
No callers yet, but these will be used by s3api in a later patch.
Change-Id: Ic0c55e307ce10b56b569c9fee728c445a2300cbd

This update implements Sigv4-streaming (chunked upload) as described in
the Amazon S3 documentation:
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
Closes-Bug: #1810026
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: ashnair <ashnair@nvidia.com>
Change-Id: I7be1ce9eb5dba7b17bdf3e53b0d05d25ac0a05b0

This will prevent client timeouts during large object copies, similar to existing support in
bulk operations and SLO manifest writes. The operator can set the frequency using the
yield_frequency inside conf file.
To use heartbeat, simply add "heartbeat=on" query parameter, which will:
- Returns a 202 Accepted response
- Sends periodic 1 byte whitespace heartbeat every yield_frequency
- Returns final status and headers
- Supports both plain text and JSON
Closes-Bug: #1738306
Change-Id: Iaf4355f2e0edbd6dde226231f3d7a61a5e53ecfa

Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: I155572bc5f4d955c918c951d6d2af9dbbae0136d

Make all statements using all caps (WARNING) and modify with print where there is stderr.write.
Fixes bug 1511139
Change-Id: I3b24c55d6ea07d9c2254f15a490fa336ad64c6c4

Tempauth fernet tokens use a secret shared among all proxies to encrypt
user group information. Because they are encrypted, clients can neither
view nor edit this information; it is an opaque bearer token similar to
the existing memcached-backed tokens (just much longer). Note that
tokens still expire after the configured token_life.
Add a new set of config options of the form
 fernet_key_<keyid> = <32 url-safe base64-encoded bytes>
Any of the configured keys will be used to attempt to decrypt tokens
starting with "ftk" and extract group information.
Another new config option
 active_fernet_key_id = <keyid>
dictates which key should be used when minting tokens. Such tokens
will start with "ftk" to distinguish them from memcached-backed tokens
(which continue to start with "tk"). If active_fernet_key_id is not
configured, memcached-backed tokens continue to be used.
Together, these allow seamless transitions from memcached-backed tokens
to fernet tokens, as well as transitions from one fernet key to another:
 1. Add a new fernet_key_<keyid> entry.
 2. Ensure all proxies have the new config with fernet_key_<keyid>.
 3. Set active_fernet_key_id = <keyid>.
 4. Ensure all proxies have the new config with the new
 active_fernet_key_id.
This is similar to the key-rotation process for the encryption feature,
except that old keys may be pruned following a token_life period.
Additionally, opportunistically compress groups before minting tokens.
Compressed tokens will begin with "zftk" but otherwise behave just like
"ftk" tokens.
Change-Id: I0bdc98765d05e91f872ef39d4722f91711a5641f

... so that eventlet.wsgi will include Connection: close headers on
Expect: 100-continue error responses from s3api and make aws clients
less unhappy.
eventlet.wsgi likes to check for hasattr('__len__', resp) when setting
connection close on expect-100 errors, but uses 'content-length' in
headers when deciding on chunked-transfer.
When we know the length we can support either interface. Also we can
imporove s3api to return error responses with the content-length known.
Change-Id: Ic504841714bd607cb9733b2de5126092a79c1094

Change-Id: I60f0b36633c2a348933fd45d348d76b256fca57a

Previously the SigV4Mixin would override S3Request signature checking
methods. This patch refactors the signature checking into sigv2 and
sigv4 helper classes i.e. moves towards composition rather than
inheritance.
No behavioural changes are intended with this patch.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thibault Person <thibault.person@ovhcloud.com>
Change-Id: Icaf86181c3bf7804c5176db48d2de5e2fc6f24d2

The method is redundant and confusingly named. Since the
Related-Change it actually returns the aggregate value of counters,
not just the count of calls made to the increment method alone.
Related-Change: I4d8a4b530465b587caced4362eb9178507701cef
Change-Id: I52585567a177f58e0f110785b44f8d238b2ac54d

Both the StatsdClient and LabeledStatsdClient implemented all of the
public interface methods for an AbstractStatsdClient. Even though the
implementations are identical the interfaces themselves were made
different WRT the "last" positional arg if provided.
This change puts the identical implementations of the public interface
in the AbstractStatsdClient class with a generic interface. For
backwards compat with the existing public interface on the
legacy-StatsdClient we have to make the old signature explicit - but we
can transparently call into the common implementation.
For consistency with legacy-Statsdclient and better docstrings this
change also "fixes" the signature for the new LabeledStatsdClient public
interface methods so they explicitly state labels must always be passed
in as a kwarg to avoid confusion accidentally passing in
`sample_rate=<dict>` or `label=<float>` as a positional arg.
Drive-by: as a side-effect of consolidating the implementations there
was now only one place to extend a fix in the FakeStatedClient interface
so that tests who assert StatsdClient method calls don't have to read
like the UUT also called some *other* public StatsdClient method; which
also cleaned up some Fake helpers that deal specifically with counters.
Drive-by: fix stub return value TypeError in a _get_hashes mock that was
caught by better faking. N.B. it doesn't matter how statsd would have
handled 'metric:None|c' because it was only a test bug.
Change-Id: I4d8a4b530465b587caced4362eb9178507701cef