openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Address an issue where `OldestAsyncManager` instances created before forking resulted in each child process maintaining its own isolated copy-on-write stats, leaving the parent process with an empty/unused instance. This caused the final `dump_recon` call at the end of `run_forever` to report no meaningful telemetry.
The fix aggregates per-disk recon stats collected by each child process. This is done by loading recon cache data from all devices, consolidating key metrics, and writing the aggregated stats back to the recon cache.
Change-Id: I70a60ae280e4fccc04ff5e7df9e62b18d916421e

- Define a single location for doc-build deps
 - As a side-effect, use constraints for api-ref builds
- Remove test-requirements.txt from those deps
Change-Id: If6cc8702e89f5110ad89ba933f55641de02550e9

... instead of self-handling subrequests manually.
Closes-Bug: #1523401
Change-Id: I85b5302c2416de1793599385b791fcd3ec3b4da0

Declare the tests to skip, rather than the tests to run. This ensures
that we pick up new bandit checks automatically.
I recently noticed a use of md5() without the usedforsecurity=False
kwarg. Confused about why this wasn't caught in the gate, I eventually
traced it back to B303 (which we explicitly enabled) being largely
superseded by B324 (which did not exist when we wrote down the tests
to enable).
Flag a bunch of false-positives with "# nosec" comments, resolve two
other errors, and skip some more-pervasive errors, to be resolved later.
Change-Id: Ia054e4f7c9e5bf29064a66933e27830adbc107d3

Change-Id: I1e9d4d5df403040d69db93a08647cd0abe1b8037

All we need is int(). Using eval() on user-provided data (or really at
all) is a Bad Idea.
Closes-Bug: #2091124
Change-Id: I39bb87f9d8e27f2f88410a087a120a0e9be1a243

Add some commentary as a reminder that whilst normal object updates to
the ContainerBroker cannot have content-type timestamp older than data
timestamp, expirer queue updates can.
Change-Id: I6d8ad06c645f25497dc15173460430fd93747afa
Related-Change: Ie4b25f1bd16def4069878983049b83de06f68e54

Related-Change: Iba43783e880e0860357ba8b9f0a11f28abf87555
Change-Id: I1e096dab9a97956bf786ccbcd37c20f9a3a5429e

The UPPER_CONSTRAINTS_FILE var has been deprecated for years.
Related-Change: I3f957187ed4f29fcf88db99cb79fb33b21a8dd8d
Change-Id: Id6201036cc619d050f86ae9c9c420388808f8b89

Previously, the OldestAsyncPendingTracker timestamp was stored as a string. This change updates it to be stored as a float.
UpgradeImpact: This will require an additional change to the recon parsers in order to process the timestamp as a float.
Change-Id: Iba43783e880e0860357ba8b9f0a11f28abf87555

Change-Id: I9473fad7c46ac03bbc71328c17e988af9d21386c

The expectation is that CI and devs would run these checks as part
of the 'pep8' environment.
Change-Id: Ib9b2e34d520baee13c4b38bc220ec7b4d2e38433

The stable branch is gone now.
Change-Id: Ib0a62256584a5c7d458ebe6740f0f477f61dd23a

Purely version bumps, and dropping the swift pin (which was added
erroneously anyway).
Change-Id: I1cbd26e6a18e01abcfb2f0a28649cfcaa6358252

Change-Id: I0fe476c56c120f8a6dab6afa2cf03e8278ceefc5

Change-Id: Ia03af1680c6230658473c0c8d444efb5bb805f58

Change-Id: Ib8196fe24f8a999315af469a435bd639378c78a9

Change-Id: Ib868258b224d97ebd1c14d5b5b98bb0718caaea9

Our API ref says "Valid values are json, xml, or plain. The default is
plain." There's no reason our tests ought to use an invalid value; it
will confuse anyone looking at tests for how to do a thing.
Note that tests were passing because invalid values are ignored, so
?format=txt behaves exactly like ?format=plain.
Change-Id: I6e119cc9c7297d8aade9736fa1d6f4a105466d77

This has been causing most/all of our tox-based jobs to run with
unconstrained dependencies (though test requirements were properly
pinned).
Change-Id: Icf1b63e39ec321887d9fe6afe7b212c813ad1d1e

Currently, when object-server serves PUT request and DiskFile
writer write file chunks to disk, there is no explicit
eventlet sleep called. When network outpace the slow disk IO,
it's possible one large and slow PUT request could cause
eventlet hub not to schedule any other green threads for a
long period of time. To improve this, this patch enable the
configurable yield parameter 'cooperative_period' into object
server controller write path.
Related-Change: I80b04bad0601b6cd6caef35498f89d4ba70a4fd4
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I1c0aba9830433f093d024b4c39cd3a3b2f0d69f1

We would like to remove any Statsd related code from logs.py. That
requires that SwiftLogAdapter no longer provides a StatsdClient
interface by default. However, for backwards compatibility the main
utils.get_logger/get_prefixed_logger entrypoints must provide a
SwiftLogAdapter that does have a StatdsClient interface.
The new utils._patch_statsd_methods helper function is therefore used
to retrospectively patch a SwiftLogAdapter instance with the
StatsdClient interface when necessary. The _patch_statsd_helper is
used in get_logger, and again when we clone a logger in
get_prefixed_logger.
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: I44694b92264066ca427bb96456d6f944e09b31c0

Verify that the reconciler is able to move objects to a container that
has, or will have, versioning enabled, and versions of the moved object
can subsequently be created.
Change-Id: I019447614ebadbb9e2cc8a18c0369bc16a89c0d9

Change-Id: I5c9af73cc66e17dc662d8c4d97cbf71ded1fed6f

A failing CORS test in the gate discovered that, when running with
eventlet==0.38.0, container and account HEAD requests returned
Content-Type of application/json to clients regardless of the requested
format. This was due to the backend HEAD response no longer having a
Content-Length header, causing the listing_formats middleware to not
modify the returned Content-Type (see Related-Change).
The Related-Change fixed the client facing issue by making
listing_formats middleware ensure the correct Content-Type is returned
to clients even when Content-Length is absent in the backend
response. The Related-Change also ensured that the 204 response to
clients always has a Content-Length header.
This patch directly fixes the problem of backend account and container
server HEADs no longer having 'Content-Length: 0' by adding it
explicitly. This violates the RFC prohibition of a 204 response having
a Content-Length header [1], but preserves Swift's historic behavior
and is consistent with the proxy-server's 204 response to clients.
[1] https://httpwg.org/specs/rfc7230.html#header.content-length
Related-Change: If724485e1425d1481d10b9255436301e346f07e8
Change-Id: Idacc59c5f43367926eff5221ee7fc417a9bc2d50

Update the 2023.1 release notes configuration to build from
unmaintained/2023.1.
Change-Id: I3e0962570145adb672bdba71f6de8f3da034b859

A failing CORS test in the gate discovered that we were responding
application/json to ?format=txt requests (which is maybe not even a
valid value for that qs param?), but only when running with
eventlet==0.38.0
This avoids the problem of backend container server HEADs no longer
having 'Content-Length: 0' by fixing the client HEAD resp headers before
we check for chunked-transfer resp.
Drive-By: refactor listing_formats to use HeaderKeyDict and always set
Content-Length explicitly
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: If724485e1425d1481d10b9255436301e346f07e8