openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Closes-Bug: #1674543
Change-Id: Ic74dbcaf6d8293ae41984d5cd61f0326c91988e2

Change-Id: I9a793e4c352cd40a073326315a6ae144a14de1e0

Make encrypter unit test assertions more explicit by using assertions
on the named attributes of FakeSwiftCall rather than assertions on
position-indexed call tuples.
Change-Id: I871ddcc4ba559e7e4c0d0e28464780c6cd669797

The encrypter middleware uses an update_footers callback to send
request footers. Previously, FakeSwift combined footers with captured
request headers in a single dict. Tests could not therefore
specifically assert that *footers* had been captured rather than
headers.
This patch modifies FakeSwift to capture footers separately for each
request. Footers are still merged with the request headers in order to
synthesise GET or HEAD response headers when a previously uploaded
object is returned.
Unfortunately the change cannot be as simple as adding another
attribute to the FakeSwiftCall namedtuple. A list of these namedtuples
is returned by FakeSwift.calls_with_headers. Some tests cast the
namedtuples to 3-tuples and will break if the length of the namedtuple
changes. Other tests access the attributes of the namedtuples by name
and will break if the list values are changed to plain 3-tuples.
Some test churn is therefore inevitable:
* FakeSwiftCall is changed from a namedtuple to a class. This prevents
 future tests assuming it is a fixed length tuple. It also supports a
 headers_and_footers property to return the combination of uploaded
 headers and footer that was previously (confusingly) returned by
 FakeSwiftCall.headers.
* A new property FakeSwift.call_list has been added which returns a
 list of FakeSwiftCalls.
* FakeSwift.calls_with_headers now returns a 3-tuple. Tests that
 previously assumed this was a namedtuple have been changed to use
 FakeSwift.call_list instead, which gives them objects with the same
 named attributes as the previous namedtuple. Tests that previously
 treated the namedtuple as a 3-tuple do not need to be changed.
* Tests that access the 'private' FakeSwift._calls have been changed
 to use FakeSwift.call_list.
Change-Id: If24b6fa50f1d67a7bbbf9a1794c70d37c41971f7

The following configuration options are deprecated:
 * expiring_objects_container_divisor
 * expiring_objects_account_name
The upstream maintainers are not aware of any clusters where these have
been configured to non-default values.
UpgradeImpact:
Operators are encouraged to remove their "container_divisor" setting and
use the default value of 86400.
If a cluster was deployed with a non-standard "account_name", operators
should remove the option from all configs so they are using a supported
configuration going forward, but will need to deploy stand-alone expirer
processes with legacy expirer config to clean-up old expiration tasks
from the previously configured account name.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Jianjian Huo <jhuo@nvidia.com>
Change-Id: I5ea9e6dc8b44c8c5f55837debe24dd76be7d6248

Add the storage_policy attribute to the metadata returned
when listing containers using the GET account API function.
The storage policy of a container is a very useful attribute
for telemetry and billing purposes, as it determines the location
and method/redundancy of on-disk storage for the objects in the
container. Ceilometer currently cannot define the storage policy as a
metadata attribute in Gnocchi as GET account, the most efficient way
of discovering all containers in an account, does not return the
storage policy for each container.
Returning the storage policy for each container in GET account
is the ideal way of resolving this issue, as it allows Ceilometer
to find all containers' storage policies without performing additional
costly API calls.
Special care has been taken to ensure the change is backwards
compatible when migrating from pre-storage policy versions
of Swift, even though those versions are quite old now.
This special handling can be removed if support for migrating
from older versions is discontinued.
Closes-bug: #2097074
Change-Id: I52b37cfa49cac8675f5087bcbcfe18db0b46d887

Previously, test_object_move_no_such_object_no_tombstone_ancient
would fail intermittently, with an assertion that two timestamps
were almost (but not quite) equal.
This probably comes down to the fact that it's passing floats as
timestamps down into FakeInternalClient's parse(); specifically,
values like 1738046018.2900746 and 1738045066.1442454 are known
to previously fail.
Just fixing the usage doesn't fix the foot-gun, though -- so fix
up parse() to be internally consistent, even if passed a float.
Change-Id: Ide1271dc4ef54b64d2dc99ef658e8340abb0b6ce

Related-Change: I8227939d04fda8db66fb2f131f2c71ce8741c7d9
Change-Id: I149a2df2d942bba02049947865b000c9cf1a89bc

Add a new tunable, `stale_worker_timeout`, defaulting to 86400 (i.e. 24
hours). Once this time elapses following a reload, the manager process
will issue SIGKILLs to any remaining stale workers.
This gives operators a way to configure a limit for how long old code
and configs may still be running in their cluster.
To enable this, the temporary reload child (which waits for the reload
to complete then closes the accept socket on all the old workers) has
grown the ability to send state to the re-exec'ed manager. Currently,
this is limited to just the set of pre-re-exec child PIDs and their
reload times, though it was designed to be reasonably extensible.
This allows the new manager to recognize stale workers as they exit
instead of logging
 Ignoring wait() result from unknown PID ...
With the improved knowledge of subprocesses, we can kick the log level
for the above message up from info to warning; we no longer expect it
to trigger in practice.
Drive-by: Add logging to ServersPerPortStrategy.register_worker_exit
that's comparable to what WorkersStrategy does.
Change-Id: I8227939d04fda8db66fb2f131f2c71ce8741c7d9

... just like we would do in a normal container. Previously, we'd try to
read a byte from the client which, due to a bug in eventlet HTTP framing,
would either hang until we hit a timeout or worse read from the next
pipelined request.
This required that we reset a (repeatedly-reused!) request in s3api to
have an empty body, or it would start triggering 411s, too.
See also: https://github.com/eventlet/eventlet/pull/985
Closes-Bug: #2081103
Change-Id: I56c1ecc4edb953c0bade8744e4bed584099f29c7

The available bandit tests change with time (e.g. the
Related-Change). We shouldn't try to maintain the list.
Related-Change: Ie668d49a56c0a6542d28128656cfd44f7c089ec4
Change-Id: I6eb106abbac28ffbb9a3f64e8aa60218cbe75682

They were removed upstream recently, so now Bandit is complaining about
the unknown test.
See https://github.com/PyCQA/bandit/pull/1212
Change-Id: Ie668d49a56c0a6542d28128656cfd44f7c089ec4

Change-Id: Ic66b9ae89837afe31929ce07cc625dfc28314ea3

Historically, the object-server would validate the ETag of an object
whenever it was streaming the complete object. This minimizes the
possibility of returning corrupted data to clients, but
- Clients that only ever make ranged requests get no benefit and
- MD5 can be rather CPU-intensive; this is especially noticeable
 in all-flash clusters/policies where Swift is not disk-constrained.
Add a new `etag_validate_pct` option to tune down this validation.
This takes values from 100 (default; all whole-object downloads are
validated) down to 0 (none are).
Note that even with etag validation turned off, the object-auditor
should eventually detect and quarantine corrupted objects. However,
transient read errors may cause clients to download corrupted data.
Hat-tip to Jianjian for all the profiling work!
Co-Authored-By: Jianjian Huo <jhuo@nvidia.com>
Change-Id: Iae48e8db642f6772114c0ae7c6bdd9c653cd035b

Add test for mutilated statsd client
Drive-by: revert unncesessary whitespace change in Related-Change.
Related-Change: I3a677bb67c5700da48f89c847f652b4610ba47c2
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Co-Authored-By: Clay Gerrard <cgerrard@nvidia.com>
Change-Id: Id262bbcf0b233f9728f55be208bee5bc146c053d

Change-Id: I14ea6bbd55512e8798e5cadf4f6e68f95961206b

The Related-Changes moved logging and statsd components into new
modules and decoupled statsd from the logs module. This patch attempts
to re-locate the related unit tests to appropriate modules.
* tests for utils functions such as get_logger should be in
 test_utils.py.
* tests for log functions such as get_swift_logger should be in
 test_logs.py.
* tests for statsd client functions should be in
 test_statsd_client.py.
* tests related to patching a SwiftLogAdapter with a StatsdClient
 interface should be in test_utils.py.
Change-Id: I3a677bb67c5700da48f89c847f652b4610ba47c2
Related-Change: I44694b92264066ca427bb96456d6f944e09b31c0
Related-Change: I8988c0add6bb4a65cc8be38f0bf527f141aac48a
Related-Change: Ie73988edf6be0e38d9004bee04ff46c906a759ff
Related-Change: I5ae2cc5c257fb8d7eab885977d9d9cf602224ec7
Related-Change: I4b5b12a3b0288b696a39903264741bc862a94ad7
Related-Change: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a

It's confusing and unecessary to have the last cycles per-device
object-updater stats reaped from recon immediately during aggregation
and make it impossible to debug stats aggregation.
Drive-by: fix some bugs with stats aggregation
Change-Id: I9df7c2d1c31646a3200614b629598576eb9e64c0

Previously, we were relying on some xfs-specific return order.
Change-Id: If9a0fdb3749a18a9479f20fb174e0c1908a783bb

Change-Id: Ia028624bded221c3bf03a8d3dac94183d4388431

This patch removes swift-multinode-rolling-upgrade-victoria job for
multiple reason:
- victoria is very old and unmaintained
- job is defined only on unmaintained branches
- py38 + CentOS Stream 8 are EOL'd and the job is based on these
Change-Id: I3d6a679e6553534e937303b5210125a6ef8365af

* Remove py2 gate jobs.
* Build non-universal, py3-only wheels.
* Specify minimum python version in package metadata.
* Clean up requirements/constraints/bindep (a little, anyway).
Change-Id: I53153c4fde043e964e1daa7bbf2089e0471dede2

Changed OS version from RHEL 7 and CentOS 7 to RHEL 9 and
CentOS Stream 9.
Changed python to python3.
Changed yum command to dnf command.
Change-Id: Ie1e815c0434255e77ef5e9103576f85d9d6490ae

Previously, it was possible for updater_workers to be a negative integer
or zero. This change enforces that updater_workers should be a positive
integer.
Change-Id: Ie40194b406aeedcf8c38a3c273ab768e2b643a5d

Change-Id: I22674f2e887bdeeffe325efd2898fb90faa4235f

H106 and H904 were already passing anyway.
Change-Id: Ic386e09e40a49b661f30ea40e2c737d59100d086

Address an issue where `OldestAsyncManager` instances created before forking resulted in each child process maintaining its own isolated copy-on-write stats, leaving the parent process with an empty/unused instance. This caused the final `dump_recon` call at the end of `run_forever` to report no meaningful telemetry.
The fix aggregates per-disk recon stats collected by each child process. This is done by loading recon cache data from all devices, consolidating key metrics, and writing the aggregated stats back to the recon cache.
Change-Id: I70a60ae280e4fccc04ff5e7df9e62b18d916421e