openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Related-Change: I86bcd7a2373cf1e7213cb716888307306b7bb50d
Change-Id: I0056bfc78f9c89b1c458943ee446499d21667fac

Change-Id: I9044e49ff30b092a97b75fa7fa133016d17cde37

Previously, we weren't converting it to a boolean, but left it as
a truth-y string.
Change-Id: Ia0349d991435492ac9d80c36c5b1123b9e310db6

... rather than importing it before it's been read in setup_package and
never getting any updated value.
Related-Change: I699d664bcef6754aac4ec6877830e745b764342f
Change-Id: I48faabc5b59f94a47dfeb871f13ddbcbaa20d349

Change-Id: Ibd3d5f83fcd31d78447a894ff4426e3e1b62e5a8

With py311 we started seeing warnings like
 DeprecationWarning: 'cgi' is deprecated and slated for removal
 in Python 3.13
The recommended replacement isn't quite up to snuff (doesn't handle
multiple parameters), but we already have a reasonably close replacement.
Add a "loose" mode to parse_content_type to make it suitable when there
may be a slash in a parameter token. Add a new utils.parse_header function
that takes advantage of the parse_content_type "loose" mode.
Closes-Bug: #2084472
Change-Id: Ie281ff90796f2d68840952c95669f264480b1b4c

We need to parse additional headers earlier, before stdlib tries to
establish message framing.
Now, TestReconstructorRebuildUTF8 can pass on py3.
Closes-Bug: #2097030
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Related-Change: https://review.opendev.org/c/openstack/swift/+/662546
Change-Id: I6aa16fda9285c9fc3816da6fbff2615bd14a020c

Change-Id: Iae07ec8d38cd747f8bfffa3c08b423bee4b6623c

There's a few places where bespoke file-like wrapper classes have been
implemented. The common methods are now inherited from
utils.InputProxy.
Make utils.FileLikeIter tolerate size=None to mean the same as size=-1
so that it is consistent with the behavior of other input streams.
Fix docstrings in FileLikeIter.
Depends-On: https://review.opendev.org/c/openstack/requirements/+/942845
Change-Id: I20741ab58b0933390dc4679c3e6b2d888857d577

Related-Change-Id: Ia1d8d9085bad5c6df744a4551eef9dcf56e6e261
Change-Id: I6c5bb824d7145c25faf37fd1fa575cedf05e499a

Change-Id: Ie59f11874daf9f166a699fa904581830942163a9
Related-Change: I02efba463a8263ca44be511c025c6c6bfbc57334

Change-Id: Ia754021714a8ed3d5d6010a946c558552821fbe4

It's always called with sha256 and raises sha256-specific errors;
we don't need to pretend to support arbitrary hashes.
Change-Id: Icff79ded067084249080e3e6f555429261eb0af0

We already prefix everything with "s3api-test-"; there's no reason to
double up the "test-" part.
Change-Id: I02efba463a8263ca44be511c025c6c6bfbc57334

Specifically, allow endpoint and region to be configured. This allows
developers to compare boto3 behaviors with TLS enabled/disabled, for
example, or AWS behaviors between different regions.
Change-Id: I4113e2fd47e5535eec8bd9487884af077e8b0318

We shouldn't allow a test to assume a strict coupling of
client/server time and the test needs to be modified so that
the functional test-suite pass for us.
Change-Id: Ia1d8d9085bad5c6df744a4551eef9dcf56e6e261

...at least, provided the client sent a X-Amz-Content-SHA256 header.
Apparently Content-MD5 is no longer strictly required by AWS? Or maybe
it never was, provided the client sent a SHA256 of the content.
This also allows us to test with newer boto3, botocore, s3transfer.
Related-Bug: #2098529
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ifbcde9820bee72d80cab0fe3e67ea0f5817df949

Depends-On: https://review.opendev.org/c/openstack/requirements/+/941708
Change-Id: I52f40bf0468236dc7ba78e08f2f9aea9336faae9

Also:
- Couple small clean-ups in account.backend and account.utils
- Make a couple test assertions more useful.
Related-Change: https://review.opendev.org/c/openstack/swift/+/940601
Change-Id: Ic14642df50592c982adfb55a0c6cfd673a5a95b8

This implementation uses abstract sockets for process notifications,
similar to systemd's notify sockets, but notifiers use a PID-specific
name from a well-known namespace and listeners are assumed to be
ephemeral.
Update swift-reload to use these instead of polling child processes to
determine when a server reload has completed. Bonus: it also acts as a
non-blocking lock to prevent two swift-reload commands from reloading a
process at the same time.
Closes-Bug: #2098405
Related-Change: Ib2dd9513d3bb7c7686e6fa35485317bbad915876
Change-Id: I5f36aba583650bddddff5e55ac557302d023ea1b

Capture a copy of the request environ as part of FakeSwiftCall and use
that to query captured txn_ids, sources rather than maintaining
separate lists for each request property.
Store captured request body as an attribute of FakeSwiftCall to
similarly avoid maintaining another list. The request body is read
after the FakeSwiftCall has been captured in case there is an error
while reading.
Drive-by: drop footers arg from FakeSwiftCall constructor since it is
never used, nor should it be used.
Related-Change: If24b6fa50f1d67a7bbbf9a1794c70d37c41971f7
Change-Id: I332ce724aa10287800cbec8ca21aacc3bbd3c22a

Closes-Bug: #1674543
Change-Id: Ic74dbcaf6d8293ae41984d5cd61f0326c91988e2

Change-Id: I9a793e4c352cd40a073326315a6ae144a14de1e0

Make encrypter unit test assertions more explicit by using assertions
on the named attributes of FakeSwiftCall rather than assertions on
position-indexed call tuples.
Change-Id: I871ddcc4ba559e7e4c0d0e28464780c6cd669797

The encrypter middleware uses an update_footers callback to send
request footers. Previously, FakeSwift combined footers with captured
request headers in a single dict. Tests could not therefore
specifically assert that *footers* had been captured rather than
headers.
This patch modifies FakeSwift to capture footers separately for each
request. Footers are still merged with the request headers in order to
synthesise GET or HEAD response headers when a previously uploaded
object is returned.
Unfortunately the change cannot be as simple as adding another
attribute to the FakeSwiftCall namedtuple. A list of these namedtuples
is returned by FakeSwift.calls_with_headers. Some tests cast the
namedtuples to 3-tuples and will break if the length of the namedtuple
changes. Other tests access the attributes of the namedtuples by name
and will break if the list values are changed to plain 3-tuples.
Some test churn is therefore inevitable:
* FakeSwiftCall is changed from a namedtuple to a class. This prevents
 future tests assuming it is a fixed length tuple. It also supports a
 headers_and_footers property to return the combination of uploaded
 headers and footer that was previously (confusingly) returned by
 FakeSwiftCall.headers.
* A new property FakeSwift.call_list has been added which returns a
 list of FakeSwiftCalls.
* FakeSwift.calls_with_headers now returns a 3-tuple. Tests that
 previously assumed this was a namedtuple have been changed to use
 FakeSwift.call_list instead, which gives them objects with the same
 named attributes as the previous namedtuple. Tests that previously
 treated the namedtuple as a 3-tuple do not need to be changed.
* Tests that access the 'private' FakeSwift._calls have been changed
 to use FakeSwift.call_list.
Change-Id: If24b6fa50f1d67a7bbbf9a1794c70d37c41971f7

The following configuration options are deprecated:
 * expiring_objects_container_divisor
 * expiring_objects_account_name
The upstream maintainers are not aware of any clusters where these have
been configured to non-default values.
UpgradeImpact:
Operators are encouraged to remove their "container_divisor" setting and
use the default value of 86400.
If a cluster was deployed with a non-standard "account_name", operators
should remove the option from all configs so they are using a supported
configuration going forward, but will need to deploy stand-alone expirer
processes with legacy expirer config to clean-up old expiration tasks
from the previously configured account name.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Jianjian Huo <jhuo@nvidia.com>
Change-Id: I5ea9e6dc8b44c8c5f55837debe24dd76be7d6248

Add the storage_policy attribute to the metadata returned
when listing containers using the GET account API function.
The storage policy of a container is a very useful attribute
for telemetry and billing purposes, as it determines the location
and method/redundancy of on-disk storage for the objects in the
container. Ceilometer currently cannot define the storage policy as a
metadata attribute in Gnocchi as GET account, the most efficient way
of discovering all containers in an account, does not return the
storage policy for each container.
Returning the storage policy for each container in GET account
is the ideal way of resolving this issue, as it allows Ceilometer
to find all containers' storage policies without performing additional
costly API calls.
Special care has been taken to ensure the change is backwards
compatible when migrating from pre-storage policy versions
of Swift, even though those versions are quite old now.
This special handling can be removed if support for migrating
from older versions is discontinued.
Closes-bug: #2097074
Change-Id: I52b37cfa49cac8675f5087bcbcfe18db0b46d887

Previously, test_object_move_no_such_object_no_tombstone_ancient
would fail intermittently, with an assertion that two timestamps
were almost (but not quite) equal.
This probably comes down to the fact that it's passing floats as
timestamps down into FakeInternalClient's parse(); specifically,
values like 1738046018.2900746 and 1738045066.1442454 are known
to previously fail.
Just fixing the usage doesn't fix the foot-gun, though -- so fix
up parse() to be internally consistent, even if passed a float.
Change-Id: Ide1271dc4ef54b64d2dc99ef658e8340abb0b6ce

Related-Change: I8227939d04fda8db66fb2f131f2c71ce8741c7d9
Change-Id: I149a2df2d942bba02049947865b000c9cf1a89bc