openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Related-Change: Icff8cf57474dfad975a4f45bf2d500c2682c1129
Change-Id: Ic2283fab0d18ea03c6beb353c6b934344606c15e

It seems unreasonable to expect timings to be accurate to sub-100ns
resolution.
Why 4 places? We already had some tests for proxy-logging that would
assertAlmostEqual to that many places.
Change-Id: Ic7a0c4a416a46eb5198d7cce103358d677ec94ab

Change-Id: Ib3fa895ea13a6703b0f146bc8833c4e635976fdd

We need to get rid of pkg_resources for py312. Fortunately,
we already use the load_pkg_resources() wrapper everywhere.
And, the tests of watchers already mock it. So, the general
mocking of pkg_resources was not doing anything, and this
patch simply deletes it.
Change-Id: Ifbdbbe18d6df6df377f0d914a097328d9c9e95ee

Some tooling out there, like Ansible, will always call to see if
object-lock is enabled on a bucket/container. This fails as Swift doesn't
understand the object-lock or the get object lock api[0].
When you use the get-object-lock-configuration to a bucket in s3 that
doesn't have it applied it returns a specific 404:
 GET /?object-lock HTTP/1.1" 404 None
 ...
 <?xml version="1.0" encoding="UTF-8"?>
 <Error>
 <Code>ObjectLockConfigurationNotFoundError</Code>
 <Message>Object Lock configuration does not exist for this bucket</Message>
 <BucketName>bucket_name</BucketName>
 <RequestId>83VQBYP0SENV3VP4</RequestId>
 </Error>'
This patch doesn't add support for get_object lock, instead it always
returns a similar 404 as supplied by s3, so clients know it's not
enabled.
Also add a object-lock PUT 501 response.
[0] https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html
Change-Id: Icff8cf57474dfad975a4f45bf2d500c2682c1129

Previously, clients could bypass fallocate_reserve checks by uploading
with `Transfer-Encoding: chunked` rather than sending a `Content-Length`
Now, a chunked transfer may still push a disk past the reserve threshold,
but once over the threshold, further PUTs and POSTs will 507. DELETEs
will still be allowed.
Closes-Bug: #2031049
Change-Id: I69ec7193509cd3ed0aa98aca15190468368069a5

Change-Id: I9c335524fb6ce391a6a7af3ff8e9600c36039d56

We already have a helper to do os.makedirs, but ignore EEXIST.
Change-Id: I6428477daac700edf3cc979260d24546505092a8

Otherwise, it'd be boto3.
Change-Id: I2961740fd4f3e914675083331f2465591d63b755

We already require swiftclient>=3.2.0, and have for years.
We can stop checking whether it's 1.x
Related-Change: I9842c9975821bda5c7d8bf2fc214480c0c0a5e96
Change-Id: I798904ab66ca10e21b4999ed7f2be74d1b63584c

Return value instead of sys.exit() if an exception occurs
RelatedChange: I4ca91e3b519a99f3096b95b286779a183e936eb7
Change-Id: I9ebbf5b9f4216c3eab5ab42a0014a750ca28980b

Moving bin script of swift-recon-cron to cli module
for unittest
Delete unused `logger` parameter in get_async_count function
Partial-Bug: #1743656
Change-Id: I4ca91e3b519a99f3096b95b286779a183e936eb7

Improve test coverage for the resuming multipart replicated GET path.
Change-Id: I7de34f443399f645f5021ed392e515f795ed7249

This is intended to be purely internal to our diskfile implementation,
allowing us to have more context about whether we're writing data,
metadata, or a tombstone when calling fallocate in DiskFileWriter.open.
Change-Id: I8c4214080a6bd73787690f75b8c150c999d948fe

Not clear on why, but sometimes no coverage report is generated;
neither term nor html output. Only seen on py311 (maybe something
about us testing with an RC?)
Closes-Bug: #2034452
Change-Id: I033b95414fa82a91a209e3a03d100a011bebe821

When using the copy-part API it is expected for s3api to write down an
empty value for X-Object-Sysmeta-S3Api-Etag on segments. This was
ostensibly to prevent writing down an unrelated S3Api-Etag when copying
a part from another MPU the copy transfers object sysmeta. We should
assume a S3Api-Etag w/o X-Static-Large-Object is non-sense, and SLO
should forever expect empty values for it's sysmeta.
Drive-By: consolidate handling of boto2 sigv4 skips
Related-Bug: #2035158
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ic6f04a5a6af8a3e65b226cff2ed6c9fce8ce1fa2

corrsponsing -> corresponding
Change-Id: I444f4a15ed8cdd5ca3999c0bbc7f7222df187fd4

We've seen a TypeError occur when evaluating what appeared to be a set
of ec_ndata + 1 successful 200 responses. I had theorized they might be
non-durable, and imagined I could artificially trigger the traceback if
I error limited all other nodes. It turns out since we already ignore
404 from handoffs this is exactly the normal 404 non-durable case and
doesn't trigger the traceback.
Change-Id: Iaf511c41358f9bd462b5717810743577be60da7e

Also clean up a couple jobs for rocky and stein; those were dropped
a while ago; it's not clear why it didn't cause some sort of error
before.
Related-Change: Ibe46aecf0f4461be59eb206bfe9063cc1bfff706
Change-Id: Ia94d4b63e5c97e75656cdf7be044b3ec2c7813e7

Add file to the reno documentation build to show release notes for
stable/2023.2.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.
Sem-Ver: feature
Change-Id: Id45b19f64267b07d9219c2b569e6cdebcef87b68

In keeping with the trend as of late, this change makes FakeSwift behave
more reliably like a real Swift backend.
Swift backend object server's grew support for ignoring Range request
headers when fetching SLO manifests in Jan-2020, and FakeSwift learned
how to mimic the real behavior in Jul-2022. This change unifies the
implementation details with a request_helper and consolidates the
behavior in FakeSwift. It also makes the modern object-server behavior
the default.
Between 2020 and 2022 there was arguably some utility defaulting to
legacy behavior, but in 2023 as we endeavor to refactor the SLO
implementation and extend it's tests: a reliable FakeSwift is paramount.
Since most of the existing tests for SLO's behavior responding to Range
requests did not reliably assert behavior across new and old swift this
change selects the most relevant tests to legacy behavior and has them
opt-in to can_ignore_range = False, while the others merely have their
backend request asserts cleaned-up to match the backend request pattern
you would expect in a production environment that's upgraded in the last
3 years. Additional technical investment may be required to ensure
older clusters can upgrade proxies before object servers w/o tracebacks
until the upgrade finishes; however it appears the existing code is
sufficiently robust despite the lack of explicit multi-inheritance
testing like was done for the legacy manifest format change in Nov-2016
(N.B. unlike rolling upgrade bugs, data is forever).
Related-Change-Id: I4ff2a178d0456e7e37d561109ef57dd0d92cbd4e
Related-Change-Id: If3861e5b9c4f17ab3b82ea16673ddb29d07820a1
Related-Change-Id: Ia6ad32354105515560b005cea750aa64a88c96f9
Change-Id: I7ebfd557b9c8ec25498c628fcf0695cd52ad78d6

The NOTICE log level (25) is added to logging when utils.monkey_patch
is executed. Depending on test execution order, this may not have
happened when some unit tests run, so LogRecords at NOTICE level may
have the default level name of 'Level 25'. FakeLogger doesn't have a
key in its lines dict for 'Level 25', so those tests would blow up.
Closes-Bug: #2034254
Related-Change: I6f35ad41414898fb7dc5da422f524eb52ff2940f
Change-Id: I2945ae725f97ee56e9c7ad580168ec8091f2049a

Change-Id: Idbca73a0b8441c27c241dfb790bc9622eab89d9a

Change-Id: Id59846e508545caf56091ebe30557a44adff1d56

A swift request was being constructed with its request method equal to
the s3api's ListingEtagMiddleware. This was benign because the swift
request is just used as a context to get container info, and its
method is never used. However, it is better to use a dummy string
value for the method. 'TEST' has been previously when an auth request
is needed prior to getting container info, so the same is employed
here.
Change-Id: Ie1c70a0ada9883cc4c16b7fc8c2d91a50f4a91b7

When a client sends a malformed http request our server returns a valid http error response with Connection: close and closes the connection.
We want to include a transaction-id and ensure we log details about about the "bad request syntax"
Change-Id: Ic0ee1e4fd4d434d442fcffa68da77e862b37d4c6