openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I9c335524fb6ce391a6a7af3ff8e9600c36039d56

We already have a helper to do os.makedirs, but ignore EEXIST.
Change-Id: I6428477daac700edf3cc979260d24546505092a8

Otherwise, it'd be boto3.
Change-Id: I2961740fd4f3e914675083331f2465591d63b755

We already require swiftclient>=3.2.0, and have for years.
We can stop checking whether it's 1.x
Related-Change: I9842c9975821bda5c7d8bf2fc214480c0c0a5e96
Change-Id: I798904ab66ca10e21b4999ed7f2be74d1b63584c

Return value instead of sys.exit() if an exception occurs
RelatedChange: I4ca91e3b519a99f3096b95b286779a183e936eb7
Change-Id: I9ebbf5b9f4216c3eab5ab42a0014a750ca28980b

Moving bin script of swift-recon-cron to cli module
for unittest
Delete unused `logger` parameter in get_async_count function
Partial-Bug: #1743656
Change-Id: I4ca91e3b519a99f3096b95b286779a183e936eb7

Improve test coverage for the resuming multipart replicated GET path.
Change-Id: I7de34f443399f645f5021ed392e515f795ed7249

When using the copy-part API it is expected for s3api to write down an
empty value for X-Object-Sysmeta-S3Api-Etag on segments. This was
ostensibly to prevent writing down an unrelated S3Api-Etag when copying
a part from another MPU the copy transfers object sysmeta. We should
assume a S3Api-Etag w/o X-Static-Large-Object is non-sense, and SLO
should forever expect empty values for it's sysmeta.
Drive-By: consolidate handling of boto2 sigv4 skips
Related-Bug: #2035158
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ic6f04a5a6af8a3e65b226cff2ed6c9fce8ce1fa2

We've seen a TypeError occur when evaluating what appeared to be a set
of ec_ndata + 1 successful 200 responses. I had theorized they might be
non-durable, and imagined I could artificially trigger the traceback if
I error limited all other nodes. It turns out since we already ignore
404 from handoffs this is exactly the normal 404 non-durable case and
doesn't trigger the traceback.
Change-Id: Iaf511c41358f9bd462b5717810743577be60da7e

In keeping with the trend as of late, this change makes FakeSwift behave
more reliably like a real Swift backend.
Swift backend object server's grew support for ignoring Range request
headers when fetching SLO manifests in Jan-2020, and FakeSwift learned
how to mimic the real behavior in Jul-2022. This change unifies the
implementation details with a request_helper and consolidates the
behavior in FakeSwift. It also makes the modern object-server behavior
the default.
Between 2020 and 2022 there was arguably some utility defaulting to
legacy behavior, but in 2023 as we endeavor to refactor the SLO
implementation and extend it's tests: a reliable FakeSwift is paramount.
Since most of the existing tests for SLO's behavior responding to Range
requests did not reliably assert behavior across new and old swift this
change selects the most relevant tests to legacy behavior and has them
opt-in to can_ignore_range = False, while the others merely have their
backend request asserts cleaned-up to match the backend request pattern
you would expect in a production environment that's upgraded in the last
3 years. Additional technical investment may be required to ensure
older clusters can upgrade proxies before object servers w/o tracebacks
until the upgrade finishes; however it appears the existing code is
sufficiently robust despite the lack of explicit multi-inheritance
testing like was done for the legacy manifest format change in Nov-2016
(N.B. unlike rolling upgrade bugs, data is forever).
Related-Change-Id: I4ff2a178d0456e7e37d561109ef57dd0d92cbd4e
Related-Change-Id: If3861e5b9c4f17ab3b82ea16673ddb29d07820a1
Related-Change-Id: Ia6ad32354105515560b005cea750aa64a88c96f9
Change-Id: I7ebfd557b9c8ec25498c628fcf0695cd52ad78d6

The NOTICE log level (25) is added to logging when utils.monkey_patch
is executed. Depending on test execution order, this may not have
happened when some unit tests run, so LogRecords at NOTICE level may
have the default level name of 'Level 25'. FakeLogger doesn't have a
key in its lines dict for 'Level 25', so those tests would blow up.
Closes-Bug: #2034254
Related-Change: I6f35ad41414898fb7dc5da422f524eb52ff2940f
Change-Id: I2945ae725f97ee56e9c7ad580168ec8091f2049a

When a client sends a malformed http request our server returns a valid http error response with Connection: close and closes the connection.
We want to include a transaction-id and ensure we log details about about the "bad request syntax"
Change-Id: Ic0ee1e4fd4d434d442fcffa68da77e862b37d4c6

Thirdy-party middlewares may compose themselves of multiple filters,
similar to what we have done with encryption and versioned_writes.
We should assume they use the get_account_info / get_container_info
APIs, but we cannot assume that they have been updated to pass our
pipeline info through all their layers.
Change-Id: Ic598715f69227f24ebf5617846a84e6bf1115a96

It adds another layer of indirection and state for the sake of labeling;
longer term it'll be easier to be explicit at the point of emission.
Related-Change: I0522b1953722ca96021a0002cf93432b973ce626
Change-Id: Ieebafb19c3fa60334aff2914ab1ae70b8f140342

Change-Id: I193874659536844d431f0c9fa9881e29392ae2b2

This reverts the fallocate- and punch_hole-related parts of commit
c78a5962b5.
Closes-Bug: #2031035
Related-Change: I3e26f8d4e5de0835212ebc2314cac713950c85d7
Change-Id: I8050296d6982f70bb64a63765b25d287a144cb8d

The S3Request class generally tries to set 's3api.backend_path' in the
original request's environ to the path cited by the response to any
swift subrequests that it issues, so that the backend path is logged
by proxy_logging middleware. For example, a multi-part upload request
will be logged usng the segments container and segment object name.
The S3Request class may also check the hash of an object as it is
uploaded using the HashingInput class. If a hash mismatch is detected,
an HTTPUnprocessableEntity exception is raised which previously caused
the setting of 's3api.backend_path' to be bypassed. This patch
therefore modifies the exception handling clause to set
's3api.backend_path' to the swift subrequest's path.
Change-Id: I0ccc6828174bc869ba0604521bdaed0ebc37a408

Change-Id: If17e10d309826b815d2b5b470a8bff071f5c4e87

FakeSwift is complex enough that it warrants some testing of its own.
Change-Id: Ib0e74c3058b74fea26696f2af80278f1072850a3

Change-Id: Ibbe4667d43aba605164a02e0c942174b0b4f841f

The existing FakeSwift implementation already supports using registered
GET responses for GET requests with a query param. It also supports
using registered GET responses for HEAD requests (if they either both
had the exact SAME matching query params, or both did not have ANY query
params). But it did not support using registered GET responses w/o
query params for HEAD requests with a query param, even though a GET
with the same query param would work.
This change makes it a little more consistent: any client or test that
makes a GET request, should be able to make a similar HEAD request and
expect consistent response headers and status.
This test infra improvement is needed as we're going to be extending
test_slo with a bunch of tests that assert consistent response headers
for both GET and HEAD requests w/ new query params.
Change-Id: Idb4020fdeee87a9164312dc9647ab0820b098ff8

While the existing named kwargs (marker, limit, prefix and etc)
are common parameters among different use cases of
direct_get_container, a new 'extra_params' kwarg is added to
support additional private parameters for each individual use case.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: If1d355fbc5d292cee4f44c0e5dc52e5df9ae990b

We added a "safety net" in swob that won't let HEAD responses return a
body. It seems like we don't currently have many unittests that both
enforce leak tracking and ALSO make HEAD requests. The way swob works
currently it's not possible for even a "well behaved client" to get a
clean assert with FakeSwift's leak tracking because the leak tracking
iterator isn't returned to the client to close and swob doesn't do
anything to close down and consume the HEAD response iterator it's
throwing away.
RelatedChange: I168e147aae7c1728e7e3fdabb7fba6f2d747d937
Change-Id: I079315ebeb772eaad1bf190fefd132b4087b4897

All it did was proxy through to NodeIter, anyway.
Change-Id: Ifec8d3a40f00141a73f6e50efe0b53b382ab2ef3

If the proxy-server gets an error (e.g. timeout) while reading data
from a backend node during an object GET then it will increment the
node's error limiting counter. If the node's error counter reaches a
threshold then the node is 'error-limited' i.e. not used for a period
of time.
When a read from a node fails the proxy will also attempt to resume
the read by searching for another useful node to replace the failed
node. Previously, the outcome of this search for a replacement node
erroneously determined whether the failed node's error counter was
incremented or not: the error counter was only incremented if a useful
replacement node was found.
For example, if a read from the last useful primary node failed, and
all handoff nodes then returned 404, the last primary node's would not
have its error counter incremented (for no good reason). However, if a
useful replacement handoff node was found, then the last primary
node's error counter would be incremented.
This patch fixes the inconsistent and confusing handling of node
errors by ensuring that the node error limiting counter is *always*
incremented when reading from a node fails, *before* the search for a
replacement node.
Note: The Related-Change fixed a bug whereby the replacement node's
error counter was incremented rather than the failed node's error
counter. This patch addresses the inconsistent incrementing of the
failed node's error counter.
Related-Change: Ibd405b79820cedaf9cb02906b8c13c8d34f4d7ec
Change-Id: I938ff7ed7bcbfb363f55821cccd92763dfce449d

Fix an AttributeError that may be raised during exception handling in
the proxy object GET path. GetterBase.node was removed by the
RelatedChange.
Change-Id: Icf77b9d6b8065814033059e33aca6e98a633dbd4
RelatedChange: I76ea042ef4b3f5cc1caa4774e35d6191f4ca548e

This patch tries to make the proxy-server GET path a little easier to
read. The changes are primarily to the GetOrHeadHandler and
ECFragGetter classes.
 - Make generator method names more uniform.
 - Use _<method> to make it more obvious which methods are internal
 to the class and which methods are the interface to the class.
 - Move the interface method to the end of the class.
 - Add some commentary and docstrings.
No intended behavioral change.
Change-Id: I3d00b1071669a42526a31588a2643f91c58ea5a8

Change-Id: I35d682e58419f141429877ac6a6964132f7e658b