openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Swift already supports that much, at least. AWS used to not support any
conditional PUTs, but that's changed somewhat recently; see
- https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/
- https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/
Drive-By: Fix retry of a CompleteMultipartUpload with changed parts; it
should 404 rather than succeed in writing the new manifest.
Change-Id: I2e57dacb342b5758f16b502bb91372a2443d0182

This update implements Sigv4-streaming (chunked upload) as described in
the Amazon S3 documentation:
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
Closes-Bug: #1810026
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: ashnair <ashnair@nvidia.com>
Change-Id: I7be1ce9eb5dba7b17bdf3e53b0d05d25ac0a05b0

Related-Change: I86bcd7a2373cf1e7213cb716888307306b7bb50d
Change-Id: I0056bfc78f9c89b1c458943ee446499d21667fac

Change-Id: I9044e49ff30b092a97b75fa7fa133016d17cde37

Previously, we weren't converting it to a boolean, but left it as
a truth-y string.
Change-Id: Ia0349d991435492ac9d80c36c5b1123b9e310db6

... rather than importing it before it's been read in setup_package and
never getting any updated value.
Related-Change: I699d664bcef6754aac4ec6877830e745b764342f
Change-Id: I48faabc5b59f94a47dfeb871f13ddbcbaa20d349

Change-Id: Ibd3d5f83fcd31d78447a894ff4426e3e1b62e5a8

Related-Change-Id: Ia1d8d9085bad5c6df744a4551eef9dcf56e6e261
Change-Id: I6c5bb824d7145c25faf37fd1fa575cedf05e499a

Change-Id: Ia754021714a8ed3d5d6010a946c558552821fbe4

We shouldn't allow a test to assume a strict coupling of
client/server time and the test needs to be modified so that
the functional test-suite pass for us.
Change-Id: Ia1d8d9085bad5c6df744a4551eef9dcf56e6e261

Add the storage_policy attribute to the metadata returned
when listing containers using the GET account API function.
The storage policy of a container is a very useful attribute
for telemetry and billing purposes, as it determines the location
and method/redundancy of on-disk storage for the objects in the
container. Ceilometer currently cannot define the storage policy as a
metadata attribute in Gnocchi as GET account, the most efficient way
of discovering all containers in an account, does not return the
storage policy for each container.
Returning the storage policy for each container in GET account
is the ideal way of resolving this issue, as it allows Ceilometer
to find all containers' storage policies without performing additional
costly API calls.
Special care has been taken to ensure the change is backwards
compatible when migrating from pre-storage policy versions
of Swift, even though those versions are quite old now.
This special handling can be removed if support for migrating
from older versions is discontinued.
Closes-bug: #2097074
Change-Id: I52b37cfa49cac8675f5087bcbcfe18db0b46d887

Change-Id: Ic66b9ae89837afe31929ce07cc625dfc28314ea3

A failing CORS test in the gate discovered that we were responding
application/json to ?format=txt requests (which is maybe not even a
valid value for that qs param?), but only when running with
eventlet==0.38.0
This avoids the problem of backend container server HEADs no longer
having 'Content-Length: 0' by fixing the client HEAD resp headers before
we check for chunked-transfer resp.
Drive-By: refactor listing_formats to use HeaderKeyDict and always set
Content-Length explicitly
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: If724485e1425d1481d10b9255436301e346f07e8

Add functional tests for copying from an SLO using PUT requests with
an x-copy-from header.
Add functional tests for copying from an SLO using part-number parameter.
Change-Id: Ic40c4b104b11de9ab09485e82543521b37ea1daa

Change-Id: I8c65c7829fbe167e0c79239d577c38e90b93b5cf

The AWS behaviour is weird: if I have my client signer always use
the casing that will be sent on the wire, I reliably get back 405s,
but if my client signer always converts to upper-case despite
sending lower-case on the wire, it's roughly a coin toss for whether
I get back a 405 or a 403 (with the string-to-sign indicating that
the server was expecting lower-case). This is true for both v2 and
v4 signatures, both in headers and in query params.
Logically, though, it seems like using what's on the wire *ought* to
be the way to go, and AWS's consistency in that case lends some
credibility.
Unfortunately, that breaks an existing functional test unless awscrt
is installed, as boto3 (or botocore? somebody) will upper-case the
request method by default, but trusts awscrt to handle things if
available. So skip the part of the test that uses a lower-cased
request method if we can't import awscrt.
Change-Id: Ia6cacc51b2744986e703469f9e5215209194de8a

Change-Id: I224e8706f1ee72b718e35a25ea417a1cf3a35938

Fixes a bug with the x-open-expired feature where our magic header
does not get copied when refetching all manifests that causes
404 on HEAD requests with part-number=N query parameter since
the object-server returns an empty response body and the
proxy needs to refetch. The fix also applies to segment GET
requests if the segments have expired.
Change-Id: If0382d433f73cc0333bb4d0319fe1487b7783e4c

Previously the in-process functional test setup attempted to insert
etag-quoter in the proxy pipeline by replacing the substring 'cache
listing_formats' with 'cache etag-quoter listing_formats'. However,
the pipeline had already been modified to have 'cache domain-remap
listing_formats', so the etag-quoter replacement failed silently.
This patch modifies the etag-quoter replacement to match just
'cache'. Some helper functions are also introduced to make the
pipeline modification more uniform and to consistently verify the
intended modifications have been achieved, or otherwise raise an
exception.
Change-Id: I640837a4da985bcbe48f9ae5eeb56385b11034eb

A print statement, presumably for debugging, crept in with the
Related-Change.
Change-Id: I0cccb960b51446a3aa58be95c8c9d06cc6b6eada
Related-Change: I69974cc8a39731c980b54137b799a36b2e63a44a

Some s3api functional tests were setting boto logging level to DEBUG,
which results in a very large amount of logging output from the test
runner, including large request bodies. This makes it extremely hard
to inspect test output.
This patch makes the quiet_boto_logging context manager revert the
logger level to its original value rather than indiscriminately
leaving it set to DEBUG.
Change-Id: I1dd9603adf9a19e89da5a461d3c6810a3432ae46

The account quota middleware was missing tests for the new quota count
API and it was also only testing the legacy API for quota bytes.
This adds the new APIs quota bytes and quota count to the functional
tests.
Change-Id: I6ebb19c90dfb1cfbe0535ed3860f2319e5153c05

Note that you can still have a "//" in the path with rest_with_last, though.
Change-Id: I171afcd67b162634189b752ff92a4f43484bc12a

Account quota metadata such as quota_bytes and quota_count are stored in
the `meta` namespace which users have access to. However, this should be
only available to reseller admins.
This patch adds support for writing the quota metadata to `sysmeta`
namespace, so that it is not accessible by users. The account policy
quota is already using `sysmeta` and has the namespace
`X-Account-Quota-*`, so we are following this pattern.
If present, `X-Account-Quota-Bytes` is always preferred. However, in
order to maintain backwards compatibility, `X-Account-Meta-Quota-Bytes`
will still be honoured if it exists and `X-Account-Quota-Bytes` is not
present.
This also adds some new "legacy" tests to validate backwards
compatibility.
Co-authored-by: Azmain Adib <adib1905@gmail.com>
Co-authored-by: Daanish Khan <daanish1337@gmail.com>
Co-authored-by: Mohammed Al-Jawaheri <mjawaheri02@gmail.com>
Co-authored-by: Nada El-Mestkawy <nadamaged05@gmail.com>
Co-authored-by: Tra Bui <trabui.0517@gmail.com>
Co-authored-by: Chris Smart <distroguy@gmail.com>
Change-Id: Icf7b26023ab5b84136ceaa103fa2797534320f1a

Previously, this would reduce the account's quota to zero, which seems
like the opposite of what the operator intended.
Now, remove the quota, similar to sending an empty quota header.
Change-Id: Ic28752d835e0b970f2baa4e68cbfcde4f500b3d4

Change-Id: Ied0ff6bea7e054fad3fe9579c85d9ae5c9c0b255

The boto library was last updated two years ago and has
rusted to the point that it's unusable on py312 -- see
https://github.com/boto/boto/issues/3951
We should transition all of these tests to boto3 equivalents,
but this should help out in the meantime.
Related-Bug: #1557260
Related-Bug: #2063367
Change-Id: If95f45371f352c6a2d16be1a3e1b64e265bccfb4

If the global configuration option 'enable_open_expired' is set
to true in the config, then the client will be able to make a
request with the header 'x-open-expired' set to true in order
to access an object that has expired, provided it is in its
grace period. If this config flag is set to false, the client
will not be able to access any expired objects, even with the
header, which is the default behavior unless the flag is set.
When a client sets a 'x-open-expired' header to a true value for a
GET/HEAD/POST request the proxy will forward x-backend-open-expired to
storage server. The storage server will allow clients that set
x-backend-open-expired to open and read an object that has not yet
been reaped by the object-expirer, even after the x-delete-at time
has passed.
The header is always ignored when used with temporary URLs.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
Change-Id: Ibe7dde0e3bf587d77e14808b169c02f8fb3dddb3

This change allows individual SLO segments to be downloaded by adding
an extra 'part-number' query parameter to the GET request. You can
also retrieve the Content-Length of an individual segment with a HEAD
request.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I7af0dc9898ca35f042b52dd5db000072f2c7512e

This has been available since py32 and was backported to py27; there
is no point in us continuing to carry the old idiom forward.
Change-Id: I21f64b8b2970e2dd5f56836f7f513e7895a5dc88

Note that there's a bit of a privilege escalation as prefix-based
tempurls can now be used to perform listings -- but only on containers
with staticweb enabled. Since having staticweb enabled was previously
pretty useless unless the container was both public and
publicly-listable, I think it's probably fine.
This also allows tempurls to be used at the container level, but only
for staticweb responses.
Change-Id: I7949185fdd3b64b882df01d54a8bc158ce2d7032

Change-Id: Ib3fa895ea13a6703b0f146bc8833c4e635976fdd

Some tooling out there, like Ansible, will always call to see if
object-lock is enabled on a bucket/container. This fails as Swift doesn't
understand the object-lock or the get object lock api[0].
When you use the get-object-lock-configuration to a bucket in s3 that
doesn't have it applied it returns a specific 404:
 GET /?object-lock HTTP/1.1" 404 None
 ...
 <?xml version="1.0" encoding="UTF-8"?>
 <Error>
 <Code>ObjectLockConfigurationNotFoundError</Code>
 <Message>Object Lock configuration does not exist for this bucket</Message>
 <BucketName>bucket_name</BucketName>
 <RequestId>83VQBYP0SENV3VP4</RequestId>
 </Error>'
This patch doesn't add support for get_object lock, instead it always
returns a similar 404 as supplied by s3, so clients know it's not
enabled.
Also add a object-lock PUT 501 response.
[0] https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html
Change-Id: Icff8cf57474dfad975a4f45bf2d500c2682c1129

Otherwise, it'd be boto3.
Change-Id: I2961740fd4f3e914675083331f2465591d63b755

We already require swiftclient>=3.2.0, and have for years.
We can stop checking whether it's 1.x
Related-Change: I9842c9975821bda5c7d8bf2fc214480c0c0a5e96
Change-Id: I798904ab66ca10e21b4999ed7f2be74d1b63584c

When using the copy-part API it is expected for s3api to write down an
empty value for X-Object-Sysmeta-S3Api-Etag on segments. This was
ostensibly to prevent writing down an unrelated S3Api-Etag when copying
a part from another MPU the copy transfers object sysmeta. We should
assume a S3Api-Etag w/o X-Static-Large-Object is non-sense, and SLO
should forever expect empty values for it's sysmeta.
Drive-By: consolidate handling of boto2 sigv4 skips
Related-Bug: #2035158
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ic6f04a5a6af8a3e65b226cff2ed6c9fce8ce1fa2

When a client sends a malformed http request our server returns a valid http error response with Connection: close and closes the connection.
We want to include a transaction-id and ensure we log details about about the "bad request syntax"
Change-Id: Ic0ee1e4fd4d434d442fcffa68da77e862b37d4c6

Related-Change: I4ecfae2bca6ffa08ad15e584579ebce707f4628d
Related-Change: I1e244c231753b8f4b6f1cf95cb0ae4c3c959ae0f
Change-Id: Ia386736b9b283858931794690538871b6e1ad9c8

Change-Id: I61f141a71eddcac600058d66ddf802306df455c1

Change-Id: Ia564f2ee70f5d04acab1c38e17d1936642a01447