openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I7bc1c60acbb0de9d999e6b33811f5e63e20bd4ac
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: I836cf1e1bdb48aab59b4f3d618f5e4f9320d80b5
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: Iafb39842731cc0f64db0f214d28262aae92150ec
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: Ibd5ec2fe65cdc454c46ecb5a490e864763f503b7
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: Id9b3c0c0d5d58ae1455847c8761b13c2a2993142
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: If484964458a8b090b9f0eb4ea37dd40476659529
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: I2152b14a116d2a3519d85e64879994337100dead
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: I8b84fa258ae0071cb171c4b5dcdb86b01980c33a
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: I3e121eee731c5e8d877786e5789903421f83bbc9
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Replace the hard-coded expected suffix hash values with values
calculated using the timestamp. That way, if the timestamp ever
changes then the test will continue to pass.
Change-Id: Id85e2f3decb386e5f865491357bc98750d287af9
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Also clarify some docstrings.
Change-Id: I14649edddd69de9f7f4fe7a9dcc45e9cf890962e
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Change-Id: I0b0fadce9e1231a45b10177d7d3e2c6bcde06a3d
Signed-off-by: Matthew Oliver <matt@oliver.net.au>

Reclaim age and sync "timestamp" args passed to
DatabaseBroker.reclaim() and TombstoneReclaimer.__init__() are
expected to be floats. This is adhered to in code, but many tests pass
a string representation of a Timestamp instance. The string has been
tolerated because it has so far always been identical to a
str(float).
However, if a Timestamp instance has an offset then its string
representation is not equivalent to str(float), and
TombstoneReclaimer.reclaim() would raise an error because the reclaim
age value in the SQL query is not quoted and does not appear to be any
recognised type.
This patch quotes the reclaim age in the TombstoneReclaimer SQL
query. This makes it tolerant of being passed string representations
of Timestamps with offsets. This is not yet expected, but,
nevertheless, text in SQL queries should be quoted.
Change-Id: Ic70689db1e4ddc6f526c283bf0cbb5862b16fd90
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>

Use Timestamp.internal and Timestamp.normal methods to better clarify
which timestamp format is used with which headers.
Use setUp provided Timestamp iters more.
Change-Id: I75b4d2fb946227fb461e8a7512f1d0f9bd3e6d9f
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

Since the Related-Change, swob's date_header_format function will
accept Timestamp instances.
Drive-by: call out the use of Timestamp.ceil() in the
date_header_format docstring.
Change-Id: If45a8172d225eea70160f5034e337feb027bda2b
Related-Change: Ie7090e8dbc90e814f74fbcbd4fba2ca40ee1fcc7
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

... that way we don't need to keep remembering that they need to use
ceil().
Change-Id: Ie7090e8dbc90e814f74fbcbd4fba2ca40ee1fcc7
Signed-off-by: Tim Burke <tim.burke@gmail.com>

Related-Change: Ia7c619fe563ab89ff5b1beb03219e763a8798cc9
Change-Id: I91723b349df5cec409ef000f0a397e180a9de7a2
Signed-off-by: Tim Burke <tim.burke@gmail.com>

There's many places where a Last-Modified header value is constructed
from a time value. This patch replaces the duplicate code with a
helper function that returns the correctly formatted header for a
given number of seconds.
Change-Id: Ia7c619fe563ab89ff5b1beb03219e763a8798cc9
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

An object server unit test was seen to fail returning a 409 when a 204
was expected. I cannot see the bug, so this patch adds some failure
context to add illumination if/when it fails again.
Change-Id: I45327cb5a56a87b8f6b9e971232fdce3b3f4bcb8
Related-Bug: #2131684
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

If a checksum header is sent with an S3 API DeleteObjects POST request
then it will be verified (unlike a CompleteMultipartUpload POST) [0].
If a checksum header is not sent with an S3 API DeleteObjects POST
request then a Content-MD5 header is required. Even when not
*required* [1], a Content-MD5 value will be verified if sent.
Previously we could return 200 when we should have sent 400 (eg, when
the client sent a sha256 but no md5 or additional checksum), or return
400 when we should have sent 200 (eg, when an additional checksum was
sent with no sha256 and the auth protocol didn't require a sha256).
[0] Related-Change: Id39fd71bc59875a5b88d1d012542136acf880019
[1] Related-Change: Ifbcde9820bee72d80cab0fe3e67ea0f5817df949
Related-Bug: #2131671
Change-Id: If25a8f0a3079558544ab15c874eda666a9f69933
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Co-authored-by: Tim Burke <tim.burke@gmail.com>

Now that we need to pass the service creds to keystone, we might as well
default secret caching by default now that they need to be provided.
This patch also adds the required s3token configuration to CI so we can use the
swift service creds to fetch s3api secrets.
As well as also configuring keystone users for cross-compatibility tests.
Change-Id: Ief0a29c4300edf2e0d52c041960d756ecc8a2677
Signed-off-by: Tim Burke <tburke@nvidia.com>

Clarify that the caught exception message is not used to construct the
resulting log message.
Change-Id: I5cdaa27d25f43865eab2f8b12cc68a5d086ef0c7
Related-Change: Ib54172131f1e3abd9b0419aa0370930c6fc82ba9
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>

We can't reliably return 404 for POST if there's a any missing primary
response, but we'll DO it ... with a quorum of 404s ... but ONLY if we
don't have proof of the object existing (i.e. a single 202 will spoil
the 404).
To offset the obvious increased server errors somewhat, and deal with
special cases when one PUT lands on a handoff - we'll *also* in this
same change add extra backend POST requests to handoffs when we observed
mixed primary 202/404 response on POST. The reasoning being if we can
just get > quorum 202 responses from *somewhere* we can actually call
the POST successful and durable despite some 404s.
A future change to the object server could in theory make the proxy more
confident about a 404 in the face of an *out-of-date* rouge 202 - IF the
object-server POST responses started included a timestamp - but this
code would still be as-correct-as-possible for the upgrade path and
there's really nothing to be done if the *only* non-202 storage server
responses are "no useful information"/missing (so again, a single 202
should STILL spoil a quorum of *missing* 404; b/c who's posting to
non-existent objects - we know it WAS there?)
Closes-Bug: #2119675
Change-Id: I43644f242d1cca416844dbd556d40f4e8cd869f9
Co-Authored-By: Jianjian Huo <jhuo@nvidia.com>
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

Recent versions of Keystone require auth tokens when accessing the
/v3/s3tokens endpoint to prevent exposure of a lot of information that
a user who just has a presigned URL should not be able to see.
UpgradeImpact
=============
The s3token middleware now requires Keystone auth credentials to be
configured. If secret_cache_duration is enabled, these credentials
should already be configured. Without these credentials, Keystone users
will no longer be able to make S3 API requests.
Closes-Bug: #2119646
Change-Id: Ie80bc33d0d9de17ca6eaad3b43628724538001f6
Signed-off-by: Tim Burke <tim.burke@gmail.com>

Update the 2024.1 release notes configuration to build from
unmaintained/2024.1.
Change-Id: I69da0bb132bf0b8e5f065a79a22c037835d097de
Signed-off-by: OpenStack Release Bot <infra-root@openstack.org>
Generated-By: openstack/project-config:roles/copy-release-tools-scripts/files/release-tools/change_reno_branch_to_unmaintained.sh

Currently we can get one proxy-logging transfer stat emission over the
duration of the upload/download. We want another stat coming out of
proxy-logging: something that gets emitted periodically as bytes are
actually sent/received so we can get reasonably accurate point-in-time
breakdowns of bandwidth usage.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: Ideecd0aa58ddf091c9f25f15022a9066088f532b
Signed-off-by: Yan Xiao <yanxiao@nvidia.com>

AFAIK this is un-needed once we ensure test_db_replicator un-patches the
object with a cleanup, but in isolation a test patching a global and
relying on someting else to clean it up looks messy/lazy - or at best
just not a pattern/style that can easily be copied elsewhere.
Related-Change: I66d4df1e2dba058b7c719a4a932234b3fc10b554
Change-Id: Ib54172131f1e3abd9b0419aa0370930c6fc82ba9
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

AFAIK this isn't causing any problems, but looks weird once we fix the
db_replicator.ReplConnection patching in Related-Change.
Related-Change: I66d4df1e2dba058b7c719a4a932234b3fc10b554
Change-Id: I45b10c71bab6933dd32c042e386f59e37279787b
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

replace a questionable reload_module with idiomatic addCleanup
Change-Id: I66d4df1e2dba058b7c719a4a932234b3fc10b554
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>

Change-Id: I1a6f85feea1e7a77966d5a5f35df12d8325c8464
Signed-off-by: Shreeya Deshpande <shreeyad@nvidia.com>

He offered his core resignation earlier this week.
Change-Id: I552ae6ae2aee29be94683e9e01b3ab7b10a4f42e
Signed-off-by: Tim Burke <tim.burke@gmail.com>

get_s3_access_key_id returns the S3 access_key_id user for the request
is_s3_req checks whether a request looks like it ought to be an S3 request
parse_path returns a wsgi string
extract_bucket_and_key extracts bucket and object from the request's PATH_INFO
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com>
Change-Id: Iaf86a07238cca6700dee736f55d4c0672cccf1b1
Signed-off-by: Shreeya Deshpande <shreeyad@nvidia.com>

Change-Id: I1e53d171faff02e2dbcbcc779ad3a47506b26853
Signed-off-by: Clay Gerrard <clay.gerrard@gmail.com>