openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I6be1a1c618e4b4fa03b34dad96f378aca01e8e08

Apparently this was already passing? Not a whole lot there...
Change-Id: I029fe2e7f4162765eee0a390793402523d0975fa

Previously we'd use two users, one admin and one unprivileged.
Ceph's s3-tests, however, assume that both users should have access to
create buckets. Further, there are different errors that may be returned
depending on whether you are the *bucket* owner or not when using
s3_acl. So now we've got:
 test:tester1 (admin)
 test:tester2 (also admin)
 test:tester3 (unprivileged)
Change-Id: I0b67c53de3bcadc2c656d86131fca5f2c3114f14

This patch added new non-voting gate job to check the s3api compatibility
via swiftstack/s3compat tool that shows the ratio of compatible S3 APIs
in the gate result for each patch. This is very useful to check the possibility
if the new incoming patch breaks S3 API compatibility unexpectedly.
Originally swift3 has this kind of the gate job but we missed the staff
while migrating from swift3 into swift upstream repo so this is the porting of that.
Note that currently the job is against to only tempauth because we don't have
custom gate jobs using keystone environment other than dsvm.
Change-Id: I6f30f74678ad35479da237361bee48c46c0ecc49

BaseDiskFileWriter will track md5 and expose upload_size, etag via the
chunks_finished method.
The BaseDiskFileReader already tracks the md5/etag via _iter_etag, for
parity we add a _chunks_etag to the BaseDiskFileReader.
Instead of returning the upload_size and hexdigest every call to write,
we return the tuple from chunks_finished.
Change-Id: I26c58719cff5fde941d0248c250a0204e0379ae5

Multipart uploads in AWS (seem to) have ETags like:
 '"' + MD5_hex(MD5(part1) + ... + MD5(partN)) + '-' + N + '"'
On the other hand, Swift SLOs have Etags like:
 MD5_hex(MD5_hex(part1) + ... + MD5_hex(partN))
(In both examples, MD5 gets the raw 16-byte digest while MD5_hex
gets the 32-byte hex-encoded digest.)
Some clients (such as aws-sdk-java) use the presence of a dash
to decide whether to perform client-side validation of downloads.
Other clients (like s3cmd) use the presence of a dash *in bucket
listings* to decide whether or not to perform additional HEAD requests
to look for MD5 metadata that can be used to compare against the MD5s
of local files.
Now we include a dash as well, to prevent spurious errors like
> Unable to verify integrity of data download. Client calculated
> content hash didn't match hash calculated by Amazon S3. The data
> may be corrupt.
or unnecessary uploads/downloads because the client assumes data has
changed that hasn't.
For new multipart-uploads via the S3 API, the ETag that is stored will
be calculated in the same way that AWS uses. This ETag will be used in
GET/HEAD responses, bucket listings, and conditional requests via the S3
API. Accessing the same object via the Swift API will use the SLO Etag;
however, in JSON container listings the multipart upload etag will be
exposed in a new "s3_etag" key.
New SLOs and pre-existing multipart-uploads will continue to behave as
before; there is no data migration or mitigation as part of this patch.
Change-Id: Ibe68c44bef6c17605863e9084503e8f5dc577fab
Closes-Bug: 1522578

Change-Id: Ib82f2494f2cc65a3f15925efe0dfeb52eb7aa22b

Change-Id: I5b0bcc6028dbe6248e0e09baf2cbb72deb011c80

Change-Id: Ibb30b23a53260a154d009b52388859c5220aca32

DiskFile already exposes a reader method that creates the DiskFileReader
instance. Add a writer method for parity.
DiskFile currently only provides a context manager create - that will
open and close the DiskFileWriter. Add explicit open and close methods
to support more flexibility in how callers manage life-cycle on their
DiskFileWriter instances.
Diskfile confusingly manages some state for DiskFileWriter (e.g. fd,
tmppath, use_linkat). Encapsulate the DiskFileWriter state to improve
readability and reduce coupling (e.g. put_succeeced).
Change-Id: If18e0041680470a9c57a08e9ea9327acba8593df

... instead of via direct_client.
When I go grepping for direct_client in my code base, I want to find the
actual users of direct_client!
Change-Id: Id55c9c8ea3ab7bb86da1c5e8f6e979c26974a260

This changes how we decided to handle HTTPBadRequest and friends.
Previously, we came to forcing everything to be bytes, because
obviously WSGI bodies can only be bytes. But it turned out to
be too much hassle, so let's revisit that decision.
We aren't going to back-convert all the bytes bodies. That is
already too much hassle too. But this patch throws in a few
examples, just to showcase what this patch does.
Change-Id: I206a9611e7f348f49e5c2d0e677fed3b0739c167

Why did we have those??? When have we *ever* broken neutron?
All those jobs ever tell me are to leave a comment saying
"recheck".
Change-Id: If32b164827d07c4b3efc88fa2794e38ca67c3b53

Change-Id: Idb7d9b3bedefc5972b467cce8922e4a673b1cab2

Change-Id: I1337712cc790baca151db2575f650da243bf09ef

Almost one year has passed since all post_as_copy related code
removed by [1], we don't have to keep the warning message for
post_as_copy setting anymore in the code tree.
1: 1e79f828ad
Change-Id: Id9eea22ed688574d84ca582584c0c207d5f01383

Change-Id: Ic5c043e8d04302adc6c771e311fa70e6aa2675d1

The new job swift-dsvm-functional replaces job
legacy-swift-dsvm-functional-identity-v3-only.
Remove legacy-swift-dsvm-functional, we do not install with Identity v2
since queens.
Change-Id: I8b353cba66b22de42ae42954ec44e5802224fc23

While s3api is the preferred method going forward, it would be nice to
allow for backwards compatibility with swift3 in the tempauth
middleware.
Change-Id: I4fd4772b1d7b173ee0faf72d1c1f1531646dde8b

The install-guide-jobs template needs to stay in project-config, remove
it from zuul file.
See also https://review.openstack.org/597618
Change-Id: Iaaa5e0b8de17114db0aab6d539e28cbcc2072cac

We definitely *do* want pep8 running on test-only changes.
Don't really need it for sample-config updates, though.
Change-Id: I363a1e91be83f238cffc4688aa74bfd7553dc900

This ensures we log subrequests.
Change-Id: I6b39d462a8c149f6d55db465f992f01eae465d57
Closes-Bug: 1486193

... and have the secrets-in-config, secrets-in-KMIP, and
secrets-in-Barbican keymasters all inherit from it.
Change-Id: I36a6ce275a086ec85e2f556fcdc0d0b94ff4e647
Related-Change: Iddc0f333861b6c1f81e181f006cd592b5eb6ea17

Note that there's no swift-dsvm-functional-identity-v3-only because
as of the Queens release, that's the *only* way Keystone gets deployed.
Change-Id: I49d963b98f3df21fea0db24c83553ef873ad73c8

This is a mechanically generated patch to switch the documentation
jobs to use the new PTI versions of the jobs as part of the
python3-first goal.
See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html
Change-Id: I686071c13c28a06092220fd6ec379bbf66662ddc
Story: #2002586
Task: #24337 

This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.
Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.
Because this patch is generated by a script, there may be some
cosmetic changes to the layout of the YAML file(s) as the contents are
normalized.
See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html
Change-Id: Ib8b04669a50a84345f2ad8456a73473aec2df2f4
Story: #2002586
Task: #24337 

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I10424a57f36b0a5e83456bb07d86f6a7444a304b