openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: Idcb1e9cacba5b959387a2bfd7a4ef5e9d502996a

... and in particular, in the filter:kmip_keymaster section. We thought
we'd done that before, but we didn't: __name__ is *just* the filter
name, and doesn't include the "filter:" prefix for the section.
Change-Id: Iede1d7450e94ba05d2610030e975f566275da88b

Change-Id: Ib9d1da8abed6be0156829e6d54ad2eb43af2516e

Use openstack-tox-cover template, this runs the cover job
in the check queue only.
Use openstack-lower-constraints-jobs template
Remove jobs that are part of the templates.
Change-Id: I76d801da15904142bda0503c5bfa9f11b4ea84f1

Change-Id: I4e2712a4a38b5cab3e31cc75bb162e73b4f5dafb

I saw GET account/container/replicated object all work,
which is not too shabby.
Change-Id: I63408274fb76a4e9920c00a2ce2829ca6d9982ca

In follow-up to the related change, mention the new
cors_expose_headers option (and other proxy-server.conf
options) in the CORS doc.
Add a test for the cors options being loaded into the
proxy server.
Improve CORS comments in docs.
Change-Id: I647d8f9e9cbd98de05443638628414b1e87d1a76
Related-Change: I5ca90a052f27c98a514a96ee2299bfa1b6d46334

This new test job adds multinode testing to swift.
It currently configures 5 nodes: test-runner, proxy,
account, container and object. It starts by checkout out
the latest release and then 'upgrades' the storage nodes
to the current patch under test. That means that it will
the test-runner and the proxy are still running the latest
release.
Future work would include a new set of tests where it will
add objects to the cluster prior to upgrading and then try
to read those objects after the upgrade. It should also be
noted that the 'upgrade' currently only means a new
'git checkout' the configuration files are not being updated.
A future patch should fix that too.
Change-Id: If8e09a082fc024257a98cf332de9a36a18d2adc5

also a drive-by fix to the bindep job--it doesn't need to install
Change-Id: Ic9b68bc60bfbf21b45a1b7f9d7ff9998e01ddd26

Change-Id: I6be1a1c618e4b4fa03b34dad96f378aca01e8e08

Change-Id: Ic7025ba79eadb39f75dd03135fae3326138b3ded

Apparently this was already passing? Not a whole lot there...
Change-Id: I029fe2e7f4162765eee0a390793402523d0975fa

User can cofigure KEEPIDLE time for sockets in TCP connection.
The default value is the old value which is 600.
Change-Id: Ib7fb166deb8a87ae4e97ba0671048b1ec079a2ef
Closes-Bug:1759606

This is useful for deallocating disk blocks as part of an alternate disk
file implementation.
Additionally, add an offset argument to the existing fallocate utility
function; this allows you to grow an existing file.
Sam always had the best descriptions:
 utils.fallocate(fd, size) allocates <size> bytes for the file referred
 to by <fd>. It allows for keeping a reserve of an additional N bytes
 or X% of the filesystem free. If neither fallocate() or
 posix_fallocate() C functions are avaialble, utils.fallocate() will
 log a warning (once only) and not actually allocate space.
 utils.punch_hole(fd, offset, length) deallocates <length> bytes
 starting at <offset> from the file referred to by <fd>. It uses the C
 function fallocate(). If fallocate() is not available, calls to
 utils.punch_hole() will raise an exception.
Since these both use the fallocate syscall, refactor that a bit and get
rid of FallocateWrapper. We add a new _LibcWrapper to do some
lazy-loading of a C function and expose whether the function is actually
available in Python, though. This allows utils.fallocate and
utils.punch_hole to keep their fancy logic pretty well-contained.
Modernized the tests for utils.fallocate() and utils.punch_hole().
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Change-Id: Ieac30a477d784905c94742ee3d0898d7e0194b39

Change-Id: Ic6d0532f9f42b4d57a12aafeec5d34baffab6e23

Previously we'd use two users, one admin and one unprivileged.
Ceph's s3-tests, however, assume that both users should have access to
create buckets. Further, there are different errors that may be returned
depending on whether you are the *bucket* owner or not when using
s3_acl. So now we've got:
 test:tester1 (admin)
 test:tester2 (also admin)
 test:tester3 (unprivileged)
Change-Id: I0b67c53de3bcadc2c656d86131fca5f2c3114f14

This patch added new non-voting gate job to check the s3api compatibility
via swiftstack/s3compat tool that shows the ratio of compatible S3 APIs
in the gate result for each patch. This is very useful to check the possibility
if the new incoming patch breaks S3 API compatibility unexpectedly.
Originally swift3 has this kind of the gate job but we missed the staff
while migrating from swift3 into swift upstream repo so this is the porting of that.
Note that currently the job is against to only tempauth because we don't have
custom gate jobs using keystone environment other than dsvm.
Change-Id: I6f30f74678ad35479da237361bee48c46c0ecc49

BaseDiskFileWriter will track md5 and expose upload_size, etag via the
chunks_finished method.
The BaseDiskFileReader already tracks the md5/etag via _iter_etag, for
parity we add a _chunks_etag to the BaseDiskFileReader.
Instead of returning the upload_size and hexdigest every call to write,
we return the tuple from chunks_finished.
Change-Id: I26c58719cff5fde941d0248c250a0204e0379ae5

Multipart uploads in AWS (seem to) have ETags like:
 '"' + MD5_hex(MD5(part1) + ... + MD5(partN)) + '-' + N + '"'
On the other hand, Swift SLOs have Etags like:
 MD5_hex(MD5_hex(part1) + ... + MD5_hex(partN))
(In both examples, MD5 gets the raw 16-byte digest while MD5_hex
gets the 32-byte hex-encoded digest.)
Some clients (such as aws-sdk-java) use the presence of a dash
to decide whether to perform client-side validation of downloads.
Other clients (like s3cmd) use the presence of a dash *in bucket
listings* to decide whether or not to perform additional HEAD requests
to look for MD5 metadata that can be used to compare against the MD5s
of local files.
Now we include a dash as well, to prevent spurious errors like
> Unable to verify integrity of data download. Client calculated
> content hash didn't match hash calculated by Amazon S3. The data
> may be corrupt.
or unnecessary uploads/downloads because the client assumes data has
changed that hasn't.
For new multipart-uploads via the S3 API, the ETag that is stored will
be calculated in the same way that AWS uses. This ETag will be used in
GET/HEAD responses, bucket listings, and conditional requests via the S3
API. Accessing the same object via the Swift API will use the SLO Etag;
however, in JSON container listings the multipart upload etag will be
exposed in a new "s3_etag" key.
New SLOs and pre-existing multipart-uploads will continue to behave as
before; there is no data migration or mitigation as part of this patch.
Change-Id: Ibe68c44bef6c17605863e9084503e8f5dc577fab
Closes-Bug: 1522578

Change-Id: Ib82f2494f2cc65a3f15925efe0dfeb52eb7aa22b

Change-Id: I5b0bcc6028dbe6248e0e09baf2cbb72deb011c80

Change-Id: Ibb30b23a53260a154d009b52388859c5220aca32

DiskFile already exposes a reader method that creates the DiskFileReader
instance. Add a writer method for parity.
DiskFile currently only provides a context manager create - that will
open and close the DiskFileWriter. Add explicit open and close methods
to support more flexibility in how callers manage life-cycle on their
DiskFileWriter instances.
Diskfile confusingly manages some state for DiskFileWriter (e.g. fd,
tmppath, use_linkat). Encapsulate the DiskFileWriter state to improve
readability and reduce coupling (e.g. put_succeeced).
Change-Id: If18e0041680470a9c57a08e9ea9327acba8593df

... instead of via direct_client.
When I go grepping for direct_client in my code base, I want to find the
actual users of direct_client!
Change-Id: Id55c9c8ea3ab7bb86da1c5e8f6e979c26974a260

This changes how we decided to handle HTTPBadRequest and friends.
Previously, we came to forcing everything to be bytes, because
obviously WSGI bodies can only be bytes. But it turned out to
be too much hassle, so let's revisit that decision.
We aren't going to back-convert all the bytes bodies. That is
already too much hassle too. But this patch throws in a few
examples, just to showcase what this patch does.
Change-Id: I206a9611e7f348f49e5c2d0e677fed3b0739c167

Why did we have those??? When have we *ever* broken neutron?
All those jobs ever tell me are to leave a comment saying
"recheck".
Change-Id: If32b164827d07c4b3efc88fa2794e38ca67c3b53