openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Since we already deducted the full length of the chunk from bytes_left
before raising the too-many-bytes error, the value in the error message
was always negative, like
 Too many bytes for /v1/AUTH_test/c%3Fwith-question-mark/dlo;
 truncating in /v1/AUTH_test/c with -27244 bytes left
In the not-enough-bytes case, we may as well mention how many more bytes
we were expecting.
Change-Id: I155ed1837d40b4aaf929452b830f03c1daab0af7

I'm not really clear on why a sqlite3.OperationalError should cause us to
retry with stale_reads_ok=True, but swift.common.exceptions.LockTimeout
*definitely* should.
Change-Id: I707dec1d11b8db80bc8fbee30662b319bf10d6a5

This lets you run something like
 $ tox -e py35 -- -s test.unit.common.test_swob
and just run the swob tests. This is handy if you've got one or two
test failures to debug or if you're trying to get another test file
passing under Python 3.
Also removed the old py34 environment. Nothing uses it.
Change-Id: I244b8903ef0a445abd028004bf02d6c581a8afe2

On Python < 3.4, there is no os.O_TMPFILE, so the check for O_TMPFILE
support was always reporting no-support. We now use utils.O_TMPFILE,
which is always defined.
Also, when opening an anonymous tempfile, you must open the directory
(not a file in the directory) with three flags or-ed together:
O_DIRECTORY, O_WRONLY, and O_TMPFILE. Calling open() with a subset of
those flags will receive either EISDIR or EINVAL. The check was trying
to open a file in the temp dir, so was always reporting no-support for
O_TMPFILE. We now correctly open the tempdir.
Change-Id: Iaf48ad9aecea73df4e794818dd15070360bff19f

<swift.common.swob.Match object at 0x7fe589571d50> isn't quite as
obvious.
Change-Id: I7197e7732a4b56084b45ce6a113bc39ce8c7bd54

Unicode characters in tempauth user names currently prevent these users
from authenticating. This manifests in two places:
1. the call to get_groups() fails to return a set that matches the user
 preventing authentication
2. a new token is generated every time for such users, as the check
 against existing memcache entries always fails to match
The patch fixes the existing behavior with respect to unicode users. As
part of it, the FakeMemcache class had to change to decode the strings
passed to it to mimic the actual memcache client behavior.
Darrell Bishop developed the patch and Timur Alperovich added the tests
for it.
Change-Id: I4ecfae2bca6ffa08ad15e584579ebce707f4628d

Seen during a retart-storm:
Traceback (most recent call last):
 File ".../swift/common/db_replicator.py", line 134, in replicate
 {'Content-Type': 'application/json'})
 File ".../httplib.py", line 1057, in request
 self._send_request(method, url, body, headers)
 File ".../httplib.py", line 1097, in _send_request
 self.endheaders(body)
 File ".../httplib.py", line 1053, in endheaders
 self._send_output(message_body)
 File ".../httplib.py", line 897, in _send_output
 self.send(msg)
 File ".../httplib.py", line 859, in send
 self.connect()
 File ".../swift/common/bufferedhttp.py", line 108, in connect
 return HTTPConnection.connect(self)
 File ".../httplib.py", line 836, in connect
 self.timeout, self.source_address)
 File ".../eventlet/green/socket.py", line 72, in create_connection
 raise err
error: [Errno 104] ECONNRESET
Traceback (most recent call last):
 File ".../swift/obj/replicator.py", line 282, in update
 '', headers=self.headers).getresponse()
 File ".../swift/common/bufferedhttp.py", line 157, in http_connect
 ipaddr, port, method, path, headers, query_string, ssl)
 File ".../swift/common/bufferedhttp.py", line 189, in http_connect_raw
 conn.endheaders()
 File ".../httplib.py", line 1053, in endheaders
 self._send_output(message_body)
 File ".../httplib.py", line 897, in _send_output
 self.send(msg)
 File ".../httplib.py", line 859, in send
 self.connect()
 File ".../swift/common/bufferedhttp.py", line 108, in connect
 return HTTPConnection.connect(self)
 File ".../httplib.py", line 836, in connect
 self.timeout, self.source_address)
 File ".../eventlet/green/socket.py", line 72, in create_connection
 raise err
error: [Errno 101] ENETUNREACH
Traceback (most recent call last):
 File ".../swift/obj/replicator.py", line 282, in update
 '', headers=self.headers).getresponse()
 File ".../swift/common/bufferedhttp.py", line 123, in getresponse
 response = HTTPConnection.getresponse(self)
 File ".../httplib.py", line 1136, in getresponse
 response.begin()
 File ".../httplib.py", line 453, in begin
 version, status, reason = self._read_status()
 File ".../httplib.py", line 417, in _read_status
 raise BadStatusLine(line)
BadStatusLine: ''
(Different transactions, of course.)
Change-Id: I07192b8d2ece2d2ee04fe0d877ead6fbfc321d86

If a middleware left of SLO wants to override the ETag for a large
object, it will need to send a X-Backend-Etag-Is-At on GETs if it wants
to be at all performant. This would work fine coming out of the object
controller (which would look at the headers in the response, figure out
what's the real conditional etag, and pass it to swob.Response), and
even encryption (which would do the same), but at SLO, we'd just replace
the ETag, flag it as a conditional response, and let swob assume the
*SLO* ETag is the conditional one.
Now, SLO will jump through the same resolve_backend_etag_is_at hoops that
other parts of the proxy have to deal with. This allows If-Match and
If-None-Match to work correctly if/when swift3 stores an S3-compatible
multipart-upload ETag.
Change-Id: Ibbf59d38d7bcc9c485b1d5305548144025d77441

https://tools.ietf.org/html/rfc7232#section-3 defines the form for
If-Match and If-None-Match as
 If-Match = "*" / 1#entity-tag
 If-None-Match = "*" / 1#entity-tag
https://tools.ietf.org/html/rfc7230#section-7 in turn defines the
1#<type> syntax as
 1#element => element *( OWS "," OWS element )
where OWS is *optional* whitespace. Our swob.Match object should respect
that optionality.
Change-Id: I6ee1c6674e0e9c156149319022fd289504bd3722

For compliance with the Project Testing Interface as described in:
https://governance.openstack.org/tc/reference/project-testing-interface.html
For more details information, please refer to:
http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html
Co-Authored-By: Nguyen Hai <nguyentrihai93@gmail.com>
Change-Id: I26dc41c7df57bf79db531c6e67e148e01c17e992

Change-Id: I1dce94ac6192746a7b90646fe4b4460a651bced1

To keep the consistency, using HTTP_NO_CONTENT instead
of 204
Change-Id: I3d4c0ab6851ce08d1ead0f3b275cb6b41fdb65ac

Sometimes, an rsync process just won't die. You can send SIGKILL, but
it isn't very effective. This is sometimes seen due to attempted I/O
on a failing disk; with some disks, an rsync process won't die until
Linux finishes the current I/O operation (whether success or failure),
but the disk can't succeed and will retry forever instead of
failing. The net effect is an unkillable rsync process.
The replicator was dealing with this by sending SIGKILL to any rsync
that ran too long, then calling waitpid() in a loop[1] until the rsync
died so it could reap the child process. This worked pretty well
unless it met an unkillable rsync; in that case, one greenthread would
end up blocked for a very long time. Since the replicator's main loop
works by (a) gathering all replication jobs, (b) performing them in
parallel with some limited concurrency, then (c) waiting for all jobs
to complete, an unkillable rsync would block the entire replicator.
There was an attempt to address this by adding a lockup detector: if
the replicator failed to complete any replication cycle in N seconds
[2], all greenthreads except the main one would be terminated and the
replication cycle restarted. It works okay, but only handles total
failure. If you have 20 greenthreads working and 19 of them are
blocked on unkillable rsyncs, then as long as the 20th greenthread
manages to replicate at least one partition every N seconds, the
replicator will just keep limping along.
This commit removes the lockup detector. Instead, when a replicator
greenthread happens upon an rsync that doesn't die promptly after
receiving SIGKILL, the process handle is sent to a background
greenthread; that background greenthread simply waits for those rsync
processes to finally die and reaps them. This lets the replicator make
better progress in the presence of unkillable rsyncs.
[1] It's a call to subprocess.Popen.wait(); the looping and sleeping
happens in eventlet.
[2] The default is 1800 seconds = 30 minutes, but the value is
configurable.
Change-Id: If6dc7b003e18ab4e8a5ed687c965025ebd417dfa

Change-Id: I0492ae1d50493585ead919904d6d9502b7738266

auditor.py currently relies on POLICY[0] object_audit_location_generator()
to yield an AuditLocation for all policies on the object-server.
The changes in this patch are :
 - object_audit_location_generator() yields AuditLocation only for the
 requested policy
 - audit_all_objects() calls object_audit_location_generator() for each
 policy
Closes-Bug: 1534212
Change-Id: Ida92ba0a5e1e486a4f7132c6539460b38c34ec87

Move the existing code in db_replicator.py to utils.py
Change-Id: I04e3f30a82c89fb7e714c5eb7225374b9a5c76f9

This is a following patch of [1]. In [1] comment, tim suggested
implementation for same behavior with less code.
This change implemented the suggestion. See [1] for more details.
[1]: https://review.openstack.org/#/c/547306/
Change-Id: Ifd8a0534fbdf41837977028c0c6ef99f1f6ac0f0

... but either way works.
Change-Id: I56b7689270e528a1d2099ff96030ce660e588c94
Related-Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361

... but either way works.
Change-Id: Ib66ee035229d0718b68450587550176c91ce9ca6
Related-Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361

ObjectReplicator.collect_jobs() takes and correctly applies the
various overrides, so there's no need to check the returned jobs
against the overrides.
Change-Id: I2a59b26410d1732a5f2c8f1f32e397d77550860e

Unit tests using O_TMPFILE only rely on the kernel version to check
for the feature. This is wrong, as some filesystem, like tmpfs, doesn't
support O_TMPFILE.
So, instead of checking kernel version, this patch actually attempts to
open a file using O_TMPFILE and see if that's supported. If not, then
the test is skipped.
Change-Id: I5d652f1634b1ef940838573cfdd799ea17b8b572

This patch ports the form_signature cli `swift-form-signature`
and it's corrisponding tests to py3.
In essence in Py3 the HMAC function expects binary strings.
Change-Id: I5dded4ceb80f0cc595403775e8f9c17873e1e37b