openstack/swift - swift - OpenDev: Free Software Needs Free Tools

This will allow functional tests to be used against it, and can then
be used for the storage-policy work as an example diskfile
implementation associated with a storage policy.
Change-Id: I47a88e70cee99225779baaed379b0c5d4c73611a

Change-Id: I7993de98ce3eb4839fa5d72d1b6ce08e4a7c1451

Apparently that disappeared and now the whole thing does on its own what this
was trying to achieve.
Change-Id: If20016cb80c855e3e8373de5d56926ef8f8e7dac

Disk usage is currently returned as bytes. This patch adds the option
to return the value with a more human-friendly suffix, eg. MB/GB/TB/PB.
Change-Id: I3d55181f7e6085e711bd6e09ea2ce0a12b290cc5

They weren't working because of the intersection between some monkey
patching in the tests and the fact that ``sqlite3.Connection`` and
``Cursor`` are Python classes on PyPy, but C-extension classes on
CPython. The result was that trying to call the superclass's init
didn't work because the name ``sqlite3.Connection`` pointed to a
different class that wasn't actually the real superclass of
``GreenDBConnection``. Using ``super`` fixes this because the method
called is now always the real superclass's.
Change-Id: Ib45b16c7499883e560877788f13285d0a47cc2cd

Remove Python 3.3 from the list of tox test environments since we do
not currently support Python 3.3 for swift.
Change-Id: I7993a511e8433b21b292f21f4775e4ef2b5c2539

Previous this code assumed that almost all arguments would be passed
as kwargs. Now the code also handles ``timeout`` occurring in
``*args``. This is needed because on PyPy's sqlite module, they are
passed in ``*args``.
Change-Id: I7ce25e7b262c6f46680e3f0eb391094de8051946

Change-Id: Id9f6c7e26ed9760372d2bfc9a1ffa3f8ae3d5861

These will allow clients to perform the minimal number of requests
required to accomplish some bulk tasks. For example, a client with
many objects to delete can learn that the cluster's limit on
deletes-per-request is, say, 128, and then batch up their deletes in
groups of 128. Without this, the client has to either discover the
limit out-of-band somehow (and get notified if it changes), or do some
sort of binary search to figure out the limit.
Similar reasoning applies to the containers-per-request value.
The errors-per-request values are included so that clients may size
their requests such that everything is attempted regardless of
failure.
I split the 'bulk' entry into 'bulk_delete' and 'bulk_upload' because,
from a client's standpoint, they're separate operations. It so happens
that Swift implements both in one piece of middleware, but clients
don't care.
Bonus fix: documented a missing config setting for the bulk middleware.
Change-Id: Ic3549aef79682fd5b798145c3545c1609aa1592b

This way, when something fails in Jenkins, you have some chance of
searching the logs for the relevant transaction.
Change-Id: I3cf606cb4963e32b5c6ac3deda08e73541b3ff7d

Add accurate timestamps to the proxy-logging middleware log lines for
the start and end of a request. We use 9 digits of precision since on
some systems the clock resolution can be as fine as 1 ns.
The goal is to allow for log processing that can use the more accurate
timestamps to correlate requests, computing number of requests in
flight at a given historical point in time.
Change-Id: I61e8784b1c455d629f1299207fc4fc7e4a134814
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Also bring unit test coverage to 100% (well, at least every line is
reported as "covered").
Change-Id: I659d0c02008368897b1307a7a5c9aaba73b80588
Signed-off-by: Peter Portante <peter.portante@redhat.com>

On OS X (and probably other Operating Systems) it isn't possible to
introspect the value of a semaphore. Account for this by skipping a
test about this.
Change-Id: I97824f9fc4e36de4f7a62c8ce53865e6977dfdfe

Mark Seger at HP has been looking at small objects, 1 and 2 KB size,
and with Rick Jones' help noticed that TCP protocol traces showed
effects from the Nagel algorithm client-to-server and
server-to-client.
This patch just addresses our WSGI server responses, but does not
address out-bound connections from the various servers.
Change-Id: I11f86df1f56fba1c6ab6084dc1f580c395f072dc
Signed-off-by: Peter Portante <peter.portante@redhat.com>

When including branch coverage results, also generate HTML reports into
a "cover" subdirectory under the directory in which .unittests resides
(i.e. known location at the top of the swift tree).
Change-Id: I493d74f38755f7bf0d7043052585efb27840b238

The "not" in front of the ismount() call got accidentally dropped in a
recent change. This patch adds it back along with a few more tests.
Note that this bug only showed up on an SAIO during probe tests because
I used actually-mounted (virtual) "disks". So keep that in mind when
building SAIOs for development/testing.
Change-Id: Ia193f3c4b73203605954036863575c22ddab6b03

When an error occurs during functional tests that use the
swift_test_client module, the reported error message includes the
method and path:
 ResponseError: 500: 'Internal Error' ('HEAD' \
 '/v1/AUTH_test/d5ce...')
Change-Id: I631cd9e83879fb644778d4ded62625483bf38045

Change-Id: Ifd28f6f14a781a67644315690491888161a7250c

Change-Id: Ibe78912cf923591bddd6a8cf0e683cd028c9c4e8

The PUT REST API has no idea how writes are performed, so when thread
pools are in use, the sleep is not necessary, though it is still
necessary when thread pools are not in use. Since the ThreadPool
object knows when threads are actually in use, it can take care of
being cooperative with the eventlet hub.
In addition, we can hide the cooperative iterator hook, given that the
only other consumer of this was the auditor, which does not need it
any longer. The only consumer of the DiskFile class that wants the
cooperative behavior is the REST API layer of the object server, which
is also using thread pools.
Change-Id: Ibc4ac672899f9a35fd68c85d7f56403c19b4f991
Signed-off-by: Peter Portante <peter.portante@redhat.com>

The documentation rightly said to use "memcache_max_connections", but
the code was looking for "max_connections", and only looking for it in
proxy-server.conf, not in memcache.conf as a fall back.
This commit brings the code coverage for the memcache middleware to
100%.
Closes-Bug: 1252893
Change-Id: I6ea64baa2f961a09d60b977b40d5baf842449ece
Signed-off-by: Peter Portante <peter.portante@redhat.com>

From a review comment on https://review.openstack.org/30051 remove
the "is not None" check, as the assignment in the try block will
never assign None as value as the int() built-in will not return it.
There was some concern of long-term maintenance of the DiskFile
class's _quarantine method raising exceptions. If that routine were
ever mistakenly changed to NOT raise an exception, subtle problems
could creep into the code (see https://review.openstack.org/53237).
We address this concern by raising an exception explicitly at the
call sites of DiskFile._quarantine().
Change-Id: I1729a2d77a6b72b4494b24a8838b47ad5272c075
Signed-off-by: Peter Portante <peter.portante@redhat.com>

In http://launchpad.net/bugs/1254405 an exception occurs when
finalizing the PUT of an object, but it is obscured by the thread pool
code, so we don't see the actual line where it originated. However,
there are two possible functions where this exception could originate:
 1. renamer()
 2. hash_cleanup_listdir()
If this is an error in renamer(), there is some other waky problem
where the temporary file has been removed. It is likely that this is a
problem where a file name from os.listdir ends up disappearing (but
even this is a rare occurence). Regardless, it is not clear that we
really want an error from hash_cleanup_listdir() from affecting the
return result of the PUT.
To that end, we squelsh OSErrors from hash_cleanup_listdir() for
now. One might argue for all errors, but since os.unlink() and
os.listdir() raise OSError today, that is probably sufficient.
Even that we use "Closes-Bug" below, it is not clear it can even be
determined that this closes that bug report.
Change-Id: I2f55df835c387e4d17cffda74c04c9994aebbe1f
Closes-Bug: 1254405
Signed-off-by: Peter Portante <peter.portante@redhat.com>