openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Fixes bug 1071790.
Change-Id: Ic0509190b9994e7d983bcf20daa64a650cc974cb

When a db is reclaimed it removes the hash dir the db files are in,
but it does not try to remove the parent suffix dir though it might
be empty now. This eventually leads to a bunch of empty suffix dirs
lying around. This patch fixes that by attempting to remove the
parent suffix dir after a hash dir reclamation.
Here's a quick script to see how bad a given drive might be:
import os, os.path, sys
if len(sys.argv) != 2:
 sys.exit('%s <mount-point>' % sys.argv[0])
in_use = 0
empty = 0
containers = os.path.join(sys.argv[1], 'containers')
for p in os.listdir(containers):
 partition = os.path.join(containers, p)
 for s in os.listdir(partition):
 suffix = os.path.join(partition, s)
 if os.listdir(suffix):
 in_use += 1
 else:
 empty += 1
print in_use, 'in use,', empty, 'empty,', '%.02f%%' % (
 100.0 * empty / (in_use + empty)), 'empty'
And here's a quick script to clean up a drive:
NOTE THAT I HAVEN'T ACTUALLY RUN THIS ON A LIVE NODE YET!
import errno, os, os.path, sys
if len(sys.argv) != 2:
 sys.exit('%s <mount-point>' % sys.argv[0])
containers = os.path.join(sys.argv[1], 'containers')
for p in os.listdir(containers):
 partition = os.path.join(containers, p)
 for s in os.listdir(partition):
 suffix = os.path.join(partition, s)
 try:
 os.rmdir(suffix)
 except OSError, err:
 if err.errno not in (errno.ENOENT, errno.ENOTEMPTY):
 print err
Change-Id: I2e6463a4cd40597fc236ebe3e73b4b31347f2309

Sometimes you just want to know what machines a given partition
should be on, or what handoffs that partition would use if needed.
We've been meaning to add this option to swift-get-nodes for quite
some time, but I just finally got to it.
Example: swift-get-nodes /etc/swift/object.ring.gz -p 123
I tried to leave as much of the existing swift-get-nodes unaltered,
so the output isn't exactly distilled to just what you'd need for
getting a partition's nodes. But it should suffice for what it is, an
admin tool.
Change-Id: I438400ddc0eecbf9c48266e7f38a2e4f0765f374

This patch ensures that the command-line arg format (boolean)
doesn't conflict with the conf file format (string) and the
proper action is taken.
Change-Id: I3284096e1a9478897e1c3246ab190b46d2590243

The proxy server's ObjectController was performing multiple GET
requests to the object server when processing requests with Range
headers. This was a workaround for a bug in the proxy server's
Controller.GETorHEAD_base method where a response code of 416 from the
object server was incorrectly treated as a bad response from the
backend, the same way a 404 or a 5xx would be.
A 416 (Requested Range Not Satisfiable) from the object server is now
considered a good response. Since the response headers from the object
server already include the X-Object-Manifest header, there's no need
to make a second request (sans Range header) to see if the object is a
manifest.
Bonus fix: updated message for status 416 to match RFC2616.
Bonus fix 2: removed a leftover debugging print() in
test/unit/proxy/test_server.py.
Fixes bug 1065869.
Change-Id: I156af0b463f02ef19a8cfe37092544a599d89b78

Change-Id: I894473994cdfea0996ad16e7619aff421f604abc

RFC2616 implicitly allows transferring utf-8 in
headers (vals) but not in header names (keys)
Swift functional tests includes testing for utf-8
header names (keys) which is non aligned with the standard.
This seem to be the case when testing file metadata.
File metadata keys are than transferred as part of
the http header keys (header names) as
 X-Object-Meta-<metadata_key>=metadata_name.
As a result metadata_keys in swift must be composed from
a subset of ASCII characters - alpha numerics,
some punctuation, not a lot else* and cannot be utf-8.
On branch bp/wsgi-application-interface-4
modified: test/functional/tests.py
Change-Id: I1932e9d4ead83089e8a31a06d28a54f5625efb9c
Fixes: bug 1068026
Implements: blueprint wsgi-application-interface

It's not sufficient to just look at swift.object-updater.successes to
see the async_pending unlink rate. There are two different spots where
unlinks happen: one when an async_pending has been successfully
processed, and another when the updater notices multiple
async_pendings for the same object. Both events are now tracked under
the same name: swift.object-updater.unlinks.
FakeLogger has now sprouted a couple of convenience methods for
testing logged metrics.
Fixed pep8 1.3.3's complaints in the files this diff touches.
Also: bonus speling and, grammar fixes in the admin guide.
Change-Id: I8c1493784adbe24ba2b5512615e87669b3d94505

The directory tests TestContainerPathsEnv creates directorys during SetUp while indicating duplicate application internet media type (MIME) via the Content-Type header.
E.g the output on the line using 'nosetests -v test.functional.tests:TestContainerPaths.testContainerListing' includes:
Host: 127.0.0.1:8080
Accept-Encoding: identity
Content-Type: application/octet-stream
Content-Length: 0
content-type: application/directory
X-Auth-Token: AUTH_tkd40f5550289d4d60af75bb1b6c6067be
On branch bp/wsgi-application-interface-3
 modified: test/functional/tests.py
Fixes: Bug #1068499
Implements Blueprint 	wsgi-application-interface
Change-Id: I49c5d1f73fff3c06d4be18b20c7209eff4e1326c

Add trove classifier to have swift listed among the
other OpenStack-related projets on PyPI to reserve
the name, even though we won't release packages there.
Change-Id: I2c6cd2e37ea55d2bc2ddbba813ee2a190b1c90a9
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>

test_client sends a wrong size indication when testing
chunked data.
For example instead of sending:
 5\r\nabcde\r\n (which is inline with the standard)
The client would send:
 0x5\r\nabcde\r\n
On branch bp/wsgi-application-interface-2
 modified: test/functional/swift_test_client.py
Change-Id: I7597dcd24adade009aead66f67674f361f3fe4fa
Fixes: Bug #1070036
Implements: Blueprint wsgi-application-interface

Add path_qs property to swob.Request. First of all this property
has webob.Request, also this property can be used in swift3 middleware
for generate canonical string, if webob will be replaced in swift3 with
swob (2b36fbd477).
Change-Id: Idf58096baaf7830dd0d624ea6c72eda1eb91ff0d

Turns out this prioritization causes stampedes right after capacity
additions (especially on smaller clusters), which means the things it
prioritizes don't get done for several passes anyway, which means the
prioritization isn't solving the problem it was designed to do, and
it causes non-productive load spikes instead. :/
Getting rid of the prioritization doesn't make things go any slower
overall, could make things actually go faster since there are less
collisions, and evens out the load.
Change-Id: I4938bcd12d9a21fad635cfe772897e4c151f8a95

Change-Id: Ia430637f24d883ee166cb6529dde03049cc06120

In general, Linux does not care about what a mount point is named as
long as it is a vaild directory name (no / or null characters).
However, that is too relaxed for swift, which will pass that mount
point name around as part of url path construction all over the
place. To make sure that the mount point name was sane from that POV,
Swift was using isalnum to verify that the mount point name was sane,
which is overly restrictive.
This patch replaces that test with a test that verifies that the name
has no characters that need to be URL encoded.
The specific use case this enables is allowing mount points to be
named according to the UUID of the filesystem that is being mounted,
which will make Swift more robust in the face of device name instability.
Change-Id: I4d49b21c1783e97c16d3f394c2171f1f80eea058

There have been a bunch of Jenkins failures lately where the StatsD
tests fail because they can't bind to their desired port. There's
nothing special about the particular port they're using, so now we let
the kernel pick an available one for us.
This also lets us get rid of a sleep() in the test that looked like an
attempt to alleviate EADDRINUSE errors, so now in the happy case, the
tests are a few fractions of a second faster.
Change-Id: Idee11349254107a59643539b1566f3588eee7ef4

Change-Id: I38e38607ba9c5cf331c216e4c81582d23c58eeca

One of the tests uses mock.call, which is only available in version
0.8.0 and up.
Fixes bug 1066274.
Change-Id: I77e4de0908f93326992fc4d4af4a7e54e60b5585

This patch merely fixes a selection of files to the point where
pep8 1.3.3 is happy. Most of the errors are indentation related to
continued lines (E126, E127, E128), bracket positions (E124) and the
use of backslash (E502).
Patch 2 fixes David's comments regarding backslash and an odd comment
 - thanks David!
Change-Id: I4fbd77ecf5395743cb96acb95fa946c322c16560

It's there to let administrators turn down the barrage of stats data
that StatsD must cope with, but it wasn't actually honored. Worse, if
the sample rate was set to e.g. 0.2, the stats would all be multiplied
by its inverse, e.g. 2. This patch actually drops packets when
sample_rate < 1, so you get correct measurements.
Fortunately, the default sample rate is 1 (i.e. drop nothing), and
multiplying by 1/1 doesn't change anything, so stats with the default
sample rate of 1.0 are, and have been, just fine.
Fixes bug 1065643.
Also, make the two touched files compliant with pep8 v1.3.3.
Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3

Change-Id: I2c1246b9bbd1d3ab22c2a035b735d937dd90da11

Change-Id: Ifd609f305ee878c39ea4e6840ed8fa9369595cfa

_get_metadata() should only look for metadata on 200 family http status codes
in the path_parts() in the internal client test file should return distinct unicode strings
with spaces for account, container, and obj
Change-Id: I74e335d3b73a59092a0eec06539f4931420061dd

fixes bug 1049017
As per the bug report, the -remove- form of deletion only works with
 meta headers and not with x-container-read and x-container-write.
This patch by Yukihiro KAWADA detects the container acls and sends
them through to the backend empty like the other metadata values.
patch2 extends metadata-helper in ContainerController tests so that
the new functionality can be tested.
patch3 changes the k.lower().startswith() commands for read/write to
a single k.lower - thanks David :)
patch4 fixes PEP8
patch5 fixes more than two hundred pep8 errors. Except one,
 where the pep8 tool complains of overindent, or underindent
depending on the position on the line. No idea how to fix that one.
patch6 fixes the remaining pep8 error - thanks Darrell
Change-Id: I36c2dd72b4636136c5ce7db414bf2b61501090ad

Change-Id: Ic263f4a77a0aa0eb40078772a567eb41a60e40f7

Fixes bug 1064249
sys.path should be extended not appended to
Change-Id: I3404a50b16b843b47f5082676029a4dd3b03f812