openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Fix bug 1067677
When we delete a versioned object, the last version will be poped out from the
versiones container. When a versioned object is expired and deleted by object-
expirer, the last version is restored but remains in the versions container
instead of getting deleted. The reason is object-expirer will set
'X-If-Delete- At' header when deleteing an object. However this is for the
current version - not for the last version. When the object-server is trying
to delete the last version, the transaction will fail with error:
X-If-Delete-At and X-Delete-At do not match. Delete the 'X-If-Delete-At'
field in the later built delete request would help to solve this issue.
This patch, without the test, was first proposed by
Zhou Yuan <yuan.zhou@intel.com>.
Change-Id: I62c97e6a0f140888497e189129825865fb6f7966

Due to a lack of zero-padding in a format string, there was a 1/16
chance that the length of the randomly-generated boundary string was
less than 32. This didn't affect client downloads, but it did give the
unit tests a 1/16 chance of failing (there's a [0-9a-f]{32} regex in a
test).
Now we left-zero-pad the boundary string to length 32, and the test
consistently passes.
Change-Id: I4ee98dab127e5674e59b078d8090470ec783e0b0

Separate the request processor initialization steps
 Load common settings from conf
 Set the logger
 Load the request processor
from the web front end server application step
On branch bp/wsgi-application-interface-7
 modified: swift/common/wsgi.py
Implements Blueprint: wsgi-application-interface
Change-Id: I25fabad8846cd644e4bf1e1bd9c2e568512e12e5

Currently, a container's info can be cached without cors data intact after
a container GET.
I made headers_to_container_info a function instead of a method and I crammed
all container metadata into container_info.
This is so e.g. staticweb can eventually re-use the same container info cache.
Fix pep8 in swift/proxy/controllers/container.py
Change-Id: I4bbb042dde79afac48395efc38bd80f0ff240e1f

This change adds multi range retrieval to OpenStack Swift. For non-
segmented data object, a client can use HTTP Range header to specify
multiple ranges to retrieve sections of the data object.
This implementation currently does not support segmented data object
multi range retrieval. When a client sends a multi range request
against a segmented data object, Swift will return HTTP status code
200. Support for segmented data multi range retrieval will be added
in near future.
This implementation is to bring Swift closer to CDMI multi range
data retrieval standard. Once support for segemented data multi
range is added, Swift will be compliant with CDMI standard in
this area.
DocImpact
Change-Id: I4ed1fb0a0a93c037ddb2f551ea62afe447945107

If an object server is down, container-sync stops syncing the container
even if the it gets object copies from "up" obejct servers.
Bug 1069910
In case the git history gets mangled, this fix was done almost entirely
by Donagh McCabe <donagh.mccabe@hp.com>.
Change-Id: Ieeadcfeb4e880fe5f08e284d7c12492bf7a29460

Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.
DocImpact
Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0

There were bugs with SCRIPT_NAME and
swift.common.wsgi.make_pre_authed* functions.
1) SCRIPT_NAME wasn't copied with PATH_INFO, which it should've been.
2) When a new path was given, SCRIPT_NAME wasn't set to an empty
 string but just omitted, which is allowed by spec, but really
 should be set.
For completeness, if SCRIPT_NAME doesn't exist in the source env, it
will be created in the new env, but as an empty string.
Change-Id: Ifbc27ed8ff357038c54df4d37de46cfd5e31372e

cleaned up pep8 (v1.3.3) in all files this patch touches
Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff

Swift proxy server, when communicating with
the back-end servers ALWAYS sends 100-continue.
Even if the length headers indicate that there
is no body (content-length: 0).
This behavior is not inline with the standard.
RFC2616:8.2.3 disallows 100-continue
 for req.content_length==0
This fix removes 100-continue during PUT requests
without a body while maintining the ability for
handoff in case of faliure.
On branch bp/wsgi-application-interface-5
 modified: swift/proxy/controllers/obj.py
Fixes: Bug #1070025
Implements Blueprint: 	wsgi-application-interface
Change-Id: Ia4eb8b886a74968cca4e8bde208641b37f2c104c

Change-Id: I78a5c109c9440df752e390698502f57d4392fb67

Fixes bug 1071790.
Change-Id: Ic0509190b9994e7d983bcf20daa64a650cc974cb

When a db is reclaimed it removes the hash dir the db files are in,
but it does not try to remove the parent suffix dir though it might
be empty now. This eventually leads to a bunch of empty suffix dirs
lying around. This patch fixes that by attempting to remove the
parent suffix dir after a hash dir reclamation.
Here's a quick script to see how bad a given drive might be:
import os, os.path, sys
if len(sys.argv) != 2:
 sys.exit('%s <mount-point>' % sys.argv[0])
in_use = 0
empty = 0
containers = os.path.join(sys.argv[1], 'containers')
for p in os.listdir(containers):
 partition = os.path.join(containers, p)
 for s in os.listdir(partition):
 suffix = os.path.join(partition, s)
 if os.listdir(suffix):
 in_use += 1
 else:
 empty += 1
print in_use, 'in use,', empty, 'empty,', '%.02f%%' % (
 100.0 * empty / (in_use + empty)), 'empty'
And here's a quick script to clean up a drive:
NOTE THAT I HAVEN'T ACTUALLY RUN THIS ON A LIVE NODE YET!
import errno, os, os.path, sys
if len(sys.argv) != 2:
 sys.exit('%s <mount-point>' % sys.argv[0])
containers = os.path.join(sys.argv[1], 'containers')
for p in os.listdir(containers):
 partition = os.path.join(containers, p)
 for s in os.listdir(partition):
 suffix = os.path.join(partition, s)
 try:
 os.rmdir(suffix)
 except OSError, err:
 if err.errno not in (errno.ENOENT, errno.ENOTEMPTY):
 print err
Change-Id: I2e6463a4cd40597fc236ebe3e73b4b31347f2309

Sometimes you just want to know what machines a given partition
should be on, or what handoffs that partition would use if needed.
We've been meaning to add this option to swift-get-nodes for quite
some time, but I just finally got to it.
Example: swift-get-nodes /etc/swift/object.ring.gz -p 123
I tried to leave as much of the existing swift-get-nodes unaltered,
so the output isn't exactly distilled to just what you'd need for
getting a partition's nodes. But it should suffice for what it is, an
admin tool.
Change-Id: I438400ddc0eecbf9c48266e7f38a2e4f0765f374

This patch ensures that the command-line arg format (boolean)
doesn't conflict with the conf file format (string) and the
proper action is taken.
Change-Id: I3284096e1a9478897e1c3246ab190b46d2590243

The proxy server's ObjectController was performing multiple GET
requests to the object server when processing requests with Range
headers. This was a workaround for a bug in the proxy server's
Controller.GETorHEAD_base method where a response code of 416 from the
object server was incorrectly treated as a bad response from the
backend, the same way a 404 or a 5xx would be.
A 416 (Requested Range Not Satisfiable) from the object server is now
considered a good response. Since the response headers from the object
server already include the X-Object-Manifest header, there's no need
to make a second request (sans Range header) to see if the object is a
manifest.
Bonus fix: updated message for status 416 to match RFC2616.
Bonus fix 2: removed a leftover debugging print() in
test/unit/proxy/test_server.py.
Fixes bug 1065869.
Change-Id: I156af0b463f02ef19a8cfe37092544a599d89b78

Change-Id: I894473994cdfea0996ad16e7619aff421f604abc

RFC2616 implicitly allows transferring utf-8 in
headers (vals) but not in header names (keys)
Swift functional tests includes testing for utf-8
header names (keys) which is non aligned with the standard.
This seem to be the case when testing file metadata.
File metadata keys are than transferred as part of
the http header keys (header names) as
 X-Object-Meta-<metadata_key>=metadata_name.
As a result metadata_keys in swift must be composed from
a subset of ASCII characters - alpha numerics,
some punctuation, not a lot else* and cannot be utf-8.
On branch bp/wsgi-application-interface-4
modified: test/functional/tests.py
Change-Id: I1932e9d4ead83089e8a31a06d28a54f5625efb9c
Fixes: bug 1068026
Implements: blueprint wsgi-application-interface

It's not sufficient to just look at swift.object-updater.successes to
see the async_pending unlink rate. There are two different spots where
unlinks happen: one when an async_pending has been successfully
processed, and another when the updater notices multiple
async_pendings for the same object. Both events are now tracked under
the same name: swift.object-updater.unlinks.
FakeLogger has now sprouted a couple of convenience methods for
testing logged metrics.
Fixed pep8 1.3.3's complaints in the files this diff touches.
Also: bonus speling and, grammar fixes in the admin guide.
Change-Id: I8c1493784adbe24ba2b5512615e87669b3d94505

The directory tests TestContainerPathsEnv creates directorys during SetUp while indicating duplicate application internet media type (MIME) via the Content-Type header.
E.g the output on the line using 'nosetests -v test.functional.tests:TestContainerPaths.testContainerListing' includes:
Host: 127.0.0.1:8080
Accept-Encoding: identity
Content-Type: application/octet-stream
Content-Length: 0
content-type: application/directory
X-Auth-Token: AUTH_tkd40f5550289d4d60af75bb1b6c6067be
On branch bp/wsgi-application-interface-3
 modified: test/functional/tests.py
Fixes: Bug #1068499
Implements Blueprint 	wsgi-application-interface
Change-Id: I49c5d1f73fff3c06d4be18b20c7209eff4e1326c

Add trove classifier to have swift listed among the
other OpenStack-related projets on PyPI to reserve
the name, even though we won't release packages there.
Change-Id: I2c6cd2e37ea55d2bc2ddbba813ee2a190b1c90a9
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>

test_client sends a wrong size indication when testing
chunked data.
For example instead of sending:
 5\r\nabcde\r\n (which is inline with the standard)
The client would send:
 0x5\r\nabcde\r\n
On branch bp/wsgi-application-interface-2
 modified: test/functional/swift_test_client.py
Change-Id: I7597dcd24adade009aead66f67674f361f3fe4fa
Fixes: Bug #1070036
Implements: Blueprint wsgi-application-interface

Add path_qs property to swob.Request. First of all this property
has webob.Request, also this property can be used in swift3 middleware
for generate canonical string, if webob will be replaced in swift3 with
swob (2b36fbd477).
Change-Id: Idf58096baaf7830dd0d624ea6c72eda1eb91ff0d

Turns out this prioritization causes stampedes right after capacity
additions (especially on smaller clusters), which means the things it
prioritizes don't get done for several passes anyway, which means the
prioritization isn't solving the problem it was designed to do, and
it causes non-productive load spikes instead. :/
Getting rid of the prioritization doesn't make things go any slower
overall, could make things actually go faster since there are less
collisions, and evens out the load.
Change-Id: I4938bcd12d9a21fad635cfe772897e4c151f8a95

Change-Id: Ia430637f24d883ee166cb6529dde03049cc06120

In general, Linux does not care about what a mount point is named as
long as it is a vaild directory name (no / or null characters).
However, that is too relaxed for swift, which will pass that mount
point name around as part of url path construction all over the
place. To make sure that the mount point name was sane from that POV,
Swift was using isalnum to verify that the mount point name was sane,
which is overly restrictive.
This patch replaces that test with a test that verifies that the name
has no characters that need to be URL encoded.
The specific use case this enables is allowing mount points to be
named according to the UUID of the filesystem that is being mounted,
which will make Swift more robust in the face of device name instability.
Change-Id: I4d49b21c1783e97c16d3f394c2171f1f80eea058