openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Ensures that the Allow header is set for 405 responses from the proxy,
as per RFC 2616.
CORS preflight requests now require both the Origin and Access-
Control-Request-Method headers to get a successful (200) response. The
draft spec defines errors as a response with a non-200 status code.
This patch sets the CORS error response code to 401 (Not Authorized).
A later patch may choose to make this configurable.
There is some ambiguity between RFC 2616 and the CORS draft spec
around what to do when a CORS request is made but the cluster has no
CORS information about the requested resource. This patch chooses to
return an error in this case because it is what would be simplest for
CORS client apps.
Further improvements to the OPTIONS verb not included in this patch
 include support of more top-level resources (eg / or /v1/) or
sending the configured constraints in the reponse body.
Change-Id: I40be059e8bbf3737dafc4e6fefa7598d05669c60

Change-Id: Ib54fa5adb7539bebfbd6644064be9d7f4d7af9db

Fix bug 1067677
When we delete a versioned object, the last version will be poped out from the
versiones container. When a versioned object is expired and deleted by object-
expirer, the last version is restored but remains in the versions container
instead of getting deleted. The reason is object-expirer will set
'X-If-Delete- At' header when deleteing an object. However this is for the
current version - not for the last version. When the object-server is trying
to delete the last version, the transaction will fail with error:
X-If-Delete-At and X-Delete-At do not match. Delete the 'X-If-Delete-At'
field in the later built delete request would help to solve this issue.
This patch, without the test, was first proposed by
Zhou Yuan <yuan.zhou@intel.com>.
Change-Id: I62c97e6a0f140888497e189129825865fb6f7966

Currently, a container's info can be cached without cors data intact after
a container GET.
I made headers_to_container_info a function instead of a method and I crammed
all container metadata into container_info.
This is so e.g. staticweb can eventually re-use the same container info cache.
Fix pep8 in swift/proxy/controllers/container.py
Change-Id: I4bbb042dde79afac48395efc38bd80f0ff240e1f

This change adds multi range retrieval to OpenStack Swift. For non-
segmented data object, a client can use HTTP Range header to specify
multiple ranges to retrieve sections of the data object.
This implementation currently does not support segmented data object
multi range retrieval. When a client sends a multi range request
against a segmented data object, Swift will return HTTP status code
200. Support for segmented data multi range retrieval will be added
in near future.
This implementation is to bring Swift closer to CDMI multi range
data retrieval standard. Once support for segemented data multi
range is added, Swift will be compliant with CDMI standard in
this area.
DocImpact
Change-Id: I4ed1fb0a0a93c037ddb2f551ea62afe447945107

If an object server is down, container-sync stops syncing the container
even if the it gets object copies from "up" obejct servers.
Bug 1069910
In case the git history gets mangled, this fix was done almost entirely
by Donagh McCabe <donagh.mccabe@hp.com>.
Change-Id: Ieeadcfeb4e880fe5f08e284d7c12492bf7a29460

There were bugs with SCRIPT_NAME and
swift.common.wsgi.make_pre_authed* functions.
1) SCRIPT_NAME wasn't copied with PATH_INFO, which it should've been.
2) When a new path was given, SCRIPT_NAME wasn't set to an empty
 string but just omitted, which is allowed by spec, but really
 should be set.
For completeness, if SCRIPT_NAME doesn't exist in the source env, it
will be created in the new env, but as an empty string.
Change-Id: Ifbc27ed8ff357038c54df4d37de46cfd5e31372e

cleaned up pep8 (v1.3.3) in all files this patch touches
Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff

Swift proxy server, when communicating with
the back-end servers ALWAYS sends 100-continue.
Even if the length headers indicate that there
is no body (content-length: 0).
This behavior is not inline with the standard.
RFC2616:8.2.3 disallows 100-continue
 for req.content_length==0
This fix removes 100-continue during PUT requests
without a body while maintining the ability for
handoff in case of faliure.
On branch bp/wsgi-application-interface-5
 modified: swift/proxy/controllers/obj.py
Fixes: Bug #1070025
Implements Blueprint: 	wsgi-application-interface
Change-Id: Ia4eb8b886a74968cca4e8bde208641b37f2c104c

Fixes bug 1071790.
Change-Id: Ic0509190b9994e7d983bcf20daa64a650cc974cb

When a db is reclaimed it removes the hash dir the db files are in,
but it does not try to remove the parent suffix dir though it might
be empty now. This eventually leads to a bunch of empty suffix dirs
lying around. This patch fixes that by attempting to remove the
parent suffix dir after a hash dir reclamation.
Here's a quick script to see how bad a given drive might be:
import os, os.path, sys
if len(sys.argv) != 2:
 sys.exit('%s <mount-point>' % sys.argv[0])
in_use = 0
empty = 0
containers = os.path.join(sys.argv[1], 'containers')
for p in os.listdir(containers):
 partition = os.path.join(containers, p)
 for s in os.listdir(partition):
 suffix = os.path.join(partition, s)
 if os.listdir(suffix):
 in_use += 1
 else:
 empty += 1
print in_use, 'in use,', empty, 'empty,', '%.02f%%' % (
 100.0 * empty / (in_use + empty)), 'empty'
And here's a quick script to clean up a drive:
NOTE THAT I HAVEN'T ACTUALLY RUN THIS ON A LIVE NODE YET!
import errno, os, os.path, sys
if len(sys.argv) != 2:
 sys.exit('%s <mount-point>' % sys.argv[0])
containers = os.path.join(sys.argv[1], 'containers')
for p in os.listdir(containers):
 partition = os.path.join(containers, p)
 for s in os.listdir(partition):
 suffix = os.path.join(partition, s)
 try:
 os.rmdir(suffix)
 except OSError, err:
 if err.errno not in (errno.ENOENT, errno.ENOTEMPTY):
 print err
Change-Id: I2e6463a4cd40597fc236ebe3e73b4b31347f2309

The proxy server's ObjectController was performing multiple GET
requests to the object server when processing requests with Range
headers. This was a workaround for a bug in the proxy server's
Controller.GETorHEAD_base method where a response code of 416 from the
object server was incorrectly treated as a bad response from the
backend, the same way a 404 or a 5xx would be.
A 416 (Requested Range Not Satisfiable) from the object server is now
considered a good response. Since the response headers from the object
server already include the X-Object-Manifest header, there's no need
to make a second request (sans Range header) to see if the object is a
manifest.
Bonus fix: updated message for status 416 to match RFC2616.
Bonus fix 2: removed a leftover debugging print() in
test/unit/proxy/test_server.py.
Fixes bug 1065869.
Change-Id: I156af0b463f02ef19a8cfe37092544a599d89b78

Change-Id: I894473994cdfea0996ad16e7619aff421f604abc

RFC2616 implicitly allows transferring utf-8 in
headers (vals) but not in header names (keys)
Swift functional tests includes testing for utf-8
header names (keys) which is non aligned with the standard.
This seem to be the case when testing file metadata.
File metadata keys are than transferred as part of
the http header keys (header names) as
 X-Object-Meta-<metadata_key>=metadata_name.
As a result metadata_keys in swift must be composed from
a subset of ASCII characters - alpha numerics,
some punctuation, not a lot else* and cannot be utf-8.
On branch bp/wsgi-application-interface-4
modified: test/functional/tests.py
Change-Id: I1932e9d4ead83089e8a31a06d28a54f5625efb9c
Fixes: bug 1068026
Implements: blueprint wsgi-application-interface

It's not sufficient to just look at swift.object-updater.successes to
see the async_pending unlink rate. There are two different spots where
unlinks happen: one when an async_pending has been successfully
processed, and another when the updater notices multiple
async_pendings for the same object. Both events are now tracked under
the same name: swift.object-updater.unlinks.
FakeLogger has now sprouted a couple of convenience methods for
testing logged metrics.
Fixed pep8 1.3.3's complaints in the files this diff touches.
Also: bonus speling and, grammar fixes in the admin guide.
Change-Id: I8c1493784adbe24ba2b5512615e87669b3d94505

The directory tests TestContainerPathsEnv creates directorys during SetUp while indicating duplicate application internet media type (MIME) via the Content-Type header.
E.g the output on the line using 'nosetests -v test.functional.tests:TestContainerPaths.testContainerListing' includes:
Host: 127.0.0.1:8080
Accept-Encoding: identity
Content-Type: application/octet-stream
Content-Length: 0
content-type: application/directory
X-Auth-Token: AUTH_tkd40f5550289d4d60af75bb1b6c6067be
On branch bp/wsgi-application-interface-3
 modified: test/functional/tests.py
Fixes: Bug #1068499
Implements Blueprint 	wsgi-application-interface
Change-Id: I49c5d1f73fff3c06d4be18b20c7209eff4e1326c

test_client sends a wrong size indication when testing
chunked data.
For example instead of sending:
 5\r\nabcde\r\n (which is inline with the standard)
The client would send:
 0x5\r\nabcde\r\n
On branch bp/wsgi-application-interface-2
 modified: test/functional/swift_test_client.py
Change-Id: I7597dcd24adade009aead66f67674f361f3fe4fa
Fixes: Bug #1070036
Implements: Blueprint wsgi-application-interface

Add path_qs property to swob.Request. First of all this property
has webob.Request, also this property can be used in swift3 middleware
for generate canonical string, if webob will be replaced in swift3 with
swob (2b36fbd477).
Change-Id: Idf58096baaf7830dd0d624ea6c72eda1eb91ff0d

In general, Linux does not care about what a mount point is named as
long as it is a vaild directory name (no / or null characters).
However, that is too relaxed for swift, which will pass that mount
point name around as part of url path construction all over the
place. To make sure that the mount point name was sane from that POV,
Swift was using isalnum to verify that the mount point name was sane,
which is overly restrictive.
This patch replaces that test with a test that verifies that the name
has no characters that need to be URL encoded.
The specific use case this enables is allowing mount points to be
named according to the UUID of the filesystem that is being mounted,
which will make Swift more robust in the face of device name instability.
Change-Id: I4d49b21c1783e97c16d3f394c2171f1f80eea058

There have been a bunch of Jenkins failures lately where the StatsD
tests fail because they can't bind to their desired port. There's
nothing special about the particular port they're using, so now we let
the kernel pick an available one for us.
This also lets us get rid of a sleep() in the test that looked like an
attempt to alleviate EADDRINUSE errors, so now in the happy case, the
tests are a few fractions of a second faster.
Change-Id: Idee11349254107a59643539b1566f3588eee7ef4

This patch merely fixes a selection of files to the point where
pep8 1.3.3 is happy. Most of the errors are indentation related to
continued lines (E126, E127, E128), bracket positions (E124) and the
use of backslash (E502).
Patch 2 fixes David's comments regarding backslash and an odd comment
 - thanks David!
Change-Id: I4fbd77ecf5395743cb96acb95fa946c322c16560

It's there to let administrators turn down the barrage of stats data
that StatsD must cope with, but it wasn't actually honored. Worse, if
the sample rate was set to e.g. 0.2, the stats would all be multiplied
by its inverse, e.g. 2. This patch actually drops packets when
sample_rate < 1, so you get correct measurements.
Fortunately, the default sample rate is 1 (i.e. drop nothing), and
multiplying by 1/1 doesn't change anything, so stats with the default
sample rate of 1.0 are, and have been, just fine.
Fixes bug 1065643.
Also, make the two touched files compliant with pep8 v1.3.3.
Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3

Change-Id: I2c1246b9bbd1d3ab22c2a035b735d937dd90da11

Change-Id: Ifd609f305ee878c39ea4e6840ed8fa9369595cfa

_get_metadata() should only look for metadata on 200 family http status codes
in the path_parts() in the internal client test file should return distinct unicode strings
with spaces for account, container, and obj
Change-Id: I74e335d3b73a59092a0eec06539f4931420061dd

fixes bug 1049017
As per the bug report, the -remove- form of deletion only works with
 meta headers and not with x-container-read and x-container-write.
This patch by Yukihiro KAWADA detects the container acls and sends
them through to the backend empty like the other metadata values.
patch2 extends metadata-helper in ContainerController tests so that
the new functionality can be tested.
patch3 changes the k.lower().startswith() commands for read/write to
a single k.lower - thanks David :)
patch4 fixes PEP8
patch5 fixes more than two hundred pep8 errors. Except one,
 where the pep8 tool complains of overindent, or underindent
depending on the position on the line. No idea how to fix that one.
patch6 fixes the remaining pep8 error - thanks Darrell
Change-Id: I36c2dd72b4636136c5ce7db414bf2b61501090ad

Change-Id: Ic263f4a77a0aa0eb40078772a567eb41a60e40f7

This fixes swob to handle "Range: bytes=-X" where X > len(content); ex.
"Range: bytes=-17" when the object has 10 bytes. Based on the RFC, the
range is satisfiable and all bytes should be returned.
It also fixes "Range: bytes=-0" to be, correctly, not satisfiable. In
addition, this case's response has Content-Length: 0 and has a zero-byte
body.
It also fixes an existing regression in swob for the case
"Range: bytes=100-" for a body of length < 100 (Content-Length was
negative and the body was returned).
The relevant RFC is 2616, section 14.35.1.
Change-Id: Ib3dc672e083173eb970c10801283813623f26e0e

made internal client handle unicode path parts by adding make_path method
fixed pep8 problems in internal client and its test
moved internal client unit test file to correct directory
Change-Id: Id1c81c9cb0db05342e4e8a8393db93552fda4647

Have swift-init warn when the running user doesn't have permissions to
signal processes. Fixes bug 1017494.
Change-Id: Icb9048ab36f1ca73bb93b11c9c2aed882d99dfa7