openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Set a utf-8 charset on default text/html responses.
Change-Id: Ic5f661bd475dca83763d4b55419ad031279e3ba1

Change-Id: I72448cd8578b53a658f151354a26066772e4b9ec

These bug fixes are lumped together because they all caused problems
with the object expirer doing its job.
There was a bug with the internal client doing listings that happened
to run across a Unicode object name for use as a marker.
There was a bug with the object expirer not utf8 encoding object
names it got from json listings, causing deletes to fail.
There was a bug with the object expirer url quoting object names when
calling the internal client's make_request, when make_request already
handles that.
Change-Id: I29fdd351fd60c8e63874b44d604c5fdff35169d4

The object server knows the size of the file on disk already since
it's counted the bytes as they were written out. There's no need to go
ask the filesystem.
This also lets object PUTs work when the filesystem is some crazy,
non-POSIX thing that doesn't always give the right answers from
fstat(). See bug 904723 for a case when this happened.
Fixes bug 904723.
Change-Id: Ibdf44ea991e0cc4f9e0997cc9ff566cb0eac5332

Change-Id: Ifc29d286a3e435fd16c1dec8b5afb7b331a89b77

Change-Id: Iee286fbe1a6b797955579bedfb2a80095c055cd8

Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4

Change-Id: I77e1d7fdcf217826402beeb7d583e3c7279c416c

Ensures that the Allow header is set for 405 responses from the proxy,
as per RFC 2616.
CORS preflight requests now require both the Origin and Access-
Control-Request-Method headers to get a successful (200) response. The
draft spec defines errors as a response with a non-200 status code.
This patch sets the CORS error response code to 401 (Not Authorized).
A later patch may choose to make this configurable.
There is some ambiguity between RFC 2616 and the CORS draft spec
around what to do when a CORS request is made but the cluster has no
CORS information about the requested resource. This patch chooses to
return an error in this case because it is what would be simplest for
CORS client apps.
Further improvements to the OPTIONS verb not included in this patch
 include support of more top-level resources (eg / or /v1/) or
sending the configured constraints in the reponse body.
Change-Id: I40be059e8bbf3737dafc4e6fefa7598d05669c60

Change-Id: Ib54fa5adb7539bebfbd6644064be9d7f4d7af9db

Using github as a package source is less reliable in the Jenkins test
framework than using pypi. This is because the Jenkins test framework
has a local pypi mirror.
Change-Id: Ia904d39ce70f5c2bd7d6edb9172f8226131074e2

content_range_header() took an optional parameter 'value_only'. If
true, it returned just the header value; if false, the whole header.
At all the call sites, value_only was either a hardcoded True or
False.
Instead, let's have two methods (one for the full header, one for the
header value) and just call the right one.
Change-Id: I3aa03aca233437e9bdbed7edf9622756e19b4c49

self.status would have been "200 OK", not 200, so the branch was never
taken. self.status_int would have been 200.
This looked like part of an attempt to honor the Range header in GET
requests. However, when I changed the code to check if self.status_int
== 200, the behavior gets weird. In the case of a GET request w/Range
header, the status gets changed from a 200 (from the underlying
app/middleware stack) into a 206. However, the response body isn't
actually changed to be of the right length. The net result is a
response w/status "206 Partial Content" that actually contains the
full content.
RFC 2616 section 14.35.2 says "A server MAY ignore the Range header."
The right thing to do is either (a) return a 206 with actual partial
content, or (b) ignore the Range header. For object GETs, there's
already code to do (a) [see commit ce274b3]. For other GETs, we should
ignore the Range header instead of returning a Frankenresponse.
This code is dead, and rightly so; let's bury it six feet deep in the
bit bucket.
Fixes bug 1068279.
Change-Id: Id648dfda9cba45012b5da097a235598fbd97c8b0

Fix bug 1067677
When we delete a versioned object, the last version will be poped out from the
versiones container. When a versioned object is expired and deleted by object-
expirer, the last version is restored but remains in the versions container
instead of getting deleted. The reason is object-expirer will set
'X-If-Delete- At' header when deleteing an object. However this is for the
current version - not for the last version. When the object-server is trying
to delete the last version, the transaction will fail with error:
X-If-Delete-At and X-Delete-At do not match. Delete the 'X-If-Delete-At'
field in the later built delete request would help to solve this issue.
This patch, without the test, was first proposed by
Zhou Yuan <yuan.zhou@intel.com>.
Change-Id: I62c97e6a0f140888497e189129825865fb6f7966

Due to a lack of zero-padding in a format string, there was a 1/16
chance that the length of the randomly-generated boundary string was
less than 32. This didn't affect client downloads, but it did give the
unit tests a 1/16 chance of failing (there's a [0-9a-f]{32} regex in a
test).
Now we left-zero-pad the boundary string to length 32, and the test
consistently passes.
Change-Id: I4ee98dab127e5674e59b078d8090470ec783e0b0

Separate the request processor initialization steps
 Load common settings from conf
 Set the logger
 Load the request processor
from the web front end server application step
On branch bp/wsgi-application-interface-7
 modified: swift/common/wsgi.py
Implements Blueprint: wsgi-application-interface
Change-Id: I25fabad8846cd644e4bf1e1bd9c2e568512e12e5

Currently, a container's info can be cached without cors data intact after
a container GET.
I made headers_to_container_info a function instead of a method and I crammed
all container metadata into container_info.
This is so e.g. staticweb can eventually re-use the same container info cache.
Fix pep8 in swift/proxy/controllers/container.py
Change-Id: I4bbb042dde79afac48395efc38bd80f0ff240e1f

This change adds multi range retrieval to OpenStack Swift. For non-
segmented data object, a client can use HTTP Range header to specify
multiple ranges to retrieve sections of the data object.
This implementation currently does not support segmented data object
multi range retrieval. When a client sends a multi range request
against a segmented data object, Swift will return HTTP status code
200. Support for segmented data multi range retrieval will be added
in near future.
This implementation is to bring Swift closer to CDMI multi range
data retrieval standard. Once support for segemented data multi
range is added, Swift will be compliant with CDMI standard in
this area.
DocImpact
Change-Id: I4ed1fb0a0a93c037ddb2f551ea62afe447945107

If an object server is down, container-sync stops syncing the container
even if the it gets object copies from "up" obejct servers.
Bug 1069910
In case the git history gets mangled, this fix was done almost entirely
by Donagh McCabe <donagh.mccabe@hp.com>.
Change-Id: Ieeadcfeb4e880fe5f08e284d7c12492bf7a29460

Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.
DocImpact
Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0

There were bugs with SCRIPT_NAME and
swift.common.wsgi.make_pre_authed* functions.
1) SCRIPT_NAME wasn't copied with PATH_INFO, which it should've been.
2) When a new path was given, SCRIPT_NAME wasn't set to an empty
 string but just omitted, which is allowed by spec, but really
 should be set.
For completeness, if SCRIPT_NAME doesn't exist in the source env, it
will be created in the new env, but as an empty string.
Change-Id: Ifbc27ed8ff357038c54df4d37de46cfd5e31372e

cleaned up pep8 (v1.3.3) in all files this patch touches
Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff

Swift proxy server, when communicating with
the back-end servers ALWAYS sends 100-continue.
Even if the length headers indicate that there
is no body (content-length: 0).
This behavior is not inline with the standard.
RFC2616:8.2.3 disallows 100-continue
 for req.content_length==0
This fix removes 100-continue during PUT requests
without a body while maintining the ability for
handoff in case of faliure.
On branch bp/wsgi-application-interface-5
 modified: swift/proxy/controllers/obj.py
Fixes: Bug #1070025
Implements Blueprint: 	wsgi-application-interface
Change-Id: Ia4eb8b886a74968cca4e8bde208641b37f2c104c

Change-Id: I78a5c109c9440df752e390698502f57d4392fb67

Fixes bug 1071790.
Change-Id: Ic0509190b9994e7d983bcf20daa64a650cc974cb

When a db is reclaimed it removes the hash dir the db files are in,
but it does not try to remove the parent suffix dir though it might
be empty now. This eventually leads to a bunch of empty suffix dirs
lying around. This patch fixes that by attempting to remove the
parent suffix dir after a hash dir reclamation.
Here's a quick script to see how bad a given drive might be:
import os, os.path, sys
if len(sys.argv) != 2:
 sys.exit('%s <mount-point>' % sys.argv[0])
in_use = 0
empty = 0
containers = os.path.join(sys.argv[1], 'containers')
for p in os.listdir(containers):
 partition = os.path.join(containers, p)
 for s in os.listdir(partition):
 suffix = os.path.join(partition, s)
 if os.listdir(suffix):
 in_use += 1
 else:
 empty += 1
print in_use, 'in use,', empty, 'empty,', '%.02f%%' % (
 100.0 * empty / (in_use + empty)), 'empty'
And here's a quick script to clean up a drive:
NOTE THAT I HAVEN'T ACTUALLY RUN THIS ON A LIVE NODE YET!
import errno, os, os.path, sys
if len(sys.argv) != 2:
 sys.exit('%s <mount-point>' % sys.argv[0])
containers = os.path.join(sys.argv[1], 'containers')
for p in os.listdir(containers):
 partition = os.path.join(containers, p)
 for s in os.listdir(partition):
 suffix = os.path.join(partition, s)
 try:
 os.rmdir(suffix)
 except OSError, err:
 if err.errno not in (errno.ENOENT, errno.ENOTEMPTY):
 print err
Change-Id: I2e6463a4cd40597fc236ebe3e73b4b31347f2309

Sometimes you just want to know what machines a given partition
should be on, or what handoffs that partition would use if needed.
We've been meaning to add this option to swift-get-nodes for quite
some time, but I just finally got to it.
Example: swift-get-nodes /etc/swift/object.ring.gz -p 123
I tried to leave as much of the existing swift-get-nodes unaltered,
so the output isn't exactly distilled to just what you'd need for
getting a partition's nodes. But it should suffice for what it is, an
admin tool.
Change-Id: I438400ddc0eecbf9c48266e7f38a2e4f0765f374

This patch ensures that the command-line arg format (boolean)
doesn't conflict with the conf file format (string) and the
proper action is taken.
Change-Id: I3284096e1a9478897e1c3246ab190b46d2590243