openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Every request of container and object will invoke account_info() or
container_info() to query the meta of account and container. The meta
will be cached in memcache with the key 'account/{$account}' or
'container/{$container}', So, if any request to update account and
container, we should delete the cache. But in the cache deletion of
account, it use the wrong key 'account/v1/{$account}'. This could lead
to inconsistency of account meta.
Change-Id: Ied116a58a2d5866ac76d75ae50f21277d66e5755

Most of the test files set the HASH_PATH_SUFFIX so you can run the test
file stand alone. This change made it easier for me to run specific
proxy tests separately.
Change-Id: I87d70367dac7f240a2b6779649f8a02cf324ae0f

Change-Id: Iab43ac342faf3722ccab1ed034f08752dbbfaae1

I know it's just TempAuth, but bug #959953 just caught my eye as
something interesting to solve.
This does a best guess on the storage URL to return for a given
request. It allows $HOST to be used in the storage URL configuration,
where $HOST will resolve to scheme://host:port. It bases the scheme
on how the server is running or on storage_url_scheme if set. The
host:port comes from the request's Host header if it exists, and
falls back to the WSGI SERVER_NAME:SERVER_PORT otherwise.
Fixes: bug #959953
DocImpact
Change-Id: Ia494bcb99a04490911ee8d2cb8b12a94e77820c5

Change-Id: I3f0db3b70ea5df323597b556682c2d1fb28afe53

Before, swift-bench would create a string of the right size and pass
that to [direct_]put_object. Uploading 5 GB objects w/concurrency of
4? Hope you've got a spare 20 GB of RAM in that machine.
Now it lazily generates chunks (of size 64 KiB) to send. In my limited
testing, this keeps RAM usage low while not impacting speed.
There's backwards-compatibility code in
direct_client.direct_put_object for any callers who are still passing
in strings. There aren't any such in Swift's codebase, though. All one
of those callers now pass an iterable.
Fixes bug 911925.
Change-Id: I40669fc00c7fb6699d8fb514e1be9b69436eca42

This should fix up the few remaining API regressions from swob, by giving all
responses a default content-type of "text/html; charset=UTF-8".
Change-Id: Ib3b859e63c5e8c4af6f7fe535947402b5df57399

It is already the case that a DELETE of a deleted object results in a
404, and GET/HEAD/POSTs to both expired and deleted objects result in
404s. However, a DELETE of an expired object resulted in a 202. This
change makes it consistent with the other verbs.
Fixes bug 1076245.
Change-Id: I793e62d72461a4fb9fb3404e10658ddcc4c3a7a6

Set a utf-8 charset on default text/html responses.
Change-Id: Ic5f661bd475dca83763d4b55419ad031279e3ba1

If the Host header is available, use it for making relative URLs absolute.
Otherwise, continue using SERVER_NAME.
Change-Id: Ifc028264ad1b122a2d5dff9d5528cb369090429f

Change-Id: I72448cd8578b53a658f151354a26066772e4b9ec

These bug fixes are lumped together because they all caused problems
with the object expirer doing its job.
There was a bug with the internal client doing listings that happened
to run across a Unicode object name for use as a marker.
There was a bug with the object expirer not utf8 encoding object
names it got from json listings, causing deletes to fail.
There was a bug with the object expirer url quoting object names when
calling the internal client's make_request, when make_request already
handles that.
Change-Id: I29fdd351fd60c8e63874b44d604c5fdff35169d4

The object server knows the size of the file on disk already since
it's counted the bytes as they were written out. There's no need to go
ask the filesystem.
This also lets object PUTs work when the filesystem is some crazy,
non-POSIX thing that doesn't always give the right answers from
fstat(). See bug 904723 for a case when this happened.
Fixes bug 904723.
Change-Id: Ibdf44ea991e0cc4f9e0997cc9ff566cb0eac5332

Change-Id: Ifc29d286a3e435fd16c1dec8b5afb7b331a89b77

Change-Id: Iee286fbe1a6b797955579bedfb2a80095c055cd8

Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4

Change-Id: I77e1d7fdcf217826402beeb7d583e3c7279c416c

Ensures that the Allow header is set for 405 responses from the proxy,
as per RFC 2616.
CORS preflight requests now require both the Origin and Access-
Control-Request-Method headers to get a successful (200) response. The
draft spec defines errors as a response with a non-200 status code.
This patch sets the CORS error response code to 401 (Not Authorized).
A later patch may choose to make this configurable.
There is some ambiguity between RFC 2616 and the CORS draft spec
around what to do when a CORS request is made but the cluster has no
CORS information about the requested resource. This patch chooses to
return an error in this case because it is what would be simplest for
CORS client apps.
Further improvements to the OPTIONS verb not included in this patch
 include support of more top-level resources (eg / or /v1/) or
sending the configured constraints in the reponse body.
Change-Id: I40be059e8bbf3737dafc4e6fefa7598d05669c60

Change-Id: Ib54fa5adb7539bebfbd6644064be9d7f4d7af9db

Using github as a package source is less reliable in the Jenkins test
framework than using pypi. This is because the Jenkins test framework
has a local pypi mirror.
Change-Id: Ia904d39ce70f5c2bd7d6edb9172f8226131074e2

content_range_header() took an optional parameter 'value_only'. If
true, it returned just the header value; if false, the whole header.
At all the call sites, value_only was either a hardcoded True or
False.
Instead, let's have two methods (one for the full header, one for the
header value) and just call the right one.
Change-Id: I3aa03aca233437e9bdbed7edf9622756e19b4c49

self.status would have been "200 OK", not 200, so the branch was never
taken. self.status_int would have been 200.
This looked like part of an attempt to honor the Range header in GET
requests. However, when I changed the code to check if self.status_int
== 200, the behavior gets weird. In the case of a GET request w/Range
header, the status gets changed from a 200 (from the underlying
app/middleware stack) into a 206. However, the response body isn't
actually changed to be of the right length. The net result is a
response w/status "206 Partial Content" that actually contains the
full content.
RFC 2616 section 14.35.2 says "A server MAY ignore the Range header."
The right thing to do is either (a) return a 206 with actual partial
content, or (b) ignore the Range header. For object GETs, there's
already code to do (a) [see commit ce274b3]. For other GETs, we should
ignore the Range header instead of returning a Frankenresponse.
This code is dead, and rightly so; let's bury it six feet deep in the
bit bucket.
Fixes bug 1068279.
Change-Id: Id648dfda9cba45012b5da097a235598fbd97c8b0

Fix bug 1067677
When we delete a versioned object, the last version will be poped out from the
versiones container. When a versioned object is expired and deleted by object-
expirer, the last version is restored but remains in the versions container
instead of getting deleted. The reason is object-expirer will set
'X-If-Delete- At' header when deleteing an object. However this is for the
current version - not for the last version. When the object-server is trying
to delete the last version, the transaction will fail with error:
X-If-Delete-At and X-Delete-At do not match. Delete the 'X-If-Delete-At'
field in the later built delete request would help to solve this issue.
This patch, without the test, was first proposed by
Zhou Yuan <yuan.zhou@intel.com>.
Change-Id: I62c97e6a0f140888497e189129825865fb6f7966

Due to a lack of zero-padding in a format string, there was a 1/16
chance that the length of the randomly-generated boundary string was
less than 32. This didn't affect client downloads, but it did give the
unit tests a 1/16 chance of failing (there's a [0-9a-f]{32} regex in a
test).
Now we left-zero-pad the boundary string to length 32, and the test
consistently passes.
Change-Id: I4ee98dab127e5674e59b078d8090470ec783e0b0

Separate the request processor initialization steps
 Load common settings from conf
 Set the logger
 Load the request processor
from the web front end server application step
On branch bp/wsgi-application-interface-7
 modified: swift/common/wsgi.py
Implements Blueprint: wsgi-application-interface
Change-Id: I25fabad8846cd644e4bf1e1bd9c2e568512e12e5

Currently, a container's info can be cached without cors data intact after
a container GET.
I made headers_to_container_info a function instead of a method and I crammed
all container metadata into container_info.
This is so e.g. staticweb can eventually re-use the same container info cache.
Fix pep8 in swift/proxy/controllers/container.py
Change-Id: I4bbb042dde79afac48395efc38bd80f0ff240e1f

This change adds multi range retrieval to OpenStack Swift. For non-
segmented data object, a client can use HTTP Range header to specify
multiple ranges to retrieve sections of the data object.
This implementation currently does not support segmented data object
multi range retrieval. When a client sends a multi range request
against a segmented data object, Swift will return HTTP status code
200. Support for segmented data multi range retrieval will be added
in near future.
This implementation is to bring Swift closer to CDMI multi range
data retrieval standard. Once support for segemented data multi
range is added, Swift will be compliant with CDMI standard in
this area.
DocImpact
Change-Id: I4ed1fb0a0a93c037ddb2f551ea62afe447945107

If an object server is down, container-sync stops syncing the container
even if the it gets object copies from "up" obejct servers.
Bug 1069910
In case the git history gets mangled, this fix was done almost entirely
by Donagh McCabe <donagh.mccabe@hp.com>.
Change-Id: Ieeadcfeb4e880fe5f08e284d7c12492bf7a29460

Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.
DocImpact
Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0

There were bugs with SCRIPT_NAME and
swift.common.wsgi.make_pre_authed* functions.
1) SCRIPT_NAME wasn't copied with PATH_INFO, which it should've been.
2) When a new path was given, SCRIPT_NAME wasn't set to an empty
 string but just omitted, which is allowed by spec, but really
 should be set.
For completeness, if SCRIPT_NAME doesn't exist in the source env, it
will be created in the new env, but as an empty string.
Change-Id: Ifbc27ed8ff357038c54df4d37de46cfd5e31372e