openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Except where the python-nose package version does not support the
"--cover-branches" option.
Change-Id: I9ea6452d9834b3ac414e14e628002f593dd4b8c8
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Use <val1> if <cond> else <val2> instead.
There're more talks:
[0] http://stackoverflow.com/questions/394809/ternary-conditional-operator-in-python
[1] http://www.python.org/dev/peps/pep-0308/
Fixes bug 1188111
Change-Id: I485e674317faf319a26ab30c2c193d8464a7e46e

- Add proxy-logging to multinode. We had it since Folsom and people
 still forget it, resulting in missing logs.
- Use correct name, for ease hit with '*' in vi at least.
Admittedly trivial changes, which I meant to hold until Leah's major
doc improvement lands, but I'm tired of keeping stuff like this in
my working repo.
Change-Id: I44f80c51d6d7329a9b696e67fcb8a895db63e497

Inject some empty lines to avoid the wall-of-text effect and to make
it a little clearer which descriptions go with which options.
Change-Id: I58914b83dad76ea5ca330903a246bee7ffaeba83

There is no known way to embed slashes into container names, even when
using URL escape sequences. Since it would seem that the more common
case would be some other character, we validate using a dash ('-')
character as the delimiter.
It should also be noted that the documentation for container listing,
found here:
http://docs.openstack.org/api/openstack-object-storage/1.0/content/s_listcontainers.html,
does not describe the use of the prefix or delimiter parameters.
DocImpact
Change-Id: Ie467a4181c6f64b24c82c72b70a37183d443dee1
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Also remove the tools directory, since it is no longer used, now that
https://review.openstack.org/30887 is merged, which renamed the
requirements files to requirements.txt and test-requirements.txt.
Fixes bug 1186625
Change-Id: Icf1ef1d2d83140420163c25257d4802282063cfa

Change-Id: I8789d6e4d22de83db9a2760d51a94eb56a48c3b5

Change-Id: I9a030b84dafe2b2300e8735052f6f15f34bc7aa7

If account autocreation is on and the proxy receives a GET request for
a nonexistent account, it'll fake up a response that makes it look as
if the account exists, but without reifying that account into sqlite
DB files.
That faked-out response was just fine as long as you wanted a
text/plain response, but it didn't handle the case of format=json or
format=xml; in those cases, the response would still be
text/plain. This can break clients, and certainly causes crashes in
swift3. Now, those responses match as closely as possible.
The code for generating an account-listing response has been pulled
into (the new) swift.account.utils module, and both the fake response
and the real response use it, thus ensuring that they can't
accidentally diverge. There's also a new probe test for that
non-divergence.
Also, cleaned up a redundant matching of the Accept header in the code
for generating the account listing.
Note that some of the added tests here pass with or without this code
change; they were added because the code I was changing (parts of the
real account GET) wasn't covered by tests.
Bug 1183169
Change-Id: I2a3b8e5d9053e4d0280a320f31efa7c90c94bb06

Our tests get rather grabby with file descriptors, mostly due to the
SysLogHandler. In recent work I'm doing, my additional tests put
things just over the 1024 limit and tests would start to fail
somewhat randomly. This alleviates the problem by mocking out one of
the bigger users of SysLogHandler, proxy/test_server.py. If you want
to verify this has an impact, you can put the following in your
proxy/test_server.py's overall teardown method and try it with and
without the mocking:
 import subprocess
 print >>sys.stderr, '%s OPEN FILES' % len(subprocess.Popen(
 ['lsof'], stdout=subprocess.PIPE).communicate()[0].split('\n'))
Change-Id: I1bd3efe46ee69d09a32c5a964f04e36e46506446

Rename tools/pip-requires to requirements.txt and tools/test-requires
to test-requirements.txt. These are standard files, and tools in the
general world are growing intelligence about them.
Change-Id: Ib3e50a811868e2969923d978ee00c4f92682aa1c
Fixes: bug #1179008 

In other places in the codebase, we cache a negative result for less
time (usually a tenth) than we cache a positive result. This commit
makes TempURL do the same.
This creates a slightly nicer experience for someone trying to set up
TempURL; if they get the initial key setup wrong, try it, then fix it,
it now takes only 6 seconds until they can use signed URLs, not
60. That's a short enough time that they may not even notice the
caching of no keys.
Change-Id: I521f023e7cddaecd07f0ce32aedd4059bd0b8ec4

If an object name ends with a /, the content-disposition filename for
a tempurl ended up as an empty string. This fix just strips any
trailing slashes before calling basename.
Also, I added a test to verify that if the filename is passed as an
overriding query parameter, that it is used in full, no rstripping,
no basenaming.
Change-Id: I37725b6ded04ed3b91cdb21132490fd857276e2f

For HEAD and GET requests on both containers and accounts APIs we now
perform parameter checking before we access the database. The unit
tests were updated to show that the parameter checking is performed
before accesses are performed.
Change-Id: Ieb753316cdccabf45022e3d83522d87d34aa6b0e
Signed-off-by: Peter Portante <peter.portante@redhat.com>

The results that are being parsed are really JSON, so using a JSON parser is
more semantically correct; in addition using eval anywhere can set a bad
example.
Change-Id: Idcd55400b2571aba0f2f377cf66a3cbf4d3af960

RFC 2616 Sec 4.4 Message Length describes how the content-length and
transfer-encoding headers interact. Basically, if chunked transfer
encoding is used, the content-length header value is ignored and if
the content-length header is present, and the request is not using
chunked transfer-encoding, then the content-length must match the body
length.
The only Transfer-Coding value we support in the Transfer-Encoding
header (to date) is "chunked". RFC 2616 Sec 14.41 specifies that if
"multiple encodings have been applied to an entity, the
transfer-codings MUST be listed in the order in which they were
applied." Since we only supported "chunked". If the Transfer-Encoding
header value has multiple transfer-codings, we return a 501 (Not
Implemented) (see RFC 2616 Sec 3.6) without checking if chunked is the
last one specified. Finally, if transfer-encoding is anything but
"chunked", we return a 400 (Bad Request) to the client.
This patch adds a new method, message_length, to the swob request
object which will apply an algorithm based on RFC 2616 Sec 4.4
leveraging the existing content_length property.
In addition to these changes, the proxy server will now notice when
the message length specified by the content-length header is greater
than the configured object maximum size and fail the request with a
413, "Request Entity Too Large", before reading the entire body.
This work flows from https://review.openstack.org/27152.
Change-Id: I5d2a30b89092680dee9d946e1aafd017eaaef8c0
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Till now if a race occurs on a request to a DB when it is
first created, the loosing party die with 500...
... and an error stating a race is in the log file.
Both removed by simply accepting the fact that the DB is in fact
created by someone else - just less work for us to do... ;)
Fixes Bug #1051998
Change-Id: I8e29bf362edf2b1a84f9bffa39c04d5b3dc241b2

When added a new devices into builder the add_dev function returns it
unique id.
blueprint argparse-in-swift-ring-builder
Change-Id: I57080bb625e812f6cea71199df907a44b332b552

Dramatic part of RingBuilder.search_devs which parse a complex format
of a search device string moved to the swift-ring-builder script.
Instead, the search_devs has a simple interface to search devices.
blueprint argparse-in-swift-ring-builder
Change-Id: If3dd77b297b474fb9a058e4693fef2dfb11fca3d

* Fixed issue with running probetests with the latest update
 of python-swiftclient that removed eventlet
* Fixed issue with replication server tests to not require hard
 coded paths
Change-Id: Ibbf727ae99c0f3893ae58e270e2f879a1f618e49

Change-Id: If932b82f4e225cfdda21f47e4215c6f4c59dae4a

Adding a '/recon/version' in order to get the swift version
Change-Id: I7b7ddbe70abb87c6a3b1010ddefa09d0acc09710

Towards moving the DiskFile class into place as the API definition for
pluggable DiskFile backends, we hide the file descriptor and the
method of writing data to disks. The mkstemp() method has been renamed
to writer(), and no longer returns an fd, but a new object that
encapsulates the state tracked for writes. This new object is then
used directly to perform the reminder of the write operations and
application of required semantics.
Change-Id: Ib37ed37b34a2ce6b442d69f83ca011c918114434
Signed-off-by: Peter Portante <peter.portante@redhat.com>

DocImpact
If account reaper has not managed to clean out an account after a long
period, it prints a message to the log (you can search your system looking
for such messages). Introduce reap_warn_after config variable to determine
when to emit the message (defaults to 30 days).
Also fix bug 1181995 (edge case where object name is an empty string)
Change-Id: Ic0dfee04742d06b6a51b59f302d7a272d7c1de92

If you have StatsD logging turned on, then the iterator returned by
Ring.get_part_nodes() will emit StatsD packets when it yields a
handoff node. That network IO may make eventlet trampoline to another
greenthread before returning from next(). Now, if that other
greenthread tries to call next() on that same iterator, it blows up
with a ValueError.
Any socket IO inside a generator's next() method can cause this. It's
easiest to reproduce with StatsD logging turned on, but logging to
syslog can trigger it too.
You can see this happen sometimes in the proxy's make_requests method
if two of the primary nodes are down. Greenthread A goes into next()
to get a handoff node, then sends a StatsD packet, and so eventlet
trampolines to Greenthread B. Now, Greenthread B also calls next() to
get a handoff node, and dies with a ValueError.
This commit wraps up concurrently-accessed iter_nodes() iterators in a
new thing called a GreenthreadSafeIterator that serializes access.
Bug 1180110
Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13

Attribute get_repl_missing_table in FakeBroker class was changed in
test_replicate_object_quarantine function and not returned back. That's
why next test cases takes not expexted values from FakeBroker.
fixes bug 1180354
Change-Id: Iba55255771e6483832c7782fcbe331e20e818f4e

Change-Id: I0dee7c109d32e6325845df9ba6e1fbf23a2d1b89

See review https://review.openstack.org/29836.
Change-Id: I8ec80202789707f723abfe93ccc9cf1e677e4dc6
Signed-off-by: Peter Portante <peter.portante@redhat.com>