openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Adding nosehtmloutput as a test dependency allows nose to output its
results to an html file. This will be used by Jenkins to save logs on
a different server.
Change-Id: I6bd4736e8cc4132db65bfe8cdfe1f3a388369fa8

- If lock_file encounters a LockTimeout a fd can be left behind
- Fixes bug 1038191
Change-Id: Ib3f45d18aeeec91eedf0b4f0492664d33b4f57fa

Change-Id: I1f91a82da064c9f0e64779f4fa8951962678831b

_make_app_iter_reader uses a queue to pass read data to
_make_app_iter, and uses an empty string as a terminator. The problem
is that it pushes the same terminator even if it fails to reads the
data. If an exception happens before pushing all data to the queue,
_make_app_iter exits without yielding all data. In that case,
proxy-server stops in the middle of sending data, and a client waits
for all data to come infinitely.
This patch uses a boolean value for the terminator instead of an empty
string to distinguish the results, and raises an exception in the
error cases. Fixes bug 890888.
Change-Id: Ie4d48025843aa1bb600446da6f4fc49f82a440bf

Change-Id: I00bf2e20c2683d0440ca425461e0ab87197ee720

bug fixes for swift 1033089
Swift 1.6.0 All in One setup uses rc.local to create a set of
directories like the following:
 /var/cache/swift /var/cache/swift2 /var/cache/swift3
 /var/cache/swift4
 /var/run/swift
without -p for mkdir command, when some of these directories
already exist, the rc.local fails so that the next set of
command won't execute. This causes swift fail to start because
of /var/run/swift does not exist. Adding -p to both mkdir
command solves this problem.
Change-Id: Ia26c9a857723747008293c0631c9acc1b58eca7e

Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays. This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7). Fixes bug 1031954.
swift.common.ring.ring.RingData is now responsible for serialization and
deserialization of its data via a new load() class method and save()
object method. The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before. So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings. THIS SHOULD BE MENTIONED PROMINENTLY IN THE
RELEASE NOTES.
I didn't want to bite of more than necessary, so I didn't mess with
builder file serialization.
Change-Id: I799b9a4c894d54fb16592443904ac055b2638e2d

Change-Id: I64647c5d613a0794f0807886a4833cd5c31c0c5e

Change-Id: I45952adfcfbc813477ee9966f05dcdeef23c683b

Change-Id: I95a750613b01bdc4d64444d7e7fae8190760d697

It wasn't logging the full path, depending on if WebOb's
path_info_pop had been called.
Change-Id: I612d4cae7a4dc1bf5164356af329496245ad85dc

This ensures req.accepts is only assigned a valid content type.
Change-Id: Idf19277959c516398d31f771cf42dec0733d1bc0

The WebOb 1.2 support patch introduced a regression where an unknown
format= query used to just default back to plain but now returns a
400. This will break anyone that uses format=text for instance, or
anything else "invalid".
Arguably, it makes the most sense to 400 in the case of an invalid
format requested, but since it is also a backwards compatibility
break, I recommend we keep the previous behavior. Retaining the
previous behavior seems less damaging than enforcing the new
behavior.
Change-Id: I6db015b33a6f3ab239b8cb4a3562ebdba1f76590

- swift-recon cli now reports all unmounted drives.
- Fixes bug 1031060
Change-Id: Ia9e6bbf05215879b602e9dcb77d007094c494dc1

Change-Id: Ib2d54da92a3f7e7325998e3e52f562a1bf538c28

A warning log line is emitted whenever the proxy has to use a handoff
node. Monitoring these warnings can indicate a problem within your
cluster; however, you can disable these log lines by setting the
proxy conf's log_handoffs to false.
While working on this, I also noticed why many proxy log lines did
not have txn_id and client_ip -- subcoroutines. Now the logger thread
locals are copied to the subcoroutines.
Change-Id: Ibac086e1b985f566c068d083620287509de35da8

Change-Id: I7e5e254660e2ce35bc7fa95435db0a987d5acd0a

Change-Id: I683633ef1c7a5df7522299a85090f1ffda0c287a

Change-Id: Ia97050ac2b0b438535f31a8eb400801c782160f9

Change-Id: I4a07ff105f2acc4174de0aef4e24cfcde323bea8

Change-Id: Ic2869ab7f728ed1a4d8b34c37f0540c582d7dbcc

Fixes bug #1024477
Change-Id: Id09bbff6bb4a5d032727d3ca53b3276b4769d178

Change-Id: If5a6d32c40de02183a2eed6e2a32d62ba38df32d

If fallocate can't be loaded from libc, bit posix_fallocate loaded
user will Swift will log warning messages. So it will be better
to write log only if both version of fallocate can't be loaded.
Change-Id: I2a9ffe4756e77b3010ccd82b33c208d6d9b9f7b7

In comments from https://review.openstack.org/8798 it was raised that it
might make sense to forbid some substrings in the name_filter
middleware.
There is now a new forbidden_regexp option for the name_filter
middleware to specify which substrings to forbid. The default is
"/\./|/\.\./|/\.$|/\.\.$" (or in a non-regexp language: the /./ and /../
substrings as well as strings ending with /. or /..).
This can be useful for extra paranoia to avoid directory traversals
(bug 1005908), or for more general filtering.
Change-Id: I39bf2de45b9dc7d3ca4d350d24b3f2276e958a62
DocImpact: new forbidden_regexp option for the name_filter middleware

don't check that number of servers equal 4.
Change-Id: I4a597eaf2b16b5b2cfc7fd706ac024357e9af2c2

Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2.
The additional following compatibility issues were addressed:
 - Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2
(49c175aec2)
 - common.constraints.check_utf8() can accept both utf8 str and unicode.
 - To convert unicode to utf-8 str if necessary.
 - Making proxy_logging can handle invalid utf-8 str
bug 888371
bug 959881
blueprint webob-support
Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496

We're still using saio:11000 in a few spots so a few things
don't work out of the box on the saio. Fixes bug #1024561
Change-Id: I226de54c2785b0d0b681c8d0cc24260adbd3d663

The probe tests were woefully out of date with all the changes that
have ocurred since they were written. I've updated most of them and
removed some that are hopeless outdated.
I also greatly improved the timing issues (hopefully completely
solved them? I ran them 25 times with no problems) and made them pep8
1.3.1 safe.
Change-Id: I8e9dbb6e7d6e04e293843b1dce1ded99d84e0348

- Rename it to keystoneauth for consistenties.
- Implements blueprint keystone-middleware.
Change-Id: I208fecdf3ee991694b4239f065032324d297fd35

swift.common.utils.validate_device_partition is a new function to check
that a device and a partition are valid. This means that they don't
contain '/' and are not '.' or '..'.
We use this new function every time we get devices and partitions from a
request.
Fix bug 1005908
Change-Id: Ia545ba8f877e85b4b576d6d7d09d890877ea6d34

In the interest of getting us to pep8 1.3.1 compliance I also fixed
all violations in the two files I was modifying anyway.
Fixes bug 1019051
Change-Id: I52eb7d59d2e2810a8cee2461f7ca869124e734e7

The X-Object-Manifest header value should be url encoded to allow for
"special" characters to be used. This fix decodes such encoding. A
corresponding fix to python-swiftclient (and any other clients) needs
to be made to url encode the header value when sending.
I also fixed any pep8 1.3.1 violations I found in the test I modified
to verify this.
Change-Id: I67f23dec1dc95ee37354dfdd9897f34758135ee6