openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Fix a PEP8 issues in doc/source/conf.py to make the code pretty.
Change-Id: I71d36ebb59d44ba145ca13e4f4827139f0466d78

Added "-p" to mkdir cmds in a "set -e" script (/etc/rc.local).
/var/log/swift gets owned by the group "adm", but isn't group writeable
by default, so the find/rm in "resetswift" fails. So I added a chmod to
add group-write for that directory tree.
Added commands for installing tox.
Change-Id: Ic141491272a83a9233cd6a6a70c3ef74411f3f51

Adding nosehtmloutput as a test dependency allows nose to output its
results to an html file. This will be used by Jenkins to save logs on
a different server.
Change-Id: I6bd4736e8cc4132db65bfe8cdfe1f3a388369fa8

Fixed a couple config gets which weren't using TRUE_VALUES (hopefully
the last ones?).
Removed a deprecated parameter from
swift.common.db.ContainerBroker.list_objects_iter() a "TODO" which dates
back to the original import of Swift into github.
Also made AccountBroker.reclaim()'s docstring match its arg names.
Change-Id: I646bdd00a3693763b55a50ba210c7ed2ba1df4cd

- If lock_file encounters a LockTimeout a fd can be left behind
- Fixes bug 1038191
Change-Id: Ib3f45d18aeeec91eedf0b4f0492664d33b4f57fa

Change-Id: I1f91a82da064c9f0e64779f4fa8951962678831b

_make_app_iter_reader uses a queue to pass read data to
_make_app_iter, and uses an empty string as a terminator. The problem
is that it pushes the same terminator even if it fails to reads the
data. If an exception happens before pushing all data to the queue,
_make_app_iter exits without yielding all data. In that case,
proxy-server stops in the middle of sending data, and a client waits
for all data to come infinitely.
This patch uses a boolean value for the terminator instead of an empty
string to distinguish the results, and raises an exception in the
error cases. Fixes bug 890888.
Change-Id: Ie4d48025843aa1bb600446da6f4fc49f82a440bf

Change-Id: I00bf2e20c2683d0440ca425461e0ab87197ee720

Fixed bug where expirer would get confused by odd deletion times.
Since this has already rolled out, I just capped things at ten 9s, or
Sat Nov 20 17:46:39 2286. I can't wait for the Y2286 world panic. :/
Change-Id: Iba10963faa344a418a1fa573d5c85f4ff864b574

bug fixes for swift 1033089
Swift 1.6.0 All in One setup uses rc.local to create a set of
directories like the following:
 /var/cache/swift /var/cache/swift2 /var/cache/swift3
 /var/cache/swift4
 /var/run/swift
without -p for mkdir command, when some of these directories
already exist, the rc.local fails so that the next set of
command won't execute. This causes swift fail to start because
of /var/run/swift does not exist. Adding -p to both mkdir
command solves this problem.
Change-Id: Ia26c9a857723747008293c0631c9acc1b58eca7e

Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays. This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7). Fixes bug 1031954.
swift.common.ring.ring.RingData is now responsible for serialization and
deserialization of its data via a new load() class method and save()
object method. The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before. So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings. THIS SHOULD BE MENTIONED PROMINENTLY IN THE
RELEASE NOTES.
I didn't want to bite of more than necessary, so I didn't mess with
builder file serialization.
Change-Id: I799b9a4c894d54fb16592443904ac055b2638e2d

Change-Id: I64647c5d613a0794f0807886a4833cd5c31c0c5e

Change-Id: I45952adfcfbc813477ee9966f05dcdeef23c683b

Change-Id: I95a750613b01bdc4d64444d7e7fae8190760d697

It wasn't logging the full path, depending on if WebOb's
path_info_pop had been called.
Change-Id: I612d4cae7a4dc1bf5164356af329496245ad85dc

This ensures req.accepts is only assigned a valid content type.
Change-Id: Idf19277959c516398d31f771cf42dec0733d1bc0

The WebOb 1.2 support patch introduced a regression where an unknown
format= query used to just default back to plain but now returns a
400. This will break anyone that uses format=text for instance, or
anything else "invalid".
Arguably, it makes the most sense to 400 in the case of an invalid
format requested, but since it is also a backwards compatibility
break, I recommend we keep the previous behavior. Retaining the
previous behavior seems less damaging than enforcing the new
behavior.
Change-Id: I6db015b33a6f3ab239b8cb4a3562ebdba1f76590

- swift-recon cli now reports all unmounted drives.
- Fixes bug 1031060
Change-Id: Ia9e6bbf05215879b602e9dcb77d007094c494dc1

Change-Id: Ib2d54da92a3f7e7325998e3e52f562a1bf538c28

A warning log line is emitted whenever the proxy has to use a handoff
node. Monitoring these warnings can indicate a problem within your
cluster; however, you can disable these log lines by setting the
proxy conf's log_handoffs to false.
While working on this, I also noticed why many proxy log lines did
not have txn_id and client_ip -- subcoroutines. Now the logger thread
locals are copied to the subcoroutines.
Change-Id: Ibac086e1b985f566c068d083620287509de35da8

Change-Id: I7e5e254660e2ce35bc7fa95435db0a987d5acd0a

Change-Id: I683633ef1c7a5df7522299a85090f1ffda0c287a

Change-Id: Ia97050ac2b0b438535f31a8eb400801c782160f9

Change-Id: I4a07ff105f2acc4174de0aef4e24cfcde323bea8

Change-Id: Ic2869ab7f728ed1a4d8b34c37f0540c582d7dbcc

Fixes bug #1024477
Change-Id: Id09bbff6bb4a5d032727d3ca53b3276b4769d178

Change-Id: If5a6d32c40de02183a2eed6e2a32d62ba38df32d

If fallocate can't be loaded from libc, bit posix_fallocate loaded
user will Swift will log warning messages. So it will be better
to write log only if both version of fallocate can't be loaded.
Change-Id: I2a9ffe4756e77b3010ccd82b33c208d6d9b9f7b7

In comments from https://review.openstack.org/8798 it was raised that it
might make sense to forbid some substrings in the name_filter
middleware.
There is now a new forbidden_regexp option for the name_filter
middleware to specify which substrings to forbid. The default is
"/\./|/\.\./|/\.$|/\.\.$" (or in a non-regexp language: the /./ and /../
substrings as well as strings ending with /. or /..).
This can be useful for extra paranoia to avoid directory traversals
(bug 1005908), or for more general filtering.
Change-Id: I39bf2de45b9dc7d3ca4d350d24b3f2276e958a62
DocImpact: new forbidden_regexp option for the name_filter middleware

don't check that number of servers equal 4.
Change-Id: I4a597eaf2b16b5b2cfc7fd706ac024357e9af2c2

Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2.
The additional following compatibility issues were addressed:
 - Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2
(49c175aec2)
 - common.constraints.check_utf8() can accept both utf8 str and unicode.
 - To convert unicode to utf-8 str if necessary.
 - Making proxy_logging can handle invalid utf-8 str
bug 888371
bug 959881
blueprint webob-support
Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496

We're still using saio:11000 in a few spots so a few things
don't work out of the box on the saio. Fixes bug #1024561
Change-Id: I226de54c2785b0d0b681c8d0cc24260adbd3d663