openstack/swift - swift - OpenDev: Free Software Needs Free Tools

This patch fixes the Swift MemcacheRing set and set_multi
interface incompatible problem with python memcache. The fix
added two extra named parameters to both set and set_multi
method. When only time or timeout parameter is present, then one
of the value will be used. When both time and timeout are present,
the time parameter will be used.
Named parameter min_compress_len is added for pure compatibility
purposes. The current implementation ignores this parameter.
To make swift memcached methods all consistent cross the board,
method incr and decr have also been changed to include a new
named parameter time.
In future OpenStack releases, the named parameter timeout will be
removed, keep the named parameter timeout around for now is
to make sure that mismatched releases between client and server
will still work.
From now on, when a call is made to set, set_multi, decr, incr
by using timeout parametner, a warning message will be logged to
indicate the deprecation of the parameter.
Fixes: bug #1095730
Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893

Original code is kind of except-pass. This will ignore the error
occurs in process of encoding path. And if such error happened
without any information in logs, developer have to debug in a
long distance from error stack. That's wasting time, so there
should be a logging message here.
Fixes: bug #1123048
Change-Id: I7941b524e936241fb8d62fad5ad90de3daffe5e7

git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check
Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4

Change supports kern.log rotation in order to avoid loss
of significant information.
There is a year change functionality added as kern.log
does not keep record of year.
There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it
Fixes Bug 1080682
Change-Id: I93436c405aff5625396514000cab774b66022dd0

Replaced GA code for cross-domain tracking.
Patchset addresses reviewer's comments
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright
Adds current year to each Sphinx-built page.
Addresses only the docs copyright attribution, not code files.
Change-Id: Ib90fd1c92c8fafce2db821bc2b17cef1377cfc1e

- Change a bit the formatting of the documention as well.
- Fix WARNING: Title underline too short. in misc.rst.
Change-Id: I2f4e36bcb5e01e984f0af0152bc5b3b9f7e942ce

differently (if at all)
Adding a swift.source to wsgi pre_auth funcs and all middleware that makes
subrequests to proxy server.
NOTE: This change will result in a change in the number of proxy logs made for
staticweb, formpost, tempurl, and any other middleware that performs sub
requests (including swauth and SOS).
Please see docs for details.
DocImpact
Change-Id: I80cf2806add1c3d34054147e2515944be340455b

Reviewed development saio and guidelines while installing
a new dev and test node.
Change-Id: I59238a4d43fa08a89be14f43681ad4693f1261dd

Change-Id: Ibe4b59a90b82742b70b00353d22aedd972bbeb59

header to a query parameter.
This is needed because query parameters show up in proxy logs and headers do
not. With this change it will be easy to determine from any log line that gets
created from the original request (of which there is currently none) that the
request was a bulk action.
Note: This is not backwards compatible with the previous method of setting a
header. Because the bulk middleware has not been included in an openstack swift
release this should be fine.
Change-Id: I0297fa2de9e491bf0b8c430c0781e2e12316ed4b

Add a new middleware implementing some basic container quotas.
Quotas are subject to several limitations: eventual consistency, the timeliness
of the cached container_info (60 second ttl by default), and it’s unable to
reject chunked transfer uploads that exceed the quota (though once the quota
is exceeded, new chunked transfers will be refused).
However, they get most of the way to container quotas fairly inexpensively.
Quotas are set by adding meta values to the container, and are validated when
set:
 X-Container-Meta-Quota-Bytes: Maximum size of the container, in bytes.
 X-Container-Meta-Quota-Count: Maximum object count of the container.
DocImpact
Change-Id: I77cfbf6dc231a2e522bd67328e4c082424a93eee

In two places, the internal client was not doing what it should with failed requests.
Change-Id: I49e566108fe920d85ca60fe76d24debe8ec7ffad

Another ring builder optimization. Profiling revealed hotspots in many
calls to min() and list.sort() in _reassign_parts(). That method didn't
get exercised in my last optimization pass because that pass targeted a
rebalance where nothing really moved around.
This time, I wrote a script which created a fresh ring, added a bunch of
devices, did the initial balance, deleted some devices, balanced, and
added some more back in.
Results from homebrew Python 2.7.3 on OS X 10.8.2 Macbook Pro
(bare-metal):
 BEFORE:
 Using part-power = 18, adding 600 devices, removing 100, then adding 300 more...
 NOT Profiling to 'initial_balance.prof'
 wall-time delta: 131.33s
 NOT Profiling to 'deleting_200_rebalance.prof'
 wall-time delta: 25.67s
 NOT Profiling to 'first_rebalance.prof'
 wall-time delta: 62.00s
 AFTER:
 Using part-power = 18, adding 600 devices, removing 100, then adding 300 more...
 NOT Profiling to 'initial_balance.prof'
 wall-time delta: 28.04s
 NOT Profiling to 'deleting_200_rebalance.prof'
 wall-time delta: 9.35s
 NOT Profiling to 'first_rebalance.prof'
 wall-time delta: 16.41s
The driver script I used is available here:
https://gist.github.com/adb982aec6f0709f1273
Change-Id: I17e270acb12b5e4d4bbb1e34d8867dea90678961

Fixes bug 1110150
Change-Id: I6c48553564b1809a7ae5de9f778d93dccbe157ae

The (account|container|object).ring.gz files contain, among other
things, a JSON-encoded dictionary. This change simply makes the JSON
serializer sort the keys of that dictionary so that two
Python-identical rings will result in two bytewise-identical ring
files. Also, to get repeatable compression, we lock down the timestamp
in the gzip output stream to a fixed value. (There's a timestamp value
in a gzip stream header; by default, gzip.GzipFile sticks time.time()
in there.)
This only works on Python 2.7; on 2.6, the 'mtime' argument to
gzip.GzipFile() is unsupported. Don't worry, serialization still works
on 2.6. It just doesn't always produce the same bytes for the same
ring.
Change-Id: Ide446413d0aeb78536883933fd0caf440b8f54ad

pyflakes itself can't be used in any automated gating way, because there are
two sets of false errors it raises. However, as an exercise, cleaning up the
'valid' ones uncovered three actual bugs. The other changes (mostly unused
variables) are included here for fun.
Command run: pyflakes swift | grep -v "undefined name '_'"
Change-Id: I18696bf047dedad1a9fdbde3463e214fba95f7c6

Seems we missed these references when committing
357b12dc2b
DocImpact
Change-Id: Ia226ce1d63e52769bc067d50ec4704cea4e11c5c

Fixes bug 1102319
DocImpact
Change-Id: I8fb0417ab9468e97ed01a6cb1e262630905e7f29

Passing a seed into rebalance makes the rebalance deterministic
which allows us to generate identical rings across disparate
nodes without having to copy the ring files around.
Change-Id: Ie5ae46ac030e61284bc501fdef9d77eeb5243afd

Some systems behave badly when they completely run out of space. To
alleviate this problem, you can set the fallocate_reserve conf value
to a number of bytes to "reserve" on each disk. When the disk free
space falls at or below this amount, fallocate calls will fail, even
if the underlying OS fallocate call would succeed. For example, a
fallocate_reserve of 5368709120 (5G) would make all fallocate calls
fail, even for zero-byte files, when the disk free space falls under
5G.
The default fallocate_reserve is 0, meaning "no reserve", and so the
software behaves exactly as it always has unless you set this conf
value to something non-zero.
Also fixed ring builder's search_devs doc bugs.
Related: To get rsync to do the same, see
https://github.com/rackspace/cloudfiles-rsync
Specifically, see this patch:
https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff
DocImpact
Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527

* in etc/proxy-server.conf-sample, there is an access_log_headers directive in proxy_logging middleware.
 but current code doesn't handle it. This patch enables proxy_logging to handle access_log_headers.
Change-Id: I03337ccc4f48625af3f578ed543da992d3eec2bd

If invoked as 'swift-ring-builder-safe' the directory containing the builder
file provided will be locked (via lock_parent_directory()). This provides a
small safe guard against multiple instances of the swift-ring-builder (or
other utilities that observe this lock) from attempting to write to or read
the builder/ring files while operations are in progress.
This is particularly useful in environments where ring management has been
automated (via Chef or custom solutions) but the operator still occasionally
needs to manually interact with the ring.
DocImpact
Change-Id: Ia362744a8151a91bfb586d01da582906726852e6

And refactor some of the code to use them.
Remove unused imports.
Change-Id: Ica479c10247fa85c740bb99cf7d1db7fbb1b2c80

Fix small problem in ratelimiting middleware.
DocImpact
Change-Id: Ide3e0b9f4887626c30cae0b97eb7e2237b1df3ed

- Fixes bug 1101143.
Change-Id: I2a82269e4aed2c8926e0aa736c38120784e25de2

- This has been confusing since the beginning of keystoneauth.
- Do not remove the feature but just print a large warning when
 used.
Change-Id: I13990d7ab6760bb6479cf4f0717067522e1ef920

Fix for bug 1095130
* Added a wrapper function around public methods to handle
 CORS actual requests. These requests need to return some
 extra headers to be valid responses to a CORS request.
 Access-Control-Expose-Headers and Access-Control-Allow-Origin.
* Added support for the CORS header Access-Control-Expose-Headers.
* Some refactoring of the OPTIONS method so the
 "is_origin_allowed" logic can be reused.
* Added a little extra detail to the CORS documentation.
DocImpact
Change-Id: I68538e472a900775427f21a8a59e738a83dcc8bc