openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Deleted unused container checks. As method
swift::common::db::AccountBroker::get_container_timestamp becomes
unused, it is deleted too, along with the corresponding tests.
Change-Id: I61de4549b0abd7103226d6a13f1d9844abaa92d3

Change-Id: I7aca3c6cafd9391031f7a10cc233f99e81ee0393

1.Correct the usage of initializing a Ring object.
2.Fix typo.
3.Align lines of print_status part. (keep origin author mind)
note: Adam Kijak has commited it at early time, but didn't put it
 in Gerrit. I import his commit and send it in right Workflow.
Fixes: bug #1131901
Change-Id: If56316cc424045f7df662d60088382f7fd40e8b1

Fix bug 1129760
Without speed limit, DB auditor will likely consume high CPU% on
storage node. That will highly impact the cluster's performance.
This patch adds two options for account/container auditor:
 - containers_per_second: Maximum containers audited per second
 - accounts_per_second: Maximum accounts audited per second
DocImpact
Change-Id: I9faa506438185a83ca77db4906969328624d015f

These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"
Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da

- Fixes bug 1119282.
- Allow middleware accessing metadata of an account without having to
 store it separately in a new memcache namespace.
- Add tests for get_container_info that was previously missed.
- Add get_account_info method based on get_container_info, a function
 for other middleware to query accounts.
- Rename container_info['count'] as container_info['object_count'].
Change-Id: I43787916c7a812cb08d278edf45370521f12c912

Mostly removed execute on non-executable files
Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9

This was an oustanding TODO for StatsD Swift metrics. A new timing
metric is tracked for (only) GET requests for accounts, containers,
and objects:
 proxy-server.<req_type>.GET.<status_int>.first-byte.timing
Also updated StatsD documentation in the Admin Guide to clarify that
timing metrics are sent in units of milliseconds.
Change-Id: I5bb781c06cefcb5280f4fb1112a526c029fe0c20

Change-Id: I2a088e6cfe354d18cb2107a42bc626c2e7822e42

This patch fixes the Swift MemcacheRing set and set_multi
interface incompatible problem with python memcache. The fix
added two extra named parameters to both set and set_multi
method. When only time or timeout parameter is present, then one
of the value will be used. When both time and timeout are present,
the time parameter will be used.
Named parameter min_compress_len is added for pure compatibility
purposes. The current implementation ignores this parameter.
To make swift memcached methods all consistent cross the board,
method incr and decr have also been changed to include a new
named parameter time.
In future OpenStack releases, the named parameter timeout will be
removed, keep the named parameter timeout around for now is
to make sure that mismatched releases between client and server
will still work.
From now on, when a call is made to set, set_multi, decr, incr
by using timeout parametner, a warning message will be logged to
indicate the deprecation of the parameter.
Fixes: bug #1095730
Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893

Original code is kind of except-pass. This will ignore the error
occurs in process of encoding path. And if such error happened
without any information in logs, developer have to debug in a
long distance from error stack. That's wasting time, so there
should be a logging message here.
Fixes: bug #1123048
Change-Id: I7941b524e936241fb8d62fad5ad90de3daffe5e7

git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check
Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4

Change supports kern.log rotation in order to avoid loss
of significant information.
There is a year change functionality added as kern.log
does not keep record of year.
There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it
Fixes Bug 1080682
Change-Id: I93436c405aff5625396514000cab774b66022dd0

Replaced GA code for cross-domain tracking.
Patchset addresses reviewer's comments
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright
Adds current year to each Sphinx-built page.
Addresses only the docs copyright attribution, not code files.
Change-Id: Ib90fd1c92c8fafce2db821bc2b17cef1377cfc1e

- Change a bit the formatting of the documention as well.
- Fix WARNING: Title underline too short. in misc.rst.
Change-Id: I2f4e36bcb5e01e984f0af0152bc5b3b9f7e942ce

differently (if at all)
Adding a swift.source to wsgi pre_auth funcs and all middleware that makes
subrequests to proxy server.
NOTE: This change will result in a change in the number of proxy logs made for
staticweb, formpost, tempurl, and any other middleware that performs sub
requests (including swauth and SOS).
Please see docs for details.
DocImpact
Change-Id: I80cf2806add1c3d34054147e2515944be340455b

Reviewed development saio and guidelines while installing
a new dev and test node.
Change-Id: I59238a4d43fa08a89be14f43681ad4693f1261dd

Change-Id: Ibe4b59a90b82742b70b00353d22aedd972bbeb59

header to a query parameter.
This is needed because query parameters show up in proxy logs and headers do
not. With this change it will be easy to determine from any log line that gets
created from the original request (of which there is currently none) that the
request was a bulk action.
Note: This is not backwards compatible with the previous method of setting a
header. Because the bulk middleware has not been included in an openstack swift
release this should be fine.
Change-Id: I0297fa2de9e491bf0b8c430c0781e2e12316ed4b

Add a new middleware implementing some basic container quotas.
Quotas are subject to several limitations: eventual consistency, the timeliness
of the cached container_info (60 second ttl by default), and it’s unable to
reject chunked transfer uploads that exceed the quota (though once the quota
is exceeded, new chunked transfers will be refused).
However, they get most of the way to container quotas fairly inexpensively.
Quotas are set by adding meta values to the container, and are validated when
set:
 X-Container-Meta-Quota-Bytes: Maximum size of the container, in bytes.
 X-Container-Meta-Quota-Count: Maximum object count of the container.
DocImpact
Change-Id: I77cfbf6dc231a2e522bd67328e4c082424a93eee

In two places, the internal client was not doing what it should with failed requests.
Change-Id: I49e566108fe920d85ca60fe76d24debe8ec7ffad

Another ring builder optimization. Profiling revealed hotspots in many
calls to min() and list.sort() in _reassign_parts(). That method didn't
get exercised in my last optimization pass because that pass targeted a
rebalance where nothing really moved around.
This time, I wrote a script which created a fresh ring, added a bunch of
devices, did the initial balance, deleted some devices, balanced, and
added some more back in.
Results from homebrew Python 2.7.3 on OS X 10.8.2 Macbook Pro
(bare-metal):
 BEFORE:
 Using part-power = 18, adding 600 devices, removing 100, then adding 300 more...
 NOT Profiling to 'initial_balance.prof'
 wall-time delta: 131.33s
 NOT Profiling to 'deleting_200_rebalance.prof'
 wall-time delta: 25.67s
 NOT Profiling to 'first_rebalance.prof'
 wall-time delta: 62.00s
 AFTER:
 Using part-power = 18, adding 600 devices, removing 100, then adding 300 more...
 NOT Profiling to 'initial_balance.prof'
 wall-time delta: 28.04s
 NOT Profiling to 'deleting_200_rebalance.prof'
 wall-time delta: 9.35s
 NOT Profiling to 'first_rebalance.prof'
 wall-time delta: 16.41s
The driver script I used is available here:
https://gist.github.com/adb982aec6f0709f1273
Change-Id: I17e270acb12b5e4d4bbb1e34d8867dea90678961

Fixes bug 1110150
Change-Id: I6c48553564b1809a7ae5de9f778d93dccbe157ae

The (account|container|object).ring.gz files contain, among other
things, a JSON-encoded dictionary. This change simply makes the JSON
serializer sort the keys of that dictionary so that two
Python-identical rings will result in two bytewise-identical ring
files. Also, to get repeatable compression, we lock down the timestamp
in the gzip output stream to a fixed value. (There's a timestamp value
in a gzip stream header; by default, gzip.GzipFile sticks time.time()
in there.)
This only works on Python 2.7; on 2.6, the 'mtime' argument to
gzip.GzipFile() is unsupported. Don't worry, serialization still works
on 2.6. It just doesn't always produce the same bytes for the same
ring.
Change-Id: Ide446413d0aeb78536883933fd0caf440b8f54ad

pyflakes itself can't be used in any automated gating way, because there are
two sets of false errors it raises. However, as an exercise, cleaning up the
'valid' ones uncovered three actual bugs. The other changes (mostly unused
variables) are included here for fun.
Command run: pyflakes swift | grep -v "undefined name '_'"
Change-Id: I18696bf047dedad1a9fdbde3463e214fba95f7c6