openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change previously hard-coded constants into config variables. This
allows deployers to tune their cluster more specifically based on
their needs. For example, a deployment that uses direct swift access
for public content may need to set a larger header value constraint to
allow for the full object name to be represented in the Content-
Disposition header (which browsers check to determine the name of a
downloaded object).
The new settings are set in the [swift-constraints] section of
/etc/swift/swift.conf. Comments were also added to this config file.
Cleaned up swift/common/constraints.py to pass pep8 1.3.3
Funtional tests now require constraints to be defined in /etc/test.conf or in
/etc/swift/swift.conf (in the case of running the functional tests against a
local swift cluster). To have any hope of tests passing, the defined
constraints must match the constraints on the tested cluster.
Removed a ton of "magic numbers" in both unit and functional tests.
Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465

Fix pep8 issues in swift/obj/expirer.py
Change-Id: Iba80aa5010f0c23e7fe1e5abc230a1c7f76687a8

Fix pep8 issues in swift/obj/auditor.py
Change-Id: Iae6fe2c7399fd52455d66ba57360e2f7d9e017f3

This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.
- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
 10.04 you'll have to pip install it, on 12.04 you can apt-get install
 it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.
Change-Id: I2e02684235aa2f4e901a00858ae037091594c545

fix some format according to pep8
Change-Id: I8ceb266d92c8997f131fa4096aaa6cfd4d585aaa

fir some format according to pep8
Change-Id: Iec8f67ea2166c955732bd598661140d4cfce6d40

format some code according to pep8
	modified: auditor.py
	modified: reaper.py
	modified: server.py
Change-Id: Ic6d8e000c711d17dcd9047cfeee7876b23ff0605

If a partition directory was a file instead of a directory, the
object-replicator would attempt to listdir() it, raise an exception, and
try again next iteration. This condition could arise after running
xfs_repair.
Now, collect_jobs() will reap any partition directories which are
actually files. Fixes bug 1045954.
Change-Id: Id65d3eab2effd61c3f6b25250611c88c907b2a16

Change-Id: Ieeee2411f84489ea106909e18e758fd170aa9bca

Change-Id: Ie3423ce90d906948a1ce2db0efe3da184e60f6e0

Change-Id: I3684a5694844dac282c0e4274b68525732c42637

Change-Id: Ie38e49915cfd233bcf00469f184cf8fce7424472

Change-Id: Ib4baac7b8b24d7fd982c71591deda3a096f470f2

If /dev/log does not exist, swift.common.utils.get_logger would fail as
it only handles the situation where /dev/log isn't a socket.
Fixes bug 1045820
Change-Id: I2c33e0fa7f88f03d5b0495779832ed4de7c496da

I did some benchmarking, and it's just a lot faster to use simplejson
to serialize account and container listings.
Change-Id: Ib0f751d9a5b6f4020031e952986811f0be090cce

Change-Id: I5423c8bf11b14f862cb321583d91991177fce5eb

Fix some pep8 issues in
 modified: test_bufferedhttp.py
 modified: test_constraints.py
 modified: test_db.py
 modified: test_db_replicator.py
 modified: test_init.py
make the code looks pretty.
Change-Id: I1c374b1ccd4f028c4e4b2e8194a6d1c201d50571

You run one or more swift-bench-client processes like this:
 $ swift-bench-client 127.0.0.1 20001
 $ swift-bench-client 127.0.0.1 20002
Then you run swift-bench with a new option, --bench-clients (-b), which is
specified once for each swift-bench-client:
 $ swift-bench -b 127.0.0.1:20001 -b 127.0.0.1:20002
You get log lines from each client (interleaved) along with a final report
for all clients:
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:44:06,148 INFO Auth version: 1.0
 127.0.0.1:20001 swift-bench-server 2012年08月25日 22:44:06,148 INFO Auth version: 1.0
 127.0.0.1:20001 swift-bench-server 2012年08月25日 22:44:12,249 INFO 83 PUTS [0 failures], 41.5/s
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:44:14,430 INFO 74 PUTS [0 failures], 34.3/s
 ...
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:45:18,942 INFO Auth version: 1.0
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:45:20,946 INFO 238 DEL [2 failures], 118.9/s
 swift-bench 2012年08月25日 22:45:27,549 INFO 2000 PUTS **FINAL** [0 failures], 56.8/s
 swift-bench 2012年08月25日 22:45:27,550 INFO 30000 GETS **FINAL** [50 failures], 974.6/s
 swift-bench 2012年08月25日 22:45:27,550 INFO 2000 DEL **FINAL** [20 failures], 237.1/s
The concurrency, PUT count, and GET count config settings are divided by
the number of bench_clients. In other words, the same volume of work is
attempted (vs. not specifying --bench-clients), but it can now span
servers and CPU cores.
Benchmark containers are created (if use_proxy = yes) and deleted (if
delete = yes), with appropriate concurrency, in the initiating
swift-bench process, not any of the swift-bench-client processes.
Change-Id: Idbf31a23093244ab357a9bf77e6031257774f24a

Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576

Change-Id: Id8872c581ed23378a8e14cbf3bf049b5c0d21577

fallocate() failures properly return HTTPInsufficientStorage from
object-server before reading from wsgi.input, allowing the proxy
server to error_limit that node.
Change-Id: Idfc293bbab2cff1e508edf58045108ca1ef5cec1

The gettext wrapper of server_type in controllers is used to translate
server_type in log. But the transfer_headers function which used to
filter the illegal metadata headers use the wrapped server_type. So, it
will cause problems that 'X-Account-Meta-' and 'X-Container-Meta-' would
be filtered when the language of operating system is not english, such
as Chinese or Japnese. So, I removed the wrapper.
Change-Id: I7d47e67b4caf5d0be171a90740f9bb12fa2cba84
Fixes: Bug #1042142 

Change-Id: Ie99250250171246e8c13e8d8c8258101bd78cce4

swift-bench now honors the environment variables, ST_AUTH, ST_USER, and
ST_KEY like python-swiftclient does.
Added --lower-object-size (or -l) command-line option which, if
specified, will turn a specified --object-size into --upper-object-size.
It will raise a ValueError if --object-size is not specified or is <=
--lower-object-size.
BenchController how handles SIGINT (KeyboardInterrupt) similarly to the
swift command-line client: the first Ctrl-C will make it fast-track to
completion (no new PUT or GET operations are started, but everything PUT
is DELETE'ed). A second Ctrl-C will immediately exit. The behavior for
SIGTERM is unchanged (a single SIGTERM will immediately terminate the
process).
Added a sample configuration file for swift-bench, with documenting
comments.
Change-Id: I6f394ad995300fc8af3d565d95c3b45559ee510a

Removed the redefinition of import in swift/object/server.py
Add my name in AUTHORS
Change-Id: Ia74df662de08fc52b92b828978d96a22b5df5597

If rate_limit_after_segment was 10 in the proxy config, then after 10
segments were coughed up by _load_next_segment() with no calls to
sleep(), the 11th segment would not trigger a sleep() call. The 12th
segment triggered a sleep(0) call, but it was only after the 13th
segment was loaded that an actual rate-limiting (non-zero) sleep got
called.
With this patch, a rate_limit_after_segment of 10 will start sleeping
the correct amount after the 11th segment.
Updated proxy-server.conf-sample with rate_limit_after_segment and
rate_limit_segments_per_sec.
Change-Id: I937c366996e6d6ab47c614d6db470e3be9657c07

This is a pretty mechanical change that moves proxy controller
classes to their own files.
Change-Id: Iabb7bc134b07193e3bcc041a5c6890a0b95c2b55

To tell when replication for a device has finished, it's important to
know when the replicator is removing objects. This was previously
handled for the object-replicator
(object-replicator.partition.delete.count.<device> and
object-replicator.partition.update.count.<device> metrics) but not the
account and container replicators.
This patch extends the existing DB removal count metrics to make them
per-device. The new metrics are:
 account-replicator.removes.<device>
 container-replicator.removes.<device>
There's also a bonus refactoring and increased test coverage of the DB
replicator code.
Change-Id: I2067317d4a5f8ad2a496834147954bdcdfc541c1

Basically, do all hashing in the replicator without a lock, then
lock briefly to rewrite the hashes file. Retry if someone else has
modified the hashes file in the mean time (which should be rare).
Also, a little refactoring.
Change-Id: I6257a53808d14b567bde70d2d18a9c58cb1e415a

Fix a PEP8 issues in doc/source/conf.py to make the code pretty.
Change-Id: I71d36ebb59d44ba145ca13e4f4827139f0466d78