openstack/swift - swift - OpenDev: Free Software Needs Free Tools

This required a bunch of whitespace-poking of the scripts in bin, but
that's all. Now every file in swift/ and bin/ is pep8-1.3.3-compliant,
so hopefully we can be done with this pep8 stuff for a good long time.
Change-Id: I44fdb41d219c57400a4c396ab7eb0ffa9dcd8db8

Change-Id: I72448cd8578b53a658f151354a26066772e4b9ec

Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.
DocImpact
Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0

cleaned up pep8 (v1.3.3) in all files this patch touches
Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff

Sometimes you just want to know what machines a given partition
should be on, or what handoffs that partition would use if needed.
We've been meaning to add this option to swift-get-nodes for quite
some time, but I just finally got to it.
Example: swift-get-nodes /etc/swift/object.ring.gz -p 123
I tried to leave as much of the existing swift-get-nodes unaltered,
so the output isn't exactly distilled to just what you'd need for
getting a partition's nodes. But it should suffice for what it is, an
admin tool.
Change-Id: I438400ddc0eecbf9c48266e7f38a2e4f0765f374

This patch ensures that the command-line arg format (boolean)
doesn't conflict with the conf file format (string) and the
proper action is taken.
Change-Id: I3284096e1a9478897e1c3246ab190b46d2590243

Change-Id: If45f77fda269ae6e251579542e70eb71bd11fe2a

Have builder.add_devs get the next id to use when adding a new device
if its not specified in the dict.
Change-Id: I5a0defab43f5cfc5d997080bfd8563bfe72368ad

Fixes Bug 1049081
Change-Id: If977080350cc5cdb6bc633b6af7d3c490ed23d46

This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.
- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
 10.04 you'll have to pip install it, on 12.04 you can apt-get install
 it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.
Change-Id: I2e02684235aa2f4e901a00858ae037091594c545

You run one or more swift-bench-client processes like this:
 $ swift-bench-client 127.0.0.1 20001
 $ swift-bench-client 127.0.0.1 20002
Then you run swift-bench with a new option, --bench-clients (-b), which is
specified once for each swift-bench-client:
 $ swift-bench -b 127.0.0.1:20001 -b 127.0.0.1:20002
You get log lines from each client (interleaved) along with a final report
for all clients:
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:44:06,148 INFO Auth version: 1.0
 127.0.0.1:20001 swift-bench-server 2012年08月25日 22:44:06,148 INFO Auth version: 1.0
 127.0.0.1:20001 swift-bench-server 2012年08月25日 22:44:12,249 INFO 83 PUTS [0 failures], 41.5/s
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:44:14,430 INFO 74 PUTS [0 failures], 34.3/s
 ...
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:45:18,942 INFO Auth version: 1.0
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:45:20,946 INFO 238 DEL [2 failures], 118.9/s
 swift-bench 2012年08月25日 22:45:27,549 INFO 2000 PUTS **FINAL** [0 failures], 56.8/s
 swift-bench 2012年08月25日 22:45:27,550 INFO 30000 GETS **FINAL** [50 failures], 974.6/s
 swift-bench 2012年08月25日 22:45:27,550 INFO 2000 DEL **FINAL** [20 failures], 237.1/s
The concurrency, PUT count, and GET count config settings are divided by
the number of bench_clients. In other words, the same volume of work is
attempted (vs. not specifying --bench-clients), but it can now span
servers and CPU cores.
Benchmark containers are created (if use_proxy = yes) and deleted (if
delete = yes), with appropriate concurrency, in the initiating
swift-bench process, not any of the swift-bench-client processes.
Change-Id: Idbf31a23093244ab357a9bf77e6031257774f24a

swift-bench now honors the environment variables, ST_AUTH, ST_USER, and
ST_KEY like python-swiftclient does.
Added --lower-object-size (or -l) command-line option which, if
specified, will turn a specified --object-size into --upper-object-size.
It will raise a ValueError if --object-size is not specified or is <=
--lower-object-size.
BenchController how handles SIGINT (KeyboardInterrupt) similarly to the
swift command-line client: the first Ctrl-C will make it fast-track to
completion (no new PUT or GET operations are started, but everything PUT
is DELETE'ed). A second Ctrl-C will immediately exit. The behavior for
SIGTERM is unchanged (a single SIGTERM will immediately terminate the
process).
Added a sample configuration file for swift-bench, with documenting
comments.
Change-Id: I6f394ad995300fc8af3d565d95c3b45559ee510a

Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays. This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7). Fixes bug 1031954.
swift.common.ring.ring.RingData is now responsible for serialization and
deserialization of its data via a new load() class method and save()
object method. The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before. So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings. THIS SHOULD BE MENTIONED PROMINENTLY IN THE
RELEASE NOTES.
I didn't want to bite of more than necessary, so I didn't mess with
builder file serialization.
Change-Id: I799b9a4c894d54fb16592443904ac055b2638e2d

Change-Id: I64647c5d613a0794f0807886a4833cd5c31c0c5e

- swift-recon cli now reports all unmounted drives.
- Fixes bug 1031060
Change-Id: Ia9e6bbf05215879b602e9dcb77d007094c494dc1

We're still using saio:11000 in a few spots so a few things
don't work out of the box on the saio. Fixes bug #1024561
Change-Id: I226de54c2785b0d0b681c8d0cc24260adbd3d663

Commit 95786e5 made swift-get-nodes only output N handoff nodes, where
N is the ring's replica count. This change leaves that behavior in
place as the default, but adds the option -a/--all to print all the
handoff nodes.
This is handy if you're consuming the output with a script, e.g. if
you're doing some analysis of the distribution of primary+handoff
nodes for a particular set of things.
Change-Id: Ie0253575bb21ccf25559a1c5fd20570a58a2d9fa

This changes the JSON output keys missing_one and missing_two to
missing_1 and missing_2 (if there are at least 2 replicas), drops
the missing_all key, and can add more missing_x keys depending on
the replica count. We will definitely need to post a prominent
notice of this change with the next release.
bug 862816
Change-Id: Ib747d944476638c33ee1d876b8f9db28953826e2

Fixes Bug #1011727
Change-Id: I6681bd85a5ddb82a1059913ae90696e5994aa906

Changed swift-get-nodes to just return up to replica_count of
more_nodes instead of all of them. When you have thousands of drives,
returning all possible handoff nodes is pretty crazy. :)
Change-Id: I864e7243726ca3ef2ac08c4ab660a05789f37878

also fixed bug where in error_log where identifier wasn't being set to anything meaningful, set it to the right thing
bug 612722
Change-Id: I53f237ea0db2a5b2b8979a7b41189faf1275e861

- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.
Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0

Expand recon middleware to include support for account and container
servers in addition to the existing object servers. Also add support
for retrieving recent information from auditors, replicators, and
updaters. In the case of certain checks (such as container auditors)
the stats returned are only for the most recent path processed.
The middleware has also been refactored and should now also handle
errors better in cases where stats are unavailable.
While new check's have been added the output from pre-existing
check's has not changed. This should allow existing 3rd party
utilities such as the Swift ZenPack to continue to function.
Change-Id: Ib9893a77b9b8a2f03179f2a73639bc4a6e264df7

RateLimit
StaticWeb
TempURL/FormPOST
Change-Id: I988e93e6f4aacb817a2e354d43a04e47516fdf88

It was loading the ring off disk once to print the primary nodes, and
then loading the whole thing off disk again to print the handoff
nodes. Changed it to only load the ring off disk once.
Change-Id: I6f4cd0af9762e1e69660c3eb20586590b5339e5f

Change-Id: I5ff2056f9f2eb99bfb98b020e3fc013332100e12

TempURL/FormPOST is now at http://gholt.github.com/swift-tempurl/
For current users of TempURL/FormPOST, this will require installing the new
package and changing the "use" line of the tempurl and formpost conf section's
to:
[filter:tempurl]
use = egg:swifttempurl#tempurl
[filter:formpost]
use = egg:swifttempurl#formpost
And then 'swift-init proxy reload'.
Change-Id: I5bddf7f9e09ee07815530a41c46ff901fc21b447

Also made tox's PEP8 check look at the scripts in bin/ to keep them
PEP8-compliant.
Change-Id: I710365ea929d7fc15578d5f742a236bad47ef28e

Added support for configuring lock file location.
added an additional error handler.
Change-Id: I3635ed9694b9fb8526e0ca0591dc731da03a7128

`swift-ring-builder X.builder [add|set_weight|set_info|remove]` can
now work on multiple devices at once.
This allows for faster bulk operations; for example, if an
administrator has to add 48 new devices to the ring, then this lets
them do it in one command invocation. Since each invocation of
swift-ring-builder follows a deserialize-operate-serialize pattern,
using the bulk form of the commands on N devices will save N-1
[de]serializations.
Change-Id: Ibf77528f28026866af3c94d0370181c8013d5bc5

On a relatively recent build of swift, all my scripts blew up with
"No tenant specified". It was caused by the fix to add --os_tenant_name,
commit ID: 208b8e85a8
This patch restores the old behavior. I tested it to work with old
swauth-based Swift, new Keystone-based Swift, in the latter case using
both -U and --os_tenant_name arguments.
Note that this patch permits to use a literal colon in the user name
with the new syntax, as long as tenant is specified. Empty tenant
names are not allowed with either syntax.
Change-Id: I7785e6981a9d6281d0421c43875ee19d61d5ff43
Bug: 982909

In the process of wrapping for profiling, a trivial wrap with def main()
ended in a ton of these:
 NameError: global name 'parser' is not defined
A closer investigation revealed an apparent confusion in argument names.
The problem was introduced by commit 985968f765
of November 2010. Note that Greg fixed st_delete() correctly in that one,
but apparently forgot the others.
Unit tests pass for both current (bad) code, and the patched code.
By the magic of Python scoping, current code works, just impossible
to wrap for a quick hack.
Change-Id: I18f2494ce2a201d04d49f99ea259d141939dcb6a

If you do specify --os_auth_url rather than using the environment variable
OS_AUTH_URL, you'll end up with different behaviour.
Change-Id: I7430c2a619487375427b1c377ce00245eff107ff

Change-Id: I080d531471d8ea57c69a918d4f6930441f1e69f6
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>

Change-Id: I5168942ad32b3461f4c46300e378b249dab6a2ee
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>

Now display's cluster disk space used, free, and total.
Change-Id: I9c48dd6badaecd64cda4a2a8a51f41f1c29d328a

- Make sure we have a tenant_name when doing get_auth for 2.0
 auth_version.
Change-Id: I02dceb03f224c8fb91972e1f0e2925e5d1a06640