openstack/swift - swift - OpenDev: Free Software Needs Free Tools

* Algorithm format_device was changed for simplicity extension new
 ip addresses parameters.
* Some prints outputs was replacement by function format_device.
Change-Id: I8565d42fcdb62eeb398c4432bb6f499c27c05cf6

Indent swift-ring-builder output was changed according
with old style (before implementation of region tier).
Change-Id: I0d1cc7acdc5baf86f343745aea6fc2120838fd36

Folks have actually been asking for this. I think they're sending a
DELETE TempURL to someone way ahead of time and the someone issues it
when they're ready. Honestly, I'm not entirely sure of the use case,
but having the set of methods configurable wouldn't hurt.
Change-Id: Ibdb48f8a72077b045eeedddfae4c0a1f56098d7a

swift-ring-builder uses outdated unbounded class method to handle
not supported command. It worked for python 2.6 or older but not
for python 2.7 or newer. This patch fixes the problem.
Change-Id: I7dbc681ef6be44f6d79ff93189ccca13c51eab74
Fixes: bug #1154882 

Using root logger will make public module log failed by unknow keyword.
Just change the logger name to module itself could dismiss this impact.
And disable logger.propagate could prevent double outputing.
Change-Id: I18696d124ebac9ca970d502558972e51de759097
Fixes: bug #1105133 

Change-Id: Icef35175682c11a406ef85accc6c6e1acbf89f4d

I broke it in 7548cb9 when a ring's replica count changed from an int
to a float.
Change-Id: I49fa7e2961a09daad0c96e19cf1e39fec248d998

The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.
Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.
swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.
Examples:
$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda
$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda
$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda
Also, some updates to ring-overview doc.
Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29

Hard coding of '/srv/node' maybe drive to some errors. But it seems
unfixable (please see talks in bug report). To avoid erros as we can,
showing a friendly notification to let user set their own device
location works better than hard coding directly.
Change-Id: I45d749dd52af8d3b051dfe98dbb9d71a5a374f97
Fixes: bug #907908 

Sometimes there is no date at the beginning of a line in kern.log.
Although it does not happen often, there should be a check ensuring
the program doesn't crash in case it happens.
Added try-except block surrounding parsing string to date format.
Change-Id: I44a101266582eea2199189a006afa1037a9bd4ea
Fixes: bug #1152658 

1.Correct the usage of initializing a Ring object.
2.Fix typo.
3.Align lines of print_status part. (keep origin author mind)
note: Adam Kijak has commited it at early time, but didn't put it
 in Gerrit. I import his commit and send it in right Workflow.
Fixes: bug #1131901
Change-Id: If56316cc424045f7df662d60088382f7fd40e8b1

Example:
$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance
This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.
In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.
Example:
$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>
$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...
Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.
Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.
Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.
DocImpact
Change-Id: I12b34dacf60350a297a46be493d5d171580243ff

git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check
Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4

Change supports kern.log rotation in order to avoid loss
of significant information.
There is a year change functionality added as kern.log
does not keep record of year.
There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it
Fixes Bug 1080682
Change-Id: I93436c405aff5625396514000cab774b66022dd0

Change-Id: Ibe4b59a90b82742b70b00353d22aedd972bbeb59

Fixes bug 1102319
DocImpact
Change-Id: I8fb0417ab9468e97ed01a6cb1e262630905e7f29

Passing a seed into rebalance makes the rebalance deterministic
which allows us to generate identical rings across disparate
nodes without having to copy the ring files around.
Change-Id: Ie5ae46ac030e61284bc501fdef9d77eeb5243afd

If invoked as 'swift-ring-builder-safe' the directory containing the builder
file provided will be locked (via lock_parent_directory()). This provides a
small safe guard against multiple instances of the swift-ring-builder (or
other utilities that observe this lock) from attempting to write to or read
the builder/ring files while operations are in progress.
This is particularly useful in environments where ring management has been
automated (via Chef or custom solutions) but the operator still occasionally
needs to manually interact with the ring.
DocImpact
Change-Id: Ia362744a8151a91bfb586d01da582906726852e6

These settings:
 1. are already exposed via config file
 2. were not exposed as command-line options, and
 3. are things I wanted on the command line while benchmarking recently.
I didn't include short options for --(get|put|delete)-concurrency
since I couldn't think of good single-letter ones, and optparse won't
take "-gc", "-pc", or "-dc".
Change-Id: I0bd7c7ae2892244eed37b8c10c9357dc7e1190d3

- Add ability to specify an alternative run_dir than the default
 /var/run/swift.
- DocImpact
Change-Id: I17677588f2c8da563b7fec2dc4fdc52da87126ed

I've been doing this with cluster-wide log searches for far too long.
This adds support for reporting the oldest replication pass
completion as well as the most recent. This is quite useful for
finding those odd replicators that have hung up for some reason and
need intervention.
Change-Id: I7fd7260eca162d6b085f3e82aaa3cf90670f2d53

Change-Id: I29f9ab73c8ca2a3c52247fb92ef951dd5b5246cb

You can now give swift-init a -k N (or --kill-wait N) option to
override the default 15 second wait for a process to die after
sending it the die signal. This is useful for boxes that are awfully
slow for whatever reason.
DocImpact
Change-Id: I328ec254f6e0ee1cd423c1d062ba4c5331bd8337

The --help message of swift-dispersion-report used to say
/etc/swift/stats.conf instead of the correct and consistent
/etc/swift/dispersion.conf - this change updates that
commandline help message.
Change-Id: I69ad64d31bb86eb0d36fcf5b17aa8bf42f646ed1

- Add two optional flags that let you limit swift-dispersion-report to only
reporting on containers OR objects.
- Also make dispersion.conf and swift-dispersion-report manpages
 current.
DocImpact
Change-Id: Iad56133cad261241db27d0e2103098e3c2f3c245

When showing the disk usage dispersion graph it is often useful to
know what those top full drives are so you can do further research.
Now you can run 'swift-recon -d --top 10' to list the top ten devices
by fullness after the usual dispersion graph.
Change-Id: I7ddb2141e55e0613f69750fabe544940192c2d48

This required a bunch of whitespace-poking of the scripts in bin, but
that's all. Now every file in swift/ and bin/ is pep8-1.3.3-compliant,
so hopefully we can be done with this pep8 stuff for a good long time.
Change-Id: I44fdb41d219c57400a4c396ab7eb0ffa9dcd8db8

Change-Id: I72448cd8578b53a658f151354a26066772e4b9ec

Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.
DocImpact
Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0

cleaned up pep8 (v1.3.3) in all files this patch touches
Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff

Sometimes you just want to know what machines a given partition
should be on, or what handoffs that partition would use if needed.
We've been meaning to add this option to swift-get-nodes for quite
some time, but I just finally got to it.
Example: swift-get-nodes /etc/swift/object.ring.gz -p 123
I tried to leave as much of the existing swift-get-nodes unaltered,
so the output isn't exactly distilled to just what you'd need for
getting a partition's nodes. But it should suffice for what it is, an
admin tool.
Change-Id: I438400ddc0eecbf9c48266e7f38a2e4f0765f374

This patch ensures that the command-line arg format (boolean)
doesn't conflict with the conf file format (string) and the
proper action is taken.
Change-Id: I3284096e1a9478897e1c3246ab190b46d2590243

Change-Id: If45f77fda269ae6e251579542e70eb71bd11fe2a

Have builder.add_devs get the next id to use when adding a new device
if its not specified in the dict.
Change-Id: I5a0defab43f5cfc5d997080bfd8563bfe72368ad

Fixes Bug 1049081
Change-Id: If977080350cc5cdb6bc633b6af7d3c490ed23d46

This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.
- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
 10.04 you'll have to pip install it, on 12.04 you can apt-get install
 it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.
Change-Id: I2e02684235aa2f4e901a00858ae037091594c545

You run one or more swift-bench-client processes like this:
 $ swift-bench-client 127.0.0.1 20001
 $ swift-bench-client 127.0.0.1 20002
Then you run swift-bench with a new option, --bench-clients (-b), which is
specified once for each swift-bench-client:
 $ swift-bench -b 127.0.0.1:20001 -b 127.0.0.1:20002
You get log lines from each client (interleaved) along with a final report
for all clients:
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:44:06,148 INFO Auth version: 1.0
 127.0.0.1:20001 swift-bench-server 2012年08月25日 22:44:06,148 INFO Auth version: 1.0
 127.0.0.1:20001 swift-bench-server 2012年08月25日 22:44:12,249 INFO 83 PUTS [0 failures], 41.5/s
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:44:14,430 INFO 74 PUTS [0 failures], 34.3/s
 ...
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:45:18,942 INFO Auth version: 1.0
 127.0.0.1:20002 swift-bench-server 2012年08月25日 22:45:20,946 INFO 238 DEL [2 failures], 118.9/s
 swift-bench 2012年08月25日 22:45:27,549 INFO 2000 PUTS **FINAL** [0 failures], 56.8/s
 swift-bench 2012年08月25日 22:45:27,550 INFO 30000 GETS **FINAL** [50 failures], 974.6/s
 swift-bench 2012年08月25日 22:45:27,550 INFO 2000 DEL **FINAL** [20 failures], 237.1/s
The concurrency, PUT count, and GET count config settings are divided by
the number of bench_clients. In other words, the same volume of work is
attempted (vs. not specifying --bench-clients), but it can now span
servers and CPU cores.
Benchmark containers are created (if use_proxy = yes) and deleted (if
delete = yes), with appropriate concurrency, in the initiating
swift-bench process, not any of the swift-bench-client processes.
Change-Id: Idbf31a23093244ab357a9bf77e6031257774f24a

swift-bench now honors the environment variables, ST_AUTH, ST_USER, and
ST_KEY like python-swiftclient does.
Added --lower-object-size (or -l) command-line option which, if
specified, will turn a specified --object-size into --upper-object-size.
It will raise a ValueError if --object-size is not specified or is <=
--lower-object-size.
BenchController how handles SIGINT (KeyboardInterrupt) similarly to the
swift command-line client: the first Ctrl-C will make it fast-track to
completion (no new PUT or GET operations are started, but everything PUT
is DELETE'ed). A second Ctrl-C will immediately exit. The behavior for
SIGTERM is unchanged (a single SIGTERM will immediately terminate the
process).
Added a sample configuration file for swift-bench, with documenting
comments.
Change-Id: I6f394ad995300fc8af3d565d95c3b45559ee510a