openstack/swift - swift - OpenDev: Free Software Needs Free Tools

When added a new devices into builder the add_dev function returns it
unique id.
blueprint argparse-in-swift-ring-builder
Change-Id: I57080bb625e812f6cea71199df907a44b332b552

Dramatic part of RingBuilder.search_devs which parse a complex format
of a search device string moved to the swift-ring-builder script.
Instead, the search_devs has a simple interface to search devices.
blueprint argparse-in-swift-ring-builder
Change-Id: If3dd77b297b474fb9a058e4693fef2dfb11fca3d

Instances of the RingBuilder class can store its data to a disk file by
the save method and load it by the load method.
blueprint argparse-in-swift-ring-builder
Change-Id: I69fdf0693ca9f520d235a795ecdd2da310dcd5d3

When swift-bench is run in direct mode, don't try to delete the
containers which weren't created.
Fixes bug 1177960.
Change-Id: Ice07e8729bb776e2b215894cf95fb80b64167a8d

Change-Id: I3d4a31111c3044da06611405ce80f208ef8a0ce3

Allow Swift daemons and servers to optionally accept a directory as the
configuration parameter. Directory based configuration leverages
ConfigParser's native multi-file support. Files ending in '.conf' in the
given directory are parsed in lexicographical order. Filenames starting with
'.' are ignored. A mixture of file and directory configuration paths is not
supported - if the configuration path is a file behavior is unchanged.
 * update swift-init to search for conf.d paths when building servers
 (e.g. /etc/swift/proxy-server.conf.d/)
 * new script swift-config can be used to inspect the cumulative configuration
 * pull a little bit of code out of run_wsgi and test separately
 * fix example config bug for the proxy servers client_disconnect option
 * added section on directory based configuration to deployment guide
DocImpact
Implements: blueprint confd
Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b

If we set device's weight to zero, currently balance will be set
special value(999.99) until zero weighted device return all
its partitions. So we cannot check balance has changed.
Thus we need to check balance or last_balance is special value.
Change-Id: I5b7db8b8e48db0c4771c51a764bda689869817d5
Fixes: bug #1171731 

Support separate replication ip address:
- Added new function in utils. This function provides ability
 to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
 Replication process uses new function now.
Replication network parameters:
- Replication network fields (replication_ip, replication_port)
 support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
 functions.
Implementation of replication servers:
- Separate replication servers use the same code as normal replication
 servers, but with replication_server parameter = True. When using a
 separate replication network, the non-replication servers set
 replication_server = False. When there is no separate replication
 network (the default case), replication_server is not included in the config.
DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network

remove duplicated key from swift-bench bin; and also add .idea to gitignore
Change-Id: I31360489716d03324b8d2bb1c51e282393069c13

* Algorithm format_device was changed for simplicity extension new
 ip addresses parameters.
* Some prints outputs was replacement by function format_device.
Change-Id: I8565d42fcdb62eeb398c4432bb6f499c27c05cf6

Indent swift-ring-builder output was changed according
with old style (before implementation of region tier).
Change-Id: I0d1cc7acdc5baf86f343745aea6fc2120838fd36

Folks have actually been asking for this. I think they're sending a
DELETE TempURL to someone way ahead of time and the someone issues it
when they're ready. Honestly, I'm not entirely sure of the use case,
but having the set of methods configurable wouldn't hurt.
Change-Id: Ibdb48f8a72077b045eeedddfae4c0a1f56098d7a

swift-ring-builder uses outdated unbounded class method to handle
not supported command. It worked for python 2.6 or older but not
for python 2.7 or newer. This patch fixes the problem.
Change-Id: I7dbc681ef6be44f6d79ff93189ccca13c51eab74
Fixes: bug #1154882 

Using root logger will make public module log failed by unknow keyword.
Just change the logger name to module itself could dismiss this impact.
And disable logger.propagate could prevent double outputing.
Change-Id: I18696d124ebac9ca970d502558972e51de759097
Fixes: bug #1105133 

Change-Id: Icef35175682c11a406ef85accc6c6e1acbf89f4d

I broke it in 7548cb9 when a ring's replica count changed from an int
to a float.
Change-Id: I49fa7e2961a09daad0c96e19cf1e39fec248d998

The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.
Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.
swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.
Examples:
$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda
$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda
$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda
Also, some updates to ring-overview doc.
Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29

Hard coding of '/srv/node' maybe drive to some errors. But it seems
unfixable (please see talks in bug report). To avoid erros as we can,
showing a friendly notification to let user set their own device
location works better than hard coding directly.
Change-Id: I45d749dd52af8d3b051dfe98dbb9d71a5a374f97
Fixes: bug #907908 

Sometimes there is no date at the beginning of a line in kern.log.
Although it does not happen often, there should be a check ensuring
the program doesn't crash in case it happens.
Added try-except block surrounding parsing string to date format.
Change-Id: I44a101266582eea2199189a006afa1037a9bd4ea
Fixes: bug #1152658 

1.Correct the usage of initializing a Ring object.
2.Fix typo.
3.Align lines of print_status part. (keep origin author mind)
note: Adam Kijak has commited it at early time, but didn't put it
 in Gerrit. I import his commit and send it in right Workflow.
Fixes: bug #1131901
Change-Id: If56316cc424045f7df662d60088382f7fd40e8b1

Example:
$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance
This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.
In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.
Example:
$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>
$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...
Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.
Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.
Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.
DocImpact
Change-Id: I12b34dacf60350a297a46be493d5d171580243ff

git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check
Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4

Change supports kern.log rotation in order to avoid loss
of significant information.
There is a year change functionality added as kern.log
does not keep record of year.
There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it
Fixes Bug 1080682
Change-Id: I93436c405aff5625396514000cab774b66022dd0

Change-Id: Ibe4b59a90b82742b70b00353d22aedd972bbeb59

Fixes bug 1102319
DocImpact
Change-Id: I8fb0417ab9468e97ed01a6cb1e262630905e7f29

Passing a seed into rebalance makes the rebalance deterministic
which allows us to generate identical rings across disparate
nodes without having to copy the ring files around.
Change-Id: Ie5ae46ac030e61284bc501fdef9d77eeb5243afd

If invoked as 'swift-ring-builder-safe' the directory containing the builder
file provided will be locked (via lock_parent_directory()). This provides a
small safe guard against multiple instances of the swift-ring-builder (or
other utilities that observe this lock) from attempting to write to or read
the builder/ring files while operations are in progress.
This is particularly useful in environments where ring management has been
automated (via Chef or custom solutions) but the operator still occasionally
needs to manually interact with the ring.
DocImpact
Change-Id: Ia362744a8151a91bfb586d01da582906726852e6

These settings:
 1. are already exposed via config file
 2. were not exposed as command-line options, and
 3. are things I wanted on the command line while benchmarking recently.
I didn't include short options for --(get|put|delete)-concurrency
since I couldn't think of good single-letter ones, and optparse won't
take "-gc", "-pc", or "-dc".
Change-Id: I0bd7c7ae2892244eed37b8c10c9357dc7e1190d3

- Add ability to specify an alternative run_dir than the default
 /var/run/swift.
- DocImpact
Change-Id: I17677588f2c8da563b7fec2dc4fdc52da87126ed

I've been doing this with cluster-wide log searches for far too long.
This adds support for reporting the oldest replication pass
completion as well as the most recent. This is quite useful for
finding those odd replicators that have hung up for some reason and
need intervention.
Change-Id: I7fd7260eca162d6b085f3e82aaa3cf90670f2d53

Change-Id: I29f9ab73c8ca2a3c52247fb92ef951dd5b5246cb

You can now give swift-init a -k N (or --kill-wait N) option to
override the default 15 second wait for a process to die after
sending it the die signal. This is useful for boxes that are awfully
slow for whatever reason.
DocImpact
Change-Id: I328ec254f6e0ee1cd423c1d062ba4c5331bd8337

The --help message of swift-dispersion-report used to say
/etc/swift/stats.conf instead of the correct and consistent
/etc/swift/dispersion.conf - this change updates that
commandline help message.
Change-Id: I69ad64d31bb86eb0d36fcf5b17aa8bf42f646ed1

- Add two optional flags that let you limit swift-dispersion-report to only
reporting on containers OR objects.
- Also make dispersion.conf and swift-dispersion-report manpages
 current.
DocImpact
Change-Id: Iad56133cad261241db27d0e2103098e3c2f3c245

When showing the disk usage dispersion graph it is often useful to
know what those top full drives are so you can do further research.
Now you can run 'swift-recon -d --top 10' to list the top ten devices
by fullness after the usual dispersion graph.
Change-Id: I7ddb2141e55e0613f69750fabe544940192c2d48

This required a bunch of whitespace-poking of the scripts in bin, but
that's all. Now every file in swift/ and bin/ is pep8-1.3.3-compliant,
so hopefully we can be done with this pep8 stuff for a good long time.
Change-Id: I44fdb41d219c57400a4c396ab7eb0ffa9dcd8db8

Change-Id: I72448cd8578b53a658f151354a26066772e4b9ec