openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I81736080fc478c2b69d5b71edd0cada39aad9400

This patch allows to import the dispersion report tool, and thus making
it more easily usable within other Python tools. This can be also used
in a follow up patch to add some tests for the report tool.
It also fixes a bug when using the "--dump-json" option - until now it
returned the policy name and made the JSON invalid.
Change-Id: Ie0d52a1a54fc152bb72cbb3f84dcc36a8dad972a

As much as anything, I'm just tired of seeing a bunch or piecemeal
fixes.
Note that we *need* to include
 from __future__ import print_function
in order to support things like
 print() # Would print "()" (the repr of an empty tuple) otherwise
 print(foo, end='') # Would SyntaxError
 print(bar, file=sys.stderr) # Would SyntaxError
Change-Id: I8fdf0740e292eb1ee785512d02e8c552781dcae1

If you have 2 swift regions served by the same keystone,
then the client cannot get the correct URL for the swift endpoint
without specifying a region_name.
Closes-Bug: 1587088
Change-Id: Iaab883386e125c3ca6b9554389e63df17267a135

swift/container/sync.py uses swift.common.internal_client.delete_object
and put_object and expected these methods raise ClientException.
But delete_object and put_object never raise the exception so this patch
raises ClientException when urllib2 library raises HTTPError.
Co-Authored-By: Eran Rom <eranr@il.ibm.com>
Closes-Bug: #1419901
Change-Id: I58cbf77988979a07998a46d9d81be84d29b0d9bf

a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.
There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.
(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)
Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a

Commit c690bcb fixed a bug in the dispersion report, but changed this
from a bare "raise" to "raise err", which loses the traceback. Not a
big deal, but worth putting back IMO.
Change-Id: Id5b72153a4b8df8e3faaf1fa3fb2040e28ba85cc

This patch fixes Swift to show message
"No objects to query. Has swift-dispersion-populate been run?"
for "swift-dispersion-report —object-only"
with no container for object dispersion.
Change-Id: I82da56709cfc296a27f5180681709bc56adbc13d
Closes-Bug: #1468120 

Fix the warning E128: "continuation line under-indented for visual
indent" of pep8.
Change-Id: Ie6c6ae341fe3d6281f2095c1d756d552fa5937f9

This provides the capability to specify a project_name,
project_domain_name and user_domain_name in /etc/swift/dispersion.conf.
If this values are set in dispersion.conf they get populated to the
swift-client. With this it is possible to have a specific dispersion
project specified, which is not the keystone default domain. Changes
were applied to swift-dispersion-populate and swift-dispersion-report.
Relevant man pages, the example dispersion.conf and the admin guide were
updated accordingly.
DocImpact
Closes-Bug: #1468374
Change-Id: I0e716f8d281b4d0f510bc568bcee4a13fc480ff7

Changes Of py3 in bin :
* https://review.openstack.org/#/c/196835/
* ConfigParser from six.moves
Change-Id: Ic0374c8e09dfd595ec12c4d31b17dad30eaa803c

The iteritems() of Python 2 dictionaries has been renamed to items() on
Python 3.
Change-Id: I4bdc064c90bab56cd60f2dca2a5a78426ffbb31c

Doesn't work for anything other than policy 0. updated to allow user
to specify policy name on cmd line (as with object-info) which
then makes populate/report work with 3x, 2x, or EC style policies
Change-Id: Ib7c298f0f6d666b1ecca25315b88539f45cf9f95
Closes-Bug: 1458688

Removes the requirement for swiftclient in swift-dispersion-report
and swift-dispersion-populate. To prevent a dependency on
keystoneclient and to avoid reinventing the wheel with an internal
keystoneclient, authentication with keystone is only supported if
swiftclient is available. If not, only auth v1 is supported.
The dependency in swift/container/sync.py has also been removed.
Implements: blueprint remove-swiftclient-dependency
Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7

swift-dispersion-report tries to avoid checking a partition more than
once, so it keeps track of partitions already queried and skips
duplicates.
swift-dispersion-report also keeps track of the number of successful
responses; it counts the number of expected replicas to find, and also
counts the number of replicas actually found, and tells the operator
if the numbers differ.
However, in the case that a partition was duplicated, the
expected-responses counter was incremented, but the actual check was
skipped, so it looked as though some copies were missing. Now we only
increment the expected-responses counter if we're actually going to
perform the check.
Change-Id: I22ac2b8066b62ca7c8ebf099c9f602118bb1a298

Partial Implements: blueprint remove-swiftclient-dependency
Change-Id: I9af7150e5d21d50e5f880e57796314b8f05822d2

Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176 

except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14

The swift-dispersion-populate and swift-dispersion-report tools now
accept a --insecure option.
Also, dispersion.conf now has a keystone_api_insecure option.
Default is obviously to use the secure path.
DocImpact
Change-Id: I4000352e547d9ce5b08ade54e0c886281caff891

For systems with very large numbers of partitions, 1% dispersion
coverage may simply be too much/take too long. This fix allows <1
values to be used for dispersion_coverage.
DocImpact
Change-Id: I5ed35b69754d55a410e66e658b3854de57c7666b

Fixes bug 1180680.
Change-Id: I1c0eaa6f934f73853e832a8e468e1b8369a21117

Change-Id: I3d4a31111c3044da06611405ce80f208ef8a0ce3

Example:
$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance
This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.
In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.
Example:
$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>
$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...
Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.
Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.
Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.
DocImpact
Change-Id: I12b34dacf60350a297a46be493d5d171580243ff

Fixes bug 1102319
DocImpact
Change-Id: I8fb0417ab9468e97ed01a6cb1e262630905e7f29

The --help message of swift-dispersion-report used to say
/etc/swift/stats.conf instead of the correct and consistent
/etc/swift/dispersion.conf - this change updates that
commandline help message.
Change-Id: I69ad64d31bb86eb0d36fcf5b17aa8bf42f646ed1

- Add two optional flags that let you limit swift-dispersion-report to only
reporting on containers OR objects.
- Also make dispersion.conf and swift-dispersion-report manpages
 current.
DocImpact
Change-Id: Iad56133cad261241db27d0e2103098e3c2f3c245

Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.
DocImpact
Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0

cleaned up pep8 (v1.3.3) in all files this patch touches
Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff

This changes the JSON output keys missing_one and missing_two to
missing_1 and missing_2 (if there are at least 2 replicas), drops
the missing_all key, and can add more missing_x keys depending on
the replica count. We will definitely need to post a prominent
notice of this change with the next release.
bug 862816
Change-Id: Ib747d944476638c33ee1d876b8f9db28953826e2

also fixed bug where in error_log where identifier wasn't being set to anything meaningful, set it to the right thing
bug 612722
Change-Id: I53f237ea0db2a5b2b8979a7b41189faf1275e861

- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.
Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0

Also made tox's PEP8 check look at the scripts in bin/ to keep them
PEP8-compliant.
Change-Id: I710365ea929d7fc15578d5f742a236bad47ef28e

Change-Id: I080d531471d8ea57c69a918d4f6930441f1e69f6
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>

Change-Id: I5168942ad32b3461f4c46300e378b249dab6a2ee
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>

Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298

Change-Id: I0f55d61c7b8de30460f17a69e5d9946494dbda6e

Change-Id: I466302f05fd17c5b5460284c372c3667d7c46d7b

Add's the configuration file option "dump_json" or command line
options [-j|--dump-json] to have swift-dispersion-report output
the report in json format. This allows the dispersion report to
be more easily consumed elsewhere.
There's also a few pep8 fixes and removal of unused imports.
Change-Id: I2374311ccbef43e6bbae24665c9584e60f3da173