openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I839c2af9e524f712e6fbeb8bf74d860af1d243e2

When the number of account/container or container/object replicas are
different, Swift had a few misbehaviors. This commit fixes them.
* On an object PUT/POST/DELETE, if there were 3 object replicas and
 only 2 container replicas, then only 2 requests would be made to
 object servers. Now, 3 requests will be made, but the third won't
 have any X-Container-* headers in it.
* On an object PUT/POST/DELETE, if there were 3 object replicas and 4
 container replicas, then only 3/4 container servers would receive
 immediate updates; the fourth would be ignored. Now one of the
 object servers will receive multiple (comma-separated) values in the
 X-Container-* headers and it will attempt to contact both of them.
 One side effect is that multiple async_pendings may be written for
 updates to the same object. They'll have differing timestamps,
 though, so all but the newest will be deleted unread. To trigger
 this behavior, you have to have more container replicas than object
 replicas, 2 or more of the container servers must be down, and the
 headers sent to one object server must reference 2 or more down
 container servers; it's unlikely enough and the consequences are so
 minor that it didn't seem worth fixing.
The situation with account/containers is analogous, only without the
async_pendings.
Change-Id: I98bc2de93fb6b2346d6de1d764213d7563653e8d

This commit makes that AuditorWorker gets a logger from ObjectAuditor class
(instead of creating new one) so the auditor uses minimum unix-sockets.
Fixes: bug #1089215
Change-Id: Ia47d862cbe7505db821784b01fcce6f22196e79f

For a part-power 18 ring with a small number of devices (12), this drops
"swift-ring-builder ... rebalance" from ~7.2 seconds to ~3.6s on OS X
with Python 2.7 (2.7 GHz i7 processor).
Under the profiler, that part-power 18 rebalance now spends only
slightly more time rebalancing the ring than it does writing out the
results.
I haven't done more extensive before/after comparison with larger
numbers of devices and larger part-power values...
Change-Id: I25735bc71da2f11617cc436d4f8e0c4f3f82bfec

- In howto install multi nodes container-sync was not included in the
 config example.
Change-Id: I2682d47c5cae28ff321ba03d70dae2fea00c276f

If the object replicator couldn't create a device's object directory
(due to permissions or whatever) it wouldn't do any work at all. This
fixes that.
Change-Id: I6a30439d036b29c9cfdb660428d13668e0dc8632

Unfortunately, SQLite truncates strings with null characters.
Additionally, XML pretty much hates them too.
Change-Id: Id9a8eaa27b841db6350d6959c202d3e3d6462b35

I've been doing this with cluster-wide log searches for far too long.
This adds support for reporting the oldest replication pass
completion as well as the most recent. This is quite useful for
finding those odd replicators that have hung up for some reason and
need intervention.
Change-Id: I7fd7260eca162d6b085f3e82aaa3cf90670f2d53

Change-Id: Ic1c67ced0169658c9aa7f78ed09bf24863cc9fb6

If a middleware (swift3, I'm looking at you), doesn't pass a Request
object into the Response constructor, Response._response_iter cannot
know to send zero bytes in the body of the HEAD response.
This patch fixes this usage of swob by making Response.__call__
helpfully reify self.request from env if it wasn't already set by the
Response object's constructor.
This fixes a bug in swift3 + swob-enabled-Swift where HEAD requests to
swift3 resulted in a response with a body in violation of the relevant
RFC and confusing clients.
Thanks to kostecky for finding the bug and describing it accurately.
Change-Id: I2bdb098052b161e1cddf1e4e482ab4dfafeb18c0

Change-Id: I0a1dc9e07f246db3e294f5c62f4ffbd3560b0afd

Change-Id: I29f9ab73c8ca2a3c52247fb92ef951dd5b5246cb

hub when it isn't.
Change-Id: I5fb05a71e8c62d84cee1da9ea05f67ffbad18b9d

This change allows other auth systems to test against the functests but not
require them to support x-storage-token (which was deprecated before swift
was even open-sourced). All other places in the code where x-storage-token is
used is only to populate x-auth-token if x-auth-token doesn't already exist.
Change-Id: Ie903d0f7135fc643353b74a61c14becf7dfc4d9d

Change-Id: If2bd1d4a850936b2e575a96073c116a8b9522602

Changing that complex dd to a functionally equivalent truncate that's easier
to read and modify.
DocImpact
Change-Id: I64404318364608e62a1d80f6a0550271eb4cd03a

Change-Id: I3742c4580d920e09de434e86b75b134c62f1732e

This class is being used at least by Ceilometer in its Swift middleware, and
since it's a general one anyway, it looks good to move it to common.utils.
This is a follow-up to Chmouel suggestion in
https://review.openstack.org/#/c/18231
Change-Id: I8d0ed8600c4152b91be9a88a3b396c3967d0add2
Signed-off-by: Julien Danjou <julien@danjou.info>

Fixes logging if logging to the console with non-swift middleware (such
as keystone) enabled. This also fixes issues with swift in devstack
Change-Id: Ib8b691b62b657a6d4ecdb1648d1fc2f3a0479982

Change-Id: I4d5ba8e58216749c2858dc1c0b8225b3619a7287

Ran dos2unix on LICENSE to convert to unit style line endings.
This makes building packages which contain the LICENSE file
a bit nicer... and matches other LICENSE files under the openstack
umbrella which have unix style line endings as well.
Change-Id: Id724d1ba402a590725a2d200f5e0599bce696e5a

Add a _get_part_nodes() method in Ring class, then call it in Ring::get_nodes() and Ring::get_part_nodes().
In order to reduce duplicate codes.
Change-Id: Id92f8dfd08388305638a2ba2ab64e77545c14cba

This functionality was lost with the swob change, but is back now.
Change-Id: I13b3154080a7c601235711b274e4899efb6adc93

You can now give swift-init a -k N (or --kill-wait N) option to
override the default 15 second wait for a process to die after
sending it the die signal. This is useful for boxes that are awfully
slow for whatever reason.
DocImpact
Change-Id: I328ec254f6e0ee1cd423c1d062ba4c5331bd8337

Change-Id: I1a2d603be983aa3291d22f7258437ee264e6c559

Change-Id: I3bad8c2995d4daf1e7798fae2c59b409608cbc45

The --help message of swift-dispersion-report used to say
/etc/swift/stats.conf instead of the correct and consistent
/etc/swift/dispersion.conf - this change updates that
commandline help message.
Change-Id: I69ad64d31bb86eb0d36fcf5b17aa8bf42f646ed1

Change-Id: I413d8bb9cbdea1f67ab2649ab0cc4a05a7a65883

This is required for a fix in keystone middleware to log to the proxy
logs. Since the keystone middleware doesn't initialize the log with
swift.commong.utils.config_logger, it doesn't have the server and
transaction id setup. This is required to fix bug #1089664
Change-Id: Icbb44076495a4d56065445c7a4d972ebede5d1e0

Using install_requires makes sure that if some code is using Swift internal
and wants to require Swift, all its dependencies have been already pulled.
This is really useful on test automation for example, otherwise, code might
be failing because on of the module listed in pip-requires is not instaled.
Also, we change the pip-requires files to use >= rather than ==, so the
requirements are easily fulfiled with distribution provided packages.
Change-Id: I65814bcd8ce798da21a5c17b4d5916a23f59e962
Signed-off-by: Julien Danjou <julien@danjou.info>