openstack/swift - swift - OpenDev: Free Software Needs Free Tools

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure
Change-Id: Ic39cc9ffb582909da3a827243b628206de5301a0

The fake_spawn() context manager wraps spawn and waits
for greenthreads to complete (such as the async_update threads).
The wait needs to be done before http_connect is un-mocked, so
the fake_spawn context manager should exit *before* any context
manager that mocks the http_connect method.
Also add fake_spawn to _test_PUT_then_POST_async_pendings()
Related-Bug: #1536376
Related-Bug: #1514111
Closes-Bug: #1555739
Change-Id: I15f36e191cfe3ee6c82b4be56e8618ec0230e328

This patch cleans up some rough edges that were left (due to
time constraints) in the original commit.
Change-Id: Id4480be8dc1b5c920c19988cb89ca8b60ace91b4
Co-Authored-By: Gerry Drudy gerry.drudy@hpe.com 

Without the correct request header, the test that should be
using an EC policy was in fact using a replication policy.
Change-Id: Id44d0d615a4fd09aadfb286425939aea6abdf7b0

With the X-Timestamp validation added in commit e619411, end users
could upload objects with
 X-Timestamp: 9999999999.99999_ffffffffffffffff
(the maximum value) and Swift would be unable to delete them.
Now, inbound X-Timestamp headers will be moved to
X-Backend-Inbound-X-Timestamp, effectively rendering them harmless.
The primary reason to allow X-Timestamp before was to prevent
Last-Modified changes for objects coming from either:
 * container_sync or
 * a migration from another storage system.
To enable the former use-case, the container_sync middleware will now
translate X-Backend-Inbound-X-Timestamp headers back to X-Timestamp
after verifying the request.
Additionally, a new option is added to the gatekeeper filter config:
 # shunt_inbound_x_timestamp = true
To enable the latter use-case (or any other use-case not mentioned), set
this to false.
Upgrade Consideration
=====================
If your cluster workload requires that clients be allowed to specify
objects' X-Timestamp values, disable the shunt_inbound_x_timestamp
option before upgrading.
UpgradeImpact
Change-Id: I8799d5eb2ae9d795ba358bb422f69c70ee8ebd2c

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure
Change-Id: I9196deac606a3d073fb1c400cbee778cbbbd1c1e

Change-Id: I2e0e671a08fb855bf53c57987c08a7eefca7078a

Just a touch of cleanup to the unittest to demonstrate the stability of the
guarantee that objects will hash to the same nodes but have different parts.
Change-Id: I4a24187755455366a6435816a138f6175ae713a4

Each object update to a container server is saved in a
pending file if the initial update attempt fails. Pending
file names were derived from the update request's x-timestamp,
which is equal to the object's data file timestamp. This meant
that updates due to an object POST used the same async pending
file as updates due to the object's PUT.
This is not so bad because the object POST update has a superset
of the metadata included in the PUT update. But there is a risk
of a race condition causing an update to be lost: the updater may
open an update file due to a PUT whuile the object server is
writing an update due to a POST to the same file name. The
updater could then unlink the file before the more recent update
for the POST is sent.
This patch changes the POST update pending file name to be derived
from the object's metadata timestamp, thus making it distinct from
the PUT update pending file name. There is no upgrade impact since
existing pending files will continue to be processed.
Change-Id: I1b093c837efe8c2a64e92075ebd5e1b93e30efb9

Change-Id: I4322978aa20ee731391f7709bbd79dee140fc703

Change-Id: I0b96dfde32b4c666eebda6e88228516dd693ef92
closes-bug:#1550569

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure
Change-Id: I96b8ff1287bf219c5f8d56a3a4868c1063a953f9

There are missing comments about storege_policy_index so appropriate
comments are added.
Change-Id: I3de3f0e6864e65918ca1a13cce70f19c23d295f5

There was a function in swift.common.utils that was importing
swob.HeaderKeyDict at call time. It couldn't import it at compilation
time since utils can't import from swob or else it blows up with a
circular import error.
This commit just moves HeaderKeyDict into swift.common.header_key_dict
so that we can remove the inline import.
Change-Id: I656fde8cc2e125327c26c589cf1045cb81ffc7e5

Change-Id: I2add370e4cfb55d1388e3a8b41f688a7f3f2c621

The existing probetests were already doing brainsplitting on the object
servers in order to introduce composite metdata timestamp reconciliation.
But they were not introducing failures at the container server level, and
therefore not covering functional testing of those servers replication
consistency repair.
This change adds a brain splitter for the container servers to additionally
exercise the consistency engine at that layer under the existing probetest so
that you can observe their correct behavior while exercising the probetests
[1].
1. I used a script like this to watch the object servers and container db's
repair themselves -> https://gist.github.com/clayg/a8077d9d29ff68bb0edd
Change-Id: Ic5bb03dffef7e30c58338cef969a105577b84620

Delete python bytecode before every test run.
Because python creates pyc files during tox runs, certain
changes in the tree, like deletes of files, or switching
branches, can create spurious errors.
Closes-Bug: #1368661
Change-Id: Iedcb400fa3b0417f5bb8e943b17758fcfb4070c6

If an EC diskfile is missing its .durable file (for example
due to a partial PUT failure) then the ssync missing check
will fail to open the file and will consider it
missing. This can result in possible reconstruction of the
fragment archive (for a sync job) and definite transmission
of the fragment archive (for sync and revert jobs), which is
wasteful.
This patch makes the ssync receiver inspect the diskfile
state after attempting to open it, and if fragments exist at
the timestamp of the sender's diskfile, but a .durable file
is missing, then the receiver will commit the diskfile at
the sender's timestamp. As a result, there is no longer any
need to send a fragment archive.
Change-Id: I4766864fcc0a3553976e8fd85bbb2fc782f04abd

This removes notes stating support for Erasure coding as beta. Questions
regarding the stability of EC are coming up regularly, and are often referring
to the docs that state EC as still in beta.
Besides this, a note marking statsd support as beta has been removed as well.
Change-Id: If4fb6a5c4cb741d42953db3cee8cb17a1d774e15

This method increases the partition power of an existing ring by one. It does
not move any data nor does it exposes a CLI command yet; it is only intended to
be used in a future version to do the actual ring modification itself.
An existing object that is currently located on partition X will be placed
either on partition 2*X or 2*X+1 after the partition power got increased. The
reason for this is the Ring.get_part() method, that does a bitwise shift to the
right.
To avoid actual data movement to different disks or even nodes, the allocation
of partitions to nodes needs to be changed. The allocation is pairwise due to
the above mentioned new partition scheme. Therefore devices are allocated like
this, with the partition being the index and the value being the device id:
OLD: 0, 3, 7, 5, 2, 1, ...
NEW: 0, 0, 3, 3, 7, 7, 5, 5, 2, 2, 1, 1, ...
If an operator stops the cluster, increases the partition power and renames &
hardlinks the existing data it is possible to do a power shift without actually
moving data. Please see the partition power spec for further details on this.
Change-Id: I063fd8077497ee8c14d9065f07b4ec0fb5cbe180
Partially-Implements: spec increasing_partition_power

Says the number of arguments does not match the number of '%'.
Change-Id: I8b5e395a07328fb9d4ac7a19f8ed2ae1637bee3b

Previously, we would only consider 204 responses successful, which would
cause some spurious gate failures, such as
http://logs.openstack.org/66/287666/3/check/gate-swift-dsvm-functional/c6d2673/console.html#_2016年03月03日_13_41_07_846
Change-Id: Ic8c300647924352a297a2781b50064f7657038b4

This patch makes a number of changes to enable content-type
metadata to be updated when using the fast-POST mode of
operation, as proposed in the associated spec [1].
* the object server and diskfile are modified to allow
 content-type to be updated by a POST and the updated value
 to be stored in .meta files.
* the object server accepts PUTs and DELETEs with older
 timestamps than existing .meta files. This is to be
 consistent with replication that will leave a later .meta
 file in place when replicating a .data file.
* the diskfile interface is modified to provide accessor
 methods for the content-type and its timestamp.
* the naming of .meta files is modified to encode two
 timestamps when the .meta file contains a content-type value
 that was set prior to the latest metadata update; this
 enables consistency to be achieved when rsync is used for
 replication.
* ssync is modified to sync meta files when content-type
 differs between local and remote copies of objects.
* the object server issues container updates when handling
 POST requests, notifying the container server of the current
 immutable metadata (etag, size, hash, swift_bytes),
 content-type with their respective timestamps, and the
 mutable metadata timestamp.
* the container server maintains the most recently reported
 values for immutable metadata, content-type and mutable
 metadata, each with their respective timestamps, in a single
 db row.
* new probe tests verify that replication achieves eventual
 consistency of containers and objects after discrete updates
 to content-type and mutable metadata, and that container-sync
 sync's objects after fast-post updates.
[1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e
Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01

This is the operational procedures guide that HPE used
to operate and monitor their public Swift systems.
It has been made publicly available.
Change-Id: Iefb484893056d28beb69265d99ba30c3c84add2b