openstack/swift - swift - OpenDev: Free Software Needs Free Tools

An operator proposing a web UX to its customers might want to allow web
browser to access some headers by default (eg: X-Storage-Policy,
 X-Container-Read, ...). This commit adds a new setting to the
proxy-server to allow some headers to be added cluster-wide to the CORS
header Access-Control-Expose-Headers.
Change-Id: I5ca90a052f27c98a514a96ee2299bfa1b6d46334

The requests library checks that the headers are either strings or
bytes. Currently, the two test_object_expirer tests fail with the
message:
InvalidHeader: Header value 1487879553 must be of type str
or bytes, not <type 'int'>
The header in question is "x-delete-at". The patch converts it to
a string, before making a Swift Client request.
Change-Id: I738697cb6b696f0e346345f75e0069048961f2ff

the "ommitted" change to "omitted".
Change-Id: I195b0de1cee6ef79e00c865fd112a34d529daec0

The patch adds documentation for the SLO raw format, specifically what
the fields are in relation to the documented format (hash vs etag,
bytes vs size_bytes, and name vs path).
Change-Id: I44c74ad406a6e55e4228f52fac623eeabbd7564f

Commit bfbf0d1e78 removed a check that was
meant to avoid to resolve a storage domain. It breaks the behavior of the
middleware as the resolution of a storage domain will return nothing, so
the global resolution will fail.
Example:
 Host header: storage.example.com
 storage_domain: [.storage.example.com]
 Host does not end with one of the storage_domains (because of the dot),
 so the middleware will loop to resolve the CNAME of storage.example.com,
 but it won't succeed because it's a storage_domain.
Closes-Bug: #1311435
Change-Id: If594b816ff2f7025521de716b32c42bf3137f5dd

Change-Id: I6c57d2b89b1b4b13f87511538fc90ba554c869fa

Change-Id: I6e200558b75ac539b59b492d13c36702443efc89

Now that we're shuffling parts before going through them, those stats no
longer make sense -- device completion would always be 100%.
Also, always use delete_partition for cleanup, so we have one place to
make improvements. This means we'll properly clean up non-numeric
directories.
Also also, put more I/O in the tpool in delete_partition.
Change-Id: Ie06bb16c130d46ccf887c8fcb252b8d018072d68
Related-Change: I69e4c4baee64fd2192cbf5836b0803db1cc71705

Change-Id: I4f68b2f84f32e2c9e567c08f7e66ce0a81b7d867

Change-Id: I4d90075bd1eb6775b9d736668aa9c7af5eb41f4e

The handoffs_first mode in the replicator has the useful behavior of
processing all handoff parts across all disks until there aren't any
handoffs anymore on the node [1] and then it seemingly tries to drop
back into normal operation. In practice I've only ever heard of
handoffs_first used while rebalancing and turned off as soon as the
rebalance finishes - it's not recommended to run with handoffs_first
mode turned on and it emits a warning on startup if option is enabled.
The handoffs_first mode on the reconstructor doesn't work - it was
prioritizing handoffs *per-part* [2] - which is really unfortunate
because in the reconstructor during a rebalance it's often *much* more
attractive from an efficiency disk/network perspective to revert a
partition from a handoff than it is to rebuild an entire partition from
another primary using the other EC fragments in the cluster.
This change deprecates handoffs_first in favor of handoffs_only in the
reconstructor which is far more useful - and just like handoffs_first
mode in the replicator - it gives the operator the option of forcing the
consistency engine to focus on rebalance. The handoffs_only behavior is
somewhat consistent with the replicator's handoffs_first option (any
error on any handoff in the replicactor will make it essentially handoff
only forever) but the option does what you want and is named correctly
in the reconstructor.
For consistency with the replicator the reconstructor will mostly honor
the handoffs_first option, but if you set handoffs_only in the config it
always takes precedence. Having handoffs_first in your config always
results in a warning, but if handoff_only is not set and handoffs_first
is true the reconstructor will assume you need handoffs_only and behaves
as such.
When running in handoffs_only mode the reconstructor will start to log a
warning every cycle if you leave it running in handoffs_only after it
finishes reverting handoffs. However you should be monitoring on-disk
partitions and disable the option as soon as the cluster finishes the
full rebalance cycle.
1. Ia324728d42c606e2f9e7d29b4ab5fcbff6e47aea fixed replicator
handoffs_first "mode"
2. Unlike replication each partition in a EC policy can have a different
kind of job per frag_index, but the cardinality of jobs is typically
only one (either sync or revert) unless there's been a bunch of errors
during write and then handoffs partitions maybe hold a number of
different fragments.
Known-Issues:
handoffs_only is not documented outside of the example config, see lp
bug #1626290
Closes-Bug: #1653018
Change-Id: Idde4b6cf92fab6c45f2c0c2733277701eb436898

It has been a real honour working with you, guys. Thanks!
Change-Id: I2668ddf546791ca36fe22d6fdd2d5e745ed14200

Change-Id: Iea629d5a08aa3d94e097fcdab28f94511b262fcf

Add the Apache License 2.0 description in ../conf.py which
is necessary.
Change-Id: Ief3767fdc22f582beb8683a9075dc25dbcb541cd

Fixed the comment in the test to match exactly what's
being removed and what the expected result is.
Also, removed that extra '/' parameter which was causing
the assert to test at the wrong directory level.
Change-Id: I2f27f0d12c08375c61047a3f861c94a3dd3915c6
Signed-off-by: Thiago da Silva <thiago@redhat.com>

Prior to the Related-Change no suffixes were written to hashes.invalid
until after initial suffix hashing created hashes.pkl - and in our probe
test the only updates to the partition occurred before replication.
Before the related change with sync_method = rsync it was possible when
starting from a clean slate to write data, and replicate from a handoff
partition without generating a hashes.invalid file in any primary.
After the Related-Change it was no longer possible to write data without
generating a hashes.invalid file; however with sync_method = rsync the
replicator could still replicate data into a partition that never
received an update directly and therefore no hashes.invalid.
When using sync_method = ssync replication updates the hashes.invalid
like any normal update to the partition and therefore all partitions
always have a hashes.invalid.
This change opts to ignores these implementation details in the probe
tests when comparing the files between synced partitions by
black-listing these known cache files and only validates that the disk
file's on disk files are in sync.
Related-Change-Id: I2b48238d9d684e831d9777a7b18f91a3cef57cd1
Change-Id: Ia9c50d7bc1a74a17c608a3c3cfb8f93196fb709d
Closes-Bug: #1663021 

Fixes the failure in TestReconstructorRevert.test_delete_propagate
introduced by Related-Change.
Related-Change-Id: Ie351d8342fc8e589b143f981e95ce74e70e52784
Change-Id: I1657c1eecc9b62320e2cf184050e0db122821139

...and refactor two extremely similar tests to use a single
helper method - the only paramerization being the existence
or not of hashes.pkl at start of the test.
Change-Id: I601218a9a031e7fc77bc53ea735e89700ec1647d
Related-Change: Ia43ec2cf7ab715ec37f0044625a10aeb6420f6e3

Change-Id: Ib7d2c16a755ae1faca5a371d5dae1e143110178f

There is a wrong word, it should be updated.
Change-Id: I17a1fed844bcd8ab3ed668a34a8471e5f6032963

Trivialfix
Change-Id: I4862d073adecf1cc5312a64795ad890eeddf774d

This only affects chunked transfers to replicated policies.
EC policies already have comparable error handling in
swift.common.utils._MultipartMimeFileLikeObject.read
See also: https://github.com/eventlet/eventlet/commit/c3ce3ee
but note that eventlet.wsgi.ChunkReadError and
swift.common.exceptions.ChunkReadError are entirely unrelated.
Change-Id: Ic9bcb73ea518c672a58a71d2b92152892f1ce2e4

Because random.randint includeds both endpoints so that
random.randint(0, 9) which is assigned in replicatore
should be [0-9]. Hence, the assertion for replication_cycle should be
*less or equal to* 9. And the replication_cycle should be mod of 10.
Change-Id: I81da375a4864256e8f3b473d4399402f83fc6aeb

- Verify .ring.gz path exist if ring file is the first argument.
- Code Refactoring:
 - swift/cli/info.parse_get_node_args()
 - Respective test cases for info.parse_get_node_args()
Closes-Bug: #1539275
Change-Id: I0a41936d6b75c60336be76f8702fd616d74f1545
Signed-off-by: Sachin Patil <psachin@redhat.com>

The reported timing can be 00:59:59 sometimes, but is still valid. This
will fail in the tests, as seen in [1].
This patch fixes this by mocking the current time, ensuring that the
first two rebalances happen at the same time.
[1] http://logs.openstack.org/97/337297/32/check/gate-swift-python27-ubuntu-xenial/46203f1/console.html#_2017年02月08日_07_28_42_589176
Change-Id: I0fd43d5bb13d0e88126f4f6ba14fb87faab6df9c

According to OpenStack Guideline[1], logged string message should be
interpolated by the logger.
[1]: http://docs.openstack.org/developer/oslo.i18n/guidelines.html#adding-variables-to-log-messages
Change-Id: I49952348c06e340c6cfc708028f2b06e02a16dfc
Closes-Bug: #1661262 

mtime and force_rewrite have a *long* tangled history starting back in
lp bug #1089140 that's been carried through many refactors.
Using force_rewrite on errors reading from the pickle has always been a
read-modify-write race; but maybe less bad than the infinite recursion
bug it fixed?
Using getmtime has always had somewhat dubious resolution for race
detection - the only way to be sure the content of the file is the same
as when we read it without locking is to open the file up and check.
Unfortunately, the ondisk data wasn't rich enough to disambiguate when
the ondisk state represented may have changed (e.g. when an invalidation
for a suffix currently being hashed is consolidated, or if all hashes
are invalid like after an error reading the hashes.pkl) - so we also add
a key with a timestamp for race detection and write down if the
dictionary has any valid suffix hashes.
Along the way, we accidentally fix a serious performance regression with
hash invalidations...
We currently rehash all invalid suffixes twice on REPLICATE calls.
First we consolidating hashes, marking all invalid suffixes as None
and then perform the first suffix rehashing.
And then also *every time* one more time again immediately as soon as we
get done with the first one we throw all that work we just did on the
floor and rehash ALL the invalid suffixes *again* a second time because
the race detector erroneously notices the hashes.pkl file has been
"modified while we were hashing".
But we're not in a race. We took the mtime before calling consolidate
hashes, and consolidate hashes modified the pickle when it wrote back the
invalid suffixes.
FWIW, since consolidate hashes operates under directory lock it can't
race - but we don't want suffix rehashing to hold the directory lock
that long so we use optimistic locking - i.e. we optimistically perform
the rehash w/o a lock and write back the update iif it hasn't changed
since read; if it has we retry the whole operation
UpgradeImpact:
If you upgrade and need to rollback - delete all hashes.pkl:
 rm /srv/node*/*/object*/*/hashes.pkl
Anything on significance achived here was blatently plagerised from the
work of others:
Co-Author: Pavel Kvasnička <pavel.kvasnicka@firma.seznam.cz>
Related-Change-Id: I64cadb1a3feb4d819d545137eecfc295389794f0
Co-Author: Alistair Coles <alistair.coles@hpe.com>
Related-Change-Id: I8f6bb89beaaca3beec2e6063299189f52e9eee51
Related-Change-Id: I08c8cf09282f737103e580c1f57923b399abe58c
Change-Id: Ia43ec2cf7ab715ec37f0044625a10aeb6420f6e3