openstack/swift - swift - OpenDev: Free Software Needs Free Tools

A make_timestamp_iter was added into setUp in the Related-Change.
There was a couple of different tests that were using timestamp iters
in different ways, consistently we apply the use of next(self.ts)
which I believe is becoming more common/standardized in unitests.
Related-Change: I074fbecb6440fb1d04279cd892d38d2acc44b47d
Change-Id: Ib6b883afec242355ae08c50c1e685a20e5efadc7

The change in preference to the Related-Change is simply to prefer
sysmeta overrides in the headers to backend overrides in the footers:
Before:
sysmeta footers > backend footers > sysmeta headers > backend headers
After:
sysmeta footers > sysmeta headers > backend footers > backend headers
This change just breaks up the tests to try to make it more obvious
what already worked and what has changed. The justification seems to
be that overrides in sysmeta headers only work on policies that don't
send backend footers, but sysmeta overrides should always have a
higher preference than backend overrides.
Related-Change: Idb40361ac72da51e1390dff690723dbc2c653a13
Change-Id: I074fbecb6440fb1d04279cd892d38d2acc44b47d

Change-Id: If764de0a28f5afb858b3e892b35fe5fa147a0650

If the curl command is used exactly as in the help, the ampersand
in the signature is interpreted as an operator and the curl
command breaks. I am aware of developers who have wasted a lot of
time because of this.
Change-Id: I6468c9a098b56db8242a2cf2c23b7a4857bd8574

* Fix bytes vs Unicode issues
* On Python 3, encode JSON to UTF-8 and decode it from UTF-8
* Open files in binary mode to avoid Unicode issues
* test_auditor: use bytes for content, open files in binary mode
Change-Id: Ifa84001493cfb57975d3b140b0d7e09020504bca

An high or increasing partition count due to storing handoffs can have
some severe side-effects, and replication might never be able to catch
up. This patch adds a note to the admin_guide how to check this.
Change-Id: Ib4e161d68f1a82236dbf5fac13ef9a13ac4bbf18

pretend_min_part_hours_passed do things like this:
self._last_part_moves[part] = 0xff
this will throw exception if self._last_part_moves is None.
this patch is to check self._last_part_moves to prevent exception.
Closes-bug: #1578835
Change-Id: Ic83c7a338b45bfcf61f5ab6100e6db335c3fa81a

About half the logic was the same, and it seems better to just implement
that once.
Change-Id: I350da34ef7a3cd0cb74f585f4691992ae64c7eab

* Replace "c = dict(a.items() + b.items())" with
 "c = dict(a); c.update(b)". It works on Python 2 and Python 3, and
 it may be a little bit more efficient on Python 2 (no need to
 create a temporary list of items).
* Replace "dict.values() + dict.values()" with
 "list(dict.values()) + list(dict.values())": on Python 3,
 dict.values() is a view which doesn't support a+b operator.
Change-Id: Id5a65628fe2fb7a02c713b758fcaa81154db28a0

Replace reload() builtin function with six.moves.reload_module() to
make the code compatible with Python 2 and Python 3.
Change-Id: I7572d613fef700b392d412501facc3bd5ee72a66

Swift EC has a strong constraint about the ring must have a number of
replicas which fits ec_k + ec_m. That is validated when servers waking
up. However, Swift has more chance to load such an invalid ring when
a request comming, calling some node iteration like get_nodes,
get_part_nodes or so, and no ring validation is there.
This patch moves ring validation from policy validate_ring into the ring
instance as validation_hook that will run at ring reload. Since this patch,
ring instance will allow to use the old ring if the reload is not fourced.
Note that the exception if invalid ring found was changed from
RingValidationError to RingLoadError because RingValidationError is a
child of RingBuilderError but the ring reload is obviously outside of
"builder".
Closes-Bug: #1534572
Change-Id: I6428fbfb04e0c79679b917d5e57bd2a34f2a0875

encrypt_header_val is used to translate a raw header value into an
encrypted value. Semantically, a header with an empty value won't be stored
and all callers seem to remove such a header before calling
encrypted_header_val.
So if no reason for returning ('', None), I think it's better to change it to
raise ValueError not to cause another error for users of the return value.
(e.g. dump_crypto_meta)
Plus this patch addes a few unit tests for those cases above.
Change-Id: Ic1237f4afb8c0e466be5ce59fe31b667c39242b0

ECStoragePolicy.fragment_size is never changed on running Swift because
it is from ec_segment_size and ec_type defined in swift.conf statically
so let's cache the value after retrieving the value from the pyeclib driver.
And more, pyeclib <= 1.2.1 (current newest) has a bug [1] to leak the reference
count of the items in the returned dict (i.e. causes memory leak) so that
this caching will be mitigation of the memory leak because this saves the call
count fewer than current as possible.
Note that the complete fix for the memory leak for pyeclib is proposed at
https://review.openstack.org/#/c/344066/
1: https://bugs.launchpad.net/pyeclib/+bug/1604335
Related-Bug: #1604335
Change-Id: I6bbaa4063dc462383c949764b6567b2bee233689

Fix copy middleware so that all client-defined object
headers that object servers allow to be persisted are copied.
For example, content-encoding and content-disposition will
now be copied.
Fix treatment of x-fresh-metadata header so that, when it is
used, new object sysmeta is applied to the object copy in the
same way as a copy without x-fresh-metadata.
Remove unnecessary passing of original request headers to
sink PUT request constructor: passing the environ is sufficient
to have the new request inherit the original's headers.
Add tests for this change and to verify that content-type
gets either copied or updated if supplied with the copy
request.
Add tests for x-fresh-metadata treatment.
Closes-Bug: #1391826
Closes-Bug: #1600247
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: I917fb0b4e831c13e04ade1c5e0b9821802dec967

This change patches the correct eventlet monkey patched httplib's
_MAXHEADERS as required by Swift.
It also makes the configparser inside the copy middleware py3
compatable.
Change-Id: I2f7dbcecbbecd7cb69b7031faa39f35bcfadcfc8

If you deploy an object server but have no rings at all (and are using
servers-per-port), then the CPU will spin as it checks for child
processes since there are actually no child processes to check.
This patch adds a sleep so that the CPU doesn't spin.
Change-Id: Iece62367aa2481a21752144b1f4477a3713282fe

Fixes generator' object has no attribute
'next' issues
Change-Id: I1f21eaed0ae7062073438503d3f6860d8b4f36c8

Change-Id: I0388f4738966bc453e922e9598ff9df60ecda4eb

Change-Id: Ic0b5445e313924c683e9889d94569c2554fd0b8b

In test_ringbuilder.py, there is one assertTrue should be
replaced with assertEqual.
Change-Id: I9a0e4a7363a5e16cc9b6df045953dfbb4f9dbd07
Closes-bug: #1604320 

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure
Change-Id: Ica92e45dbc3f83ec806e092fd3ea5ada636ab7e9

The object replicator can log some junk about the cluster ip instead
of the replication ip in some specific error log lines that can make
you think either you're crazy or your rings are crazy.
... in this case it was just the logging was crazy - so fix that.
Change-Id: Ie5cbb2d1b30feb2529c17fc3d72af7df1aa3ffdd

Change-Id: I9b98da30e5f934164e490beb4d6cde840f08832a

For example: a>b and a<=c is equal to b<a<=c
Change-Id: Iae1532f0946c6d4aa7321f3957820b486869c59f

Change-Id: I3aff0de418b52b1f16f5863c95ad2700678143ae

For example: a < b and b <= c is equal to a < b <= c
Change-Id: I91ceb194bce60f6160ebdf0aadf0e8f0d7a35975

I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I80ec96e0b729bef38213a6be4ff4b6eb65c7612d

I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type)
or assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: If6aad8681aab7c9a41d65a4f449d8abbe3e64616

I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type)
or assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I9531c9939aa7c2dac127b5dc865b8d396dab318f

Reduced source code by extracting duplicated code
(swift/cli/ringbuilder.py)
Change-Id: Ibd000df1dc9042e31b65b000199dff4a645e63b4