openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: Idbff951506ee2f3b288eda00217c902314393877

Change-Id: I1f26483dbc5935052ec8df0c874ee38070b84d5b

While we use requests-mock, we don't need the pytest plugin.
Change-Id: If14e4d2c1af2efcbc99e9b6fe10973a7eb94d589

Below new metrics will be added:
memcached.[method].timeout.timing
memcached.[method].conn_err.timing
memcached.[method].errors.timing
Also, MemcacheRing method and time spent in the function will
be logged when Timeout or other exceptions are raised.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I4d4f20b92b85255ac8bf66f2c830e691e64bbe47

Create a unit test to verify client timeout for multiple requests
Change-Id: I974e01cd2cb18f4ea87c3966dbf4b06bff22ed39

Change-Id: I9fdc74d26fd830f463c077c912cdcf00eaab1dfa

As it was, it would hide issues in the logging or ratelimiter
implementations.
Change-Id: I9e557442401ef17b753f45b9e1cb181e71784ccf

Change-Id: Ie2a8e4eced6688e5a98aa37c3c7b0c13fd2ddeee

Change-Id: Ib3c9b274bbd2e643f3febbdf54a8a43f4775944b

Replicated, unencrypted metadata is written down differently on py2
vs py3, and has been since we started supporting py3. Fortunately,
we can inspect the raw xattr bytes to determine whether the pickle
was written using py2 or py3, so we can properly read legacy py2 meta
under py3 rather than hitting a unicode error.
Closes-Bug: #2012531
Change-Id: I5876e3b88f0bb1224299b57541788f590f64ddd4

User provided keys are need to debug those tracebacks/timeouts when
clients talking to memcached, in order to associate those failures
with specific memcache usages within swift services.
Change-Id: I07491bb4ebc3baa13cf09f64a04a61011d561409

Previously swift.common.utils monkey patched logging.thread,
logging.threading, and logging._lock upon import with eventlet
threading modules, but that is no longer reasonable or necessary.
With py3, the existing logging._lock is not patched by eventlet,
unless the logging module is reloaded. The existing lock is not
tracked by the gc so would not be found by eventlet's
green_existing_locks().
Instead we group all monkey patching into utils function and apply
patching consistently across daemons and WSGI servers.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Closes-Bug: #1380815
Change-Id: I6f35ad41414898fb7dc5da422f524eb52ff2940f

... and clean up WatchDog start a little.
If this pattern proves useful we could consider extending it.
Change-Id: Ia85f9321b69bc4114a60c32a7ad082cae7da72b3

This effects both daemon config parsing and paste-deploy config parsing
when using conf.d. When the WSGI servers were loaded from a flat file
they have always been case-sensitive. This difference was surprising
(who wants anything case-insensitive?) and potentially dangerous for
values like RECLAIM_AGE.
UpgradeImpact:
Previously the option keys in swift's configuration .ini files were
sometimes parsed in a case-insensitive manner, so you could use
CLIENT_TIMEOUT and the daemons would recognize you meant client_timeout.
Now upper-case or mixed-case option names, such as CLIENT_TIMEOUT or
Client_Timeout, will be ignored.
Change-Id: Idd8e552d9fe98b84d7cee1adfa431ea3ae93345d

The log message phrase 'ChunkWriteTimeout fetching fragments'
implies that the timeout has occurred
while getting a fragment (from the backend object server)
when in fact the timeout has occurred
waiting to yield the fragment to the app iter.
Hence, changing message to 'ChunkWriteTimeout feeding fragments'
Change-Id: Ic0813e6a9844da1130091d27e3dbe272ea871d11

Closes-Bug: #2017021
Change-Id: If422f99a77245b35ab755857f9816c1e401a4e22

Closes-Bug: #2016463
Change-Id: Id500a2429b7412823970a06e3e82b1d1646c70b8

This puts replication info in antique rings loaded with metadata_only=True.
Closes-Bug: #1696837
Change-Id: Idf263a7f7a984a1307bd74040ac8f8bb1651bc79

Using an antiquated distro to keep us on an EOL'ed version of python
that *just so happens* to have had support dropped for it around the
same time virtualenv removed the ability to create py27 environments
doesn't seem like the right way to address the problem.
Make sure we also include the tox<4 pin that we're using project-wide
so we don't get some nonsense about "Multiple top-level packages
discovered in a flat-layout".
Related-Change: https://review.opendev.org/c/openstack/swift/+/881035
Change-Id: I32a161e555179ca34d306ac37e4097611853e36b

Simplify ECFragGetter by removing code that guards against the policy
fragment_size being None or zero.
Policy fragment_size must be > 0: the fragment_size is based on the
ec_segment_size, which is verified as > 0 when constructing an EC
policy. This is asserted by test_parse_storage_policies in
test.unit.common.test_storage_policy.TestStoragePolicies.
Also, rename client_chunk_size to fragment_size for clarity.
Change-Id: Ie1efaab3bd0510275d534b5c023cb73c98bec90d

Refactor and add some targeted unit tests. No behavioral change.
Change-Id: I153528b8a1709f3756c261cf3eb2acfd5de10f9c

The test claimed to assert that ChunkWriteTimeouts are logged, but
the test would in fact pass if the timeouts were not logged.
Change-Id: Ic9d119858397e8aeccaf7f89487f9e62f16ee453

Change-Id: Ia87caebc9a3c70602209daeda97fc588f4bafc76

Change-Id: I3d5981cfc8799c45817b24fdaecc6cbbc0534ec8

`much_older` has to be much older than `older`, or the test gets
flakey. See
- test_cleanup_ondisk_files_reclaim_non_data_files,
- test_cleanup_ondisk_files_reclaim_with_data_files, and
- test_cleanup_ondisk_files_reclaim_with_data_files_legacy_durable
for a more standard definition of "much_older".
Closes-Bug: #2017024
Change-Id: I1eaa501827f4475ddc0c20d82cf0a6d4a5e98f75