openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Their 2.9.0 release is known-broken for py27-py35.
Change-Id: I40c1724fa673ac252f5052ac85006788ba69d5c7

Currently, when object-server serves GET request and DiskFile
reader iterate over disk file chunks, there is no explicit
eventlet sleep called. When network outpace the slow disk IO,
it's possible one large and slow GET request could cause
eventlet hub not to schedule any other green threads for a
long period of time. To improve this, this patch add a
configurable sleep parameter into DiskFile reader, which
is 'cooperative_period' with a default value of 0 (disabled).
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I80b04bad0601b6cd6caef35498f89d4ba70a4fd4

Object GET requests with a truthy X-Newest header are not resumed if a
backend request times out. The GetOrHeadHandler therefore uses the
regular node_timeout when waiting for a backend connection response,
rather than the possibly shorter recoverable_node_timeout. However,
previously while reading data from a backend response the
recoverable_node_timeout would still be used with X-Newest requests.
This patch simplifies GetOrHeadHandler to never use
recoverable_node_timeout when X-Newest is truthy.
Change-Id: I326278ecb21465f519b281c9f6c2dedbcbb5ff14

Both GetOrHeadHandler (used for replicated policy GETs) and
ECFragGetter (used for EC policy GETs) have _get_next_response_part
methods that are very similar. This patch replaces them with a single
method in the common GetterBase superclass.
Both classes are modified to use *only* the Request instance passed to
their constructors. Previously their entry methods
(GetOrHeadHandler.get_working_response and
ECFragGetter.response_parts_iter) accepted a Request instance as an
arg and the class then variably referred to that or the Request
instance passed to the constructor. Both instances must be the same
and it is therefore safer to only allow the Request to be passed to
the constructor.
The 'newest' keyword arg is dropped from the GetOrHeadHandler
constructor because it is never used.
This refactoring patch makes no intentional behavioral changes, apart
from the text of some error log messages which have been changed to
differentiate replicated object GETs from EC fragment GETs.
Change-Id: I148e158ab046929d188289796abfbbce97dc8d90

Change-Id: Ib790be26a2b990f313484f9ebdc99b8dc14613c9

... in document_iters_to_http_response_body.
We seemed to be relying a little too heavily upon prompt garbage
collection to log client disconnects, leading to failures in
test_base.py::TestGetOrHeadHandler::test_disconnected_logging
under python 3.12.
Closes-Bug: #2046352
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I4479d2690f708312270eb92759789ddce7f7f930

Update the yoga release notes configuration to build from
unmaintained/yoga.
Change-Id: I3ef2117e0e00c2a1dc02ab018baae04ebfeb7214

This has been available since py32 and was backported to py27; there
is no point in us continuing to carry the old idiom forward.
Change-Id: I21f64b8b2970e2dd5f56836f7f513e7895a5dc88

Last time we did this was nearly 4 years ago; drag ourselves into
something approaching the present. Address a few new pyflakes issues
that seem reasonable to enforce:
 E275 missing whitespace after keyword
 E231 missing whitespace after ','
 E721 do not compare types, for exact checks use `is` / `is not`,
 for instance checks use `isinstance()`
Main motivator is that the old hacking kept us on an old version
of flake8 et al., which no longer work with newer Pythons.
Change-Id: I54b46349fabb9776dcadc6def1cfb961c123aaa0

We've observed a root container suddenly thinks it's unsharded when it's
own_shard_range is reset. This patch blocks a remote osr with an epoch
of None from overwriting a local epoched OSR.
The only way we've observed this happen is when a new replica or handoff
node creates a container and it's new own_shard_range is created without
an epoch and then replicated to older primaries.
However, if a bad node with a non-epoched OSR is on a primary, it's
newer timestamp would prevent pulling the good osr from it's peers. So
it'll be left stuck with it's bad one.
When this happens expect to see a bunch of:
 Ignoring remote osr w/o epoch: x, from: y
When an OSR comes in from a replica that doesn't have an epoch when
it should, we do a pre-flight check to see if it would remove the epoch
before emitting the error above. We do this because when sharding is
first initiated it's perfectly valid to get OSR's without epochs from
replicas. This is expected and harmless.
Closes-bug: #1980451
Change-Id: I069bdbeb430e89074605e40525d955b3a704a44f

Change-Id: I0357939cf3a12712e6719c257705cf565e3afc8b

This is a more-intuitive name for what's going on and it's been working
well for us in the reconstructor.
Change-Id: Id935de4ca9eb6f38b0d587eaed8d13c54bd89d60

From https://docs.python.org/3/whatsnew/3.12.html :
 sum() now uses Neumaier summation to improve accuracy and
 commutativity when summing floats or mixed ints and floats.
At least, I *think* that's what was causing the ring builder failures.
Partial-Bug: #2046352
Change-Id: Icae2f1e3e95f216d214636bd5a6d1f40aacab20d

If the proxy timed out while reading a replicated policy multi-part
response body, it would transform the ChunkReadTimeout to a
StopIteration. This masks the fact that the backend read has
terminated unexpectedly. The document_iters_to_multipart_byteranges
would complete iterating over parts and send a multipart terminator
line, even though no parts may have been sent.
This patch removes the conversion of ChunkReadTmeout to StopIteration.
The ChunkReadTimeout that is now raised prevents the
document_iters_to_multipart_byteranges 'for' loop completing and
therefore stops the multi-part terminator line being sent. It is
raised from the GetOrHeadHandler similar to other scenarios that raise
ChunkReadTimeouts while the resp body is being read.
A ChunkReadTimeout exception handler is removed in the
_iter_parts_from_response method. This handler was previously never
reached (because StopIteration rather than ChunkReadTimeout was raised
from _get_next_response_part), but if it were reached (i.e. with this
change) then it would repeat logging of the error and repeat
incrementing the node's error counter.
This change in the GetOrHeadHandler mimics a similar change in the
ECFragGetter [1].
[1] Related-Chage: I0654815543be3df059eb2875d9b3669dbd97f5b4
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I6dd53e239f5e7eefcf1c74229a19b1df1c989b4a

The proxy should NOT read or write to memcache when handling a
container GET that explicitly requests 'shard' or 'object' record
type. A request for 'shard' record type may specify 'namespace'
format, but this request is unrelated to container listings or object
updates and passes directly to the backend.
This patch also removes unnecessary JSON serialisation and
de-serialisation of namespaces within the proxy GET path when a
sharded object listing is being built. The final response body will
contain a list of objects so there is no need to write intermediate
response bodies with a list of namespaces.
Requests that explicitly specify record type of 'shard' will of
course still have the response body with serialised shard dicts that
is returned from the backend.
Change-Id: Id79c156432350c11c52a4004d69b85e9eb904ca6

Related-Change-Id: I9b5cc6d5fb69a2957b8c4846ce1feed8c115e6b6
Change-Id: I5dda9767c1c66597291211a087f7c917ba990651

Unit test changes only:
- Add tests for some resuming replicated GET scenarios.
- Add test to cover resuming GET fast_forward "failing" when range
 read is complete.
- Add test to verify different node_timeout for account and container
 vs object controller getters.
- Refactor proxy.test_server.py tests to split out different
 scenarios.
Drive-by: remove some ring device manipulation setup that's not needed.
Change-Id: I38c7fa648492c9bd2173ecf92f89e423bee4abf3
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Seems since somewhere around sqlite 3.40+ our in tests malformed sqlite
db isn't malformed anymore. I don't actually know how it was malformed
but looking in a hex editor it seems to have a bunch of null
truncated in the middle of the file. Which maybe isn't an issue anymore?
Instead I've gone and messed up what looks like to be the marker before
defining the test table data at the end of file, so from:
 00001FF0 00 00 00 00 00 00 00 00 00 00 00 03 01 02 0F 31 ...............1
 ^^
To:
 00001FF0 00 00 00 00 00 00 00 00 00 00 00 FF 01 02 0F 31 ...............1
 ^^
Basically FF'ed the start of the data marker (at least what I'm calling
it).
Closes-Bug: #2051067
Change-Id: I2a10adffa39abbf7e97718b7228de298209140f8

See https://github.com/eventlet/eventlet/pull/890
It's a relatively minor breakage, triggered by clients hitting a funny
corner of the HTTP spec, but it does prevent our unit tests from passing
and may surprise some clients.
Change-Id: Id29ba545e4ac7887c63fa62b75469988a2c6773c

Change-Id: I744416ffaf245c278ebce6350854b99c0eed88e3

Change-Id: Ia35743d3a59bfc456c45d28340f81a45289ceb6c

Recently, upper-constraints updated eventlet. Unfortunately, there
was a bug which breaks our unit tests which was not discovered during
the cross-project testing because the affected unit tests require an
XFS temp dir. The requirements change has since been reverted, but we
ought to have tests that cover the problematic behavior that will
actually run as part of cross-project testing.
See https://github.com/eventlet/eventlet/pull/826 for the eventlet
change that introduced the bug; it has since been fixed on master in
https://github.com/eventlet/eventlet/pull/890 (though we still need
https://review.opendev.org/c/openstack/swift/+/905796 to be able to
work with eventlet master).
Change-Id: I4a6d79317b65f746ee29d2d25073b8c3859cd6a0

Change-Id: I94e8cfd154aa058d91255efc87776224a919f572

See https://github.com/eventlet/eventlet/pull/826 and its follow-up,
https://github.com/eventlet/eventlet/pull/890
Change-Id: I7dff5342013a3f31f19cb410a9f3f6d4b60938f1

When looking at containers and accounts it's sometimes nice to know who
they've been replicating with. This patch adds a `--sync|-s` option to
swift-{container|account}-info which will also dump the incoming and
outgoing sync tables:
 $ swift-container-info /srv/node3/sdb3/containers/294/624/49b9ff074c502ec5e429e7af99a30624/49b9ff074c502ec5e429e7af99a30624.db -s
 Path: /AUTH_test/new
 Account: AUTH_test
 Container: new
 Deleted: False
 Container Hash: 49b9ff074c502ec5e429e7af99a30624
 Metadata:
 Created at: 2022年02月16日T05:34:05.988480 (1644989645.98848)
 Put Timestamp: 2022年02月16日T05:34:05.981320 (1644989645.98132)
 Delete Timestamp: 1970年01月01日T00:00:00.000000 (0)
 Status Timestamp: 2022年02月16日T05:34:05.981320 (1644989645.98132)
 Object Count: 1
 Bytes Used: 7
 Storage Policy: default (0)
 Reported Put Timestamp: 1970年01月01日T00:00:00.000000 (0)
 Reported Delete Timestamp: 1970年01月01日T00:00:00.000000 (0)
 Reported Object Count: 0
 Reported Bytes Used: 0
 Chexor: 962368324c2ca023c56669d03ed92807
 UUID: f33184e7-56d5-4c74-9d2e-5417c187d722-sdb3
 X-Container-Sync-Point2: -1
 X-Container-Sync-Point1: -1
 No system metadata found in db file
 No user metadata found in db file
 Sharding Metadata:
 Type: root
 State: unsharded
 Incoming Syncs:
 Sync Point	Remote ID 	Updated At
 1 	ce7268a1-f5d0-4b83-b993-af17b602a0ff-sdb1	2022年02月16日T05:38:22.000000 (1644989902)
 1 	2af5abc0-7f70-4e2f-8f94-737aeaada7f4-sdb4	2022年02月16日T05:38:22.000000 (1644989902)
 Outgoing Syncs:
 Sync Point	Remote ID	Updated At
 Partition	294
 Hash 	49b9ff074c502ec5e429e7af99a30624
As a follow up to the device in DB ID patch we can see that the replicas
at sdb1 and sdb4 have replicated with this node.
Change-Id: I23d786e82c6710bea7660a9acf8bbbd113b5b727

With the Related-Change, container servers can return a list Namespace
objects in response to a GET request. This patch modifies the proxy
to take advantage of this when fetching namespaces. Specifically,
the proxy only needs Namespaces when caching 'updating' or 'listing'
shard range metadata.
In order to allow upgrades to clusters we can't just send
'X-Backend-Record-Type = namespace', as old container servers won't
know how to respond. Instead, proxies send a new header
'X-Backend-Record-Shard-Format = namespace' along with the existing
'X-Backend-Record-Type = shard' header. Newer container servers will
return namespaces, old container servers continue to return full
shard ranges and they are parsed as Namespaces by the new proxy.
This patch refactors _get_from_shards to clarify that it does not
require ShardRange objects. The method is now passed a list of
namespaces, which is parsed from the response body before the method
is called. Some unit tests are also refactored to be more realistic
when mocking _get_from_shards.
Also refactor the test_container tests to better test shard-range and
namespace responses from legacy and modern container servers.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Jianjian Huo <jhuo@nvidia.com>
Related-Change: If152942c168d127de13e11e8da00a5760de5ae0d
Change-Id: I7169fb767525753554a40e28b8c8c2e265d08ecd

The proxy-server makes GET requests to the container server to fetch
full lists of shard ranges when handling object PUT/POST/DELETE and
container GETs, then it only stores the Namespace attributes (lower
and name) of the shard ranges into Memcache and reconstructs the list
of Namespaces based on those attributes. Thus, a namespaces GET
interface can be added into the backend container-server to only
return a list of those Namespace attributes.
On a container server setup which serves a container with ~12000
shard ranges, benchmarking results show that the request rate of the
HTTP GET all namespaces (states=updating) is ~12 op/s, while the
HTTP GET all shard ranges (states=updating) is ~3.2 op/s.
The new namespace GET interface supports most of headers and
parameters supported by shard range GET interface. For example,
the support of marker, end_marker, include, reverse and etc. Two
exceptions are: 'x-backend-include-deleted' cannot be supported
because there is no way for a Namespace to indicate the deleted state;
the 'auditing' state query parameter is not supported because it is
specific to the sharder which only requests full shard ranges.
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: If152942c168d127de13e11e8da00a5760de5ae0d