openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Related-Change-Id: I9b5cc6d5fb69a2957b8c4846ce1feed8c115e6b6
Change-Id: I5dda9767c1c66597291211a087f7c917ba990651

Unit test changes only:
- Add tests for some resuming replicated GET scenarios.
- Add test to cover resuming GET fast_forward "failing" when range
 read is complete.
- Add test to verify different node_timeout for account and container
 vs object controller getters.
- Refactor proxy.test_server.py tests to split out different
 scenarios.
Drive-by: remove some ring device manipulation setup that's not needed.
Change-Id: I38c7fa648492c9bd2173ecf92f89e423bee4abf3
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Seems since somewhere around sqlite 3.40+ our in tests malformed sqlite
db isn't malformed anymore. I don't actually know how it was malformed
but looking in a hex editor it seems to have a bunch of null
truncated in the middle of the file. Which maybe isn't an issue anymore?
Instead I've gone and messed up what looks like to be the marker before
defining the test table data at the end of file, so from:
 00001FF0 00 00 00 00 00 00 00 00 00 00 00 03 01 02 0F 31 ...............1
 ^^
To:
 00001FF0 00 00 00 00 00 00 00 00 00 00 00 FF 01 02 0F 31 ...............1
 ^^
Basically FF'ed the start of the data marker (at least what I'm calling
it).
Closes-Bug: #2051067
Change-Id: I2a10adffa39abbf7e97718b7228de298209140f8

Recently, upper-constraints updated eventlet. Unfortunately, there
was a bug which breaks our unit tests which was not discovered during
the cross-project testing because the affected unit tests require an
XFS temp dir. The requirements change has since been reverted, but we
ought to have tests that cover the problematic behavior that will
actually run as part of cross-project testing.
See https://github.com/eventlet/eventlet/pull/826 for the eventlet
change that introduced the bug; it has since been fixed on master in
https://github.com/eventlet/eventlet/pull/890 (though we still need
https://review.opendev.org/c/openstack/swift/+/905796 to be able to
work with eventlet master).
Change-Id: I4a6d79317b65f746ee29d2d25073b8c3859cd6a0

Change-Id: I94e8cfd154aa058d91255efc87776224a919f572

See https://github.com/eventlet/eventlet/pull/826 and its follow-up,
https://github.com/eventlet/eventlet/pull/890
Change-Id: I7dff5342013a3f31f19cb410a9f3f6d4b60938f1

When looking at containers and accounts it's sometimes nice to know who
they've been replicating with. This patch adds a `--sync|-s` option to
swift-{container|account}-info which will also dump the incoming and
outgoing sync tables:
 $ swift-container-info /srv/node3/sdb3/containers/294/624/49b9ff074c502ec5e429e7af99a30624/49b9ff074c502ec5e429e7af99a30624.db -s
 Path: /AUTH_test/new
 Account: AUTH_test
 Container: new
 Deleted: False
 Container Hash: 49b9ff074c502ec5e429e7af99a30624
 Metadata:
 Created at: 2022年02月16日T05:34:05.988480 (1644989645.98848)
 Put Timestamp: 2022年02月16日T05:34:05.981320 (1644989645.98132)
 Delete Timestamp: 1970年01月01日T00:00:00.000000 (0)
 Status Timestamp: 2022年02月16日T05:34:05.981320 (1644989645.98132)
 Object Count: 1
 Bytes Used: 7
 Storage Policy: default (0)
 Reported Put Timestamp: 1970年01月01日T00:00:00.000000 (0)
 Reported Delete Timestamp: 1970年01月01日T00:00:00.000000 (0)
 Reported Object Count: 0
 Reported Bytes Used: 0
 Chexor: 962368324c2ca023c56669d03ed92807
 UUID: f33184e7-56d5-4c74-9d2e-5417c187d722-sdb3
 X-Container-Sync-Point2: -1
 X-Container-Sync-Point1: -1
 No system metadata found in db file
 No user metadata found in db file
 Sharding Metadata:
 Type: root
 State: unsharded
 Incoming Syncs:
 Sync Point	Remote ID 	Updated At
 1 	ce7268a1-f5d0-4b83-b993-af17b602a0ff-sdb1	2022年02月16日T05:38:22.000000 (1644989902)
 1 	2af5abc0-7f70-4e2f-8f94-737aeaada7f4-sdb4	2022年02月16日T05:38:22.000000 (1644989902)
 Outgoing Syncs:
 Sync Point	Remote ID	Updated At
 Partition	294
 Hash 	49b9ff074c502ec5e429e7af99a30624
As a follow up to the device in DB ID patch we can see that the replicas
at sdb1 and sdb4 have replicated with this node.
Change-Id: I23d786e82c6710bea7660a9acf8bbbd113b5b727

With the Related-Change, container servers can return a list Namespace
objects in response to a GET request. This patch modifies the proxy
to take advantage of this when fetching namespaces. Specifically,
the proxy only needs Namespaces when caching 'updating' or 'listing'
shard range metadata.
In order to allow upgrades to clusters we can't just send
'X-Backend-Record-Type = namespace', as old container servers won't
know how to respond. Instead, proxies send a new header
'X-Backend-Record-Shard-Format = namespace' along with the existing
'X-Backend-Record-Type = shard' header. Newer container servers will
return namespaces, old container servers continue to return full
shard ranges and they are parsed as Namespaces by the new proxy.
This patch refactors _get_from_shards to clarify that it does not
require ShardRange objects. The method is now passed a list of
namespaces, which is parsed from the response body before the method
is called. Some unit tests are also refactored to be more realistic
when mocking _get_from_shards.
Also refactor the test_container tests to better test shard-range and
namespace responses from legacy and modern container servers.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Jianjian Huo <jhuo@nvidia.com>
Related-Change: If152942c168d127de13e11e8da00a5760de5ae0d
Change-Id: I7169fb767525753554a40e28b8c8c2e265d08ecd

The proxy-server makes GET requests to the container server to fetch
full lists of shard ranges when handling object PUT/POST/DELETE and
container GETs, then it only stores the Namespace attributes (lower
and name) of the shard ranges into Memcache and reconstructs the list
of Namespaces based on those attributes. Thus, a namespaces GET
interface can be added into the backend container-server to only
return a list of those Namespace attributes.
On a container server setup which serves a container with ~12000
shard ranges, benchmarking results show that the request rate of the
HTTP GET all namespaces (states=updating) is ~12 op/s, while the
HTTP GET all shard ranges (states=updating) is ~3.2 op/s.
The new namespace GET interface supports most of headers and
parameters supported by shard range GET interface. For example,
the support of marker, end_marker, include, reverse and etc. Two
exceptions are: 'x-backend-include-deleted' cannot be supported
because there is no way for a Namespace to indicate the deleted state;
the 'auditing' state query parameter is not supported because it is
specific to the sharder which only requests full shard ranges.
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: If152942c168d127de13e11e8da00a5760de5ae0d

The 'states' argument of get_shard_ranges() should be a list of ints,
but previously just a single int was tolerated. This was unnecessary
and led to inconsistent usage across call sites.
We'd like similar ContainerBroker methods, such as the anticipated
get_namespaces() [1], to have an interface consistent with
get_shard_ranges(), but not continue the unnecessary pattern of
supporting both a list and a single int argument for 'states'. This
patch therefore normalises all call sites to pass a list and
deprecates support for just a single int.
[1] Related-Change: If152942c168d127de13e11e8da00a5760de5ae0d
Change-Id: I056cefbf0894dbc68b9a6eb3d76ec4dc0a72de0d

Be willing to accept shards instead of objects when querying containers.
If we receive shards, be willing to query them looking for the object.
Change-Id: I0d8dd42f81b97dddd6cf8910afaef4ba85e67d27
Partial-Bug: #1925346 

The setUp method creates a timestamp iterator, so let's use it
consistently in all the tests.
Change-Id: Ibd06b243c6db93380b99227ac79157269a64b28a

Change-Id: Ic89141c0dce619390c2be8a01d231f9ff8e2056c

Previously, our tests would not just fail, but segfault on recent
eventlet releases. See https://github.com/eventlet/eventlet/issues/864
and https://github.com/python/cpython/issues/113631
Fortunately, it looks like we can just avoid actually monkey-patching
to dodge the bug.
Closes-Bug: #2047768
Change-Id: I0dc22dab05bc00722671dca3f0e6eb1cf6e18349

When logging a request, if the request environ has a
swift.proxy_logging_status item then use its value for the log
message status int.
The swift.proxy_logging_status hint may be used by other middlewares
when the desired logged status is different from the wire_status_int.
If the proxy_logging middleware detects a client disconnect then any
swift.proxy_logging_status item is ignored and a 499 status int is
logged, as per current behaviour. i.e.:
 * client disconnect overrides swift.proxy_logging_status and the
 response status
 * swift.proxy_logging_status overrides the response status
If the proxy_logging middleware catches an exception then the logged
status int will be 500 regardless of any swift.proxy_logging_status
item.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I9b5cc6d5fb69a2957b8c4846ce1feed8c115e6b6

If a method is not allowed, real swift proxy server app will return an
HTTPMethodNotAllowed response, whereas FakeSwift would previously
*raise* HTTPNotImplemented. S3Api deliberately sends requests with
method 'TEST' which is not allowed/implemented. To workaround the
difference in real and fake swift behaviour, FakeSwift was configured
to allow the 'TEST' method, and then in some tests an
HTTPMethodNotAllowed response was registered for 'TEST' requests!
This patch modifies FakeSwift to return an HTTPMethodNotAllowed
response to the incoming request when the request method is not
allowed.
It is no longer necessary for FakeSwift to support extending the
default list of allowed methods.
Change-Id: I550d0174e14a5d5a05d26e5cbe9d3353f5da4e8a

We remove s3api.FakeSwift and replace it with the "normal" FakeSwift.
Additionally the @s3acl decorator is removed and replaced with an
inheritance based pattern. This simplifies maintenance using more
familiar patterns and improves debugging.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I55b596a42af01870b49fda22800f7a1293163eb8

The ?format=raw TestCase has it's own manifest setup and doesn't do any
segment validation. It's manifests are not suitable for use in other
TestCases.
Change-Id: Idf4b72bb59b8bf7232236ca544a3317b6e2e08fd

The Namespace class grew account/container properties to make them
easier to use in the proxy and subjected to similar consistency
requirements as the ShardRange's properties in the related change.
There are no new assertions added in this change, it merely
consolidates the py2/py3 validating helper which was duplicated
between the Namespace and ShardRange TestCases.
Related-Change-Id: Iebb09d6eff2165c25f80abca360210242cf3e6b7
Change-Id: Ide7f1dd3d9c664fb57c47dcd50edb44ae90ff5f9

When constructing an object listing from container shards, the proxy
would previously return the X-Backend-Record-Type header with the
value 'shard' that is returned with the initial GET response from the
root container. It didn't break anything but was plainly wrong.
This patch removes the header from object listing responses to request
that did not have the header. The header value is not set to 'object'
because in a request that value specifically means 'do not recurse
into shards'.
Change-Id: I94c68e5d5625bc8b3d9cd9baa17a33bb35a7f82f

ShardRange.name is required to have the form <account/container>. We'd
like to be able to replace ShardRange instances with the Namespace
superclass but still have the convenience of the account and container
accessors.
The name is stored as a single attribute and split when accessing via
the account and container getters, rather than splitting into two
attributes in the name setter, to minimise the overhead of
constructing Namespace instances. Where performance can be critical
(e.g. fetching the entire set of namespaces from a container server)
the number of Namespace instances constructed can be much greater than
the number whose account and container properties are used. The author
found that splitting in the account and container getters became more
efficient than splitting in the name setter when the rate of
constructing instances was ~2x greater than the rate of calling the
account and container getters.
The account and container property setters are removed from the
ShardRange class. The name setter is removed from the Namespace class.
These setter were never used.
Change-Id: Iebb09d6eff2165c25f80abca360210242cf3e6b7

Modern tests mostly consistently use a debug_logger when available and I
find the assertions a little more intuative to read than Mock's
assert_calls helpers.
Related-Change-Id: I13a24c4a0cd4045ab2fffb18067674b3e3bf742b
Change-Id: Ia9eeae6ed231a9be504967208efa3a0124f8cf09

Add a TestCase for the GetOrHeadHandler class and relocate existing
unit tests for that class from TestFuncs.
No changes other than lines moving.
Change-Id: I13a24c4a0cd4045ab2fffb18067674b3e3bf742b

During an MPU complete, the s3api first PUTs an SLO manifest and then
DELETEs the upload marker in <bucket>+segments. If the proxy's clock
is slow relative to the proxy that created the upload marker, the DELETE
will fail with a 409.
Previously the 409 would be returned to the client, which for some
clients (e.g. aws cli) does not trigger a retry. Both the manifest and
upload marker would remain, which would cause the upload to continue
to be listed as if still "in progress". Worse, since the complete
failed, the client might delete the segments, leaving the manifest in
place but with no segments.
This patch adds a "look-before-you-leap" pre-check on the upload
marker timestamp. If the marker is found to be in the future then
neither the manifest PUT nor the marker DELETE are attempted, and the
client receives a 503 response. If the pre-check passes but somehow
the marker DELETE still fails with a 409, the client will now receive
a 503 that will hopefully trigger a retry.
Closes-Bug: #2045046
Change-Id: Ie2e0cb75425e00cff533014af6b6fafad89bff94

Change-Id: I303a1ff97325654478c2a47af056deda37696b7b

Add sub-type timing metrics for container PUT/GET, includes below new metrics:
Container PUT: PUT_object, PUT_shard, PUT_container
Container GET: GET_object, GET_shard
Change-Id: Ifa2341b11b8244cad29fca574a13cf31134a4a0d

...and rename to _get_updating_shard_ranges.
The method is only used by the ObjectController, and it is only used
to fetch shard ranges in the 'updating' states.
Also relocate the associated unit tests.
Change-Id: I083e0c6898bf93d8a0dc593acd9723827e55508e

The per-service option was deprecated almost 4 years ago[1].
[1] 4601548dab
Change-Id: I45f7678c9932afa038438ee841d1b262d53c9da8

Create new helper functions to set and get namespaces in cache. Use
these in both the object and container controllers when caching
namespaces for updating and listing state shard ranges respectively.
Add unit tests for the new helper functions.
No intentional behavioural changes.
Change-Id: I6833ec64540fa19f658f0ee78952ecb43b49f169

The proxy_logging middleware needs an X-Backend-Storage-Policy-Index
header to populate the storage policy field in logs, and will look in
both request and response headers to find it.
Previously, the s3api middleware would indiscriminately copy the
X-Backend-Storage-Policy-Index from swift backend requests into the
S3Request headers [1]. This works for logging but causes the header
to leak between backend requests [2] and break mixed policy
multipart uploads. This patch sets the X-Backend-Storage-Policy-Index
header on s3api responses rather than requests.
Additionally, the middleware now looks for the
X-Backend-Storage-Policy-Index header in the swift backend request
*and* response headers, in the same way that proxy_logging would
(preferring a response header over a request header). This means that
a policy index is now logged for bucket requests, which only have
X-Backend-Storage-Policy-Index header in their response headers.
The s3api adds the value from the *final* backend request/response
pair to its response headers. Returning the policy index from the
final backend request/response is consistent with swift.backend_path
being set to that backend request's path i.e. proxy_logging will log
the correct policy index for the logged path.
The FakeSwift helper no longer looks in registered object responses
for an X-Backend-Storage-Policy-Index header to update an object
request. Real Swift object responses do not have an
X-Backend-Storage-Policy-Index header. By default, FakeSwift will now
update *all* object requests with an X-Backend-Storage-Policy-Index as
follows:
 - If a matching container HEAD response has been registered then
 any X-Backend-Storage-Policy-Index found with that is used.
 - Otherwise the default policy index is used.
Furthermore, FakeSwift now adds the X-Backend-Storage-Policy-Index
header to the request *after* the request has been captured. Tests
using FakeSwift.calls_wth_headers() to make assertions about captured
headers no longer need to make allowance for the header that FakeSwift
added.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Closes-Bug: #2038459
[1] Related-Change: I5fe5ab31d6b2d9f7b6ecb3bfa246433a78e54808
[2] Related-Change: I40b252446b3a1294a5ca8b531f224ce9c16f9aba
Change-Id: I2793e335a08ad373c49cbbe6759d4e97cc420867

Change-Id: I8dac31e302031d4318890bb87bf5a26889371766

This patch reorganizes the SLO read response handling. The main goal
was to push the response header replacement for both GET/HEAD SLO and
multipart-manifest=get paths all into a common return path. A new
RespAttrs primitive is used to carry around some metadata details from
requests made in SLO. The authors hope these changes make the code more
easily readable and easier to modify.
Drive-By: add new "friendly_close" function in common.utils so we can
drain empty/error responses more confidently (and use it in swob and
request_helpers).
Drive-By: the tests added in the Related-Change discovered a 500 on
If-[Un]Modified-Since conditional GET requests - it probably wasn't
important, but this refactor fixed it on accident as a side effect.
Closes-Bug: #2040178
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Ashwin Nair <nairashwin952013@gmail.com>
Related-Change-Id: I54094f3d2098f56b755ec19cc9315d06a6ca8b15
Change-Id: Idc84e70539fc7480b6ecb86e2f0da904baf2c727