openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Previously FakeSwift would raise a KeyError for a call that had no
matching registered response *before* capturing the call. This
prevents tests reliably asserting that only expected calls have been
made. In particular, if the code being tested handles the KeyError
gracefully then it was possible to write quite reasonable test
assertions that passed despite unexpected calls being made.
Tests in test/unit/container/test_reconciler.py seem to rely on this
behaviour, so this patch adds a capture_unexpected_calls option for
FakeSwift which defauts to True. This allows the reconciler tests to
opt out of the new stricter call capturing.
Change-Id: Idc6b6b5a2b665538e861700f5d0996fc39368f5b

Apparently errno.EUCLEAN is not available on OS X, which can complicate
running unit tests.
Change-Id: Iaa3d7756949b4a67d4afe8a53b242ed9f41e9374

Of the last 100 successful runs, the average runtime has been ~4800s -- we
still get too many timeouts.
Change-Id: Ic5ba02198d51f938605787281488efccb25c0242

We've seen significant delays in the arm pipeline lately, with some changes
waiting nine or more hours for jobs to run. We should do what we can to
alleviate demand for arm64 nodes; I'm not sure any other projects have four
arm jobs on every change.
Change-Id: I7929ec3b856d90201c4cc4bf201e634446d3ad88

If py38 and py312 pass, it is reasonable to assume py39, py310, and py311
would, too.
Change-Id: Iefbfafe7c1005c9843a97d54c05398d4f0a3d2b0

We introduced this a while back, but forgot to add it then.
Related-Change: Ia13ee5da3d1b5c536eccaadc7a6fdcd997374443
Change-Id: Ib65ddf50d7f5c3e27475626000943eb18e65c73a

This is a path, not a request. See the parent class, Putter, and
all the callers of MIMEPutter.connect().
Related-Change: I12a6e41150f90de746ce03623032b83ed1987ee1
Change-Id: I42c7dcf2b4a3bf8bd023fdc1ce7d454ce01885a0

When Expirer is iterating task containers and doing listings, it's
possible one of the container nodes hosting the task container may
become overloaded (or have a really low backend ratelimit set).
Object-expirer should expect UnexpectedResponse and continue to try
and list the task objects in the next container.
And if the task container doesn't exist, expirer should not try to
delete the non-existent containers, before continue to work on
the next container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Id1966fa22725a02471e2d7c5a42fb243b1cfcf6a

Previously the object-expirer would not respect an internal_client_conf_path option if its config was loaded from a "legacy" config filepath (e.g. **/object-expirer.conf). Legacy object-expirer config expected config sections for an internal-client to be included in the same file. However, "modern" object-expirer config files (e.g. **/object-server.conf) expect internal-client config to be loaded from a separate file specified by internal_client_conf_path and defaulting to 'internal_client_conf_path'.
"Legacy" expirer config is still used in production clusters but it is useful to use a shared internal-client config. This patch therefore makes the internal_client_conf_path option always be respected regardless of the path of the object-expirer conf file. If 'internal_client_conf_path' is not specified, "modern" config will continue to use the default '/etc/swift/internal-client.conf', but "legacy" config will default to the path of the expirer conf file (i.e. the same as the previous behavior).
Related-Change: Ib21568f9b9d8547da87a99d65ae73a550e9c3230
Change-Id: I24ec702cd2ed074ca9df084cefc896418cece394

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
Change-Id: I0e73e5548efbf93e433507b63fb41e34148b81a2

The datetime.utcfromtimestamp() is deprecated in Python 3.12.
Replace datetime.utcfromtimestamp() with datetime.fromtimestamp().
Change-Id: I01d6b94de394413aa13a045ab2c36504e65a6f5a
Signed-off-by: Takashi Natsume <takanattie@gmail.com>

py2 http responses don't have a 'headers' attribute, so don't try to
use it.
Unfortunately the unit test infrastructure's FakeConn class does
have a 'headers' attribute (whose value doesn't match the result of
calling its 'getheaders()' method!), so this bug was not caught by
tests in the Related-Change.
Related-Change: I96f28ab0b2b5f9374c399e8905ee240e7b093f8b
Change-Id: I2cd820280b8c69cafc5730183903c9d379d8dde5

Previously the in-process functional test setup attempted to insert
etag-quoter in the proxy pipeline by replacing the substring 'cache
listing_formats' with 'cache etag-quoter listing_formats'. However,
the pipeline had already been modified to have 'cache domain-remap
listing_formats', so the etag-quoter replacement failed silently.
This patch modifies the etag-quoter replacement to match just
'cache'. Some helper functions are also introduced to make the
pipeline modification more uniform and to consistently verify the
intended modifications have been achieved, or otherwise raise an
exception.
Change-Id: I640837a4da985bcbe48f9ae5eeb56385b11034eb

A print statement, presumably for debugging, crept in with the
Related-Change.
Change-Id: I0cccb960b51446a3aa58be95c8c9d06cc6b6eada
Related-Change: I69974cc8a39731c980b54137b799a36b2e63a44a

Some s3api functional tests were setting boto logging level to DEBUG,
which results in a very large amount of logging output from the test
runner, including large request bodies. This makes it extremely hard
to inspect test output.
This patch makes the quiet_boto_logging context manager revert the
logger level to its original value rather than indiscriminately
leaving it set to DEBUG.
Change-Id: I1dd9603adf9a19e89da5a461d3c6810a3432ae46

Maintaining testing requirements for our py36 gate jobs
since we have been remiss in updating them regularly. This
will help us test swift which operators might be running.
Change-Id: I58fa780460f7ac84a02d508fd95770bada991326

Related-Bug: #2080600
Change-Id: Id5b0bf6efcb2c16e10334d2edb268194fa1ec008

Add file to the reno documentation build to show release notes for
stable/2024.2.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2024.2.
Sem-Ver: feature
Change-Id: I9a384cc42eecc5ffd6f24fa58653a672190017df

We used to have a fairly bimodal distribution of successful ARM test
times: roughly 2/3 of the time, tests would take 800-1800s (~15-30min),
and 1/3 of the time they would take more like 2000-5400s (~30-90min).
Starting around August, the bimodal distribution is gone, with only the
longer expected test times remaining. Timeouts have increased, leaving
the ARM jobs frequently failing spuriously. Increase the timeouts to
get better signal from the tests.
Change-Id: I56165dc6d877bd9e56d7160dd28c6c60f339d730

Change-Id: I374abcdd93664322cb440cd083f1ea4edf04d730

The RelatedChange introduced a _make_internal_client() method with an
unused argument 'is_legacy_conf'. This patch completes the original
intention i.e. for the selection of internal client config path to
also be moved to the new method and use the 'is_legacy_conf' arg.
Change-Id: I5075cb446a15edc7f47e83f6aa038c626bd1dd82
RelatedChange: Ia6e1e6a8b58a8476fa16a3c7d45e620c6d7f88e4

If the nodepool provider uses NAT'd floating IPs, the public IP won't
actually be available locally for binding. This has caused a bunch of
failures on raxflex lately.
Change-Id: I5669a814377242e9939a09a42bb36642358b85b3

The account quota middleware was missing tests for the new quota count
API and it was also only testing the legacy API for quota bytes.
This adds the new APIs quota bytes and quota count to the functional
tests.
Change-Id: I6ebb19c90dfb1cfbe0535ed3860f2319e5153c05

We don't need to go counting slashes or catching IndexErrors;
we've already done all the path-parsing we need to do.
This also makes it clear that stat_type can never be None.
Change-Id: I25f91b1943b91c7429219c3b5a4280abe9bef5b3

Note that you can still have a "//" in the path with rest_with_last, though.
Change-Id: I171afcd67b162634189b752ff92a4f43484bc12a

Related-Change: I526bbcc59c9eb5923c3784d5d06bc38998cb48db
Change-Id: I820339934f895557ebcde4bb32168b0cfc494948

Account quota metadata such as quota_bytes and quota_count are stored in
the `meta` namespace which users have access to. However, this should be
only available to reseller admins.
This patch adds support for writing the quota metadata to `sysmeta`
namespace, so that it is not accessible by users. The account policy
quota is already using `sysmeta` and has the namespace
`X-Account-Quota-*`, so we are following this pattern.
If present, `X-Account-Quota-Bytes` is always preferred. However, in
order to maintain backwards compatibility, `X-Account-Meta-Quota-Bytes`
will still be honoured if it exists and `X-Account-Quota-Bytes` is not
present.
This also adds some new "legacy" tests to validate backwards
compatibility.
Co-authored-by: Azmain Adib <adib1905@gmail.com>
Co-authored-by: Daanish Khan <daanish1337@gmail.com>
Co-authored-by: Mohammed Al-Jawaheri <mjawaheri02@gmail.com>
Co-authored-by: Nada El-Mestkawy <nadamaged05@gmail.com>
Co-authored-by: Tra Bui <trabui.0517@gmail.com>
Co-authored-by: Chris Smart <distroguy@gmail.com>
Change-Id: Icf7b26023ab5b84136ceaa103fa2797534320f1a

Previously, this would reduce the account's quota to zero, which seems
like the opposite of what the operator intended.
Now, remove the quota, similar to sending an empty quota header.
Change-Id: Ic28752d835e0b970f2baa4e68cbfcde4f500b3d4

Change-Id: Ied0ff6bea7e054fad3fe9579c85d9ae5c9c0b255

- SHA256 mismatches should trip XAmzContentSHA256Mismatch errors,
 not BadDigest. This should include ClientComputedContentSHA256 and
 S3ComputedContentSHA256 elements.
- BadDigest responses should include ExpectedDigest elements.
- Fix a typo in InvalidDigest error message.
- Requests with a v4 authorization header require a sha256 header,
 rejecting with InvalidRequest on failure (and pretty darn early!).
- Requests with a v4 authorization header perform a
 looks-like-a-valid-sha256 check, rejecting with InvalidArgument
 on failure.
- Invalid SHA256 should take precedence over invalid MD5.
- v2-signed requests can still raise XAmzContentSHA256Mismatch errors
 (though they *don't* do the looks-like-a-valid-sha256 check).
- If provided, SHA256 should be used in calculating canonical request
 for v4 pre-signed URLs.
Change-Id: I06c2a16126886bab8807d704294b9809844be086