openstack/swift - swift - OpenDev: Free Software Needs Free Tools

PasteDeploy version 3.0 (2022年10月16日) dropped support for python2 (as
well as <3.7), which causes our py2 tests to fail. So cap the version
here at 2.1.1, the last which is compatible with our tests.
Even doing this doesn't stop pip install swift pulling in a newer
PasteDeploy in the
tools/playbooks/saio_single_node_setup/make_rings.yaml playbook
(causing the probes test on CentOS-7 to fail); so handle CentOS 7
explicitly.
Change-Id: If69ae0f8eac8fe8ff7d5e4f4f1bff6d0ea9e7a8b
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

ansible_ssh_user has been deprecated since Ansible 2.0 [1]
[1] https://github.com/ansible/ansible/blob/stable-2.0/CHANGELOG.md
Change-Id: If1ef202b41af124637091a3527320b80734948fe

Change-Id: I70dec389c1ac608a9d2767d162bd5edc315e564b

Now that OpenStack writ large has dropped support for py36,
we're going to have to manage our constraints ourselves
more, just like we did for py27.
Change-Id: I0973be6692e3a9b871eb61cbf759e13644a25107

Change-Id: I7eed34ab19a97b2c561e4aac8c95a8a5e1689266

As part of that, the ceph test runner needed up-rev'ing to run under
py3. As a result, the known-failures shifted.
Trim the on-demand rolling upgrade jobs list -- now that it's running
py3, we only expect it to pass for train and beyond.
Also, pin smmap version on py2 -- otherwise, the remaining experimental
jobs running on centos-7 fail.
Change-Id: Ibe46aecf0f4461be59eb206bfe9063cc1bfff706

Change-Id: I49d8480fd0c775249d8b586f9dea026448b8910d

Although these errors are not important, they affect the code specification.
Change-Id: Ibc0fe151ccd5a33c552bb72a3cd200642a0d8131

We've recently started seeing some failures in the gate related to these
projects, and they have final py2-supporting versions.
Change-Id: If81fc352c8b2b1f03f3fa7b79c56dfcf981ced70

Some selenium webdrivers (e.g. Chrome, Firefox) support a headless
option so we can expose that as an option for the test runner too.
Use this in an attempt to fix "Error: cannot open display: :99" errors
seen in the gate.
Related-Bug: #1918864
Change-Id: I2a549ce829eb0bc38406575582202e1d8dd1a0e2

Drive-bys:
* Only copy outputs when there are outputs to copy
* Print tracebacks when there's an issue running a selenium driver
Change-Id: I0807af4525a13a30baf27ada40eeabe311b44296

This adds support for presigned GET URLs, at least.
Note that there is no support yet for preflight requests, so a whole
bunch of other CORS stuff *doesn't* work (yet). This was just an easy
first step.
Change-Id: I43150a630a2a7620099e6bfecaed3bbe958ba423

If you've got selenium installed (and working), the whole thing can be
automated pretty well; run main.py, wait while some windows pop up (or
use xvfb-run to run things on a virtual display), then check out what
tests were run on which browsers and whether any of them failed. Exit
code is the number of failed tests.
Includes tests against:
- Account
- Containers, with various ACLs/CORS settings
- Objects
- /info
- SLOs
- DLOs
- Symlinks
Include a gate job that runs the tests in firefox.
Areas for future work:
- Install chromium and chromedriver in the gate; tests should
 automatically pick up on the fact that it's available
- Capture the web browser's console logs, too, so we can get
 more info when things go wrong
Change-Id: Ic1d3a062419f1133c6e2f00a598867d567358c9f

On every sharder cycle up update in progress recon stats for each sharding
container. However, we tend to not run it one final time once sharding
is complete because the DB state is changed to SHARDED and therefore the
in_progress stats never get their final update.
For those collecting this data to monitor, this makes sharding/cleaving shards
never complete.
This patch, adds a new option `recon_shared_timeout` which will now
allow sharded containers to be processed by `_record_sharding_progress()`
after they've finished sharding for an amount of time.
Change-Id: I5fa39d41f9cd3b211e45d2012fd709f4135f595e

Change-Id: I952690558486a1d981985eae0fe9d7e07ff30e17

Long term, though, we should look at moving this in-tree if we really care about it.
Change-Id: I0a25a6e395e5cf2bb39fa5b349418384eb513963

Change-Id: Iae86d8838854023010686d3d4bed3befe6160ca5

That way we avoid POST_FAILUREs when the real problem was in run.
Change-Id: I9eb84d1c794d58f0af3b7d78d3bc4660c1823dc8

...and bump up their timeout, since that seems more likely to happen if
we have to retry.
Change-Id: Ie05521f6cd146234dc5615c96ad19681b43e9110

This should help make tests that rely on up-to-date listings more reliable.
Change-Id: Ib888c84fa629cc78f48a8251eda98c0fa51242c0

Change-Id: If14f53993e6a0554345b327415812423b6b57434
Related-Change: If5dd79f926fa51a58b3a732b212b484a7e9f00db

We've seen failures with probe tests lately where dnspython 2.0.0 is
getting installed even though it doesn't support py2 anymore. I think
using latest pip should be better about noticing that and installing the
last 1.x release intead?
Change-Id: I6eda54ccd2792effadb334ce9324887132b62b6f

Note that existing SAIOs with 60xx ports should still work fine.
Change-Id: If5dd79f926fa51a58b3a732b212b484a7e9f00db
Related-Change: Ie1c778b159792c8e259e2a54cb86051686ac9d18

Hopefully this will fix the currently-broken probe test gate?
Depends-On: https://review.opendev.org/#/c/736070/
Change-Id: Ib652534b35236fdb6bcab131c7dc08a079bf72f6

crudini seems to have trouble on py3 -- still not sure *why* it's using
py3 for the losf job, though...
Change-Id: Id98055994c8d59e561372417c9eb4aec969afc6a

Change-Id: I0bdd3a1d044f8f99873f6270ca821862bb994d72

Depends-On: https://review.opendev.org/#/c/571021/
Change-Id: I3ac3288cd61b745ce7dbf2bded8eade026d0418f

* Stop specifying logbufs=8; that's the default
* Stop including nodiratime with noatime; the latter implies the former
Nothing wrong with being explicit, I suppose, but may as well keep the
mount options to what we can easily explain: we want noatime because
Swift does not use atime, so we don't want to lose any performance to
tracking atime.
Change-Id: I1e52b4368ad7eb375964eee5132bc50297536355

Change-Id: I3cbbcd2ea9ced0923bee4a6b0783e4cf5e82e95b

Change-Id: I3833843cd8d23c498a2afe6c68a3f0afe26343c0

Also, correct the version that we check out when upgrading from stable
branches.
Change-Id: Ie733bc50466c66d6e6eb5c6bd42e42a05ef88798

Change-Id: If93325f2651a02f98f9d480c10bf7b849cc9617e

XFS no longer supports nobarrier mount option.
It has been deprecated for a long time[1] and removed in
recent kernel versions resulting in an error when trying to
mount: "kernel: XFS (loop0): unknown mount option [nobarrier]."
[1] - https://patchwork.kernel.org/patch/9486549/
Change-Id: Iaa9208fb20545ae9ac990f0e180899108d983123

Give storage nodes more time to complete requests for multi-node upgrade
and probetests.
Also slightly decouple probetests from default configs.
Change-Id: I334ef517d833916a3b7be3151a812d4f9c66a6e1

Previously, we'd install development versions of Swift as root, causing
later tox runs as zuul to fail on a permissions error because the
generated egg-info (at least) was locked down.
Change-Id: Ia688790f8b23ed1cf76947b5809c208df5dee8bb

...and drop openstack-cover-jobs since it is no longer necessary.
Change-Id: Iad12e64f41bbdd8b8e005b0fb14a7e2108fa3bcf

Note that eventlet 0.22.0+ closes connections between requests when
it stops accepting connections.
Partial-Bug: #1792615
Change-Id: Ia8d9ab95e2aad40e8d797acc3423a917e809ffdb

This new test job adds multinode testing to swift.
It currently configures 5 nodes: test-runner, proxy,
account, container and object. It starts by checkout out
the latest release and then 'upgrades' the storage nodes
to the current patch under test. That means that it will
the test-runner and the proxy are still running the latest
release.
Future work would include a new set of tests where it will
add objects to the cluster prior to upgrading and then try
to read those objects after the upgrade. It should also be
noted that the 'upgrade' currently only means a new
'git checkout' the configuration files are not being updated.
A future patch should fix that too.
Change-Id: If8e09a082fc024257a98cf332de9a36a18d2adc5

Previously we'd use two users, one admin and one unprivileged.
Ceph's s3-tests, however, assume that both users should have access to
create buckets. Further, there are different errors that may be returned
depending on whether you are the *bucket* owner or not when using
s3_acl. So now we've got:
 test:tester1 (admin)
 test:tester2 (also admin)
 test:tester3 (unprivileged)
Change-Id: I0b67c53de3bcadc2c656d86131fca5f2c3114f14

This patch added new non-voting gate job to check the s3api compatibility
via swiftstack/s3compat tool that shows the ratio of compatible S3 APIs
in the gate result for each patch. This is very useful to check the possibility
if the new incoming patch breaks S3 API compatibility unexpectedly.
Originally swift3 has this kind of the gate job but we missed the staff
while migrating from swift3 into swift upstream repo so this is the porting of that.
Note that currently the job is against to only tempauth because we don't have
custom gate jobs using keystone environment other than dsvm.
Change-Id: I6f30f74678ad35479da237361bee48c46c0ecc49

Note that there's no swift-dsvm-functional-identity-v3-only because
as of the Queens release, that's the *only* way Keystone gets deployed.
Change-Id: I49d963b98f3df21fea0db24c83553ef873ad73c8

Latest version of CentOS should not need this anymore
Change-Id: Ief83c39420cfb4509f16d4e357d704025b116134

Change-Id: Ib08c2c4c685eb30a4c804ece0b667575c04d2ca0