openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Change-Id: I9044e49ff30b092a97b75fa7fa133016d17cde37

Depends-On: https://review.opendev.org/c/openstack/requirements/+/941708
Change-Id: I52f40bf0468236dc7ba78e08f2f9aea9336faae9

* Remove py2 gate jobs.
* Build non-universal, py3-only wheels.
* Specify minimum python version in package metadata.
* Clean up requirements/constraints/bindep (a little, anyway).
Change-Id: I53153c4fde043e964e1daa7bbf2089e0471dede2

Change-Id: I9473fad7c46ac03bbc71328c17e988af9d21386c

The main motivation here is that mock.call becomes a namedtuple and you
can say `m.call_args_list[0].args` instead of `m.call_args_list[0][0]`
Change-Id: Ibb1a64ef0bfdebf06d26636cdb6ea191c10705f7

Last time we did this was nearly 4 years ago; drag ourselves into
something approaching the present. Address a few new pyflakes issues
that seem reasonable to enforce:
 E275 missing whitespace after keyword
 E231 missing whitespace after ','
 E721 do not compare types, for exact checks use `is` / `is not`,
 for instance checks use `isinstance()`
Main motivator is that the old hacking kept us on an old version
of flake8 et al., which no longer work with newer Pythons.
Change-Id: I54b46349fabb9776dcadc6def1cfb961c123aaa0

nose has not seen active development for many years now. With py310, we
can no longer use it due to import errors.
Also update lower contraints
Closes-Bug: #1993531
Change-Id: I215ba0d4654c9c637c3b97953d8659ac80892db8

Change-Id: Ic8159ec505a5b5ca241729e157195a60c73b8cd8

In the process, move from using os-testr to stestr.
Change-Id: Ib6651aab8b009605bf0855ec217003936c0fae6e

* Drop osc from test-requirements (and lower-constraints)
 I'm not clear on where/how we use it; I think it was a hold-over from
 swift3?
* Pin python-keystoneclient in our py2-constraints
 Something sees to have changed with the pip resolver that means it
 keeps trying to install a newer, py3-only version for our py2 jobs.
Change-Id: Ie37ac077517e1ece5fa6bf163d1ab5e316ced509

The repo is Python using both Python 2 and 3 now, so update hacking to
version 2.0 which supports Python 2 and 3. Note that latest hacking
release 3.0 only supports version 3.
Fix problems found.
Remove hacking and friends from lower-constraints, they are not needed
for installation.
Change-Id: I9bd913ee1b32ba1566c420973723296766d1812f

Reno is only needed for building release notes. Reno has now dropped py2
support, so attempts to install all requirements for py27 jobs fail with
the latest reno release.
Reno is already correctly listed in doc/requirements.txt, so this just
removes the unnecessary entry in test-requirements.txt.
Need-by: https://review.opendev.org/705994
Change-Id: Id1c54915ebe43f2b46f611a95109d2ab0d02d34d
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

Adds the scaffolding required for tests to use boto3 and converts the
test_bucket.py tests to the new interface. Follow on patches will
convert the other tests to use the boto3 library.
Notable changes: we no longer try to reach for the equivalent of
`boto.make_request()` and instead rely on the boto3/botocore event
system to mutate requests as necessary (or to disable pre-flight
validators).
Partial-Bug: 1557260
Change-Id: I3d77ef4a6b878c49ebfa0c8b8647d7199d87601e

I just took these from the current versions at
https://github.com/openstack/requirements/blob/master/lower-constraints.txt
so we can get the the change we *actually* want, which is to black-list
a known-bad version of eventlet.
Change-Id: I9842c9975821bda5c7d8bf2fc214480c0c0a5e96

See https://review.openstack.org/#/c/571007/ and
https://review.openstack.org/#/c/570999/ for context.
Change-Id: Ib6045de3cb9097ed7e6d48646109f35612cbbc74

This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
 deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>

For compliance with the Project Testing Interface as described in:
https://governance.openstack.org/tc/reference/project-testing-interface.html
For more details information, please refer to:
http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html
Co-Authored-By: Nguyen Hai <nguyentrihai93@gmail.com>
Change-Id: I26dc41c7df57bf79db531c6e67e148e01c17e992

As part of the docs migration work[0] for Pike we need to switch to use
the openstackdocstheme.
Fix one display problem with wrong section markup in the index file.
[0]https://review.openstack.org/#/c/472275/
Change-Id: Ide31218a7f37ba5d959de99cab48fc6513bf426f

...and fix up the one warning that's crept in.
Change-Id: I3985d027f0ac2119ceaeb4daba5964f937de6cea

This should give the necessary tooling to set up the right
links in the right places so Swift has release notes listed
alongside other projects.
Change-Id: I4e5f1ce1fcfbb2943036c821a24a0b4a3a2d9fc8

The testing requirements should not cause extra packaging
work for deployers (since these aren't required to deploy).
This patch brings it in line with the latest global requirements,
except for python-swiftclient itself (which I kept with no version
specifier). By being in line with global requirements, this is a
more accurate list with respect to how things are actually tested
in the gate.
Change-Id: Iddc92d9bb135144d38c3e5aec2e95df5787e9b5b

This commit updates the Swift Bandit file to the new style
introduced in Bandit 1.0. In response to the struggle with
getting a Bandit config file working and kept up to date we
introduced a simplified version in Bandit 1.0.
This commit updates Swift's bandit.yaml to use the new version.
Change-Id: Ida5dd08f4ea72a377346f2159caeb2f3741d4980

This adds swift-specific contents from the OpenStack Installation Guide
in the swift repo per [1]. A separate change will remove the swift contents
from the OpenStack Installation Guide for Newton per [2].
The swift install-guide structure is based on the Install Guide
Cookiecutter [3].
Also adds tox.ini environment for install-guide and adds
openstackdocs-theme to test-requirements.txt.
[1] http://specs.openstack.org/openstack/docs-specs/specs/newton/project-specific-installguides.html
[2] http://specs.openstack.org/openstack/docs-specs/specs/newton/installguide.html
[3] http://git.openstack.org/cgit/openstack/installguide-cookiecutter/
Change-Id: I59b92eebaf5acc657b97bcf10d9ff2cf2db05885
Partially-Implements: blueprint projectspecificinstallguides
Depends-On: Ifebc65b188c4f2ba35b61c0deae5ec24401df7f9

This brings in RST plus YAML files, migrated from the source for [0].
The migration explanation is found on the openstack-dev mailing list [1].
Project instruction is in the OpenStack Documentation Contributor Guide [2].
A patch for publishing this source is in [3].
The conf.py and the tox environment are standard across other projects.
[0]http://developer.openstack.org/api-ref-objectstorage-v1.html
[1]http://lists.openstack.org/pipermail/openstack-dev/2016-May/093765.html
[2]http://docs.openstack.org/contributor-guide/api-guides.html
[3]https://review.openstack.org/#/c/313015/
Change-Id: Ifebc65b188c4f2ba35b61c0deae5ec24401df7f9

The api-ref document needs a newer sphinx version, allow a 1.2 Sphinx
version to be used - like it's used in global-requirements.txt.
Change-Id: I9183cc56753fbe7e41206c6a9081899df5c3919a
Needed-By: Ifebc65b188c4f2ba35b61c0deae5ec24401df7f9

Defcore uses Tempest, which uses Test Repository.
This change makes it easier for Defcore to pull functional
tests from Swift and run them. Additionally, using testr
allows tests to be run in parallel.
Concurrency set to 1 for now, >1 causes failures for
reasons that are still TBD.
With switch to ostestr all the server logs are being sent to stdout
which makes it completely unreadable. Suppressing the logs by default
now with a flag to enable it if desired.
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Robert Collins <rbtcollins@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5
Related-Bug: 1177924

This change adds a basic bandit config for Swift. It can be invoked
by running the tox environment for bandit;
 tox -e bandit
This is an initial step for using bandit with Swift
and it should be revisited to improve the testing as more is learned
about the specific needs of the Swift code base.As per now some tests
are excluded as they were used on purpose.
https://wiki.openstack.org/wiki/Security/Projects/Bandit
Implements: blueprint swift-bandit
Change-Id: I621be9a68ae9311f3a6eadd1636b05e646260cf2

Replace the whitelist of flake8 checks (select) with a blacklist
(ignore). It makes possible to disable a single check, which was not
possible before. This new approach permits to enable new tests more
easily and see which checks are currently disabled.
Only new checks are disabled, this change doesn't run less checks than
before. Currently, many checks are disabled, but following changes will
fix these new warnings.
When the "select" option of flake8 is used in tox.ini, the "ignore"
option is ignored. It is not possible to use "select" and "ignore"
options at the same time. See the code in pep8 (flake8 is based on
pep8):
d48eef0710/pep8.py (L1808)
The purpose of flake8 is to enhance the coding style, but also to help
to detect common bugs. Blacklisting checks permits to detect new bugs
when upgrading the tools. For example, new checks were recently added to
help porting code to Python 3.
This change permits to use pbr 1.0 and newer which is required to
support Python 3. Previously, hacking 0.8.1 was used, but hacking 0.8 is
incompatible with pbr 1.0 and newer (hacking 0.8 requires pbr<1.0).
pbr 1.0 is required to use environment markers on requirements to
install dnspython on Python 2 and dnspython3 on Python 3.
Change-Id: I41d63b9467083d7606ad18aaa398ca7738b27fae

When functional tests are run in tox and an exception is raised when connecting
to Swift (for example: Swift not running, missing python-keystoneclient package
used by python-swiftclient) 0 tests are executed, but tox returns a success.
An exception is raised during tests, caused by a missing python-keystoneclient
in python-swiftclient. Instead of adding python-keystoneclient as a dependency
in python-swiftclient the package is added to the test-requirements.txt in Swift
itself. Note that adding python-keystoneclient to the test-requirements in
python-swiftclient is not sufficient (it's already in there).
The exception in setup_package() is catched by the openstack.nose_plugin, thus
disabling this plugin for now as well.
Also fixing two test errors seen on the gate regarding the tempurl middleware.
There was also an update to tox, environment variables were no longer passed
with versions >= 2.0 (http://tox.readthedocs.org/en/latest/changelog.html).
Swift test environment variables have been added to the passenv to re-enable the
former behavior, as well as environment variables required to pass proxy
settings.
This also led to skipped tempauth tests, and together with the missing
python-keystoneclient no tests were executed.
Related-Bug: 1461440
Related-Bug: 1455102
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: Ideea071017d04912c60ed0bc76532adbb446c31d

Change-Id: I64ae9191863564e278a35d42ec9cd743a233028e
Closes-Bug: 1365061

Currently the theme used by the swift developer docs are out of
date, it should be using oslosphinx, to provide a similar look
and feel between all openstack related projects.
Change-Id: Id7c226cdc13c6c4f3b5082b1ef4dfe09966b21ec

mock >= 1.0 changed the behavior for iterable sideeffects. Any
exceptions in an iterable side_effect will now be raised instead of
returned.
With mock < 1.0 tests are failing, because test for the
internal client (TestSimpleClient.test_get_with_retries) use an
iterable sideeffect that raise an URLError on the first request
and pass on the second request.
Change-Id: I568b68c33ff599d7a4f590e2a0eb8caeee4b4d4e

The flake8 command (when run from the command line or via vim-flake8) seems to
bypass the the normal pep8 [W]arning and [E]rror as well as py[F]lake checks
if the project level tox.ini includes a select, but doesn't include them in
it's select option (tested on flake8 v2.0 and v2.1). Somehow if hacking is
installed these checks always run even if they're not explicitly listed in the
select list - but adding them so flake8 works on our project even without
hacking doesn't seem to cause any problems.
Also The [flake8] section's "select" option in tox.ini doesn't seem to run all
checks when presented with the space-comma seperated list. I had to pull out
the bare except check (H201), because our hacking fix for the "H201 # noqa"
hasn't been released to pypi yet (maybe hacking version 0.9.0?).
As far as I know the Swift core development team is fully onboard with pep8,
and the flake8 checks are mostly like "unused import" stuff - so I *think*
this change is more about making the tool test what we want it too and not
changing or adding any new requirements or expectations.
Change-Id: Ia911c63745100cce182d5c5eb67b77e31c1e9b18

Removes the requirement for swiftclient in swift-dispersion-report
and swift-dispersion-populate. To prevent a dependency on
keystoneclient and to avoid reinventing the wheel with an internal
keystoneclient, authentication with keystone is only supported if
swiftclient is available. If not, only auth v1 is supported.
The dependency in swift/container/sync.py has also been removed.
Implements: blueprint remove-swiftclient-dependency
Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7

They should be determined by the hacking dependency
implicitely.
Change-Id: I8805f7b0ef9f2b1f59be342e9c6d032330707688

Sync the global requirements to pin sphinx. This addresses an issue
where Sphinx 1.2 is not building documents correctly and causing
check/gate to fail.
We also had to adjust the pip command used.
Change-Id: I8894c0199db845e90e5086a7c0e6bb7c7a26b5a0

Instead of blacklisting Hacking globally,
only blacklist those that currently occur frequently
(for a later followup patch), and fix the rest. In
detail:
H101 Use TODO(NAME)
H201 no 'except:' at least use 'except Exception:'
H231 octal number 022 should be written as 0o22
H401 docstring should not start with a space
H701 Empty localization string
Change-Id: Ib3b3d56b68d1cf15d3b67ac9749fcbdb876dc52a

Rename tools/pip-requires to requirements.txt and tools/test-requires
to test-requirements.txt. These are standard files, and tools in the
general world are growing intelligence about them.
Change-Id: Ib3e50a811868e2969923d978ee00c4f92682aa1c
Fixes: bug #1179008