openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Following up on patch 417209.
Fast post is now the default mode, so there's
no more need for a fast-post specific testenv.
Change-Id: I6faf8fcfa0a1d96aaf0f5e0ad2106b2b416da22f
Signed-off-by: Thiago da Silva <thiago@redhat.com>

Additionally, emit deprecation warnings when running POST-as-COPY
Change-Id: I11324e711057f7332577fd38f9bff82bdc6aac90

Currently swift ignores a lot of the Hacking style guide. This patch
enables the H401 and H403 checks and fixes any violations. With this
we can get a little closer to following the OpenStack style guidelines.
Change-Id: I5109a052f2ceb2e6a9a174cded62f4231156d39b

The bandit gate is passing, let's make it voting.
Since bandit is a security linter, add it to the linter target - called
pep8 - and run it there. This will save the CI systems spawning an extra
test machine for this.
Needed-By: I90e5401ee8e46a2a153adb62d0f99a8a88a686b8
Change-Id: I75eeefa2441466da60b9b206769a2e64e9c1e7a9

Change default py3x to py35 in tox.ini to support python3.5,
and fix the error when run tox test for Swfit.
Closes-Bug: #1649773
Change-Id: I8e26183ffe270518503da8b5aa394e99ae4cd8c6

Fixes: NameError: name 'file' is not defined
file() is removed in python3.x using open()
 will solve it.
Fixes: AttributeError: 'generator' object has no
attribute 'next'
Python3 it is __next__()
Change-Id: Ib2cbdfbbedb86f6e01239d2ee991677727b0548b

Installed my workstation with Ubuntu Xenial and then hit a few
snags running tox.
python3-dev is needed to build the python3 envs. Did not actually
test python3-devel (rpm) as I don't have such a system.
While python2.7 is the default, tox (on Xenial) itself uses
python3.5, hence 3.5 becomes the basepython and a number
of envs don't work.
Driveby, removed warning about /bin/echo in py3pep8
Change-Id: Ie880246dd757570b8e444da0e643f3c50093bdc5

Follow up to [1]. The --allow-insecure switch is no longer
required for netifaces.
[1] Related-Change: I7c60be623b4340ee34ae1aa520f17b303348811d
Change-Id: If893cdeff68ae5b279ea7055c078b0c1247ae7f2

This shorten shebang in infra, because we are hitting 128 bytes limit.
Change-Id: I02477d81b836df71780942189d37d616944c4dce

This should give the necessary tooling to set up the right
links in the right places so Swift has release notes listed
alongside other projects.
Change-Id: I4e5f1ce1fcfbb2943036c821a24a0b4a3a2d9fc8

Decode header to latin1 on python3
encode header to utf-8 on python2.
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Change-Id: I10f205a05bb3a566e52a597d9315b3a8b8c14664

Change-Id: I7c60be623b4340ee34ae1aa520f17b303348811d

F812 list comprehension redefines <variable> from line ...
While the current violations were benign, this sort of code can easily
lead to subtle bugs. Seems worth checking, especially given how cheap it
is to bring existing code in line with it.
Change-Id: Ibdcf9f93b85a1f1411198001df6bdbfa8f92d114

Change-Id: Ia78fbb32aa6f45aa6578734dc43f3262e717bab8

This commit updates the Swift Bandit file to the new style
introduced in Bandit 1.0. In response to the struggle with
getting a Bandit config file working and kept up to date we
introduced a simplified version in Bandit 1.0.
This commit updates Swift's bandit.yaml to use the new version.
Change-Id: Ida5dd08f4ea72a377346f2159caeb2f3741d4980

* add NamedTemporaryFile() wrapper
* use byte strings for binary data in tests
* tox.ini: run test_splice on Python 3.4 and Python 3.5
Change-Id: I042d739fbf29433733bf0c2154749bc90b210416

Use tox substitution to avoid duplicating testenv:py34 commands in
testenv:py35, to not have to maintain the whitelist of Python 3 tests
in two different places.
Write also the list of tests in a newline to be able to more easily
add new unit tests.
Change-Id: I6e7f238f1c5d3fc9b6560918dcbb93e9dd8ec084

Avoids a warning when running tox -e api-ref
Change-Id: Ib02849075e6424f1db84499fd7500d7bb76dde67

Running functional tests in the in-process mode uses
the default value for the pipeline. This patch adds support
to specify the SWIFT_TEST_IN_PROCESS_CONF_LOADER variable
to point to a labeled function that changes the proxy
configuration for the functional test.
The patch also adds a new tox environment
func-in-process-encryption
which runs with the environment variable
SWIFT_TEST_IN_PROCESS_CONF_LOADER=encryption
The motivation for this change is to put support in place for an
upstream CI job that will functionally test using encryption
middleware in the pipeline. The gate job is proposed at:
https://review.openstack.org/#/c/348292/
Change-Id: I15c4b20f1d2be57ae21c69c614f6a9579145bee9

As much as anything, I'm just tired of seeing a bunch or piecemeal
fixes.
Note that we *need* to include
 from __future__ import print_function
in order to support things like
 print() # Would print "()" (the repr of an empty tuple) otherwise
 print(foo, end='') # Would SyntaxError
 print(bar, file=sys.stderr) # Would SyntaxError
Change-Id: I8fdf0740e292eb1ee785512d02e8c552781dcae1

Just having a `flake8 --filename=swift* bin` command is insufficient.
Change-Id: Ia2a5c364e52d9972d31e6b5e22366503894b720d

This adds swift-specific contents from the OpenStack Installation Guide
in the swift repo per [1]. A separate change will remove the swift contents
from the OpenStack Installation Guide for Newton per [2].
The swift install-guide structure is based on the Install Guide
Cookiecutter [3].
Also adds tox.ini environment for install-guide and adds
openstackdocs-theme to test-requirements.txt.
[1] http://specs.openstack.org/openstack/docs-specs/specs/newton/project-specific-installguides.html
[2] http://specs.openstack.org/openstack/docs-specs/specs/newton/installguide.html
[3] http://git.openstack.org/cgit/openstack/installguide-cookiecutter/
Change-Id: I59b92eebaf5acc657b97bcf10d9ff2cf2db05885
Partially-Implements: blueprint projectspecificinstallguides
Depends-On: Ifebc65b188c4f2ba35b61c0deae5ec24401df7f9

Very few of our tests can actually be run under py3. The ones that can
should still pass on py35, though.
Change-Id: Iaf9aaa296e3b21aa0ee513c479a50f3796787f32

This brings in RST plus YAML files, migrated from the source for [0].
The migration explanation is found on the openstack-dev mailing list [1].
Project instruction is in the OpenStack Documentation Contributor Guide [2].
A patch for publishing this source is in [3].
The conf.py and the tox environment are standard across other projects.
[0]http://developer.openstack.org/api-ref-objectstorage-v1.html
[1]http://lists.openstack.org/pipermail/openstack-dev/2016-May/093765.html
[2]http://docs.openstack.org/contributor-guide/api-guides.html
[3]https://review.openstack.org/#/c/313015/
Change-Id: Ifebc65b188c4f2ba35b61c0deae5ec24401df7f9

Change-Id: I4ce3068f79563e4d4296c6e1078bc12f0cf84c96
Related-Bug: 1559431

Add an other-requirements.txt file containing a cross-platform list of
dependencies needed for running included tox-based tests. Also include a
tox environment for convenience calling the bindep[*] utility to list
any missing system requirements.
Document bindep and other-requirements usage.
This change is self-testing.
For other-requirements.txt see also
http://docs.openstack.org/infra/manual/drivers.html#package-requirements
[*] http://docs.openstack.org/infra/bindep/
Change-Id: Iea6f5fecba3b7cb9f6dac7029c0f17fc31fc0e3c

Change-Id: I4322978aa20ee731391f7709bbd79dee140fc703

Change-Id: I2add370e4cfb55d1388e3a8b41f688a7f3f2c621

Delete python bytecode before every test run.
Because python creates pyc files during tox runs, certain
changes in the tree, like deletes of files, or switching
branches, can create spurious errors.
Closes-Bug: #1368661
Change-Id: Iedcb400fa3b0417f5bb8e943b17758fcfb4070c6

tox was taking python3 as it's base version which lead to
tox pep8 check failing on local env:
http://paste.openstack.org/show/486183/
Change-Id: Ia822ec5b5e12a1fc30b676f1e6d95e287b9b3368

Running functional tests in the in-process mode uses
the default value for proxy-server object_post_as_copy,
which is True. This patch adds support for an environment
variable to change this value to False so that the fast-post
mode is tested when running in-process functional tests.
The patch also adds a new tox environment func-in-process-fast-post
which forces in-process functional testing with
object_post_as_copy=False.
The motivation for this change, apart from enabling configurable
local testing, is to put support in place for an upstream CI job
that will actually functionally test the fast-post mode, which is
otherwise only covered by unit tests.
There are currently two gate jobs that run the functional tests:
- gate-swift-dsvm-functional runs the tests *twice* against a
 devstack swift service, once using tempauth and once using
 keystoneauth. The devstack swift service uses the default
 object_post_as_copy=True.
- gate-swift-tox-func runs the func tests in in-process mode
 which also uses tempauth and object_post_as_copy=True. This
 duplicates one of the config scenarios above.
With this change either the gate-swift-tox-func job or a new job
could run the functional tests using object_post_as_copy=False.
Change-Id: Ia37f6df1dc38e44ef7404fbf0a52f6fc22fae0c2

Defcore uses Tempest, which uses Test Repository.
This change makes it easier for Defcore to pull functional
tests from Swift and run them. Additionally, using testr
allows tests to be run in parallel.
Concurrency set to 1 for now, >1 causes failures for
reasons that are still TBD.
With switch to ostestr all the server logs are being sent to stdout
which makes it completely unreadable. Suppressing the logs by default
now with a flag to enable it if desired.
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Robert Collins <rbtcollins@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5
Related-Bug: 1177924

Caching is enabled by default from pip version 6.0
More info:
https://testrun.org/tox/latest/config.html#confval-downloadcache=path
https://pip.pypa.io/en/stable/reference/pip_install/#caching
Change-Id: I9451a0f0dee5c5a3c0ca0a52f58bd353602661a2

Add a py34 test environment to tox.ini using a whitelist
of tests which pass on Python 3. Currently, only test_exceptions.py
of test/unit/common/ is run on Python 3
Change-Id: I5c8a9fe7b9635f1acab8f401908975fc6fc58c7a

All usages of assertEquals and assertNotEquals are fixed now, so let's enable
the H234 check to avoid regressions in the future.
Change-Id: I2c2ccb3b268cf9eb11f2db045378ab125a02bc31

Now you can run
 $ tox -e pep8 path/to/file.py [path/to/file2.py [...]]
to run pep8 against just those files[1]. This is quite a bit faster
than a full pep8 run, and the faster feedback is nice when you're
fiddling with some formatting to placate pep8.
Of course, you can still run "tox -e pep8" to check the whole source
tree, just as before this commit.
[1] It'll still run against bin/swift* as well, but that's still a lot
faster than running against all our .py files.
Change-Id: I81b4363fb95a34ff0f5c346b2b24f2047154f502