openstack/swift - swift - OpenDev: Free Software Needs Free Tools

Ubuntu 18.04 ships 3.6.5; may as well have a testenv handy for it.
Change-Id: Ibea05bc879685fab148cde1b19510e83f0639f5d

See https://review.openstack.org/#/c/571007/ and
https://review.openstack.org/#/c/570999/ for context.
Change-Id: Ib6045de3cb9097ed7e6d48646109f35612cbbc74

This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
 deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>

There were a couple of cleanups in swob as part of this. First,
status lines are always native str objects (as PEP 3333 wants), rather
than being encoded to bytes under py3. Second, _resp_body_property
now works (only) with bytestrings from the app iter.
In gatekeeper, we now deal with dict.items() returning an object of
type "dict_items" in py3, not a list. Also fixed a NameError caused by
py2's list comprehensions leaking variables to function scope where
py3's don't.
Change-Id: I6da8eceb91edb2b47aa345d61b825c7199a5569b

Also a drive by HASH_PATH_PREFIX/SUFFIX bytesting fix which was
breaking ring and relinker tests.
Change-Id: I1db8f102361770cfed2b12f689789d1e8c21c3e4

This lets you run something like
 $ tox -e py35 -- -s test.unit.common.test_swob
and just run the swob tests. This is handy if you've got one or two
test failures to debug or if you're trying to get another test file
passing under Python 3.
Also removed the old py34 environment. Nothing uses it.
Change-Id: I244b8903ef0a445abd028004bf02d6c581a8afe2

For compliance with the Project Testing Interface as described in:
https://governance.openstack.org/tc/reference/project-testing-interface.html
For more details information, please refer to:
http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html
Co-Authored-By: Nguyen Hai <nguyentrihai93@gmail.com>
Change-Id: I26dc41c7df57bf79db531c6e67e148e01c17e992

This patch ports the form_signature cli `swift-form-signature`
and it's corrisponding tests to py3.
In essence in Py3 the HMAC function expects binary strings.
Change-Id: I5dded4ceb80f0cc595403775e8f9c17873e1e37b

Change-Id: I6bc70039c1b99b8a5bf5a30e9c6877d4f0aa638d

Bring under test
 - test/unit/cli/test_dispersion_report.py
 - test/unit/cli/test_info.py and
 - test/unit/cli/test_relinker.py
I've verified that swift-*-info (at least) behave reasonably under
py3, even swift-object-info when there's non-utf8 metadata on the
data/meta file.
Change-Id: Ifed4b8059337c395e56f5e9f8d939c34fe4ff8dd

Note that we're punting on configuring socket buffer sizes (for now)
Change-Id: I285a9b521fd0af381a227e0e824bc391817547f4

Change-Id: I7f04b3977971f0581b04180e5372686d8186346f

Change-Id: I7030280a8495628df9ed8edcc8abc31f901da72e

I can't imagine us *not* having a py3 proxy server at some point, and
that proxy server is going to need a ring.
While we're at it (and since they were so close anyway), port
* cli/ringbuilder.py and
* common/linkat.py
* common/daemon.py
Change-Id: Iec8d97e0ce925614a86b516c4c6ed82809d0ba9b

Since commit 1e79f828, we don't need to test with post_as_copy=True
any more since we haven't got post_as_copy at all.
Change-Id: I9c96ce0b812d877bbe11bdb50eb160d6ffa5933d

Change-Id: I966f849804cd5eeeae566bc7e732c7bf56bc9eaf

Add support for a 2+1 EC policy to be optionally used as default
policy when running in process functional tests.
The EC policy may be selected by setting the env var:
 SWIFT_TEST_IN_PROCESS_CONF_LOADER=ec tox
when running .functests, or by using the new tox test env:
 tox -e func-ec
Change-Id: I02e3553a74a024efdab91dcd609ac1cf4e4f3208

Following up on patch 417209.
Fast post is now the default mode, so there's
no more need for a fast-post specific testenv.
Change-Id: I6faf8fcfa0a1d96aaf0f5e0ad2106b2b416da22f
Signed-off-by: Thiago da Silva <thiago@redhat.com>

Additionally, emit deprecation warnings when running POST-as-COPY
Change-Id: I11324e711057f7332577fd38f9bff82bdc6aac90

Currently swift ignores a lot of the Hacking style guide. This patch
enables the H401 and H403 checks and fixes any violations. With this
we can get a little closer to following the OpenStack style guidelines.
Change-Id: I5109a052f2ceb2e6a9a174cded62f4231156d39b

The bandit gate is passing, let's make it voting.
Since bandit is a security linter, add it to the linter target - called
pep8 - and run it there. This will save the CI systems spawning an extra
test machine for this.
Needed-By: I90e5401ee8e46a2a153adb62d0f99a8a88a686b8
Change-Id: I75eeefa2441466da60b9b206769a2e64e9c1e7a9

Change default py3x to py35 in tox.ini to support python3.5,
and fix the error when run tox test for Swfit.
Closes-Bug: #1649773
Change-Id: I8e26183ffe270518503da8b5aa394e99ae4cd8c6

Fixes: NameError: name 'file' is not defined
file() is removed in python3.x using open()
 will solve it.
Fixes: AttributeError: 'generator' object has no
attribute 'next'
Python3 it is __next__()
Change-Id: Ib2cbdfbbedb86f6e01239d2ee991677727b0548b

Installed my workstation with Ubuntu Xenial and then hit a few
snags running tox.
python3-dev is needed to build the python3 envs. Did not actually
test python3-devel (rpm) as I don't have such a system.
While python2.7 is the default, tox (on Xenial) itself uses
python3.5, hence 3.5 becomes the basepython and a number
of envs don't work.
Driveby, removed warning about /bin/echo in py3pep8
Change-Id: Ie880246dd757570b8e444da0e643f3c50093bdc5

Follow up to [1]. The --allow-insecure switch is no longer
required for netifaces.
[1] Related-Change: I7c60be623b4340ee34ae1aa520f17b303348811d
Change-Id: If893cdeff68ae5b279ea7055c078b0c1247ae7f2

This shorten shebang in infra, because we are hitting 128 bytes limit.
Change-Id: I02477d81b836df71780942189d37d616944c4dce

This should give the necessary tooling to set up the right
links in the right places so Swift has release notes listed
alongside other projects.
Change-Id: I4e5f1ce1fcfbb2943036c821a24a0b4a3a2d9fc8

Decode header to latin1 on python3
encode header to utf-8 on python2.
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Change-Id: I10f205a05bb3a566e52a597d9315b3a8b8c14664

Change-Id: I7c60be623b4340ee34ae1aa520f17b303348811d

F812 list comprehension redefines <variable> from line ...
While the current violations were benign, this sort of code can easily
lead to subtle bugs. Seems worth checking, especially given how cheap it
is to bring existing code in line with it.
Change-Id: Ibdcf9f93b85a1f1411198001df6bdbfa8f92d114

Change-Id: Ia78fbb32aa6f45aa6578734dc43f3262e717bab8

This commit updates the Swift Bandit file to the new style
introduced in Bandit 1.0. In response to the struggle with
getting a Bandit config file working and kept up to date we
introduced a simplified version in Bandit 1.0.
This commit updates Swift's bandit.yaml to use the new version.
Change-Id: Ida5dd08f4ea72a377346f2159caeb2f3741d4980

* add NamedTemporaryFile() wrapper
* use byte strings for binary data in tests
* tox.ini: run test_splice on Python 3.4 and Python 3.5
Change-Id: I042d739fbf29433733bf0c2154749bc90b210416

Use tox substitution to avoid duplicating testenv:py34 commands in
testenv:py35, to not have to maintain the whitelist of Python 3 tests
in two different places.
Write also the list of tests in a newline to be able to more easily
add new unit tests.
Change-Id: I6e7f238f1c5d3fc9b6560918dcbb93e9dd8ec084

Avoids a warning when running tox -e api-ref
Change-Id: Ib02849075e6424f1db84499fd7500d7bb76dde67

Running functional tests in the in-process mode uses
the default value for the pipeline. This patch adds support
to specify the SWIFT_TEST_IN_PROCESS_CONF_LOADER variable
to point to a labeled function that changes the proxy
configuration for the functional test.
The patch also adds a new tox environment
func-in-process-encryption
which runs with the environment variable
SWIFT_TEST_IN_PROCESS_CONF_LOADER=encryption
The motivation for this change is to put support in place for an
upstream CI job that will functionally test using encryption
middleware in the pipeline. The gate job is proposed at:
https://review.openstack.org/#/c/348292/
Change-Id: I15c4b20f1d2be57ae21c69c614f6a9579145bee9

As much as anything, I'm just tired of seeing a bunch or piecemeal
fixes.
Note that we *need* to include
 from __future__ import print_function
in order to support things like
 print() # Would print "()" (the repr of an empty tuple) otherwise
 print(foo, end='') # Would SyntaxError
 print(bar, file=sys.stderr) # Would SyntaxError
Change-Id: I8fdf0740e292eb1ee785512d02e8c552781dcae1

Just having a `flake8 --filename=swift* bin` command is insufficient.
Change-Id: Ia2a5c364e52d9972d31e6b5e22366503894b720d

This adds swift-specific contents from the OpenStack Installation Guide
in the swift repo per [1]. A separate change will remove the swift contents
from the OpenStack Installation Guide for Newton per [2].
The swift install-guide structure is based on the Install Guide
Cookiecutter [3].
Also adds tox.ini environment for install-guide and adds
openstackdocs-theme to test-requirements.txt.
[1] http://specs.openstack.org/openstack/docs-specs/specs/newton/project-specific-installguides.html
[2] http://specs.openstack.org/openstack/docs-specs/specs/newton/installguide.html
[3] http://git.openstack.org/cgit/openstack/installguide-cookiecutter/
Change-Id: I59b92eebaf5acc657b97bcf10d9ff2cf2db05885
Partially-Implements: blueprint projectspecificinstallguides
Depends-On: Ifebc65b188c4f2ba35b61c0deae5ec24401df7f9