openstack/nova - nova - OpenDev: Free Software Needs Free Tools

This flips the tox.ini to always use constraints installation for all
targets. It drops the extra -constraints targets in the process. This
makes it so there is no developer change required to operate in our
new world order.
Change-Id: I857032002c2ca873510ce7c39138e0e35cc5d3b3

Validate that translations are ok using msgfmt, so that no invalid
ones can be imported.
Closes-Bug: #1536214
Change-Id: Ic1f106684270a8f2197e0fb2a67d6fc00535ee65

Just like in the docs target, we should remove the releasenotes/build
directory between runs of the releasenotes target to avoid stale
sphinx results corrupting the current run results.
Change-Id: Ia9e0409344671e04148308e08c26e8796227f0f7
Closes-Bug: #1534640 

This reverts commit ef8335185a
Now that Ibf84ee0a604da94f709a0eeb436efe988d21274c is merged
on stable/liberty we should be OK again.
Change-Id: I1a56426a86747bdcd1a9179ecbff46cf54a98202
Related-Bug: #1534613 

b2acc9fa86 on stable/liberty introduced a
sphinx warning which was not treated as an error in stable/liberty nova
releasenotes tox runs so it slipped in, but it is treated as an error
in master branch nova releasenotes tox runs, and since reno scans the
branches, master is broken because of the error in stable/liberty.
Plus, reno scans the git history, so right now it's not obvious how to
travel back in time and fix the stable/liberty change.
This change reduces the level of control we have by not treating reno warnings
as errors. This is not fun but it's the only way we can still land
release notes until an appropriate fix is merged (in reno or nova).
Again, this is a temporary patch and we consider reverting it once bug 1534613
is fixed.
Change-Id: Iedb69c66632383e056840e3294a90cda651a8778
Related-Bug: #1534613 

The commands used by constraints need at least tox 2.0. Update to
reflect reality, which should help with local running of constraints
targets.
Change-Id: I27348462a21daab479d76b24b48e3a4e017504cc

This commit removes the direct call to discovery in the tox py34 job.
This was added as a temporary step to enable importing of everything
and is now include by standard in the use of ostestr. There isn't any
reason to keep it around anymore.
Change-Id: I117fbc2108b285aa148cb447c1c01145c43031dd

Caching is enabled by default from pip version 6.0
More info:
https://testrun.org/tox/latest/config.html#confval-downloadcache=path
https://pip.pypa.io/en/stable/reference/pip_install/#caching
Change-Id: I83b5333f4f4b1600b9bbd508b5e0bc04c0068836

Due to testrepository bug 1212909, if you run py27 before py34, py34
fails with "db type could not be determined". The workaround is to
delete the .testrepository directory and try again. As long as you run
py34 before py27 you're OK.
Change-Id: I13c4f9a39b97573e78fe397efc7d87f78bac4a58

api-samples generation is no longer straight forward, now that tox is
blocking environment changes, and that PYTHONHASHSEED is randomized in
each run.
This creates a tox target to make it easy to generate the api-samples.
Change-Id: Ie88597fecee5ce81fb29ce2cfb73b15d60f3f389

Per http://lists.openstack.org/pipermail/openstack-dev/2015-November/080521.html,
we need to make sure that there are no warnings for reno.
Change-Id: I50ce202f82e3213d07c3989780833e18b4c2bf08

Add html for sphinx invocation so that files end in this directory
as usual. This is the usual place for sphinx documentation.
Change-Id: Iae7a3b91d407c06b7f4d2dce909a15c38ea72b9b

Since reno is the new tool for Relnotes [1], we need to add it
to Nova and provide some Sphinx docs for Liberty and Mitaka.
Change-Id: Ibd74e62b6f36076dbec17dd146632fd42aad6eb2

Uses openstackdocstheme to match other content
Has a dependent change in project-config also so that
file will build to developer.openstack.org/compute
at https://review.openstack.org/#/c/231000/
Change-Id: Ic060a1e79e4b2f8695cb788ff4df018e0cfd3286

The fixture in the o.vo library allows us to remove duplicate test
code in Nova, so let's use that.
Change-Id: I896e7ee53856955e0338e7c83cd112d43b27367d

Tox does not currently support using factors with sections, and work
towards enabling this has stalled, as such we will need to duplicate
sections adding -constraints to facilitate running sections with the
constrained install_command.
Implements Blueprint: Requirements-Management
Change-Id: Icbbb78cfcd074b0050e60c54557637af723f9b92

Tox installs the current tree (either via an sdist or as a develop
install) always. Our requirements.txt are reflected into the metadata
of that tree, so we shouldn't use requirements.txt directly - it just
means that we'd be hiding from ourselves issues like e.g. using URLs
which can't be reflected properly.
Change-Id: Id096f1d1971cb8a19f9e86ba57957b39e00bd6f5

os-testr has a blacklist we can use to specify which
tests to avoid. It uses "testr run --parallel" so tests
are run in parallel as well. We also get rid of the
huge list of tests in tox.ini into a separate file.
<soap_box>
The reason for using a black list is that when new tests
are being added, they are automatically either fixed to
work under python34 or added to a well known list of
tests so whoever is working on python34 can try to fix
those tests over a period of time. Hoping really that
this black list shrinks over time quickly. This is also
a pool of low hanging tasks that folks when they have
some time to pull a few tests from here and fix them.
</soapbox>
Bumps number of tests executed from 5606 to 7206
Depends-On: Ib5e682d0380cf7bc5e288a1e4d125379b452fa92
Change-Id: Ib641bb0f7553eb7704b419b9d00f86174f6d831d

Couple of tiny fixes and we essentially double our py34
coverage.
Change-Id: I77355043d613b8b3b6789950302c54b16b690049

flake8 rule in tox.ini has following
exclude rule is exclude = .venv,.git,.tox,dist,doc,*openstack/common*
which excludes api/openstack/common.py
Change-Id: I39b82cf90048970e7bfc6be77de2e1543130f81b
Closes-Bug: 1498634

This adds a pip install command to tox.ini that is only used when the
tox env is passed with the 'constraints' factor appended onto it.
As such this will not effect developer workflows or current unit tests.
The initial use of this will be in a non-voting job, to verify that the
constrained checks with tox are stable. DevStack is already running
constrained jobs, as such problems are no expected.
To run a tox with pip using constraints on a developer system a
developer should run the desired tox environment with -constraints.
For example: $(tox -epy27-constraints)
Pip will pull the current version of the upper-constraints.txt file down
from the git.openstack.org, however this method can be overriden to use
a local file setting the environment variable "UPPER_CONSTRAINTS_FILE"
to the local path or a different URL, it is passed directly to pip.
This is currently not enabled in the default tox run, however it is
possible to enable it as a default by adding it to 'envlist' in tox.ini
Depends-On: I17ac389f78af241917b6da7f049085f2b13d30f2
Change-Id: I8ea1710a4e3a287405978b467e0b2dfbb025b795
Implements Blueprint: Requirements-Management

doc/source/api and doc/build are build artifacts from building the docs
using the tox -e docs target. These are ignored in .gitignore since we
don't care about them from build to build - but leaving them around can
actually make the docs build fail if modules no longer exist but are
still sitting in these directories.
So this change just cleans those up each time you run tox -e docs.
Closes-Bug: #1488637
Change-Id: I646ae0b77a472c425d23815316e804b09bbe867d

If someone makes a change that breaks "tox -e genconfig"
it is not caught today by any of the gate/check jobs.
We should fail at least the docs job by running
oslo-config-generator to make sure it works.
Change-Id: I63fede4a7b5f358f2212383e2870cc64c4f9f1e4

`cmp` method was removed in Python 3, so it would be nice to use __lt__,
__le__, __eq__, __ne__, __gt__ and __ge__ instead, which are Py2 and Py3
compatible.
Change-Id: I1c89da0831b77b73f55d8681fd7d946535cc89b5

Now that we require at least pbr 1.3 and that has
commit 61a3a1ecdf99c680e509742fdc6c441b499af130 in it,
let's remove the workaround.
Change-Id: I4074b05a823d32f5db0051062527b0db4d379b62
Related-Bug: #1473401 

Currently the tox py34 target uses testtools.run to run
a subset of our test harness. We need to be able to use
pretty_tox.sh just like py27 as we make progress with py34
support.
The first step is to make sure we can discover all the
tests using:
python -m subunit.run discover -t . ./nova/tests/ --list
So, we need to fix a bunch of things for the discovery
to work including updating to a new version of websockify.
In the xen code, we should keep the original import and
add except for py34 as xen uses an older python that does
not work with six.moves.
Depends-On: Ib4ef2e79b28b7180e564b3d6dc2188456c66c08a
Change-Id: I88b6746da6136a7386a173f6cacd42f0b470deee

nova/pci/stats.py:
 need to specify a key parameter which is a function that returns
 a sorting key. since the original intention with pool_cmp was to
 sort by number of items in each pool, we just use key parameter
 with a lambda
nova/scheduler/filters/trusted_filter.py
nova/scheduler/host_manager.py
 needed list() so we could delete stuff inside the loop in py3
nova/tests/unit/scheduler/test_filter_scheduler.py
 needed the import of test as some tests were failing in py3,
 added noqa as test is not explicitly referenced within the
 module
nova/tests/unit/scheduler/test_scheduler_options.py
 need BytesIO for py3, StringIO will fail in many tests
nova/tests/unit/scheduler/test_scheduler_utils.py
 message is not available in py3, so we need to adjust
 use test.nested already added in a previous commit
nova/test.py
 if we have a key which is a list, treat it just like a set
nova/utils.py
 use reduce from six and explicitly typcast version_int as
 an integer
Change-Id: Ic862873f38767f12b990a030b5812ae50a44c4d0

The 'tox -epep8' command takes a long time as it checks
every single file in the Nova git repository (~1,400 at
time of writing).
This makes tox use a simple wrapper around flake8 which
can be told to restrict the check to only files changed
in the current command. This can be invoked in a simple
manner with 'tox -epep8 -- -HEAD'. Since most commits
only touch a handful of files, this will usually be
far faster than checking all 1,400 source files.
To check an entire branch for bisectability it can be
automated via
 git rebase -i master -x 'tox -epep8 -- -HEAD'
Change-Id: I157d1ccb883ca02402eee51fd7d6a50f86079389

* Fix sort(): dictionaries are no more comparable on Python 3, use a key
 function building a sorted list of the dictionary items
* Replace __builtin__ with six.moves.builtins
* get_hash_str() now accepts Unicode: Unicode is encoded to UTF-8
 before hashing the string. Mention also the hash method (MD5) in the
 docstring.
* LastBytesTestCase: use binary files instead of text files. Use also a
 context manager on the TemporaryFile to ensure that the temporary file
 is closed (even on error).
* Functions: use the __code__ attribute instead of func_code, and
 __closure__ instead of func_closure. These attributes are also
 available on Python 2.6.
* Replace unichr() with six.unichr()
* SafeTruncateTestCase(): write directly the chinese Unicode character
 instead of using safe_decode() (which is not reliable, it depends on
 the locale encoding!)
* sanitize_hostname(): On Python 3, decode the hostname from Latin1 to
 get back Unicode. Add also a comment explaining the purpose of the
 conversion to Latin1.
* last_bytes(): replace the hardcoded constant 22 with errno.EINVAL and
 add a comment explaining how the function works.
* tox.ini: add nova.tests.unit.test_utils to Python 3.4
Blueprint nova-python3
Change-Id: I96d9b0581ceaeccf9c35e0c9bada369e9d19fd15

This includes 2 classes of fixes (and 1 skip) for mock.
The first is the change in allowed values for assert_has_calls -
https://github.com/testing-cabal/mock/issues/263
The second is a yet unsolved bug around the use of autospec
https://github.com/testing-cabal/mock/issues/264
The skip is because something has changed with mock.open that is
causing the vhduils test to fail. We don't know why, but it's one test
to skip.
This also includes a brute force fix for tox -e docs, because pip is
no longer respecting the sphinx pin in test requirements. This has to
be landed with the other changes because they won't work without it.
Change-Id: Id835d080a1ada52cbd3f24dad9bab5eeb2f29a54
Partial-Bug: 1473401

Fix bytes vs Unicode issues:
* ssh_encrypt_text() now encodes text to UTF-8 if text type is Unicode.
* Encode Unicode to ASCII before encoding it to base64
* On Python 3, decode binascii.hexlify() and base64.encodestring()
 output from ASCII to get Unicode (str type in Python 3) is required.
* convert_from_sshrsa_to_pkcs8(): reuse binascii.hexlify() instead of
 a loop using struck.unpack() and "%02X" format.
* Replace str.encode('hex') with binascii.hexlify(str)
* Replace string.strip(text) with text.strip()
* Replace StringIO.StringIO with six.StringIO
* Add test on the output type of ssh_encrypt_text() and decrypt_text()
* Call utils.execute() with binary=True to not decode stdout/stderr
 (to get them as raw bytes).
* Replace reduce() with six.moves.reduce()
* convert_version_to_str(): replace a/b with a//b to get integer
 division
* tox.ini: add the following tests to Python 3.4
 - nova.tests.unit.compute.test_keypairs
 - nova.tests.unit.test_crypto
Blueprint nova-python3
Change-Id: I83d927166c0864020b205ac7495473795da7830d

The desired OS_TEST_DBAPI_ADMIN_CONNECTION string is now reflected
in the oslo.db 1.12.0 release's default behavior, and so does not
need to be set any longer to achieve the same opportunistic backend
discovery for tests.
Change-Id: I1af372f6f18038e8fc63076b4e399d0ae86f5a23

* FakeNovaException_Remote: on Python 3, define a __str__() method
 instead of __unicode__() method. str(exc) now calls __str__() now
 Pyton 3, __unicode__() is no more used.
* tox.ini: add nova.tests.unit.test_exception to Python 3.
Blueprint nova-python3
Change-Id: Ia3e3f253aa57b75a47f3ba044d2c539f9ce48aea

This change adds a basic bandit config for Nova. It can be invoked
by running the tox environment for bandit;
 tox -e bandit
This is intended as a starting point for using bandit with Nova
and it should be revisited to improve the testing as more is learned
about the specific needs of the Nova code base.
Tox is configured to only show results for high and medium severity
results.
https://wiki.openstack.org/wiki/Security/Projects/Bandit
Change-Id: I3026b81317f0a6322acfc94784899a7453af586f

All tests in nova/tests/unit/objects/test*.py now run with
python3.4 tox target.
* Fix imports in limits.py and urlmap.py
* Add a list() as keys()/values() return iterator in
 availability_zones.py, flavors.py and instance_numa_topology.py
* contextlib.nested is not present in python3, so whip up an
 alternative using ExitStack(). Directly using ExitStack() wont
 work for us.
* Add a few assertJsonEqual in the test cases
* Ensure fingerprinting generates the same exact value in both
 python27 and python34
Blueprint nova-python3
Change-Id: I848c48475189c4b4ad8151e14509020ae7d110a4

With environment markers this is not needed, and it should fix
requirement syncing.
Change-Id: I9e32764417852ea65b28efe3c05d39865ebc5744

* Replace dict(obj.iteritems()) with dict(obj) where obj is a dictionary
 or an oslo.db object
* Replace obj.iteritems() with obj.items() where obj is a dictionary:
 obj.items() works on Python 2 and Python 3.
* Replace filter() and map() with list-comprehension using if when a list is
 expected. On Python 3, filter() and map() return an iterator.
* Replace obj.keys() with list(obj.keys()) when a list is expected.
 Replace obj.keys()[0] with list(obj.keys())[0]. On Python 3,
 dict.keys() now returns an iterator.
* Replace (int, long) with six.integer_types
* Get the name of a function using the __name__ attribute, instead of
 the func_name attribute. The func_name attribute was removed
 in Python 3.
* InstanceTypeTestCase: use dict.update() to combine dictionaries
 instead of using dict.items()+reduce()+dict()
* matchers.py: replace StringIO.StringIO with six.StringIO.
* tox.ini: add nova.tests.unit.db.test_db_api to Python 3.4
Blueprint nova-python3
Change-Id: Iae4f6d971818d5b38e838a83753a0707e954bb04

Oslo team is actively getting rid of the old style generator.py
in openstack/common/config. So we should switch to the newer
oslo-config-generator which is the way forward. For simplicity's
sake, i used one single list_opts in opts.py. However we can
split this up and add entrypoints in setup.cfg if we decide to
do that instead.
blueprint oslo-config-generator
Change-Id: Ib7c72af7be47df08983708ac6da0ce103518b78c

When a development environment is under a proxy, tox is failed even if
environment variables of the proxy are set.
This patch fix this problem.
Change-Id: If0bc42891a0714cc9b37b9c211388db20555ea7e