openstack/nova - nova - OpenDev: Free Software Needs Free Tools

The docs job is not run in the gate, add the linting command from
the docs environment that checks for valid JSON files to the pep8
environement so that jobs run it.
Add a simple comment to both invocations to explain what it does.
Note that pep8 is our general linting target and thus should include
these kind of tests.
Change-Id: Ibb2ad29ca59b86fef41ebfe0ec3b8e8ea38e6d72

OpenStack CI has now enabled constraints for releasenotes, use them -
remove the special install command.
Note that venv still cannot use constraints, this needs more changes
in OpenStack CI.
Change-Id: I125cb906f89c1de4227dc53521e1a60241ace997
Depends-On: If5d6b11bd795091eef949001cca6e8d4ec723ef2

At least once a cycle we get contributions from new folks that attempt
to update a bunch of pep8 formatting. They believe that this is in
good faith helping, but don't realize we really aren't likely to
accept that.
Update the text here to explain that more clearly, so that people
don't keep falling down this rabbit hole.
Change-Id: Iec383a18cb3d64b9655e5bd1d82cabd3e785686c

This patch introduces a new tools/releasenotes_tox.sh script for the
tox -e releasenotes test environment. It will detect when there are
uncommitted release notes and print a reminder to commit them.
Change-Id: I54af12d7eb20b37263e23443900745884cffacea

OpenStack CI setup has been changed so that api jobs can
use constraints, remove the extra install_command.
Change-Id: I1eb8c5b74adf28e8b6267f25b116341c540729dd
Depends-On: I8f45a53429b9fcbf3689a268f096afdf5f32f461

OpenStack CI supports now constraints in the coverage job - as a first
job to check that recent changes for zuul-cloner work correctly.
Use constraints for cover environment. If this works successfully, the
other environments can be updated in followups.
Change-Id: I6069bee0a89d38f6a3ec57418e7bda9850dc6f64

This simplifies our tox file and reduces duplication.
Change-Id: I085899e00a249fb850c81737b1088bb82ce820ba

There's no reason to create these files and cleaning them up takes
precious time during each test run. Don't do it.
The benefits of this can be seen in running both the full test suite
and part of the test suite. For the former case - running the full
test suite:
 $ time ( tox -e py27 )
Without this change:
 real 12m48.003s
 user 38m27.039s
 sys 1m2.786s
With this change:
 real 11m53.921s
 user 39m33.360s
 sys 1m1.686s
For the latter case - running part of the full test suite:
 $ time ( tox -e py27 -- nova.tests.unit.test_configdrive2 )
Without this change:
 real 2m0.686s
 user 1m3.080s
 sys 0m12.950s
With this change:
 real 1m37.098s
 user 1m16.559s
 sys 0m11.846s
The actual results can vary massively, but this change is always
faster.
Change-Id: Iffd759e368878f39c015258935d8b37de09d9745

Infra will start running python35 jobs soon [1], targeting to replace
the current python34 jobs. This is due to the change from Ubuntu Trusty
to Xenial, where only python3.5 will be available.
So we add a py35 environment with the same definitions for skipping
not-yet-working checks as we have in place for py34 already.
[1] https://review.openstack.org/336272
Change-Id: I017478fafa31121f40c95d73e49ac4dced80c7a2

This adds the entry point and config file necessary for using the
oslo.policy sample generation script. It also adds a tox target to
simplify the usage of it.
The command is also run in the docs tox target since the sample file
can be considered documentation.
Change-Id: I18a1fa567fc04b0e0a37c02f20024b2ce483f060
Partially-Implements: bp policy-in-code

For py27 you can simply run specific tests by calling
 tox -e py27 nova.tests.unit.test_something
Add a similiar option for the py34 environment. This will skip the
blacklist of tests that would otherwise be ignored for python3.
Change-Id: I9bf5901afed4abf592e1aaff2f4820b9e3abc671

We pass -W option to sphinx-build to generate api-ref now.
The comment looks no longer correct.
Change-Id: Ie5dfaf46e9d3369c32c43da07c02501b224fb092

The StandardLogging fixture defaults to log level INFO and has
the ability to log at level DEBUG if the OS_DEBUG environment
variable is set to a true value. Let tox copy OS_DEBUG to the
test environment to allow debug log capture for a command line
like:
 OS_DEBUG=True tox -epy27
Change-Id: Ibe51653f652c99aee56f702be571130cbb7f913d

Two years ago the translation files have been split into several
files, separating the log messages of different log levels from each
other, like X.pot, X-log-warning.pot, X-log-info.pot, and so on.
However, the setup.py command `compile_catalogs`, that comes from the
babel package and compiles the corresponding .po files into .mo
files, only supported one file per python package. This means that
during packaging `compile_catalogs` never compiled the X-log-*.po
files, so the corresponding translations were always missing.
Since babel 2.3 the domain can be set to a space separated list of
domains. This change adds the the additional log level files to the
domain list.
The obsolete check that .po and .pot files are valid is removed from
tox.ini.
Change-Id: I0d9cb16ac50caae570c57a1db6bed656bf13c3dc
Closes-Bug: #1536226 

This removes the final warnings from the api-ref code base, and flips
the sphinx switch to enforce warnings as errors.
It also adds code to the rest_parameters extension to make it more
clear where a problem is when it comes to included parameters. This
puts us in a place where we can start doing the file per file look at
content.
Part of bp:api-ref-in-rst
Change-Id: Ic2c99d96d6addcafa00b9f16785c2fe59b1798d3

Oslo team won't create new libary to include functions of module cliutils.
Nova only use two methods in cmd/manage.py, so copy these two methods
to nova/utils.py and clean up oslo-incubator related stuff.
Note:There is tiny change of method validate_args to make it Python 3
compatible.
Change-Id: Ibef5076f9556fa3224c2ca32e8c7fe2c4c1f739b

This adds sample API content for versions and servers resources,
including the parameters.yaml that is needed in both of these cases.
It also makes a new tox.ini target 'api-ref' for building these docs,
which will be used as part of the publishing pipeline.
Change-Id: I310ed352dc5dd81d01f2fd5f1a2fab662c29f0dc

Error on warnings in building the API guide and fix any problem
encountered:
* tox.ini: Enable tread warnings as errors
* index.rst: Add unreferenced file extra_specs_and_properties as
 hidden file so that it does not show up in index. It is still
 built and will be referenced from other files.
* general_info.rst: Remove reference to non-existing file
* server_concept: Add missing empty line after code directives
Also fix a typo and remove extra EOL whitespace.
Change-Id: Ie766f8d2f7d82b7b5b02541b342f9829ca4550a0

The venv environment is used by infra scripts to build tarballs and
documents - and used by both check and post jobs. In post jobs, we
currently do not support constraints, so add the non-constraints install
command here so that nova-docs (publishing of documents to
docs.openstack.org) and the tarball jobs work again.
Change-Id: I2e5c0a4d739296b8b9d4ef7f58b69e1eff3c9575

Because we are constrained by the constraints mechanism we no longer
need to install all of test-requirements to get hacking at the right
version, because we'll only install the upper-constraints version.
Change-Id: I877ffec045cc43a32f97be8afe453d4755507217

The dependency mentioned in the TODO has been resolved, thus the TODO
can now be addressed.
Change-Id: I306f83bb9474371a4aa8277a2a5e1e258471b400

pep8 doesn't work with python3 on our codebase. If someone is on a
platform that defaults to python => python3, pep8 won't work for
them. This is actually really easy to fix with a single line in tox.
Change-Id: I7a888e4f7cc828638a9d61d2249a854ba1f3cb7b
Closes-Bug: #1456955 

This flips the tox.ini to always use constraints installation for all
targets. It drops the extra -constraints targets in the process. This
makes it so there is no developer change required to operate in our
new world order.
Change-Id: I857032002c2ca873510ce7c39138e0e35cc5d3b3

Validate that translations are ok using msgfmt, so that no invalid
ones can be imported.
Closes-Bug: #1536214
Change-Id: Ic1f106684270a8f2197e0fb2a67d6fc00535ee65

Just like in the docs target, we should remove the releasenotes/build
directory between runs of the releasenotes target to avoid stale
sphinx results corrupting the current run results.
Change-Id: Ia9e0409344671e04148308e08c26e8796227f0f7
Closes-Bug: #1534640 

This reverts commit ef8335185a
Now that Ibf84ee0a604da94f709a0eeb436efe988d21274c is merged
on stable/liberty we should be OK again.
Change-Id: I1a56426a86747bdcd1a9179ecbff46cf54a98202
Related-Bug: #1534613 

b2acc9fa86 on stable/liberty introduced a
sphinx warning which was not treated as an error in stable/liberty nova
releasenotes tox runs so it slipped in, but it is treated as an error
in master branch nova releasenotes tox runs, and since reno scans the
branches, master is broken because of the error in stable/liberty.
Plus, reno scans the git history, so right now it's not obvious how to
travel back in time and fix the stable/liberty change.
This change reduces the level of control we have by not treating reno warnings
as errors. This is not fun but it's the only way we can still land
release notes until an appropriate fix is merged (in reno or nova).
Again, this is a temporary patch and we consider reverting it once bug 1534613
is fixed.
Change-Id: Iedb69c66632383e056840e3294a90cda651a8778
Related-Bug: #1534613 

The commands used by constraints need at least tox 2.0. Update to
reflect reality, which should help with local running of constraints
targets.
Change-Id: I27348462a21daab479d76b24b48e3a4e017504cc

This commit removes the direct call to discovery in the tox py34 job.
This was added as a temporary step to enable importing of everything
and is now include by standard in the use of ostestr. There isn't any
reason to keep it around anymore.
Change-Id: I117fbc2108b285aa148cb447c1c01145c43031dd

Caching is enabled by default from pip version 6.0
More info:
https://testrun.org/tox/latest/config.html#confval-downloadcache=path
https://pip.pypa.io/en/stable/reference/pip_install/#caching
Change-Id: I83b5333f4f4b1600b9bbd508b5e0bc04c0068836

Due to testrepository bug 1212909, if you run py27 before py34, py34
fails with "db type could not be determined". The workaround is to
delete the .testrepository directory and try again. As long as you run
py34 before py27 you're OK.
Change-Id: I13c4f9a39b97573e78fe397efc7d87f78bac4a58

api-samples generation is no longer straight forward, now that tox is
blocking environment changes, and that PYTHONHASHSEED is randomized in
each run.
This creates a tox target to make it easy to generate the api-samples.
Change-Id: Ie88597fecee5ce81fb29ce2cfb73b15d60f3f389

Per http://lists.openstack.org/pipermail/openstack-dev/2015-November/080521.html,
we need to make sure that there are no warnings for reno.
Change-Id: I50ce202f82e3213d07c3989780833e18b4c2bf08

Add html for sphinx invocation so that files end in this directory
as usual. This is the usual place for sphinx documentation.
Change-Id: Iae7a3b91d407c06b7f4d2dce909a15c38ea72b9b

Since reno is the new tool for Relnotes [1], we need to add it
to Nova and provide some Sphinx docs for Liberty and Mitaka.
Change-Id: Ibd74e62b6f36076dbec17dd146632fd42aad6eb2

Uses openstackdocstheme to match other content
Has a dependent change in project-config also so that
file will build to developer.openstack.org/compute
at https://review.openstack.org/#/c/231000/
Change-Id: Ic060a1e79e4b2f8695cb788ff4df018e0cfd3286

The fixture in the o.vo library allows us to remove duplicate test
code in Nova, so let's use that.
Change-Id: I896e7ee53856955e0338e7c83cd112d43b27367d

Tox does not currently support using factors with sections, and work
towards enabling this has stalled, as such we will need to duplicate
sections adding -constraints to facilitate running sections with the
constrained install_command.
Implements Blueprint: Requirements-Management
Change-Id: Icbbb78cfcd074b0050e60c54557637af723f9b92

Tox installs the current tree (either via an sdist or as a develop
install) always. Our requirements.txt are reflected into the metadata
of that tree, so we shouldn't use requirements.txt directly - it just
means that we'd be hiding from ourselves issues like e.g. using URLs
which can't be reflected properly.
Change-Id: Id096f1d1971cb8a19f9e86ba57957b39e00bd6f5