openstack/nova - nova - OpenDev: Free Software Needs Free Tools

This adds a new tox target that uses sitepackages=False. This is
handy for people running unit tests on a devstack box, where the
site-packages oslo.config interferes.
Change-Id: I1101730f18550de0b391d12bcc5841cb671434a2

check_uptodate.sh re-generates a nova.conf.sample and then compares
it, which was fine. It stopped being fine when we started pulling in
options from oslo, or other modules made releases and then the gate
broke until we fixed the config sample.
Change-Id: I04b2b52a9e46dcf5f4f06e639327f22b827ccd0d

This check indicates on comments in which multi line docstring should
start without a leading new line. This change fixed all violators of
said check.
Change-Id: Ic7357b8c7420767dba611f6fcee07b7700f3aea8

* tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
introduced originally during the testr migration in an attempt to be
conservative about the possibility that locale settings in the
calling environment could cause consistency problems for test runs.
In actuality, this should be unnecessary and any place where it does
cause issues ought to be considered an actual bug. Also, having
these in the configuration actively causes older pip to have
problems with non-ASCII content in some package metadata files under
Python 3, so drop it now.
Change-Id: I5353dd318ce0bfdc85d6468750fb4c2a2690cde1
Closes-Bug: #1277495 

This reverts commit de59c55e77.
The netaddr developers have agreed to go back to uploading releases
to PyPI, so this workaround is no longer necessary. For details see
https://github.com/drkjam/netaddr/issues/57#issuecomment-31796111 .
Change-Id: Ieeb3e8e549d9e4b53dba0165277fa2423d2eba01

* Stop ignoring F403 as we have no cases of it anyway
* Cleanup the comments to make it clear what is ignored on purpose and
 what is ignored until we clean up the code.
Change-Id: I2500e2db70b9c71fbed781d8f7b7c80eee45140f

* tox.ini(testenv.install_command): Use the --allow-external and
--allow-insecure options so that pip 1.5 and later will assent to
retrieve the netaddr package even though it's not hosted on PyPI.
The --allow-insecure option is aliased to a clearer
--allow-unverified wording in 1.5, but the old form is being used to
avoid breaking users of 1.4.x and will be valid at least through
1.6.x according to comments in the pip source.
Partial-Bug: #1266513
Change-Id: I005e752c8cb9bca78b1ebe777eb5230e9bf05bd8

Apache2 license says that each source file should include the license
header. However, some non empty files don't have it. And we already have
a hacking rule (H102) for this.
This commit fixes them and enables the rule.
Change-Id: I68fa0e1ee43ed28d6b470c0ffff963c095be8283

When running unit-tests concurrently using testr, we might
need to use interprocess file locks (e.g. for running various
tests using the same DB in MySQL/PostgreSQL).
At the same time file locks are only usable when they are placed
to one dir, and we have a test fixture that overrides the lock_path
config value for each test case with a new temporary directory.
The solution is to create the one temprorary directory for locks
to be used by all tests before running tests and then remove it
after running tests. The corresponding helper has been already
put to openstack.common.lockutils module, we just need to reuse
it properly, i.e. change the way unit tests are run in tox.ini.
Closes-Bug: #1261728
Change-Id: I76f95a9f7fdd31c15e6cf4fd6316c7569284f780

E125 overreaches on what pep8 specifies. See
https://github.com/jcrocholl/pep8/issues/126
In practice E125 can be a pain as compliance requires
manual indenting if using emacs and refactoring tools
don't always get it right.
There's a bit of a discussion about it here (on a tempest
changeset):
https://review.openstack.org/#/c/36788/
Change-Id: Ic73ab3c4a47f33de9145e0c7db2d8674230c2fe0

Change-Id: I19060db8622e5747ab19a11a8eafe3144d61abe9

See I62ce43a330d7ae94eda4c7498782a655e63747fa for the gorey details on
why this exists.
As of this fix:
https://github.com/eventlet/eventlet/pull/34
which was released in eventlet 0.13, we no longer need the patch.
This has now been removed from oslo-incubator, so this is really just
syncing that removal.
Change-Id: I84267f3c6726cb2e750f615e107c48b12c6ed353

Removed plugins/xenserver from tox.ini excludes list, ran tox -e pep8, and
fixed violations.
Fixes: bug 1197893
Change-Id: I8e0116304b07d54c49a01202b3ce6d3426097bb4

H803 checks for a period in commit messages, something the community
clearly rejected as a sensible check: periods in commit messages
should neither be rejected or required.
Change-Id: I866928c638d23a307f4c11b7cb947e88f90f1ab2

H302 is already part of the OpenStack style guide, and we only had a few
violations. This fixes and gates on H302, so there is one less thing
for a reviewer to worry about.
gettext import line is ignored using hacking 0.7's import_exceptions
option.
Reason for import only modules 'The namespace management convention is
simple. The source of each identifier is indicated in a consistent way;
x.Obj says that object Obj is defined in module x.'
http://google-styleguide.googlecode.com/svn/trunk/pyguide.html
Change-Id: I53e8039f1dd2ebf8ab5fcf5a561feb3ca3363107

tox 1.6 was released, which means that we can now take advantage of the
feature we added to it - which is using setup.py develop to install the
code into the virtualenv. The logic was taken from run_tests.sh - so the
performance issues around using tox vs. using install_venv should now be
gone.
run_tests.sh is still needed to handle the ui switching between
parallel, single and debug modes to testr.
Change-Id: Ida5e440d1bdb9f8e9031277ea53a02d2ef171438

Change-Id: Ide587d3311c22f698a82eeee210c5bdbe24eb484

Remove session parameter from:
 agent_build_get_by_triple()
 bw_usage_update()
They are never called with session parameter
Rename aggregate_get_query() -> _aggregate_get_query()
because it is private method
Enable N309
implement blueprint db-session-cleanup
Change-Id: Idd43d230105cb140d8825a251f03a58ea28cce90

Wildcard imports make reading code unnecessarily confusing because they
make it harder to see where a functions comes from. We had two types of
wildcard imports in the code. Unneeded ones in test files that are just
removed, and some that we actually want which are kept using the '#
noqa' comment to tell flake8 to skip specific checks (such as H303)
for that line.
Change-Id: Id4705011579659fd74a4aaa05ac541e9694c483e

Also fixed all violators of this check.
Change-Id: Id87ff6f44ab76fc59b18d5da739df475400e1259

Also fixed violations thereof.
Change-Id: I581a33385fc47a6c92c8933e0ed63fd688806c03

Previous _ was monkey patched into builtins whenever
certain modules were imported. This removes that and
simply imports it when it is needed.
Change-Id: I0af2c6d8a230e94440d655d13cab9107ac20d13c

According to the commit message in
560e5cd84e we ignore E712 because '
With this change we ignore E712 since it is normal to use
"column == True" in sqlalchemy.'
Add note in tox.ini explaining this.
Change-Id: I393e13b1430aae46f958311875678bd99e17b13c

So we can start using new hacking 0.6 checks.
Since hacking 0.6 pins pep8, pyflakes and flake8 they can be removed
from test-requires. For any unfixed checks list them in tox.ini
Fix a H103, 'Header does not match Apache 2.0 License notice' that
leaked in
Change-Id: I652fe3270f1c37afdd386bfae7c4b27b6abcfa06

The sample generator tool in Oslo is located in tools/config therefore
this patch moves the files in tools/conf to tools/config. The reason
Oslo chose 'config' is because the generator.py module is located in
the openstack.common.config package and update.py looks for the
corresponding directory in tools when it syncs the 'config' package.
./update.sh --module config --base nova --dest-dir $NOVA_HOME --nodep
Since every thing moved to tools/config, changes had to happen on
the tox.ini file and check_update.sh
Change-Id: Icbd467888cc7f16fa3694ed2b93548d0285461b5

We added this check recently to Ceilometer, and it helps a lot to keep this
file always up to date and not falling behind.
Change-Id: I5ad2c6366032a39c2e55dd11a16b9f1780ab9ed0

Since we have site-packages=True in nova, the existence of system
packages of the wrong version can cause nova to not install what
you expect into the tox virtualenv. Adding -U into the deps list
causes it to get appended into the command line, which causes pip
to do what we want and install what we asked for.
Change-Id: I4f388e07caf31ab870f22d261e856a205ebb36c0

We need to exclude the plugins/xenserver directory
because the python2.4 code can't use as in except clauses.
The existing exclude directory accidentally included some
code in nova/api/openstack/compute/plugins and the related
tests. This code is now included in the flake8 tests,
and the current violations have been fixed up.
Fixes bug 1197858
Change-Id: Ib5f1b65d5d793f0f5854e898ca3ce751e43ef233

E125 continuation line does not distinguish itself from next logical
line.
E125 makes code more readable, and fixing all the existing issues allows
us to gate on this, making this one less thing reviewers have to check
for.
Fix was made using https://github.com/hhatto/autopep8
Change-Id: Ie569fd74fb84d220ceb87e37b1b1f39143bba80f

Add a HACKING check to enforce that public db/api and db/sqlalchemy/api
methods to not accept a 'session' parameter.
This check is initially disabled, since it is failing ~24 times right
now, but will be enabled once bp/db-session-cleanup is complete.
Change-Id: Ib89eea58555032dd142d4e21e62d66e2726f0d06

Multi-line doc-strings should end on a new, separate line.
Change-Id: I4cf0cfe92b634ef77971863a4df41ef43531bc20

Delete last bits of bin/. With the move to entrypoints these aren't
needed anymore.
Update CONF.bindir to default to
 os.path.join(sys.prefix, 'local', 'bin')
Part of blueprint entrypoints-plugins
Change-Id: I95250d3779433e7b85aaa889a873b16c86a7d2be

Introduce py33 to tox.ini to make testing with
python3 easier.
Change-Id: I3f966efc1a845aaa8e317d810bd6a1366acd323c
Signed-off-by: Chuck Short <chuck.short@canonical.com>

Add posargs to flake8 call in tox.ini, with this you can pass arguments
directly into flake8 using the following notation
 tox -epep8 -- --FLAKE8-ARG
Change-Id: I1296eac3df46438ef050bf99ca33eb38d3b02efa

Instead of ignoring all E12x errors, ignore only the failing ones,
so its easy to fix each one separately.
Change-Id: Id365f4d61be56b51823ead8c7c1948e034ea609b

F823 local variable 'blah' (defined in enclosing scope on line xx)
referenced before assignment
Change-Id: I721e1339ef023558032980bbf59e8dc44b391d52

F812 list comprehension redefines
List comprehension don't have scope so redefining variables can lead to
unexpected behaviors
Change-Id: I85484dfcd4fa391195ea7be75aadc7897da4739c

Instead of ignoring all Fxxx errors, ignore only the failing ones, so
its easy to fix each one separately.
Change-Id: I4ccc1d38247fb76b6dad4388b76c6e8d696f081f

We already pass the test, so just remove from ignore list
Change-Id: I28bfeb99d9841532133a3825f7336e7b712d6561

Rename tools/pip-requires to requirements.txt and tools/test-requires
to test-requirements.txt. These are standard files, and tools in the
general world are growing intelligence about them.
Change-Id: I68ece8406fb1d4e082a42db8e76e17b1aaa7e775
Fixes: bug #1179008 

Remove a bunch of local custom scripts. Replace with configurable
external tools.
Use local hacking checks for nova specifics.
Change-Id: I75a01375ba0ec36d2ff05abc47abe0a3f225eda5

Enable the use of site-packages in tox which will allow the use of the
system install of libvirt while testing.
Hardcode the libvirt host UUID for tests that check this UUID when
system libvirt is being used. Without this hardcoding eight tests
would fail when using the system libvirt install.
Partially fixes bug #1113181
Change-Id: I59c5fbd45639962c0963298203c39759b6ca2d11

In order to support running unit tests on RHEL 6.x we need to patch
eventlet with contrib/redhat-eventlet.patch. We already
have support for this in the tools/install_venv_common.py but we need
to make a couple changes to allow tox to consume this:
1) Sync in the latest intall_venv_common.py from oslo. This changes
 patch to use the -N option (ignore already applied patches) and makes
 it safe to call the patching function more than once.
2) Add a new patch_tox_venv.py script in tools.
3) Update tox.ini to call patch_tox_venv.py before it runs tests and
coverage.
Change-Id: I3e2b07c3f718e4aede5c5f231ff0cdb7721ec885

The cfg API is now available via the oslo-config library, so switch to
it and remove the copied-and-pasted version.
Add the 2013.1b4 tarball to tools/pip-requires - this will be changed
to 'oslo-config>=2013.1' when oslo-config is published to pypi. This
will happen in time for grizzly final.
Add dependency_links to setup.py so that oslo-config can be installed
from the tarball URL specified in pip-requires.
Remove the 'deps = pep8==1.3.3' from tox.ini as it means all the other
deps get installed with easy_install which can't install oslo-config
from the URL.
Make tools/hacking.py include oslo in IMPORT_EXCEPTIONS like it already
does for paste. It turns out imp.find_module() doesn't correct handle
namespace packages.
Retain dummy cfg.py file until keystoneclient middleware has been
updated (I18c450174277c8e2d15ed93879da6cd92074c27a).
Change-Id: I4815aeb8a9341a31a250e920157f15ee15cfc5bc