openstack/nova - nova - OpenDev: Free Software Needs Free Tools

This change mainly fixes incorrect use of backticks
but also adress some other minor issues like unbalanced
backticks, incorrect spacing or missing _ in links.
This change add a tox target to run sphinx-lint
as well as adding it to the relevent tox envs to enforce
it in ci. pre-commit is leveraged to install and execute
sphinx-lint but it does not reqiure you to install the
hooks locally into your working dir.
Change-Id: Ib97b35c9014bc31876003cef4362c47a8a3a4e0e

Nova Hyper-V driver is not tested in OpenStack upstream and no maintianers.
This driver has been marked as deprecated in Antelope release. It has dependency
on the OpenStack Winstacker project which has been retired[1].
As discussed in vPTG[2], removing the HyperV driver, tests, and its config.
[1] https://review.opendev.org/c/openstack/governance/+/886880
[2] https://etherpad.opendev.org/p/nova-caracal-ptg#L301
Change-Id: I568c79bae9b9736a20c367096d748c730ed59f0e

tox now always recreates an env although the env is shared using envdir
options.
~~~
$ tox -e genpolicy
genpolicy: recreate env because env type changed from
{'name': 'genconfig', 'type': 'VirtualEnvRunner'} to
{'name': 'genpolicy', 'type': 'VirtualEnvRunner'}
~~~
According to the maintainer of tox, this functionality is not intended
to be supported.
https://github.com/tox-dev/tox/issues/425#issuecomment-1011944293
There is a note not to remove the option for bandit integration job but
the note is no longer valid since the functionality itself no longer
works with recent tox.
Change-Id: I2ba52a9ecc5cc24dee98837e0513897cba2eb046

Add mypy so we can run these checks as part of our linting step. Note
that mypy is the last item in the list of checks since flake8 is going
to catch e.g. syntax issues which should be addressed first.
This also allows us to drop use of mypywrap: rather than wrapping mypy,
we can make use of mypy's configuration to specify the files we wish to
run checks on.
Change-Id: Ie51401ead4e31ccd70ab833c59cb397f802ff188
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

pre-commit is an effective linter runner. No need to duplicate things.
We are able to remove our homegrown "is this valid JSON" check by using
the one built into pre-commit. Beautiful!
Change-Id: I89398d4a1c5d4e7205f5f9c499cba0b4ff90fa46
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This chnage adds the pre-commit config and
tox targets to run codespell both indepenetly
and via the pep8 target.
This change correct all the final typos in the
codebase as detected by codespell.
Change-Id: Ic4fb5b3a5559bc3c43aca0a39edc0885da58eaa2

We leak due to running background operations like server create, rebuild
and in one case a sleeping claim. So this patch removes the leaks by
making sure that the background operations stop before the test
finishes.
As there is no more leak in functional test too this patch makes the
leak an error there too.
Change-Id: I6905999050e8d09b772837034a212c534e9c3226

This change add a global greenpool which is used to manage
the greenthreads created via nova.utils.spawn(_n).
A test fixture is also added to use an isolated greenpool
which will raise an exception if a greenthread is leaked.
the fixture will optionally raise if greenlets are leaked.
This is enabled by unit test by default and is configurable
for functional tests.
This change removes all greenthread leaks from the unit
and functional tests that were detected. 7 functional
tests still leak greenlets but they have no obvious
cause. as such greenlet leaks are not treated as errors
for funtional tests by default. Greenthread leaks
are always treated as errors.
Set NOVA_RAISE_ON_GREENLET_LEAK=1|true|yes when invoking
tox to make greenlet leaks an error for functional tests.
Change-Id: I73b4684744b340bfb80da08537a745167ddea106

When the upper constraint usage moved to install_command in tox.ini, it
was not removed from docs target's deps list, where it's now not
necessary, and even could cause issues when overriding
TOX_CONSTRAINTS_FILE.
This patch replaces the old UPPER_CONSTRAINTS_FILE to the "new"
TOX_CONSTRAINTS_FILE in the install_command, too.
Change-Id: I9c27caa869d72e06e84bcf84ceedb586ed3bc8e9

This makes us able to run functional on python 3.11. Without this,
tox will happily (and silently) run the default venv, which is unit
tests.
Change-Id: I544a29ae78814f9a454daba8c1978f7ab2c2505c

Nova functional tests import the placement test fixture
and which require the new fix change merged in placement
- https://review.opendev.org/c/openstack/placement/+/869525
the above change is released in openstack-placement 9.0.0.0b1
version
- https://review.opendev.org/c/openstack/releases/+/870989
bumping that version in tox.ini as functional test env use
the placement from PyPi.
Change-Id: Ib0217f85f9d2782f1a742b19f6ec56a082946a2b

It seems that with tox 4.2.6 the missing interpreter error was fixed but
the generative testenv feature is broken and the
[testenv:functional{,-py38,-py39,-py310}] format is leads to missing
interpreter error. It is due to a conflict between basepython = python3
and the version fragment in the generative target suppressed by
ignore_basepython_conflict = true.
This patch removes basepython = python3 assuming that developers already
switched for python3 in their environment as python2.7 is EOL.
Also we took the opportunity to add the global constraints via the
install_command instead of deps as deps is not used during the
installation of the editable package.
Change-Id: I258a7c13434b29402804181dea275b42d5539df0

As per 2023.1 testing runtime[1], we need to test on Ubuntu
Jammy (which will be taken care by tempest and devstack patches
to move base jobs to Jammy) and at least single job to run on
Ubutnu Focal (for smooth upgrade). Also, python 3.10 testing is
voting now.
This commit adds a new job to run on focal which can be removed
in future cycle when testing runtime drop the requirement of Focal
testing. Also, make python 3.10 functional and unit test job as voting
(openstack-tox-py310 is running as part of generic template so we do
not need to explicitly add that)
[1] https://governance.openstack.org/tc/reference/runtimes/2023.1.html
Change-Id: Ia43f73dba00b0b5932939bcc7d11b97a83072ee3

Add the following hacking rule.
* N372: Don't use the setDaemon method.
 Use the daemon attribute instead.
Change-Id: Idb45421205f76d2d3b0576bd0504d261ed249edd
Related-Bug: 1987191
Signed-off-by: Takashi Natsume <takanattie@gmail.com>

This change extends the guest xml parsing such that
the source device path can be extreacted from interface
elements of type vdpa.
This is required to identify the interface to remove when
detaching a vdpa port from a domain.
This change fixes a latent bug in the libvirt fixutre
related to the domain xml generation for vdpa interfaces.
Change-Id: I5f41170e7038f4b872066de4b1ad509113034960

The PowerVM driver was deprecated in November 2021 as part of change
Icdef0a03c3c6f56b08ec9685c6958d6917bc88cb. As noted there, all
indications suggest that this driver is no longer maintained and may be
abandonware. It's been some time and there's still no activity here so
it's time to abandon this for real.
This isn't as tied into the codebase as the old XenAPI driver was, so
removal is mostly a case of deleting large swathes of code. Lovely.
Change-Id: Ibf4f36136f2c65adad64f75d665c00cf2de4b400
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

Since we dropped support for python 2 [1], we no longer need to use the
mock library, which existed to backport py3 functionality into py2.
Change Ib44b5bff657c8e76c4f701e14d51a4efda3f6d32 cut over to importing
the stock mock, which must be done by saying::
 from unittest import mock
...because if you say::
 import mock
...you will be using the third party mock library instead, which may or
may not be installed.
This commit adds hacking check N371 to enforce the former.
[1] https://review.opendev.org/#/c/687954/
Change-Id: I71439580e80d33cff62aba807df2b35164a47cbe

These are currently non-voting since we don't care about this stuff for
Zed. It does get us ready for a 3.10-having future, however.
Change-Id: I7740dafd6523eca27fa4e725d7eaf8558e434779
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

As discussed in TC PTG[1] and TC resolution[2], we are
dropping the lower-constraints.txt file and its testing.
We will keep lower bounds in the requirements.txt file but
with a note that these are not tested lower bounds and we
try our best to keep them updated.
[1] https://etherpad.opendev.org/p/tc-zed-ptg#L326
[2] https://governance.openstack.org/tc/resolutions/20220414-drop-lower-constraints.html#proposal
Change-Id: Ifbc383f6d4c858189cee55e67b4d4274d9c3358a

When tox 'docs' target is called, first it installs the dependencies
(listed in 'deps') in 'installdeps' phase, then it installs nova (with
its requirements) in 'develop-inst' phase. In the latter case 'deps' is
not used so that the constraints defined in 'deps' are not used.
This could lead to failures on stable branches when new packages are
released that break the build. To avoid this, the simplest solution is
to pre-install requirements, i.e. add requirements.txt to 'docs' tox
target.
Change-Id: I4471d4488d336d5af0c23028724c4ce79d6a2031

Spotted this in a review recently. We don't want people using six
anymore.
Change-Id: Ie107a95bc06390ab519d3b3af9b07103a9a14316
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

Install these via tox.ini when required. Note that we don't add them to
'test-requirements.txt' since packagers consume those and would be
forced to package these dependencies or modify this file.
A small bug in the tox file is corrected: you can't share an environment
directory if the dependencies in that directory are different. As such,
pep8 must go in its own directory again, not the 'shared' directory.
Change-Id: Iffb9ff2e300213ec01cccaf3b967e73331c9c9ff
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This is a follow up change to I168fffac8002f274a905cfd53ac4f6c9abe18803
which added a hackaround to enable our tests to pass with
fasteners>=0.15 which was upgraded recently as part of a
openstack/requirements update.
The ReaderWriterLock from fasteners (and thus lockutils) cannot work
correctly with eventlet patched code, so this adds a wrapper containing
the aforementioned hackaround along with a hacking check to do our best
to ensure that future use of ReaderWriterLock will be through the
wrapper.
Change-Id: Ia7bcb40a21a804c7bc6b74f501d95ce2a88b09b5

Yoga testing runtime is updated now
- https://governance.openstack.org/tc/reference/runtimes/yoga.html
which needs to test py38 and py39. Unit tests update are
handled by the job template change in openstack-zuul-job and
this commit makes changes to fucntional job to run py39 as voting
and updating the metdata in setup file.
Change-Id: I314fd61f20f2c3551f6231d191b7dcaaefabd2b5

Well, sort of. We enable them but immediately filter out the ones we're
actually seeing, the rationale being that we can address these in a
piecemeal fashion without the risk of introducing new issues.
There's a lot more to be done here. However, the work done in oslo.db
[1] should provide a guide for how to resolve the outstanding issues.
[1] https://review.opendev.org/q/topic:%2522sqlalchemy-20%2522+project:openstack/oslo.db
Change-Id: Iafe726d3819031c357460fd131bb2bb58babb4e2
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

autopep8 is a code formating tool that makes python code pep8
compliant without changing everything. Unlike black it will
not radically change all code and the primary change to the
existing codebase is adding a new line after class level doc strings.
This change adds a new tox autopep8 env to manually run it on your
code before you submit a patch, it also adds autopep8 to pre-commit
so if you use pre-commit it will do it for you automatically.
This change runs autopep8 in diff mode with --exit-code in the pep8
tox env so it will fail if autopep8 would modify your code if run
in in-place mode. This allows use to gate on autopep8 not modifying
patches that are submited. This will ensure authorship of patches is
maintianed.
The intent of this change is to save the large amount of time we spend
on ensuring style guidlines are followed automatically to make it
simpler for both new and old contibutors to work on nova and save
time and effort for all involved.
Change-Id: Idd618d634cc70ae8d58fab32f322e75bfabefb9d

We have placement-nova-tox-functional-py38 job defined and run
on placement gate[1] to run the nova functional test excluding
api and notification _sample_tests, and db-related tests but that
job skip those tests via tox_extra_args which is not right way
to do as we currently facing error when tox_extra_args is included
in tox siblings task
- c02c28a982
- https://zuul.openstack.org/build/a8c186b2c7124856ae32477f10e2b9a4
Let's define a new tox env which can exclude the required test
in stestr command itself.
[1] bd5b19c00e/.zuul.yaml (L83)
Change-Id: I20d6339a5203aed058f432f68e2ec1af57030401

As the bug and fix If71620e808744736cb4fe3abda76d81a6335311b showed
it is dangerous to forget instantiating the Mock class before it is
used in the test as changes on the class directly leaks out from the
test. In almost all the cases using Mock class directly is a bug and the
author original intention is to use an instance instead, just forgot
about the parents. So this patch adds two new hacking rules:
N367: catches the case when Mock class is aliased in the test:
 self.mock_mystuff = mock.Mock
N368: catches when mock.patch instructed to use the Mock class as
replacement value during patching:
 mock.patch('Bar.foo', new=mock.Mock)
For N367 the previous patch removed the last hit. For N368 this patch
removes the two hits exists.
Change-Id: Id42ca571b1569886ef47aa350369e9d2068e77bc
Related-Bug: #1936849 

Some things that were missed in previous patches and are thrown together
here:
- Add alembic as an explicit dependency (we were getting it transitively
 from oslo.db). We also bump the sqlalchemy dependency to a 1.4.x
 release, which is the minimum supported by our chosen version of
 alembic (more on this below)
- Remove tooling related to the old migrations
- Fix the tox whitelisting of the flaky MySQL tests
On the SQLAlchemy front, we opt for 1.4.13. Technically alembic should
support anything from 1.4.0, however, with SQLAlchemy >= 1.4.0, < 1.4.13
we see errors like the following in some tests:
 sqlalchemy.exc.InvalidRequestError: Entity namespace for
 "count(instance_mappings.id)" has no property "queued_for_delete"
There's nothing specific about this in the release notes for 1.4.13 [1]
but it definitely fixes things.
[1] https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.13
Change-Id: I4c8eb13f11aa7471c26a5ba326319aef245c9836
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This reverts commit 89e321b7b5.
Reason for revert: The previous change require develper to install multiple spcific version python 
locally to run the test envs and it arbitrally pins lower constratins to the 
oldest python we support but lower contratis shoudl work with the newwst python we support too.
on fedora or other operating systems that ship with default python version that exceed our supported
versions test can be run with tox -e py38 if required.
similarly our updated constratins shoudl work with the older python.
Change-Id: I358ff8128746c9d0c584a03381e55817c4aace3f

Declare specific versions of Python for environments that don't
specifically encode a version. We use the highest version officially
supported, Python 3.8, for most environments except lower-constraints,
which uses the lowest version, Python 3.6. This is necessary for users
on Fedora, where the default Python version is Python 3.9.
This will unfortunately introduce a small amount of busy work whenever
we change supported Python versions, as tox will need to be updated, but
that's a small price to pay for the usability wins that Fedora users
will see.
Change-Id: I240d5b8aa3eb1925b97af57e2644be61bfc106c2
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Depends-On: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/801773 

This currently runs in the 'check' pipeline, as part of the pep8 job,
which causes otherwise perfectly valid backports to report as failing
CI. There's no reason a stable core shouldn't be encouraged to review
these patches: we simply want to prevent them *merging* before their
parent(s). Resolve this conflict by moving the check to separate voting
job in the 'gate' pipeline as well as a non-voting job in the 'check'
pipeline to catch more obvious issues.
Change-Id: Id3e4452883f6a3cf44ff58b39ded82e882e28c23
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This reverts commit 531fa4cd04.
Change-Id: Ia095a9ace86694fd431f2a1c8f59516ed1eb0d7e

Just removing mypy and then we will revert this patch once
I035d6dc752eaa83105cc12797765b304d843e1f7 is merged.
Change-Id: Iaa02e9d35419bf7100cef1014217210c1d5b6855

The patch bumps min version of tox to 3.18.0 in order to
replace whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020年07月23日
Change-Id: Idb189fc46d729dabe609cc39ec88d278d384424d
Signed-off-by: Takashi Natsume <takanattie@gmail.com>

I4f551dc4b57905cab8aa005c5680223ad1b57639 introduced the environment
variable to disable the check-cherry-pick.sh script but forgot to allow
it to be passed into the pep8 tox env.
Change-Id: Ie8a672fd21184c810bfe9c0e3a49582189bf2111

This significantly speeds up our doc build process. This requires a
newer version of 'sphinx-feature-classification' and some tweaks to our
own in-tree extensions. While we're here, we drop the '-d DOCTREE_DIR'
parameter since it's of no use when we blast away our previously built
docs each time we build.
Change-Id: I679da65d44c40880f720df8a2f06286a19eb8d22
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

Introduce API microversion 2.88, which makes the following changes to
a number of 'os-hypervisors'. Specifically, the following fields are
dropped from both the '/os-hypervisors/detail' (detailed list) and
'/os-hypervisors/{hypervisor_id}' (show) APIs:
- current_workload
- cpu_info
- vcpus
- vcpus_used
- free_disk_gb
- local_gb
- local_gb_used
- disk_available_least
- free_ram_mb
- memory_mb
- memory_mb_used
- running_vms
In addition, the '/os-hypervisors/statistics' API, which provided a
summary of the above stats but for all hypervisors in the deployment, is
dropped entirely.
Finally, the '/os-hypervisors/{hypervisor}/uptime' API, which provided a
similar response to the '/os-hypervisors/{hypervisor}' API but with an
additional 'uptime' field, has been removed in favour of including this
field in the primary '/os-hypervisors/{hypervisor}' API.
A small tweak to 'tox.ini' that allows us to share some venvs is
included.
Part of blueprint modernize-os-hypervisors-api
Change-Id: I515e484ade6c6455f82a3067940a418a0d7d965a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

In a heavily IO deprived CI VM the db migration tests could take a
significant amount of time and eventually time out. This patch moves the
tests into the same test executor worker process to spread the load
generated by these test in time until a final solution is found. For
example we hope that [1] will help eventually to decrease the load.
[1] https://review.opendev.org/q/topic:bp/compact-db-migrations-wallaby
Change-Id: I6ce930fa86c82da1008089791942b1fff7d04c18
Related-Bug: #1823251 

Nova does not run gabbi test since placement is moved to a separate
git repository. So the gabbi related tox.ini comment is removed.
Change-Id: Ic324e3e32fa03478895b32fa583e805ee6c721e2

This patch adds both a tox target and a zuul job to run functional tests
with python3.9
Depends-On: https://review.opendev.org/760932
Change-Id: I672904e9bfb45a66a82331063c7d49c4bc0439df