openstack/nova - nova - OpenDev: Free Software Needs Free Tools

This patch switches the default concurrency mode to native threading
for the services that gained native threading support in Flamingo:
nova-scheduler, nova-api, and nova-metadata.
The OS_NOVA_DISABLE_EVENTLET_PATCHING env variable still can be used to
explicitly switch the concurrency mode to eventlet by
 OS_NOVA_DISABLE_EVENTLET_PATCHING=false
We also ensure that the cover, docs, py3xx and functional tox targets
are still running with eventlet while py312-threading kept running
with native threading.
Change-Id: I86c7f31f19ca3345218171f0abfa8ddd4f8fc7ea
Signed-off-by: Balazs Gibizer <gibi@redhat.com>

We have a centralized nova.utils.cooperative_yield() instead of
time.sleep(0). It is better as it allows to turn off the sleep calls
when the service runs in threaded mode.
Change-Id: I625daec79ee5b7f8b92116f450e21f997cef0546
Signed-off-by: Balazs Gibizer <gibi@redhat.com>

The py312-threading tox target will run the currently working unit tests
with threading mode. We have an exclude list, those tests are
failing or hanging. Also the current test list might still have unstable
tests.
This also adds a non voting zuul job to run the new target.
Change-Id: Ibf41fede996fbf2ebaf6ae83df8cfde35acb2b7e
Signed-off-by: Balazs Gibizer <gibi@redhat.com>

The end goals is to be able to run at least some of the unit tests
without eventlet. But there are things preventing that for now.
We need to make sure that the oslo.sevice backed is not initialized to
eventlet by any early import code before our monkey_patch module can do
the selective backed selection based on the env variable.
The nova.tests.unit module had some import time code execution that is
forcing imports that initialize the oslo.service backend too early,
way before nova would do it in normal execution. We could remove
objects.register_all() from nova/tests/unit/__init__.py as it seems
tests are passing without it. Still that would not be enough so I
eventually decide to keep it.
The other issue is that the unit test discovery imports all modules
under nova.tests.unit and that eventually imports oslo.messaging and
that also forces oslo.service backend selection.
So we injected an early call to our smart monkey_patch module to preempt
that. This does not change the imported modules as monkey_patch module
imported anyhow via nova.test module. Just changed the order to allow
oslo.service backend selection explicitly.
After this patch the unit test can be run via
 OS_NOVA_DISABLE_EVENTLET_PATCHING=true tox -e py312
Most of the test will pass but there are a bunch of test timing out or
hanging.
Change-Id: I210cb6a30deaee779d55f88f0f57584c65b0dc05
Signed-off-by: Balazs Gibizer <gibi@redhat.com>

OpenStack recently dropped python3.9 support for global requirements[1]
as it was removed from the Flamingo supported runtimes[2].
So this patch removes 3.9 support from nova too.
[1]https://review.opendev.org/c/openstack/requirements/+/948285
[2]https://governance.openstack.org/tc/reference/runtimes/2025.2.html#python
Change-Id: I8aea971d7972959c32d5175926cbaddb21839f8e

Add a new hacking check to prevent adding Eventlet concurrency
primitives where the stdlib one is equivalent.
Right now only eventlet/greenthread.sleep is checked but as we progress
with the transformation we will add more primitives.
Change-Id: I1310bb7a8c8133364060eb8814226cab52219171

... because Python 3.8 is no longer supported.
Change-Id: I0a8d26c47acd98d39e609e93da5817a9a37a1031

This environment was used by SQLAlchemy 1.4 and is no longer necessary
since SQLAlchemy was bumped to 2.0 .
Change-Id: If27f720f2f3c0ee89b2c43bd206429b9398895e1

This change enable the tox -e docs env to work for python
installed using the nix package manager.
Change-Id: I84d865e00979e84abc541ccbbc7c0ed43fcb8357

change I6de86f3e3e283ba404f927ea4c8164f791df3989
added the py312 funtional job definition but did not
update the tox.ini to define it.
As a result it is runing the unit tests not the functional tests.
This change simpley corrects that.
Change-Id: Id6ee76e0190469ac09baf0bc56a9022317c6f881

This change mainly fixes incorrect use of backticks
but also adress some other minor issues like unbalanced
backticks, incorrect spacing or missing _ in links.
This change add a tox target to run sphinx-lint
as well as adding it to the relevent tox envs to enforce
it in ci. pre-commit is leveraged to install and execute
sphinx-lint but it does not reqiure you to install the
hooks locally into your working dir.
Change-Id: Ib97b35c9014bc31876003cef4362c47a8a3a4e0e

Nova Hyper-V driver is not tested in OpenStack upstream and no maintianers.
This driver has been marked as deprecated in Antelope release. It has dependency
on the OpenStack Winstacker project which has been retired[1].
As discussed in vPTG[2], removing the HyperV driver, tests, and its config.
[1] https://review.opendev.org/c/openstack/governance/+/886880
[2] https://etherpad.opendev.org/p/nova-caracal-ptg#L301
Change-Id: I568c79bae9b9736a20c367096d748c730ed59f0e

tox now always recreates an env although the env is shared using envdir
options.
~~~
$ tox -e genpolicy
genpolicy: recreate env because env type changed from
{'name': 'genconfig', 'type': 'VirtualEnvRunner'} to
{'name': 'genpolicy', 'type': 'VirtualEnvRunner'}
~~~
According to the maintainer of tox, this functionality is not intended
to be supported.
https://github.com/tox-dev/tox/issues/425#issuecomment-1011944293
There is a note not to remove the option for bandit integration job but
the note is no longer valid since the functionality itself no longer
works with recent tox.
Change-Id: I2ba52a9ecc5cc24dee98837e0513897cba2eb046

Add mypy so we can run these checks as part of our linting step. Note
that mypy is the last item in the list of checks since flake8 is going
to catch e.g. syntax issues which should be addressed first.
This also allows us to drop use of mypywrap: rather than wrapping mypy,
we can make use of mypy's configuration to specify the files we wish to
run checks on.
Change-Id: Ie51401ead4e31ccd70ab833c59cb397f802ff188
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

pre-commit is an effective linter runner. No need to duplicate things.
We are able to remove our homegrown "is this valid JSON" check by using
the one built into pre-commit. Beautiful!
Change-Id: I89398d4a1c5d4e7205f5f9c499cba0b4ff90fa46
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This chnage adds the pre-commit config and
tox targets to run codespell both indepenetly
and via the pep8 target.
This change correct all the final typos in the
codebase as detected by codespell.
Change-Id: Ic4fb5b3a5559bc3c43aca0a39edc0885da58eaa2

We leak due to running background operations like server create, rebuild
and in one case a sleeping claim. So this patch removes the leaks by
making sure that the background operations stop before the test
finishes.
As there is no more leak in functional test too this patch makes the
leak an error there too.
Change-Id: I6905999050e8d09b772837034a212c534e9c3226

This change add a global greenpool which is used to manage
the greenthreads created via nova.utils.spawn(_n).
A test fixture is also added to use an isolated greenpool
which will raise an exception if a greenthread is leaked.
the fixture will optionally raise if greenlets are leaked.
This is enabled by unit test by default and is configurable
for functional tests.
This change removes all greenthread leaks from the unit
and functional tests that were detected. 7 functional
tests still leak greenlets but they have no obvious
cause. as such greenlet leaks are not treated as errors
for funtional tests by default. Greenthread leaks
are always treated as errors.
Set NOVA_RAISE_ON_GREENLET_LEAK=1|true|yes when invoking
tox to make greenlet leaks an error for functional tests.
Change-Id: I73b4684744b340bfb80da08537a745167ddea106

When the upper constraint usage moved to install_command in tox.ini, it
was not removed from docs target's deps list, where it's now not
necessary, and even could cause issues when overriding
TOX_CONSTRAINTS_FILE.
This patch replaces the old UPPER_CONSTRAINTS_FILE to the "new"
TOX_CONSTRAINTS_FILE in the install_command, too.
Change-Id: I9c27caa869d72e06e84bcf84ceedb586ed3bc8e9

This makes us able to run functional on python 3.11. Without this,
tox will happily (and silently) run the default venv, which is unit
tests.
Change-Id: I544a29ae78814f9a454daba8c1978f7ab2c2505c

Nova functional tests import the placement test fixture
and which require the new fix change merged in placement
- https://review.opendev.org/c/openstack/placement/+/869525
the above change is released in openstack-placement 9.0.0.0b1
version
- https://review.opendev.org/c/openstack/releases/+/870989
bumping that version in tox.ini as functional test env use
the placement from PyPi.
Change-Id: Ib0217f85f9d2782f1a742b19f6ec56a082946a2b

It seems that with tox 4.2.6 the missing interpreter error was fixed but
the generative testenv feature is broken and the
[testenv:functional{,-py38,-py39,-py310}] format is leads to missing
interpreter error. It is due to a conflict between basepython = python3
and the version fragment in the generative target suppressed by
ignore_basepython_conflict = true.
This patch removes basepython = python3 assuming that developers already
switched for python3 in their environment as python2.7 is EOL.
Also we took the opportunity to add the global constraints via the
install_command instead of deps as deps is not used during the
installation of the editable package.
Change-Id: I258a7c13434b29402804181dea275b42d5539df0

As per 2023.1 testing runtime[1], we need to test on Ubuntu
Jammy (which will be taken care by tempest and devstack patches
to move base jobs to Jammy) and at least single job to run on
Ubutnu Focal (for smooth upgrade). Also, python 3.10 testing is
voting now.
This commit adds a new job to run on focal which can be removed
in future cycle when testing runtime drop the requirement of Focal
testing. Also, make python 3.10 functional and unit test job as voting
(openstack-tox-py310 is running as part of generic template so we do
not need to explicitly add that)
[1] https://governance.openstack.org/tc/reference/runtimes/2023.1.html
Change-Id: Ia43f73dba00b0b5932939bcc7d11b97a83072ee3

Add the following hacking rule.
* N372: Don't use the setDaemon method.
 Use the daemon attribute instead.
Change-Id: Idb45421205f76d2d3b0576bd0504d261ed249edd
Related-Bug: 1987191
Signed-off-by: Takashi Natsume <takanattie@gmail.com>

This change extends the guest xml parsing such that
the source device path can be extreacted from interface
elements of type vdpa.
This is required to identify the interface to remove when
detaching a vdpa port from a domain.
This change fixes a latent bug in the libvirt fixutre
related to the domain xml generation for vdpa interfaces.
Change-Id: I5f41170e7038f4b872066de4b1ad509113034960

The PowerVM driver was deprecated in November 2021 as part of change
Icdef0a03c3c6f56b08ec9685c6958d6917bc88cb. As noted there, all
indications suggest that this driver is no longer maintained and may be
abandonware. It's been some time and there's still no activity here so
it's time to abandon this for real.
This isn't as tied into the codebase as the old XenAPI driver was, so
removal is mostly a case of deleting large swathes of code. Lovely.
Change-Id: Ibf4f36136f2c65adad64f75d665c00cf2de4b400
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

Since we dropped support for python 2 [1], we no longer need to use the
mock library, which existed to backport py3 functionality into py2.
Change Ib44b5bff657c8e76c4f701e14d51a4efda3f6d32 cut over to importing
the stock mock, which must be done by saying::
 from unittest import mock
...because if you say::
 import mock
...you will be using the third party mock library instead, which may or
may not be installed.
This commit adds hacking check N371 to enforce the former.
[1] https://review.opendev.org/#/c/687954/
Change-Id: I71439580e80d33cff62aba807df2b35164a47cbe

These are currently non-voting since we don't care about this stuff for
Zed. It does get us ready for a 3.10-having future, however.
Change-Id: I7740dafd6523eca27fa4e725d7eaf8558e434779
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

As discussed in TC PTG[1] and TC resolution[2], we are
dropping the lower-constraints.txt file and its testing.
We will keep lower bounds in the requirements.txt file but
with a note that these are not tested lower bounds and we
try our best to keep them updated.
[1] https://etherpad.opendev.org/p/tc-zed-ptg#L326
[2] https://governance.openstack.org/tc/resolutions/20220414-drop-lower-constraints.html#proposal
Change-Id: Ifbc383f6d4c858189cee55e67b4d4274d9c3358a

When tox 'docs' target is called, first it installs the dependencies
(listed in 'deps') in 'installdeps' phase, then it installs nova (with
its requirements) in 'develop-inst' phase. In the latter case 'deps' is
not used so that the constraints defined in 'deps' are not used.
This could lead to failures on stable branches when new packages are
released that break the build. To avoid this, the simplest solution is
to pre-install requirements, i.e. add requirements.txt to 'docs' tox
target.
Change-Id: I4471d4488d336d5af0c23028724c4ce79d6a2031

Spotted this in a review recently. We don't want people using six
anymore.
Change-Id: Ie107a95bc06390ab519d3b3af9b07103a9a14316
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

Install these via tox.ini when required. Note that we don't add them to
'test-requirements.txt' since packagers consume those and would be
forced to package these dependencies or modify this file.
A small bug in the tox file is corrected: you can't share an environment
directory if the dependencies in that directory are different. As such,
pep8 must go in its own directory again, not the 'shared' directory.
Change-Id: Iffb9ff2e300213ec01cccaf3b967e73331c9c9ff
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This is a follow up change to I168fffac8002f274a905cfd53ac4f6c9abe18803
which added a hackaround to enable our tests to pass with
fasteners>=0.15 which was upgraded recently as part of a
openstack/requirements update.
The ReaderWriterLock from fasteners (and thus lockutils) cannot work
correctly with eventlet patched code, so this adds a wrapper containing
the aforementioned hackaround along with a hacking check to do our best
to ensure that future use of ReaderWriterLock will be through the
wrapper.
Change-Id: Ia7bcb40a21a804c7bc6b74f501d95ce2a88b09b5

Yoga testing runtime is updated now
- https://governance.openstack.org/tc/reference/runtimes/yoga.html
which needs to test py38 and py39. Unit tests update are
handled by the job template change in openstack-zuul-job and
this commit makes changes to fucntional job to run py39 as voting
and updating the metdata in setup file.
Change-Id: I314fd61f20f2c3551f6231d191b7dcaaefabd2b5

Well, sort of. We enable them but immediately filter out the ones we're
actually seeing, the rationale being that we can address these in a
piecemeal fashion without the risk of introducing new issues.
There's a lot more to be done here. However, the work done in oslo.db
[1] should provide a guide for how to resolve the outstanding issues.
[1] https://review.opendev.org/q/topic:%2522sqlalchemy-20%2522+project:openstack/oslo.db
Change-Id: Iafe726d3819031c357460fd131bb2bb58babb4e2
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

autopep8 is a code formating tool that makes python code pep8
compliant without changing everything. Unlike black it will
not radically change all code and the primary change to the
existing codebase is adding a new line after class level doc strings.
This change adds a new tox autopep8 env to manually run it on your
code before you submit a patch, it also adds autopep8 to pre-commit
so if you use pre-commit it will do it for you automatically.
This change runs autopep8 in diff mode with --exit-code in the pep8
tox env so it will fail if autopep8 would modify your code if run
in in-place mode. This allows use to gate on autopep8 not modifying
patches that are submited. This will ensure authorship of patches is
maintianed.
The intent of this change is to save the large amount of time we spend
on ensuring style guidlines are followed automatically to make it
simpler for both new and old contibutors to work on nova and save
time and effort for all involved.
Change-Id: Idd618d634cc70ae8d58fab32f322e75bfabefb9d

We have placement-nova-tox-functional-py38 job defined and run
on placement gate[1] to run the nova functional test excluding
api and notification _sample_tests, and db-related tests but that
job skip those tests via tox_extra_args which is not right way
to do as we currently facing error when tox_extra_args is included
in tox siblings task
- c02c28a982
- https://zuul.openstack.org/build/a8c186b2c7124856ae32477f10e2b9a4
Let's define a new tox env which can exclude the required test
in stestr command itself.
[1] bd5b19c00e/.zuul.yaml (L83)
Change-Id: I20d6339a5203aed058f432f68e2ec1af57030401

As the bug and fix If71620e808744736cb4fe3abda76d81a6335311b showed
it is dangerous to forget instantiating the Mock class before it is
used in the test as changes on the class directly leaks out from the
test. In almost all the cases using Mock class directly is a bug and the
author original intention is to use an instance instead, just forgot
about the parents. So this patch adds two new hacking rules:
N367: catches the case when Mock class is aliased in the test:
 self.mock_mystuff = mock.Mock
N368: catches when mock.patch instructed to use the Mock class as
replacement value during patching:
 mock.patch('Bar.foo', new=mock.Mock)
For N367 the previous patch removed the last hit. For N368 this patch
removes the two hits exists.
Change-Id: Id42ca571b1569886ef47aa350369e9d2068e77bc
Related-Bug: #1936849 

Some things that were missed in previous patches and are thrown together
here:
- Add alembic as an explicit dependency (we were getting it transitively
 from oslo.db). We also bump the sqlalchemy dependency to a 1.4.x
 release, which is the minimum supported by our chosen version of
 alembic (more on this below)
- Remove tooling related to the old migrations
- Fix the tox whitelisting of the flaky MySQL tests
On the SQLAlchemy front, we opt for 1.4.13. Technically alembic should
support anything from 1.4.0, however, with SQLAlchemy >= 1.4.0, < 1.4.13
we see errors like the following in some tests:
 sqlalchemy.exc.InvalidRequestError: Entity namespace for
 "count(instance_mappings.id)" has no property "queued_for_delete"
There's nothing specific about this in the release notes for 1.4.13 [1]
but it definitely fixes things.
[1] https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.13
Change-Id: I4c8eb13f11aa7471c26a5ba326319aef245c9836
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This reverts commit 89e321b7b5.
Reason for revert: The previous change require develper to install multiple spcific version python 
locally to run the test envs and it arbitrally pins lower constratins to the 
oldest python we support but lower contratis shoudl work with the newwst python we support too.
on fedora or other operating systems that ship with default python version that exceed our supported
versions test can be run with tox -e py38 if required.
similarly our updated constratins shoudl work with the older python.
Change-Id: I358ff8128746c9d0c584a03381e55817c4aace3f