openstack/openstack-ansible-os_nova - openstack-ansible-os_nova - OpenDev: Free Software Needs Free Tools

qemu-system on debian derivative OS is a meta-package which installs
qemu-system-* for all architecures understood by qemu.
This is different from redhat type OS where the qemu-kvm package
installed with dnf only installs the qemu-system-* binary matching
the host architecture.
This gives two problems, first there is inconsistency in openstack-ansible
deployments between redhat and debian OS. Second, there is a potentially
unexpected emulation of architectures when launching VM on a cloud
with a mix of compute architectures when a full set of qemu-system-*
binaries is available on a compute node. The compute node becomes a
candidate for scheduling any of the supported architectures and a
very specific configuration is needed both from the operator and end
user to ensure that VM are run on a native architecture or emulated as
required.
This patch changes the installation so that redhat and debian compute nodes
only have the native qemu-system binary installed.
A new feature should be introduced to openstack-ansible in the future
to explicitly control installation of non-native qemu-system-* binaries
and write the config options for controlling emulation.
Change-Id: I1c876c7968efb7f24880f1a6e96ba6b7264ddc94

RDO packages for nova does depend on python3-openvswitch,
which makes it required to install OVS on computes regardless
of everything else.
We also clean out pre-rhel9 variable files as they're not needed anymore
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/872896
Change-Id: I3e31254b7dd1c0ff3cb46153cefce6f6cadd52aa

At the moment we don't provide any option rather then use memcached
backend. With that we also hardocde list of packages that should be
installed inside virtualenv for selected backend.
Adding bmemcached requirement to oslo_cache.memcache_pool [1] gives us
opportunity to refactor this bit of deployment and allow to be more
flexible in backend selection and requirements installation for it.
[1] https://review.opendev.org/c/openstack/oslo.cache/+/854628
Change-Id: I48e193ef29e56aa8639511c5b5dcddc70f5e1198

This configuration option has been observed to result in file
descriptor leaks in certain circumstances. A variable is added
here so that it can be easily overridden.
Change-Id: I7de034307da9352e6f5d1f5f175a330fb8c86463
Related-Bug: #1961603 

libvirtd.socket does monitor libvirtd.service and trigger service restart
when it spot that service is down.
However in order to enable tcp and tls sockets, we need libvirt
to be stopped.
Currently race condition can happen, when we stop libvirt, but it's
started by socket before we enable tls one.
To overcome this we stop socket along with service.
Change-Id: Iacc093311036fb8d6559a0e32252579303a639ba

Change-Id: I7d09f6f5f5d66ecbe29fd3969d586eb416c98589

After qemu has been updated in osbpo repo, extra requirement has
appeared, that is available only from backports repo.So we add it
to nova_backports_packages and limit apt_package_pinning
only to Debian Buster.
Change-Id: I284fbd7f8587886502ecc54adfe7314fb80967fd

When Nova and/or Cinder are using Ceph as backend, qemu will need
to open a connection and two threads for each and every Ceph OSD.
Since all connections occur in the same qemu process, this may result
in hitting default max open files limit.
Thus in case of more then 10 volumes are attached
to the same instance and are used actively can end up in blocked
IO operations inside VMs.
We increase these limits by default when RBD backend is used.
Change-Id: Ib3081280cdbae1eb2235083c95c27e2efd0b413e

All references to Gentoo, SUSE, Debian stretch and Centos-7 are removed.
Conditional tasks, ternary operators and variables are simplified where possible
OS specific variables files are generalised where possible
Change-Id: I4a68549bf85fd322ea344139869916aae3275377

See https://github.com/ansible/ansible/issues/73654
Change-Id: I3cf2a30e0929835a84f0502bc4e87522b688b538

Some extra packages needs to be installed from Debian updates
repository, so we pin them with higher priority.
Change-Id: I49c75dd11d6c4e8d37fe013b7ffdfd56ff193fcd

We use the same condition, which defines against what host some "service"
tasks should run against, several times. It's hard to keep it the same
across the role and ansible spending additional resources to evaluate
it each time, so it's simpler and better for the maintenance to set
a boolean variable which will say for all tasks, that we want to run
only against signle host, if they should run or not now.
Change-Id: I78d44704a0224b2c73832a607e84ea8d9499f8c2

Change-Id: If5f5640b80c62fa249a0706b6f64d751b04293a6

Add 'nova_scheduler_extra_filters' to allow operators to enable
additional scheduler filters without overriding the entire list of
filters as in [0].
This also reduces the burden on ops to maintain the list of overridden
default schedulers because of things like [1].
[0] 3886dbc40d/openstack_deploy/user_variables.yml (L51-L69)
[1] https://review.openstack.org/#/c/596502/
Change-Id: I9ab6bcbef2b496df7f6ecf11a7d8f5f7891aeeca

libvirt-python 6.7.0 makes nova-compute with [1]
Since urls can't be used with constraints, we're bumping
libvirt-python right in the requirements.
[1] http://paste.openstack.org/show/797571/
Change-Id: Ie05c8c2b7e0afe1aa7f84468e22f80debb689232

Until now the necessary pip packages differ between nova-api and
compute hosts, when novnc is used, because 'websockify' was only
added to nova-api containers. As a result, the venv is rebuilt twice
for each run. With this change 'websockify' is always added to
the needed pip packages.
Change-Id: I3f284e6d32f4aa614f311d3e4e754830be2f11a6

We shouldn't need this anymore since interpreter patch has been merged
https://review.opendev.org/#/q/topic:interpreter+(status:open+OR+status:merged)
Change-Id: Ibc8c9b92c8c3c705964fbc85019b357913583ca7

netcat-openbsd is required on Debian/Ubuntu in order for live migration
to function. If it is not installed, you get an error that looks like:
libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova@hv1/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa: End of file while reading data:
sh: 1: nc: not found: Input/output error
2020年05月19日 22:13:32.553 4511 ERROR nova.virt.libvirt.driver [-] [instance: b5a313c0-326c-4e3f-bec4-22926289999f] Migration operation has aborted
This was originally fixed in [1] and then accidentally reverted in a
package cleanup[2].
[1] https://review.opendev.org/#/c/666759/
[2] https://review.opendev.org/#/c/662585/
Closes-Bug: #1833644
Change-Id: Ia2747142e6253b286d36d8c71a263ef7ed01341d

UEFI booting support was removed on Debian/Ubuntu was broken by the
removal of ovmf and qemu-efi package installs during a package list
cleanup[1].
It looks like suse/redhat did not reecive the same cleanup effects,
as the UEFI bootloader packages are still installed there, so no
fix is needed on those OS families.
[1] https://review.opendev.org/#/c/662585/
Change-Id: I8ce72e4a0c754a745b1abf42030e41d6084f1361

This removes superfluous packages and moves others to align with
what vars/{debian,suse}.yml does.
Change-Id: I88bc42eaa30d59250003339241380e3e50b64e3b

Unfortunatelly CentOS 7 do not have libvirt and guestfs libraries
for py3, so isntead of symlinking them, we have to install devel package
and build inside venv.
Change-Id: Ie678e44c2369347cf0816cbcc4269de8eba7b963

Return required CentOS packages to build corret venv and
to have virsh CLI tool installed.
Change-Id: I9656f9cf65f371a9640c387b5e5d815bf2062a5f

Had issues where QEMU packages were not in sync and got an error:
... Note: only modules from the same build can be loaded. ...
The qemu package no longer lists dependencies on qemu-system, qemu-user,
qemu-utils in apt show.
Qemu-utils is recommended by other packages but I am unsure as to whether
this will guarantee an upgrade in all circumstances.
While qemu-system does list dependencies on the arm, mips, ppc... packages
they are unversioned and therefore do not keep in step.
Change-Id: I8ea4681a58a54bdc6a24ff053b11dfee5cc7af59

In openSUSE 15.x genisoimage was replaced by mkisofs. Although
some projects have added autodetection, nova needs to be explicitly
told.
Also remove deprecated qemu-kvm (wrapper around qemu-system-x86_64).
Change-Id: If0cbd60790935141d52465abe61b40058f1829b1

In [1] many spurious packages were removed, and libxml2 was probably
a dependancy of a removed package. Add libxml2 as an explicit
requirement.
[1] https://review.opendev.org/662585
Change-Id: I6160b53ccfb161b8b8a161971795fd4962d86201

In [1] this package was moved to the 'devel' list mistakenly, it is
required on the compute host always.
In addition, due to infra. issues with Debian, this patch marks the
jobs as non-voting[2].
[1] https://review.opendev.org/662585
[2] https://review.opendev.org/686396
Change-Id: Iaa08e18d9da1a55a27241e4b5629eccb7969e2e0

- Created nova_compute_packages for compute common packages
All operating systems:
- bridge-utils because os-vif uses iproute2
- postgres headers because OpenStack doesn't support it
- python libraries as they are already installed
- python-openstackclient as it's not used in run-time
Debian removals:
- qemu-kvm because it's just a legacy wrapper for qemu-system-x86
- qemu-utils recommended by qemu-system-x86
- qemu-system-{arm,mips,misc,ppc,sparc,x86} required by qemu-system
- qemu-{system,user,utils} required by qemu
- vlan as nova no longer uses vconfig
- python3-{guestfs,libvirt} as python3 support is not there yet
RedHat removals:
- libvirt-daemon-config-{nwfilter,qemu} required by libvirt-daemon-kvm
- qemu-img-ev required by qemu-kvm-ev
- iscsi-initiator-utils required by libvirt-daemon-driver-storage-iscsi
SUSE removals:
- open-iscsi required by libvirt-daemon-driver-storage-iscsi
- libvirt-daemon-driver-storage-core required by libvirt-daemon-driver-storage-*
- device-mapper required by libvirt-daemon-driver-storage-disk
- nfs-utils required by libvirt-daemon-driver-storage-core
- qemu-block-{curl,dmg,ssh} as that block storage driver is not used
Change-Id: If81939f95042460dd83234160f44426692189376

The RedHat-based operating systems such as CentOS have a pretty
stable list of packages, therefore, we don't need to pin it by
version and we can instead move to a much more generic redhat.yml
which will support a bigger range of systems.
Change-Id: I5eaa3cb6b3bcae96ff8ebebb368eb4a6fdfd1bae

For distro package based deployments, install the packaged nova console
proxies and console clients packages instead of git repos.
For debian/ubuntu installs, use the console client packages directly so
that the service configs of console proxy services are managed by OSA.
Change-Id: I27577ea263d38555e73ead078526cff1c3114b5a

Change-Id: Ibc5f8f9f73905c4c04a254e589cd13fe2254521b

Move service to use uWSGI role instead of iternal task for uwsgi
deployment. This aims to ease the maintenance of uWSGI and speedup
metal deployments as the same uwsgi environment will be used
across all services.
Depends-On: https://review.opendev.org/678200/
Depends-On: https://review.opendev.org/678055/
Change-Id: If5aeeefb93c8ba3c1368970de61ea1300218f637

This driver has been retired [1] and tests are now failing becasue the
nova-lxd repo master branch is now empty.
[1] https://review.opendev.org/#/c/672283/
Change-Id: I9906ede54f6b41972a03bfa1d39ba5f99c6235ed

The qemu-efi package will pull in the x86_64 and aarch64 uefi firmware
ensuring both architectures are supported on debian/ubuntu
Change-Id: Id2f266def07e0ba6967746738efd07cff628a19a

This patch aims to migrate service from usage of regular syslog files
to journald. We also disable uwsgi logging, since it dublicates
requests that are logged by service itself.
Change-Id: If1eda4d803661a0b924941aecd1867302391a5f4

Compute hosts require netcat in order to perform live migrations.
Closes-Bug: #1833644
Change-Id: Ie0caab0b10e878722f1804f67808f1d83d504c59

Depends-On: https://review.opendev.org/666058
Depends-On: https://review.opendev.org/666298
Change-Id: I0ac8593477cd127c33594fd8f1c209f08b138794

The debian packages for individual nova services provide service
configs which automatically start after install, leading to all nova
services running on each compute host. Instead only install the
python3-nova library package and rely on the service files OSA manages.
The nova_service_distro_packages can be cleaned up further since most
packages are provided as dependencies. uwsgi packages are only required
on hosts running nova-api-os-compute or nova-api-metadata, so an
additional key, 'nova_api' has been added for these hosts under
'nova_service_extra_distro_packages'.
Beginning in the Stein release, Ubuntu distro packages are now using
Python3. This requires additionally installing and using the uwsgi python3
plugin.
Change-Id: Id0c7b57f0119c5213abdf3505e4644680a78d55e

The PowerVM driver has been untested for a few years as well
as broken due to using the wrong driver as well. It's not possible
to test it within our infrastructure neither has there been any
contributions to improve/fix it.
This patch drops the support for it.
Change-Id: Iea84648c7ff65b5a4d8b97957fb92716578f714d

In order to boot UEFI instances successfully, the OVMF image
must be installed and located at the path that is hardcoded
inside Nova.
This patch takes care of installing the OVMF images and sets
up links to point towards the hardcoded path for operating
systems that don't install at the expected path.
Change-Id: I62f4aa26ffd0d2f3ba4aa19fbdb72578ef0223e4
Closes-Bug: #1825386
Closes-Bug: #1831279 

Since we no longer install python-keystoneclient[1], there is no
longer a need to install python-httplib2.
Change-Id: I7ef1dac07656bec527b631fdcb621e66b769ccd2

Placement is now a separate service and has been removed from nova[1]
and implemented as a separate service in OSA[2]. As such, the nova
in-tree placement API is now disabled and can be removed.
[1] https://review.opendev.org/#/c/618215/
[2] https://review.opendev.org/#/c/656891/
Depends-On: I01df855d8b9255d24efe16ab053ffa491fda351f
Depends-On: I25ce79b9e3492cb075c50beed90ea2b78055e474
Change-Id: I2f5d4c2f1fc1418bc4f4c4baa1ce3c90b88208fc

As far as we are linking libvirt-python inside venv, we do not need
libvirt-devel anymore and it might not be installed.
Change-Id: I289e04f5fe4f441e82cdd236b2ff865476a4f8d2

This patch adds support for this role to be able to deploy on
Debian Stretch.
Change-Id: I1058238e2467cc257ec48ba84fb89995079e9be7
Needed-By: I9a92b73c419a0dc1cca40dacfef75de61a61db94