openstack/openstack-ansible-os_nova - openstack-ansible-os_nova - OpenDev: Free Software Needs Free Tools

Change-Id: I75bdecd4a2452b56b19561432e0b77791f111c95

In order to reduce divergance with ansible-lint rules, we apply
auto-fixing of violations.
In current patch we replace all kind of truthy variables with
`true` or `false` values to align with recommendations along with
alignment of used quotes.
Change-Id: Ie1737a7f88d783e39492c704bb6805c89a199553

Support is removed in oslo.messaging so we remove support in
openstack-ansible roles.
Change-Id: I13f77bb8b63b3cc3d198dcbf918a6708f7d9d80e

This change matches an earlier modification to os_neutron
Currently we symlink /etc/<service> to empty directory at pre-stage,
and filling it with config only during post_install. This means,
that policies and rootwrap filters are not working properly until
playbook execution finish. Additionally, we replace sudoers file
with new path in it, which makes current operations impossible for
the service, since rootwrap can not gain sudo privileges.
With this change we move symlinking and rootwrap steps to handlers,
which means that we will do replace configs while service is stopped.
During post_install we place all of the configs inside the venv,
which is versioned at the moment.
This way we minimise downtime of the service while performing upgrades
Closes-Bug: #2056180
Change-Id: I9c8212408c21e09895ee5805011aecb40b689a13

With ansible-core 2.16 a breaking changes landed [1] to some filters
making their result returned in arbitrary order. With that, we were
relying on them to always return exactly same ordered lists.
With that we need to ensure that we still have determenistic behaviour
where this is important.
[1] https://github.com/ansible/ansible/issues/82554
Change-Id: If26ec122b8defaa1dc1a44f8d6cb2510982cfdf7

The qemu-efi package does not exist on Ubuntu Noble, so instead
install the specific package for the host architecture.
Change-Id: Id91cafc9c2f234bd5f18017a99f757f2bd751b35

The default value for heartbeat_in_pthread has been reverted in
oslo.messaging to False [1] and backported back to Yoga.
At the moment this setting brings intermittent issues during live
migrations of instances and some other operations. So makes sense
to align it with default value.
[1] https://review.opendev.org/c/openstack/oslo.messaging/+/852251
Change-Id: I5601726095ff19620de2d87220efad191cf7cb6d

Instead of evaluating same condition of my_ip in multiple places across
the role this patch suggests doing this once in vars and using the
resulting variable afterwards.
This not only reduce amount of evaluations made throughout the role runtime,
but also solves possible corner cases where some syntax may go off.
Closes-Bug: #2052884
Change-Id: I454b53713ecacf844ac14f77b6d1e1adc1322c0e

As of today we do not have any means of Blazar integration with Nova,
while we do provide roles for Blazar installation for a while now. This
patch aims to bring in more native integration and remove necessity
of overrides for such deployment.
Related-Bug: #2048048
Co-Authored-By: Alexey Rusetsky <fenuks@fenuks.ru>
Change-Id: Ica50a5504de1b1604f72123751cbb3f45c85ab46

For quite some time, we relate usage of --by-service flag for
nova-manage cell_v2 discover_hosts command to the used nova_virt_type.
However, we run db_post_setup tasks only once and delegating to the
conductor host. With latest changes to the logic, when this task in
included from the playbook level it makes even less sense, since
definition of nova_virt_type for conductor is weird and wrong.
Instead, we attempt to detect if ironic is in use by checking hostvars
of all compute nodes for that. It will include host_vars, group_vars,
all sort of extra variables, etc.
Thus, ironic hosts should be better discovered now with nova-manage
command.
Related-Bug: #2034583
Change-Id: I3deea859a4017ff96919290ba50cb375c0f960ea

This is a dummy package with almost no content and no dependancies.
It does not exist on debian 12.
Change-Id: Ibb330238e728af257d46812e64a58fc71a424a1f

This change implements and enables by default quorum support
for rabbitmq as well as providing default variables to globally tune
it's behaviour.
In order to ensure upgrade path and ability to switch back to HA queues
we change vhost names with removing leading `/`, as enabling quorum
requires to remove exchange which is tricky thing to do with running
services.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/875399
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/873618
Change-Id: I792595dac8b651debcd364cd245145721575a516

When Nova is deployed with a mix of x86 and arm systems
(for example), it may be necessary to deploy both 'novnc' and
'serialconsole' proxy services on the same host in order to
service the mixed compute estate.
This patch introduces a list which defines the required proxy
console types.
Change-Id: I93cece8babf35854e5a30938eeb9b25538fb37f6

With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.
With that we also update metdata to reflect current state.
Depends-On: https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/888223
Change-Id: I730ae569f199fc8542a5a61beb149f459465d7e2

CentOS has upgraded their libivrt to version 9.3, where libvirt-daemon
is not installed as a dependency anymore. So we need to explicitly
isntall this package to restore functionality.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2209936
Change-Id: Ic6f2606b5a478c7a891c25bd131ad351a19699bc

This is required by qemu-system-x86 but only recommended by
qemu-system-arm. Without the file /usr/lib/ipxe/efi-virtio.rom
from ipxe-qemu it is not possible to boot a VM on arm
hosts.
This patch ensures that ipxe-qemu is always installed.
Change-Id: I27fd98a1568bda8bea3d88c3f18b44a080982d0e

At the moment we don't restart services if systemd unit file is changed.
We knowingly prevent systemd_service role handlers to execute
by providing `state: started` as otherwise service will be restarted twice.
With that now we ensure that role handlers will also listen for systemd
unit changes.
Change-Id: I4273d2fbcbff3028e693e3274093c1afebdcfca2

qemu-system on debian derivative OS is a meta-package which installs
qemu-system-* for all architecures understood by qemu.
This is different from redhat type OS where the qemu-kvm package
installed with dnf only installs the qemu-system-* binary matching
the host architecture.
This gives two problems, first there is inconsistency in openstack-ansible
deployments between redhat and debian OS. Second, there is a potentially
unexpected emulation of architectures when launching VM on a cloud
with a mix of compute architectures when a full set of qemu-system-*
binaries is available on a compute node. The compute node becomes a
candidate for scheduling any of the supported architectures and a
very specific configuration is needed both from the operator and end
user to ensure that VM are run on a native architecture or emulated as
required.
This patch changes the installation so that redhat and debian compute nodes
only have the native qemu-system binary installed.
A new feature should be introduced to openstack-ansible in the future
to explicitly control installation of non-native qemu-system-* binaries
and write the config options for controlling emulation.
Change-Id: I1c876c7968efb7f24880f1a6e96ba6b7264ddc94

RDO packages for nova does depend on python3-openvswitch,
which makes it required to install OVS on computes regardless
of everything else.
We also clean out pre-rhel9 variable files as they're not needed anymore
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/872896
Change-Id: I3e31254b7dd1c0ff3cb46153cefce6f6cadd52aa

At the moment we don't provide any option rather then use memcached
backend. With that we also hardocde list of packages that should be
installed inside virtualenv for selected backend.
Adding bmemcached requirement to oslo_cache.memcache_pool [1] gives us
opportunity to refactor this bit of deployment and allow to be more
flexible in backend selection and requirements installation for it.
[1] https://review.opendev.org/c/openstack/oslo.cache/+/854628
Change-Id: I48e193ef29e56aa8639511c5b5dcddc70f5e1198

This configuration option has been observed to result in file
descriptor leaks in certain circumstances. A variable is added
here so that it can be easily overridden.
Change-Id: I7de034307da9352e6f5d1f5f175a330fb8c86463
Related-Bug: #1961603 

libvirtd.socket does monitor libvirtd.service and trigger service restart
when it spot that service is down.
However in order to enable tcp and tls sockets, we need libvirt
to be stopped.
Currently race condition can happen, when we stop libvirt, but it's
started by socket before we enable tls one.
To overcome this we stop socket along with service.
Change-Id: Iacc093311036fb8d6559a0e32252579303a639ba

Change-Id: I7d09f6f5f5d66ecbe29fd3969d586eb416c98589

After qemu has been updated in osbpo repo, extra requirement has
appeared, that is available only from backports repo.So we add it
to nova_backports_packages and limit apt_package_pinning
only to Debian Buster.
Change-Id: I284fbd7f8587886502ecc54adfe7314fb80967fd

When Nova and/or Cinder are using Ceph as backend, qemu will need
to open a connection and two threads for each and every Ceph OSD.
Since all connections occur in the same qemu process, this may result
in hitting default max open files limit.
Thus in case of more then 10 volumes are attached
to the same instance and are used actively can end up in blocked
IO operations inside VMs.
We increase these limits by default when RBD backend is used.
Change-Id: Ib3081280cdbae1eb2235083c95c27e2efd0b413e

All references to Gentoo, SUSE, Debian stretch and Centos-7 are removed.
Conditional tasks, ternary operators and variables are simplified where possible
OS specific variables files are generalised where possible
Change-Id: I4a68549bf85fd322ea344139869916aae3275377

See https://github.com/ansible/ansible/issues/73654
Change-Id: I3cf2a30e0929835a84f0502bc4e87522b688b538

Some extra packages needs to be installed from Debian updates
repository, so we pin them with higher priority.
Change-Id: I49c75dd11d6c4e8d37fe013b7ffdfd56ff193fcd

We use the same condition, which defines against what host some "service"
tasks should run against, several times. It's hard to keep it the same
across the role and ansible spending additional resources to evaluate
it each time, so it's simpler and better for the maintenance to set
a boolean variable which will say for all tasks, that we want to run
only against signle host, if they should run or not now.
Change-Id: I78d44704a0224b2c73832a607e84ea8d9499f8c2

Change-Id: If5f5640b80c62fa249a0706b6f64d751b04293a6

Add 'nova_scheduler_extra_filters' to allow operators to enable
additional scheduler filters without overriding the entire list of
filters as in [0].
This also reduces the burden on ops to maintain the list of overridden
default schedulers because of things like [1].
[0] 3886dbc40d/openstack_deploy/user_variables.yml (L51-L69)
[1] https://review.openstack.org/#/c/596502/
Change-Id: I9ab6bcbef2b496df7f6ecf11a7d8f5f7891aeeca

libvirt-python 6.7.0 makes nova-compute with [1]
Since urls can't be used with constraints, we're bumping
libvirt-python right in the requirements.
[1] http://paste.openstack.org/show/797571/
Change-Id: Ie05c8c2b7e0afe1aa7f84468e22f80debb689232

Until now the necessary pip packages differ between nova-api and
compute hosts, when novnc is used, because 'websockify' was only
added to nova-api containers. As a result, the venv is rebuilt twice
for each run. With this change 'websockify' is always added to
the needed pip packages.
Change-Id: I3f284e6d32f4aa614f311d3e4e754830be2f11a6

We shouldn't need this anymore since interpreter patch has been merged
https://review.opendev.org/#/q/topic:interpreter+(status:open+OR+status:merged)
Change-Id: Ibc8c9b92c8c3c705964fbc85019b357913583ca7

netcat-openbsd is required on Debian/Ubuntu in order for live migration
to function. If it is not installed, you get an error that looks like:
libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+ssh://nova@hv1/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa: End of file while reading data:
sh: 1: nc: not found: Input/output error
2020年05月19日 22:13:32.553 4511 ERROR nova.virt.libvirt.driver [-] [instance: b5a313c0-326c-4e3f-bec4-22926289999f] Migration operation has aborted
This was originally fixed in [1] and then accidentally reverted in a
package cleanup[2].
[1] https://review.opendev.org/#/c/666759/
[2] https://review.opendev.org/#/c/662585/
Closes-Bug: #1833644
Change-Id: Ia2747142e6253b286d36d8c71a263ef7ed01341d

UEFI booting support was removed on Debian/Ubuntu was broken by the
removal of ovmf and qemu-efi package installs during a package list
cleanup[1].
It looks like suse/redhat did not reecive the same cleanup effects,
as the UEFI bootloader packages are still installed there, so no
fix is needed on those OS families.
[1] https://review.opendev.org/#/c/662585/
Change-Id: I8ce72e4a0c754a745b1abf42030e41d6084f1361

This removes superfluous packages and moves others to align with
what vars/{debian,suse}.yml does.
Change-Id: I88bc42eaa30d59250003339241380e3e50b64e3b

Unfortunatelly CentOS 7 do not have libvirt and guestfs libraries
for py3, so isntead of symlinking them, we have to install devel package
and build inside venv.
Change-Id: Ie678e44c2369347cf0816cbcc4269de8eba7b963

Return required CentOS packages to build corret venv and
to have virsh CLI tool installed.
Change-Id: I9656f9cf65f371a9640c387b5e5d815bf2062a5f

Had issues where QEMU packages were not in sync and got an error:
... Note: only modules from the same build can be loaded. ...
The qemu package no longer lists dependencies on qemu-system, qemu-user,
qemu-utils in apt show.
Qemu-utils is recommended by other packages but I am unsure as to whether
this will guarantee an upgrade in all circumstances.
While qemu-system does list dependencies on the arm, mips, ppc... packages
they are unversioned and therefore do not keep in step.
Change-Id: I8ea4681a58a54bdc6a24ff053b11dfee5cc7af59

In openSUSE 15.x genisoimage was replaced by mkisofs. Although
some projects have added autodetection, nova needs to be explicitly
told.
Also remove deprecated qemu-kvm (wrapper around qemu-system-x86_64).
Change-Id: If0cbd60790935141d52465abe61b40058f1829b1

In [1] many spurious packages were removed, and libxml2 was probably
a dependancy of a removed package. Add libxml2 as an explicit
requirement.
[1] https://review.opendev.org/662585
Change-Id: I6160b53ccfb161b8b8a161971795fd4962d86201

In [1] this package was moved to the 'devel' list mistakenly, it is
required on the compute host always.
In addition, due to infra. issues with Debian, this patch marks the
jobs as non-voting[2].
[1] https://review.opendev.org/662585
[2] https://review.opendev.org/686396
Change-Id: Iaa08e18d9da1a55a27241e4b5629eccb7969e2e0

- Created nova_compute_packages for compute common packages
All operating systems:
- bridge-utils because os-vif uses iproute2
- postgres headers because OpenStack doesn't support it
- python libraries as they are already installed
- python-openstackclient as it's not used in run-time
Debian removals:
- qemu-kvm because it's just a legacy wrapper for qemu-system-x86
- qemu-utils recommended by qemu-system-x86
- qemu-system-{arm,mips,misc,ppc,sparc,x86} required by qemu-system
- qemu-{system,user,utils} required by qemu
- vlan as nova no longer uses vconfig
- python3-{guestfs,libvirt} as python3 support is not there yet
RedHat removals:
- libvirt-daemon-config-{nwfilter,qemu} required by libvirt-daemon-kvm
- qemu-img-ev required by qemu-kvm-ev
- iscsi-initiator-utils required by libvirt-daemon-driver-storage-iscsi
SUSE removals:
- open-iscsi required by libvirt-daemon-driver-storage-iscsi
- libvirt-daemon-driver-storage-core required by libvirt-daemon-driver-storage-*
- device-mapper required by libvirt-daemon-driver-storage-disk
- nfs-utils required by libvirt-daemon-driver-storage-core
- qemu-block-{curl,dmg,ssh} as that block storage driver is not used
Change-Id: If81939f95042460dd83234160f44426692189376

The RedHat-based operating systems such as CentOS have a pretty
stable list of packages, therefore, we don't need to pin it by
version and we can instead move to a much more generic redhat.yml
which will support a bigger range of systems.
Change-Id: I5eaa3cb6b3bcae96ff8ebebb368eb4a6fdfd1bae