openstack/devstack - devstack - OpenDev: Free Software Needs Free Tools

AlmaLinux 10 has been introduced in OpenDev to increase hardware
coverage - it supports x86-64-v2 (compared to v3 required by
CentOS Stream 10 and Rocky Linux 10)
Change-Id: I5c91f2166bfce51cadef9c22a22a6031223604c7
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>

With [1] fips based jobs which runs on 9-stream started
to fail as os_CODENAME not applicable on those. This
patch adds fallback as before.
Moving fips jobs to 10-stream/rocky requires some more work
due to [2]
[1] https://review.opendev.org/c/openstack/devstack/+/960342
[2] https://fedoraproject.org/wiki/Changes/RemoveFipsModeSetup
Change-Id: I6d7ba4f5698e9b4837b29662b0b7f883b3c5de35
Signed-off-by: Yatin Karel <ykarel@redhat.com>

Change-Id: Iaad9eb034348d559809108d254601d51719ff3e0
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>

This patch includes changes required to run devstack
on CentOS Stream 10 which has been already published
in official repos by CentOS team [1]:
- Add RDO deps repository for CS10.
- remove xinetd package from installation for swift. Note that
 rsync-daemon is installed which should work fine.
- Use python3-distro to identify the distro
- Add devstack-single-node-centos-10-stream nodeset
- Add devstack-platform-centos-10-stream job to the check
 pipeline.
Closes https://issues.redhat.com/browse/RDO-379
[1] https://mirror.stream.centos.org/10-stream/
Change-Id: I33a6c5530482c28a24f2043cd4195e7bcd46427d
Signed-off-by: Cyril Roelandt <cyril@redhat.com>
Signed-off-by: Sean Mooney <work@seanmooney.info>

This will allow to pass env variables from zuul job definitions to
to systemd service files via the local conf.
The first use case of this is to pass
OS_NOVA_DISABLE_EVENTLET_PATCHING=true to nova services that already
supports running in native threading mode instead of with Eventlet.
During the Eventlet removal effort this will allow us to have separate
jobs testing the same service in different concurrency mode.
Change-Id: I675043e42006286bb7e1190ea9462fb8d8daa38c

This is another occurrence of the issue fixed in bug
1786259 with change I30bf655f which occurs when there
are multiple IPv6 gateways present.
Before this change:
$ source openrc
+++++functions-common:get_default_host_ip:776 ip -f inet6 addr show 100
Device "100" does not exist.
This is because the ip route command returns:
default proto ra metric 100 expires 1497sec pref medium
	nexthop via fe80::4e16:fc01:298c:98ed dev ens3 weight 1
	nexthop via fe80::4e16:fc01:2983:88aa dev ens3 weight 1
Related-Bug: #1786259
Change-Id: I7729730df66a4dc7ee11df1d23b19b9c0794b575

OpenStackClient has a significant amount of startup overhead, which
adds a non-trivial amount of time to each devstack run because it makes
a lot of OSC calls. This change uses the OSC service from [0] to run
a persistent process that handles openstack calls. This removes most
of the startup overhead and in my local testing removes about three
minutes per devstack run.
Currently this is implemented as an opt-in feature. There are likely a
lot of edge cases in projects that use a devstack plugin so turning it
on universally is going to require boiling the ocean. I think getting
this in and enabled for some of the major projects should give us a lot
of the benefit without the enormous effort of making it 100% compatible
across all of OpenStack.
Depends-On: https://review.opendev.org/c/openstack/nova/+/918689
Depends-On: https://review.opendev.org/c/openstack/ironic/+/918690
Change-Id: I28e6159944746abe2d320369249b87f1c4b9e24e
0: http://lists.openstack.org/pipermail/openstack-dev/2016-April/092546.html 

The ubuntu-xenial labels are going to disappear from opendev as that
image is EOL and will we deleted. Clean up our zuul config.
Update some example reference as well.
Change-Id: Id04110f7c871caa1739ff2b62e9796be4fb9aa00

This change add a new lib/host-mem file and moves the existing
ksm support to a new configure_ksm function.
Additional support for ksmtuned is added with a new flag
"ENABLE_KSMTUNED" which defaults to true.
This change also adds support for zswap. zswap is disabled
by default. When enabled on ubuntu lz4 will
be used as the default compressor and z3fold as the zpool.
On non debian distros the compressor and zpool are not set.
The default values should result in very low overhead although
the zstd compressor may provide better overall performance in ci
or with slow io due to the higher compression ratio.
Additionally memory and network sysctl tunings are optionally applied
to defer writes, prefer swapping and optimise tcp connection
startup and keepalive. The sysctl tunings are disabled by default
The base devstack job has been modifed to enable zram and sysctl
tuning.
Both ksm and zswap are wrapped by a tune_host function
which is now called very early in devstack to ensure
they are configured before any memory/network intensive
operations are executed.
The ci jobs do not enable this functionality by default.
To use this functionaltiy define
 ENABLE_SYSCTL_MEM_TUNING: true
 ENABLE_SYSCTL_NET_TUNING: true
 ENABLE_ZSWAP: true
in the devstack_localrc section of the job vars.
Change-Id: Ia5202d5a9903492a4c18b50ea8d12bd91cc9f135

This reverts commit 67630d4c52.
Reason for revert: Seeing random failures across jobs as sometimes
'keyring_pass.cfg' gets duplicated keys and that makes executions
of any openstackclient command to fail until the file is removed.
This should be handled before re enabling the token caching again.
Change-Id: I3d2fe53a2e7552ac6304c30aa2fe5be33d77df53
Related-Bug: #2042943 

SDK uses python keyring library to enable token caching. Normally this
is requiring a proper desktop (interactive) session, but there are some
backend plugins working in non-interactive mode. Store cache in an
unencrypted file on FS (this is not worse than storing passwords in
plaintext).
Change-Id: I42d698f15db5918443073fff8f27b926126d1d0f

Allows for testing of basic sharding configuration.
Change-Id: Idfb2bd1822898d95af8643d69d97d9a76b4d64cc
Needed-By: https://review.opendev.org/c/openstack/ironic/+/894460 

Since we are python3 only for openstack we create a single python3
virtualenv to install all the packages into. This gives us the benefits
of installing into a virtualenv while still ensuring coinstallability.
This is a major change and will likely break many things.
There are several reasons for this. The change that started this effort
was pip stopped uninstalling packages which used distutils to generate
their package installation. Many distro packages do this which meant
that pip installed packages and distro packages could not coexist in the
global install space. More recently git has made pip installing repos as
root more difficult due to file ownership concerns.
Currently the switch to the global venv is optional, but if we go down
this path we should very quickly remove the old global installation
method as it has only caused us problems.
Major hurdles we have to get over are convincing rootwrap to trust
binaries in the virtualenvs (so you'll notice we update rootwrap
configs).
Some distros still have issues, keep them using the old setup for now.
Depends-On: https://review.opendev.org/c/openstack/grenade/+/880266
Co-Authored-By: Dr. Jens Harbott <frickler@offenerstapel.de>
Change-Id: If9bc7ba45522189d03f19b86cb681bb150ee2f25

This patch: https://review.opendev.org/c/openstack/devstack/+/882299
provides functionality, that commit hash can be passed as last arugment,
however when GIT_DEPTH is set, it fails, as in:
timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git ./ovn
--depth 1 --branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
fatal: Remote branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
not found in upstream origin
Closes-Bug: #2023020
Change-Id: I748354964a133e028e12458cc9014d6d014cbdb9

git_clone assumes a branch or a tag is passed as the last argument, and
it fails when a commit hash is passed, as in:
timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git
/opt/stack/ovn --branch 36e3ab9b47e93af0599a818e9d6b2930e49473f0
Cloning into '/opt/stack/ovn'...
fatal: Remote branch 36e3ab9b47e93af0599a818e9d6b2930e49473f0 not found
in upstream origin
Change-Id: Id1328d7cba418fa7c227ae9db4fe83c09fd06035

devstack-base is changed to descend from
openstack-multinode-fips which is defined in
project-config.
This allows jobs to execute the enable_fips playbook
to enable FIPS mode on the node, but only if they
opt-in by setting enable_fips to True. Otherwise,
this is a no-op.
Change-Id: I5631281662dbd18056ffba291290ed0978ab937e

This patch includes changes required to run devstack on RHEL 9.
- en_US.utf8 is provided by glibc-langpack-en
- iptables command is provided by iptables-nft
- Use /etc/os-release to identify the distro in RHEL 9 as it doesn't
 provide lsb_release command.
- CRB repository name is different from CentOS 9
Change-Id: I8f6d9263b24f9c2cf82e09258e2d14d7766ad337

We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.
Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc

openEuler 20.03 LTS SP2 support was removed from devstack in last
few months due to its python version is too old and the CI job
always fail. And openEuler 20.03 LTS SP2 was out of maintainer in May
2022 by openEuler community.
The newest LTS version was released in March 2022 called 22.03 LTS.
This release will be maintained for at least 2 years. And the python
version is 3.9 which works well for devstack.
This Patch add the openEuler distro support back. And add the related
CI job to make sure its works well.
Change-Id: I99c99d08b4a44d3dc644bd2e56b5ae7f7ee44210

Adding job and nodeset to run tempest-full-py3 in Rocky Linux 9 instance
Change-Id: I6fb390bfeec436b50a3ddc18d154bbce3f3b1975

Because adding the role is idempotent, we can save doing the initial
check for role assignment. Also simplify the output matching by using
osc's filters where appropriate.
Co-Authored-By: Jens Harbott <harbott@osism.tech>
Change-Id: If2a661cc565a43a7821b8f0a10edd97de08eb911

Similar to other functions, this uses "--or-show" to avoid double
calls.
Co-Authored-By: Jens Harbott <harbott@osism.tech>
Change-Id: I548f9acd812687838e04b705f86f3b70d2b10caf

Right now we don't officialy support LinuxMint as our
documentation says [1], it seems LinuxMint is a relict
and got forgotten over time.
This patch removes LinuxMint from the code in order not to
confuse users.
[1] https://docs.openstack.org/devstack/latest/
Closes-Bug: #1983427
Change-Id: Ie1ced25f89389494b28a7b2e9bb1c4273e002dd5

Signed-off-by: Yadnesh Kulkarni <ykulkarn@redhat.com>
Change-Id: I9cf3cd8921347eacc1effb2b197b97bc6ff3e0df

This patch reduces memory usage on the Cinder Volume and Backup services
by tuning glibc.
The specific tuning consist on disabling the per thread arenas and
disabling dynamic thresholds.
The Cinder Backup service suffers from high water mark memory usage and
uses excessive memory. As an example just after 10 restore operations
the service uses almost 1GB of RAM and does not ever free it afterwards.
With this patch the memory consumption of the service is reduced down to
almost 130MB. If we add a revert from Cinder (Change-Id
I43a20c8687f12bc52b014611cc6977c4c3ca212c) it goes down to 100MB during
my tests.
This glibc tuning is not applied to all Python services because I
haven't done proper testings on them and at first glance they don't seem
to have such great improvements.
Related-bug: #1908805
Change-Id: Ic9030d01468b3189350f83b04a8d1d346c489d3c

Currently, neutron tunnel endpoints must be IPv4 addresses,
i.e. $HOST_IP, although IPv6 endpoints are supported by most
drivers.
Create a TUNNEL_IP_VERSION variable to choose which host IP
to use, either HOST_IP or HOST_IPV6, and configure it in the
OVS and Linuxbridge agent driver files. The default is still
IPv4, but it can be over-ridden by specifying TUNNEL_ENDPOINT_IP
accordingly.
This behaves similar to the SERVICE_IP_VERSION option, which
can either be set to 4 or 6, but not 4+6 - the tunnel overhead
should be consistent on all systems in order not to have MTU
issues.
Must set the ML2 overlay_ip_version config option to match
else agent tunnel sync RPC will not work.
Must set the OVN external_ids:ovn-encap-ip config option to
the correct address.
Updated 'devstack-ipv6-only' job definition and verification role
that will set all services and tunnels to use IPv6 addresses.
Closes-bug: #1619476
Change-Id: I6034278dfc17b55d7863bc4db541bbdaa983a686

The job is broken since it is running with python3.7 and most services
now require at least python3.8.
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Ie21f71acffabd78c79e2b141951ccf30a5c06445

This updates each devstack service library, to use it as the
default value for service-specific RBAC configuration.
Change-Id: I41061d042206c411ee3dd94ce91098e612af7ae7

This is necessary for more consistent behavior across multiple
distro versions. Apparently somewhere along the way, git started
looking at the current user's home directory instead of $HOME.
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
Change-Id: I941ef5ea90970a0901236afe81c551aaf24ac1d8

git commit [1] introduced a new behaviour to work around a CVE that
disallows any git operations in directories not owned by the current
user.
This may seem unrelated to installation, but it plays havoc with PBR,
which calls out to git to get to get revision history. So if you are
"pip install"-ing from a source tree you don't own, the PBR git calls
in that tree now fail and the install blows up.
This plays havoc with our model. Firstly, we checkout all code as
"stack" then install it globally with "sudo" (i.e. root) -- which
breaks. We also have cases of essentially the opposite -- checkouts
we have installed as root, but then run tox in them as a regular user;
tox wants to install the source in its venv but now we have another
user conflict.
This uses the only available configuration option to avoid that by
globally setting the source directories we clone as safe. This is an
encroachment of the global system for sure, but is about the only
switch available at the moment. For discussion of other approaches,
see [2].
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
[1] 8959555cee
[2] https://review.opendev.org/c/openstack/devstack/+/837636
Change-Id: Ib9896a99b6d6c4d359ee412743ce30512b3c4fb7

After patch [1] new RBAC policies changed in the way that SYSTEM_ADMIN
user isn't anymore allowed to e.g. create resources in behalf of some
projects. Now PROJECT_ADMIN needs to create such resources instead.
So this patch basically reverts most of the changes which were done
in [2] some time ago.
It also introduces new entry in the clouds.yaml file -
"devstack-admin-demo" which is "admin" user in the "demo" project as
it's needed to create some resouces in the demo project now.
Additionally, because of bug [3] this patch changes way how IPv6
external gateway IP is found using Neutron API. This change may be
reverted in the future when bug [3] will be fixed.
[1] https://review.opendev.org/c/openstack/neutron/+/821208
[2] https://review.opendev.org/c/openstack/devstack/+/797450
[3] https://bugs.launchpad.net/neutron/+bug/1959332
Depends-On: https://review.opendev.org/c/openstack/neutron/+/826828
Closes-Bug: #1959196
Change-Id: I32a6e8b9b59269a8699644b563657363425f7174

This is needed so we can set keystone into enforcing secure RBAC.
This also adjusts lib/glance, which already partially used
devstack-system-admin.
Change-Id: I6df8ad23a3077a8420340167a748ae23ad094962

openEuler is an open-source Linux based operating system. The current
openEuler kernel is based on Linux and supports multi arch, such as X86_64
and aarch64. It fully unleashes the potential of computing chips. As an
efficient, stable, and secure open-source OS built by global open-source
contributors, openEuler applies to database, big data, cloud computing,
and AI scenarios. openEuler is using RPM for package management.
Note:
Currently there is no available package for uwsgi-plugin-python3 and ovn, so that
openEuler needs manually install them from source.
Website: https://www.openeuler.org/en/
Change-Id: I169a0017998054604a63ac6c177d0f43f8a32ba6
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

This patch includes changes required to run devstack on CentOS Stream 9
which has been already published in official repos by CentOS team [1]:
- Add RDO deps repository for CS9.
- remove xinetd package from installation for swift. Note that
 rsync-daemon is installed which should work fine.
- Replace genisoimage by xorriso in CS9.
- Use /etc/os-release to identify the distro in CS9 as it doesn't
 provide lsb_release command.
- Use pip from rpm package instead of from get-pip.py as done in Fedora.
- Add non-voting job devstack-platform-centos-9-stream to the check
 pipeline.
Change-Id: Ic67cddabd5069211dc0611994b8b8360bcd61bef

This commit formalizes some additional users to act as different project
users and updates clouds.yaml file so they're easy to use.
It creates:
 - a reader on the demo project
 - a reader on the alt_demo project
 - a member on the alt_demo project
With the adoption of secure RBAC personas, these are useful for using
OpenStack APIs as that work continues.
Change-Id: I3237a771275311377313b7d7d80ac059ac69d031

Keystone has supported system-scope since Queens and we already make
sure we create a cloud profile for system-admin in
/etc/openstack/clouds.yaml.
This commit ensures keystone creates a couple of new users to model
system-member and system-reader personas. Doing this by default in
devstack makes it easier for people to use.
We've already taken a similar approach in tempest by setting up the
various system personas for tempest clients to use.
Change-Id: Iceb7c5f517db20072e121dc7538abaa888423c67

If ERROR_ON_CLONE is set to True which is case for
all the devstack based job, devstack does not clone the
repo instead raise error. From current error message, it
is difficult to know that ERROR_ON_CLONE is True until we
traceback the code or check devstack-base job set ERROR_ON_CLONE
to True.
Current error message is like:
-------
+ functions-common:git_clone:560 : echo
'The /opt/stack/oslo.limit project was not found; if this is a gate job, add'
 The /opt/stack/oslo.limit project was not found; if this is a gate job, add
+ functions-common:git_clone:561 : echo 'the project to the
 $PROJECTS variable in the job definition.'
the project to the $PROJECTS variable in the job definition.
+ functions-common:git_clone:562 : die 562 'Cloning not
 allowed in this configuration'
--------
Adding ERROR_ON_CLONE info in error message will help to
know the reason of devstack not cloning the repo.
Change-Id: I9e9852f046fefb299b4ef4446323e9c86437212f

Change-Id: I9fb6f010842a495c838d468b47dc5081596f41a2