openstack/devstack - devstack - OpenDev: Free Software Needs Free Tools

Adding job and nodeset to run tempest-full-py3 in Rocky Linux 9 instance
Change-Id: I6fb390bfeec436b50a3ddc18d154bbce3f3b1975

Because adding the role is idempotent, we can save doing the initial
check for role assignment. Also simplify the output matching by using
osc's filters where appropriate.
Co-Authored-By: Jens Harbott <harbott@osism.tech>
Change-Id: If2a661cc565a43a7821b8f0a10edd97de08eb911

Similar to other functions, this uses "--or-show" to avoid double
calls.
Co-Authored-By: Jens Harbott <harbott@osism.tech>
Change-Id: I548f9acd812687838e04b705f86f3b70d2b10caf

Right now we don't officialy support LinuxMint as our
documentation says [1], it seems LinuxMint is a relict
and got forgotten over time.
This patch removes LinuxMint from the code in order not to
confuse users.
[1] https://docs.openstack.org/devstack/latest/
Closes-Bug: #1983427
Change-Id: Ie1ced25f89389494b28a7b2e9bb1c4273e002dd5

Signed-off-by: Yadnesh Kulkarni <ykulkarn@redhat.com>
Change-Id: I9cf3cd8921347eacc1effb2b197b97bc6ff3e0df

This patch reduces memory usage on the Cinder Volume and Backup services
by tuning glibc.
The specific tuning consist on disabling the per thread arenas and
disabling dynamic thresholds.
The Cinder Backup service suffers from high water mark memory usage and
uses excessive memory. As an example just after 10 restore operations
the service uses almost 1GB of RAM and does not ever free it afterwards.
With this patch the memory consumption of the service is reduced down to
almost 130MB. If we add a revert from Cinder (Change-Id
I43a20c8687f12bc52b014611cc6977c4c3ca212c) it goes down to 100MB during
my tests.
This glibc tuning is not applied to all Python services because I
haven't done proper testings on them and at first glance they don't seem
to have such great improvements.
Related-bug: #1908805
Change-Id: Ic9030d01468b3189350f83b04a8d1d346c489d3c

Currently, neutron tunnel endpoints must be IPv4 addresses,
i.e. $HOST_IP, although IPv6 endpoints are supported by most
drivers.
Create a TUNNEL_IP_VERSION variable to choose which host IP
to use, either HOST_IP or HOST_IPV6, and configure it in the
OVS and Linuxbridge agent driver files. The default is still
IPv4, but it can be over-ridden by specifying TUNNEL_ENDPOINT_IP
accordingly.
This behaves similar to the SERVICE_IP_VERSION option, which
can either be set to 4 or 6, but not 4+6 - the tunnel overhead
should be consistent on all systems in order not to have MTU
issues.
Must set the ML2 overlay_ip_version config option to match
else agent tunnel sync RPC will not work.
Must set the OVN external_ids:ovn-encap-ip config option to
the correct address.
Updated 'devstack-ipv6-only' job definition and verification role
that will set all services and tunnels to use IPv6 addresses.
Closes-bug: #1619476
Change-Id: I6034278dfc17b55d7863bc4db541bbdaa983a686

The job is broken since it is running with python3.7 and most services
now require at least python3.8.
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Ie21f71acffabd78c79e2b141951ccf30a5c06445

This updates each devstack service library, to use it as the
default value for service-specific RBAC configuration.
Change-Id: I41061d042206c411ee3dd94ce91098e612af7ae7

This is necessary for more consistent behavior across multiple
distro versions. Apparently somewhere along the way, git started
looking at the current user's home directory instead of $HOME.
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
Change-Id: I941ef5ea90970a0901236afe81c551aaf24ac1d8

git commit [1] introduced a new behaviour to work around a CVE that
disallows any git operations in directories not owned by the current
user.
This may seem unrelated to installation, but it plays havoc with PBR,
which calls out to git to get to get revision history. So if you are
"pip install"-ing from a source tree you don't own, the PBR git calls
in that tree now fail and the install blows up.
This plays havoc with our model. Firstly, we checkout all code as
"stack" then install it globally with "sudo" (i.e. root) -- which
breaks. We also have cases of essentially the opposite -- checkouts
we have installed as root, but then run tox in them as a regular user;
tox wants to install the source in its venv but now we have another
user conflict.
This uses the only available configuration option to avoid that by
globally setting the source directories we clone as safe. This is an
encroachment of the global system for sure, but is about the only
switch available at the moment. For discussion of other approaches,
see [2].
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
[1] 8959555cee
[2] https://review.opendev.org/c/openstack/devstack/+/837636
Change-Id: Ib9896a99b6d6c4d359ee412743ce30512b3c4fb7

After patch [1] new RBAC policies changed in the way that SYSTEM_ADMIN
user isn't anymore allowed to e.g. create resources in behalf of some
projects. Now PROJECT_ADMIN needs to create such resources instead.
So this patch basically reverts most of the changes which were done
in [2] some time ago.
It also introduces new entry in the clouds.yaml file -
"devstack-admin-demo" which is "admin" user in the "demo" project as
it's needed to create some resouces in the demo project now.
Additionally, because of bug [3] this patch changes way how IPv6
external gateway IP is found using Neutron API. This change may be
reverted in the future when bug [3] will be fixed.
[1] https://review.opendev.org/c/openstack/neutron/+/821208
[2] https://review.opendev.org/c/openstack/devstack/+/797450
[3] https://bugs.launchpad.net/neutron/+bug/1959332
Depends-On: https://review.opendev.org/c/openstack/neutron/+/826828
Closes-Bug: #1959196
Change-Id: I32a6e8b9b59269a8699644b563657363425f7174

This is needed so we can set keystone into enforcing secure RBAC.
This also adjusts lib/glance, which already partially used
devstack-system-admin.
Change-Id: I6df8ad23a3077a8420340167a748ae23ad094962

openEuler is an open-source Linux based operating system. The current
openEuler kernel is based on Linux and supports multi arch, such as X86_64
and aarch64. It fully unleashes the potential of computing chips. As an
efficient, stable, and secure open-source OS built by global open-source
contributors, openEuler applies to database, big data, cloud computing,
and AI scenarios. openEuler is using RPM for package management.
Note:
Currently there is no available package for uwsgi-plugin-python3 and ovn, so that
openEuler needs manually install them from source.
Website: https://www.openeuler.org/en/
Change-Id: I169a0017998054604a63ac6c177d0f43f8a32ba6
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

This patch includes changes required to run devstack on CentOS Stream 9
which has been already published in official repos by CentOS team [1]:
- Add RDO deps repository for CS9.
- remove xinetd package from installation for swift. Note that
 rsync-daemon is installed which should work fine.
- Replace genisoimage by xorriso in CS9.
- Use /etc/os-release to identify the distro in CS9 as it doesn't
 provide lsb_release command.
- Use pip from rpm package instead of from get-pip.py as done in Fedora.
- Add non-voting job devstack-platform-centos-9-stream to the check
 pipeline.
Change-Id: Ic67cddabd5069211dc0611994b8b8360bcd61bef

This commit formalizes some additional users to act as different project
users and updates clouds.yaml file so they're easy to use.
It creates:
 - a reader on the demo project
 - a reader on the alt_demo project
 - a member on the alt_demo project
With the adoption of secure RBAC personas, these are useful for using
OpenStack APIs as that work continues.
Change-Id: I3237a771275311377313b7d7d80ac059ac69d031

Keystone has supported system-scope since Queens and we already make
sure we create a cloud profile for system-admin in
/etc/openstack/clouds.yaml.
This commit ensures keystone creates a couple of new users to model
system-member and system-reader personas. Doing this by default in
devstack makes it easier for people to use.
We've already taken a similar approach in tempest by setting up the
various system personas for tempest clients to use.
Change-Id: Iceb7c5f517db20072e121dc7538abaa888423c67

If ERROR_ON_CLONE is set to True which is case for
all the devstack based job, devstack does not clone the
repo instead raise error. From current error message, it
is difficult to know that ERROR_ON_CLONE is True until we
traceback the code or check devstack-base job set ERROR_ON_CLONE
to True.
Current error message is like:
-------
+ functions-common:git_clone:560 : echo
'The /opt/stack/oslo.limit project was not found; if this is a gate job, add'
 The /opt/stack/oslo.limit project was not found; if this is a gate job, add
+ functions-common:git_clone:561 : echo 'the project to the
 $PROJECTS variable in the job definition.'
the project to the $PROJECTS variable in the job definition.
+ functions-common:git_clone:562 : die 562 'Cloning not
 allowed in this configuration'
--------
Adding ERROR_ON_CLONE info in error message will help to
know the reason of devstack not cloning the repo.
Change-Id: I9e9852f046fefb299b4ef4446323e9c86437212f

Change-Id: I9fb6f010842a495c838d468b47dc5081596f41a2

The Secure RBAC effort has updated Ironic such that it
can support a mode where it is scope enforcing for all
interactions with the API. Due to the design, and operating
nature of Ironic's API, services speaking with it must
authenticate with a system scope to have a full picture
of the universe.
In this case, we need to update the nova configuration
accordingly such that the compute service understands
how to talk to ironic so that it can see the nodes under
management.
Ironic will likely update this again at a later point in
time to enable a "hybrid" mixed-mode as the operating model
and related permissions *should* allow nova to use a project
scoped "owner" account with Ironic, in order to access
and command nodes to deploy. But at this time, we're
focusing on the exclusive operating mode.
Change-Id: I1946725ce08c495178c419eaf38829f921c91bbe
Needed-By: https://review.opendev.org/c/openstack/ironic/+/778957 

The XenAPI driver was removed during the Victoria release [1], while the
libvirt+xen driver has been removed in the Wallaby release [2]. Remove
references to Xen from DevStack since its all a no-op now.
[1] I42b302afbb1cfede7a0f7b16485a596cd70baf17
[2] I73305e82da5d8da548961b801a8e75fb0e8c4cf1
Change-Id: If7055feb88391f496a5e5e4c72008bf0050c5356
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

When deploying on the centos 8 stream variant
the output of "lsb_release -i -s" is
CentOSStream instead of CentOS
This breaks the is_fedora function in devstack
preventing package installation and removal.
Change-Id: I39ccefbd06f46adf5077f8d8001f37d3b190f040

This function has been deprecated for a long time, let's finally
remove it. It is only generating a warning anyway.
Change-Id: I7bd440adf2ce8283e3ad3d5d09e6b2b877e2b42e

Previously pyc files were only cleaned if clean.sh was run.
with this change a new clean_pyc_files function was introduced
with the logic that was previously in clean.sh but it is now
invoked from unstack.sh
With the previous behavior you could not stack with horizon
enabled then unstack and stack again due to the presence of pyc
files that were owned by root.
By moving the clean to unstack in stead of clean.sh you can
now stack, unstack and stack again without hitting the pyc issue.
since unstack is invoked by clean the existing clean.sh behavior has
not changed in practice except for the fact the pyc files are
removed sooner in the process.
This change also removes support for findutils < 4.2.3
Ubuntu 12.04 and CentOS 6 both have 4.4.2 since they were
released 8 years ago and are now EOL its fair to assume
that all modern distros have 4.2.3+
https://repology.org/project/findutils/versions
Change-Id: I13c9aad9be7e0930a0d875b7d382090caf0b8982

There are a few dependencies which are really os-brick-specific.
They are listed in its bindep.txt file, but os-brick is usually
installed from pip, so its bindep.txt file is not available.
As those dependencies are needed by the various services
which use os-brick, move them to their own dependency file
(with the addition of the new lsscsi, required by the next
os-brick stable release) and make sure that file is parsed
when installing the services which require os-brick.
Side note: there should be a way to avoid this duplication
also for pip-installed libraries (normal services can use
files/ or even bindep, but in this case the source is not always
available), (temporarily?) duplicate them, as it has been
the case for the other os-brick dependencies already listed here.
Change-Id: I9ab6e215dbef9ebdb1946da2f9a40ce020ecc95b

While RHEL 7 is detected as RedHatEnterpriseServer, RHEL 8 is
RedHatEnterprise.
 $ lsb_release -i -s
 RedHatEnterprise
Change-Id: I3d750d808c6ebea9c230f0508cdbc016415b9922

Most libs maintain their own system packages in a local bindep.txt file.
We don't currently use those when installing packages from source, which
can result in broken package installs.
This adds a flag to always attempt to install bindep packages if the
bindep.txt file exists. If a file cannot be found, it will just emit a
warning and carry on.
Change-Id: Ia0570f837b8af1c3fee0a314b026a4a7ed27e6a9
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

Keystone no longer has any special functionality hidden behind the admin
endpoint. Stop referencing it in consumers, so it can later be dropped
completely.
Change-Id: I04a5d77908005268cc7c59e7e9ddeea70f6732e2

New cirros with recent fixes for metadata service
has been released. [1]
Let's update the image version in gate.
This also includes a fix to pass image name to SDK tests.
[1] e40bcd2964
Change-Id: I6ccd646f1c22a99bed0bebf6e363d2190241b667
Co-authored-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

We no longer support platforms with Yum on master. Cleanup old
references and convert to dnf.
We don't need any of the failure wrapper stuff as dnf runs in strict
mode by default.
There seem to be a few callers out there, so we'll leave it called
yum_install for now.
Change-Id: Ie71a48fd85b00a97a14bf260cd013b18af4cce06

This was added in 2012 with I89677fd54635e82b10ab674ddeb9ffb3f1a755f0,
but I can not see it being used anywhere currently.
It's use of virtualenv's has become problematic in a python2
deprecated world, but since it is not used, remove it to avoid further
confusion.
Change-Id: I65d44d24f449436ca6229928eee2c5a021793055

A huge part of the logs is irrelevant bash aliases captured by the
openstack client timing wrapper from the run of "openstack complete",
which is only helpful on interactive systems where you'll interact
with the command line. Call it directly to avoid capturing the logs.
While we're here, turn off tracing inside the oscwrap function, which
is called frequently. It's not useful for debugging.
Change-Id: I1cb5399fe7ee6f0e547a9cfff70396aa2007632e

Remove nova cells v1 support, which also allows/necessitates removing
support for nova networks (which was only supported with cells v1) and
nova-consoleauth (which was required by cells v1 but is unnecessary
otherwise).
The Depends-On isn't really necessary, but it's here to make sure this
doesn't merge until we _really_ have killed cells v1.
I honestly expected this patch would be bigger.
Change-Id: I90316208d1af42c1659d3bee386f95e38aaf2c56
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Depends-On: Ib0e0b708c46e4330e51f8f8fdfbb02d45aaf0f44

On SUSE Linux Enterprise distributions, lsb_release -i typically
returns "SUSE" not "SUSE LINUX" as the vendor string.
To avoid duplication of the same regular expressions in multiple
places, add is_opensuse() and is_sle() helper functions, and modify
is_suse to invoke those.
This may also be helpful in the future for distinguishing some corner
cases where things are handled differently between openSUSE and SLE.
Change-Id: I43bf163bc963758ddbb6289928837f5f6512f265

This allows plugins to specify their binary dependencies in bindep
format.
Some thinking on the implementation: this is in contrast to the
files/[deb|rpm] installation, which is called from the external
install_prereqs.sh script. This script being an externally callable
entry-point is really an artifact of the days when we would build
snapshot images for CI and wanted to pre-cache downloads. These days
we use the mirror system to keep packages close to CI nodes. Thus
rather than expand install_prereqs.sh to also be installing
virtualenvs and python dependencies, this seems to fit better as a
separate internal phase of stack.sh.
Documentation is updated
Change-Id: Icbdfbf97c17c906a7ae86f43e80eb2c445816228

This adds a -bindep option to the key development library install
functions. With this option the bindep.txt file will be referenced
and the relevant packages installed.
Change-Id: I856f1f59fca49b6020920d8f859b797f3b904300

Keystone is moving more things to require a system scoped token to
work. Getting one of those requires that domain and project information
are not set.
Change-Id: I2e1640e9f9ef6cdf56bef49d1ae8f0591570c3e6

Volume API v2 has been deprecated for a long time.
There is no reason to use volume v2 in clouds.yaml by default.
This commit also drops "--os-identity-api-version 3" from
write_clouds_yaml in functions -common as "3" is the default value
of tools/update_clouds_yaml.py. They are hardcoded in DevStack
so there is no reason to pass it.
Change-Id: Ie84026a3d19f7711fc781b7012355096c7ff6b5a