openstack/devstack - devstack - OpenDev: Free Software Needs Free Tools

git_clone assumes a branch or a tag is passed as the last argument, and
it fails when a commit hash is passed, as in:
timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git
/opt/stack/ovn --branch 36e3ab9b47e93af0599a818e9d6b2930e49473f0
Cloning into '/opt/stack/ovn'...
fatal: Remote branch 36e3ab9b47e93af0599a818e9d6b2930e49473f0 not found
in upstream origin
Change-Id: Id1328d7cba418fa7c227ae9db4fe83c09fd06035

devstack-base is changed to descend from
openstack-multinode-fips which is defined in
project-config.
This allows jobs to execute the enable_fips playbook
to enable FIPS mode on the node, but only if they
opt-in by setting enable_fips to True. Otherwise,
this is a no-op.
Change-Id: I5631281662dbd18056ffba291290ed0978ab937e

We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.
Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc

openEuler 20.03 LTS SP2 support was removed from devstack in last
few months due to its python version is too old and the CI job
always fail. And openEuler 20.03 LTS SP2 was out of maintainer in May
2022 by openEuler community.
The newest LTS version was released in March 2022 called 22.03 LTS.
This release will be maintained for at least 2 years. And the python
version is 3.9 which works well for devstack.
This Patch add the openEuler distro support back. And add the related
CI job to make sure its works well.
Change-Id: I99c99d08b4a44d3dc644bd2e56b5ae7f7ee44210

Adding job and nodeset to run tempest-full-py3 in Rocky Linux 9 instance
Change-Id: I6fb390bfeec436b50a3ddc18d154bbce3f3b1975

Because adding the role is idempotent, we can save doing the initial
check for role assignment. Also simplify the output matching by using
osc's filters where appropriate.
Co-Authored-By: Jens Harbott <harbott@osism.tech>
Change-Id: If2a661cc565a43a7821b8f0a10edd97de08eb911

Similar to other functions, this uses "--or-show" to avoid double
calls.
Co-Authored-By: Jens Harbott <harbott@osism.tech>
Change-Id: I548f9acd812687838e04b705f86f3b70d2b10caf

Right now we don't officialy support LinuxMint as our
documentation says [1], it seems LinuxMint is a relict
and got forgotten over time.
This patch removes LinuxMint from the code in order not to
confuse users.
[1] https://docs.openstack.org/devstack/latest/
Closes-Bug: #1983427
Change-Id: Ie1ced25f89389494b28a7b2e9bb1c4273e002dd5

Signed-off-by: Yadnesh Kulkarni <ykulkarn@redhat.com>
Change-Id: I9cf3cd8921347eacc1effb2b197b97bc6ff3e0df

This patch reduces memory usage on the Cinder Volume and Backup services
by tuning glibc.
The specific tuning consist on disabling the per thread arenas and
disabling dynamic thresholds.
The Cinder Backup service suffers from high water mark memory usage and
uses excessive memory. As an example just after 10 restore operations
the service uses almost 1GB of RAM and does not ever free it afterwards.
With this patch the memory consumption of the service is reduced down to
almost 130MB. If we add a revert from Cinder (Change-Id
I43a20c8687f12bc52b014611cc6977c4c3ca212c) it goes down to 100MB during
my tests.
This glibc tuning is not applied to all Python services because I
haven't done proper testings on them and at first glance they don't seem
to have such great improvements.
Related-bug: #1908805
Change-Id: Ic9030d01468b3189350f83b04a8d1d346c489d3c

Currently, neutron tunnel endpoints must be IPv4 addresses,
i.e. $HOST_IP, although IPv6 endpoints are supported by most
drivers.
Create a TUNNEL_IP_VERSION variable to choose which host IP
to use, either HOST_IP or HOST_IPV6, and configure it in the
OVS and Linuxbridge agent driver files. The default is still
IPv4, but it can be over-ridden by specifying TUNNEL_ENDPOINT_IP
accordingly.
This behaves similar to the SERVICE_IP_VERSION option, which
can either be set to 4 or 6, but not 4+6 - the tunnel overhead
should be consistent on all systems in order not to have MTU
issues.
Must set the ML2 overlay_ip_version config option to match
else agent tunnel sync RPC will not work.
Must set the OVN external_ids:ovn-encap-ip config option to
the correct address.
Updated 'devstack-ipv6-only' job definition and verification role
that will set all services and tunnels to use IPv6 addresses.
Closes-bug: #1619476
Change-Id: I6034278dfc17b55d7863bc4db541bbdaa983a686

The job is broken since it is running with python3.7 and most services
now require at least python3.8.
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Ie21f71acffabd78c79e2b141951ccf30a5c06445

This updates each devstack service library, to use it as the
default value for service-specific RBAC configuration.
Change-Id: I41061d042206c411ee3dd94ce91098e612af7ae7

This is necessary for more consistent behavior across multiple
distro versions. Apparently somewhere along the way, git started
looking at the current user's home directory instead of $HOME.
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
Change-Id: I941ef5ea90970a0901236afe81c551aaf24ac1d8

git commit [1] introduced a new behaviour to work around a CVE that
disallows any git operations in directories not owned by the current
user.
This may seem unrelated to installation, but it plays havoc with PBR,
which calls out to git to get to get revision history. So if you are
"pip install"-ing from a source tree you don't own, the PBR git calls
in that tree now fail and the install blows up.
This plays havoc with our model. Firstly, we checkout all code as
"stack" then install it globally with "sudo" (i.e. root) -- which
breaks. We also have cases of essentially the opposite -- checkouts
we have installed as root, but then run tox in them as a regular user;
tox wants to install the source in its venv but now we have another
user conflict.
This uses the only available configuration option to avoid that by
globally setting the source directories we clone as safe. This is an
encroachment of the global system for sure, but is about the only
switch available at the moment. For discussion of other approaches,
see [2].
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
[1] 8959555cee
[2] https://review.opendev.org/c/openstack/devstack/+/837636
Change-Id: Ib9896a99b6d6c4d359ee412743ce30512b3c4fb7

After patch [1] new RBAC policies changed in the way that SYSTEM_ADMIN
user isn't anymore allowed to e.g. create resources in behalf of some
projects. Now PROJECT_ADMIN needs to create such resources instead.
So this patch basically reverts most of the changes which were done
in [2] some time ago.
It also introduces new entry in the clouds.yaml file -
"devstack-admin-demo" which is "admin" user in the "demo" project as
it's needed to create some resouces in the demo project now.
Additionally, because of bug [3] this patch changes way how IPv6
external gateway IP is found using Neutron API. This change may be
reverted in the future when bug [3] will be fixed.
[1] https://review.opendev.org/c/openstack/neutron/+/821208
[2] https://review.opendev.org/c/openstack/devstack/+/797450
[3] https://bugs.launchpad.net/neutron/+bug/1959332
Depends-On: https://review.opendev.org/c/openstack/neutron/+/826828
Closes-Bug: #1959196
Change-Id: I32a6e8b9b59269a8699644b563657363425f7174

This is needed so we can set keystone into enforcing secure RBAC.
This also adjusts lib/glance, which already partially used
devstack-system-admin.
Change-Id: I6df8ad23a3077a8420340167a748ae23ad094962

openEuler is an open-source Linux based operating system. The current
openEuler kernel is based on Linux and supports multi arch, such as X86_64
and aarch64. It fully unleashes the potential of computing chips. As an
efficient, stable, and secure open-source OS built by global open-source
contributors, openEuler applies to database, big data, cloud computing,
and AI scenarios. openEuler is using RPM for package management.
Note:
Currently there is no available package for uwsgi-plugin-python3 and ovn, so that
openEuler needs manually install them from source.
Website: https://www.openeuler.org/en/
Change-Id: I169a0017998054604a63ac6c177d0f43f8a32ba6
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>

This patch includes changes required to run devstack on CentOS Stream 9
which has been already published in official repos by CentOS team [1]:
- Add RDO deps repository for CS9.
- remove xinetd package from installation for swift. Note that
 rsync-daemon is installed which should work fine.
- Replace genisoimage by xorriso in CS9.
- Use /etc/os-release to identify the distro in CS9 as it doesn't
 provide lsb_release command.
- Use pip from rpm package instead of from get-pip.py as done in Fedora.
- Add non-voting job devstack-platform-centos-9-stream to the check
 pipeline.
Change-Id: Ic67cddabd5069211dc0611994b8b8360bcd61bef

This commit formalizes some additional users to act as different project
users and updates clouds.yaml file so they're easy to use.
It creates:
 - a reader on the demo project
 - a reader on the alt_demo project
 - a member on the alt_demo project
With the adoption of secure RBAC personas, these are useful for using
OpenStack APIs as that work continues.
Change-Id: I3237a771275311377313b7d7d80ac059ac69d031

Keystone has supported system-scope since Queens and we already make
sure we create a cloud profile for system-admin in
/etc/openstack/clouds.yaml.
This commit ensures keystone creates a couple of new users to model
system-member and system-reader personas. Doing this by default in
devstack makes it easier for people to use.
We've already taken a similar approach in tempest by setting up the
various system personas for tempest clients to use.
Change-Id: Iceb7c5f517db20072e121dc7538abaa888423c67

If ERROR_ON_CLONE is set to True which is case for
all the devstack based job, devstack does not clone the
repo instead raise error. From current error message, it
is difficult to know that ERROR_ON_CLONE is True until we
traceback the code or check devstack-base job set ERROR_ON_CLONE
to True.
Current error message is like:
-------
+ functions-common:git_clone:560 : echo
'The /opt/stack/oslo.limit project was not found; if this is a gate job, add'
 The /opt/stack/oslo.limit project was not found; if this is a gate job, add
+ functions-common:git_clone:561 : echo 'the project to the
 $PROJECTS variable in the job definition.'
the project to the $PROJECTS variable in the job definition.
+ functions-common:git_clone:562 : die 562 'Cloning not
 allowed in this configuration'
--------
Adding ERROR_ON_CLONE info in error message will help to
know the reason of devstack not cloning the repo.
Change-Id: I9e9852f046fefb299b4ef4446323e9c86437212f

Change-Id: I9fb6f010842a495c838d468b47dc5081596f41a2

The Secure RBAC effort has updated Ironic such that it
can support a mode where it is scope enforcing for all
interactions with the API. Due to the design, and operating
nature of Ironic's API, services speaking with it must
authenticate with a system scope to have a full picture
of the universe.
In this case, we need to update the nova configuration
accordingly such that the compute service understands
how to talk to ironic so that it can see the nodes under
management.
Ironic will likely update this again at a later point in
time to enable a "hybrid" mixed-mode as the operating model
and related permissions *should* allow nova to use a project
scoped "owner" account with Ironic, in order to access
and command nodes to deploy. But at this time, we're
focusing on the exclusive operating mode.
Change-Id: I1946725ce08c495178c419eaf38829f921c91bbe
Needed-By: https://review.opendev.org/c/openstack/ironic/+/778957 

The XenAPI driver was removed during the Victoria release [1], while the
libvirt+xen driver has been removed in the Wallaby release [2]. Remove
references to Xen from DevStack since its all a no-op now.
[1] I42b302afbb1cfede7a0f7b16485a596cd70baf17
[2] I73305e82da5d8da548961b801a8e75fb0e8c4cf1
Change-Id: If7055feb88391f496a5e5e4c72008bf0050c5356
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

When deploying on the centos 8 stream variant
the output of "lsb_release -i -s" is
CentOSStream instead of CentOS
This breaks the is_fedora function in devstack
preventing package installation and removal.
Change-Id: I39ccefbd06f46adf5077f8d8001f37d3b190f040

This function has been deprecated for a long time, let's finally
remove it. It is only generating a warning anyway.
Change-Id: I7bd440adf2ce8283e3ad3d5d09e6b2b877e2b42e

Previously pyc files were only cleaned if clean.sh was run.
with this change a new clean_pyc_files function was introduced
with the logic that was previously in clean.sh but it is now
invoked from unstack.sh
With the previous behavior you could not stack with horizon
enabled then unstack and stack again due to the presence of pyc
files that were owned by root.
By moving the clean to unstack in stead of clean.sh you can
now stack, unstack and stack again without hitting the pyc issue.
since unstack is invoked by clean the existing clean.sh behavior has
not changed in practice except for the fact the pyc files are
removed sooner in the process.
This change also removes support for findutils < 4.2.3
Ubuntu 12.04 and CentOS 6 both have 4.4.2 since they were
released 8 years ago and are now EOL its fair to assume
that all modern distros have 4.2.3+
https://repology.org/project/findutils/versions
Change-Id: I13c9aad9be7e0930a0d875b7d382090caf0b8982

There are a few dependencies which are really os-brick-specific.
They are listed in its bindep.txt file, but os-brick is usually
installed from pip, so its bindep.txt file is not available.
As those dependencies are needed by the various services
which use os-brick, move them to their own dependency file
(with the addition of the new lsscsi, required by the next
os-brick stable release) and make sure that file is parsed
when installing the services which require os-brick.
Side note: there should be a way to avoid this duplication
also for pip-installed libraries (normal services can use
files/ or even bindep, but in this case the source is not always
available), (temporarily?) duplicate them, as it has been
the case for the other os-brick dependencies already listed here.
Change-Id: I9ab6e215dbef9ebdb1946da2f9a40ce020ecc95b

While RHEL 7 is detected as RedHatEnterpriseServer, RHEL 8 is
RedHatEnterprise.
 $ lsb_release -i -s
 RedHatEnterprise
Change-Id: I3d750d808c6ebea9c230f0508cdbc016415b9922

Most libs maintain their own system packages in a local bindep.txt file.
We don't currently use those when installing packages from source, which
can result in broken package installs.
This adds a flag to always attempt to install bindep packages if the
bindep.txt file exists. If a file cannot be found, it will just emit a
warning and carry on.
Change-Id: Ia0570f837b8af1c3fee0a314b026a4a7ed27e6a9
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

Keystone no longer has any special functionality hidden behind the admin
endpoint. Stop referencing it in consumers, so it can later be dropped
completely.
Change-Id: I04a5d77908005268cc7c59e7e9ddeea70f6732e2

New cirros with recent fixes for metadata service
has been released. [1]
Let's update the image version in gate.
This also includes a fix to pass image name to SDK tests.
[1] e40bcd2964
Change-Id: I6ccd646f1c22a99bed0bebf6e363d2190241b667
Co-authored-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

We no longer support platforms with Yum on master. Cleanup old
references and convert to dnf.
We don't need any of the failure wrapper stuff as dnf runs in strict
mode by default.
There seem to be a few callers out there, so we'll leave it called
yum_install for now.
Change-Id: Ie71a48fd85b00a97a14bf260cd013b18af4cce06

This was added in 2012 with I89677fd54635e82b10ab674ddeb9ffb3f1a755f0,
but I can not see it being used anywhere currently.
It's use of virtualenv's has become problematic in a python2
deprecated world, but since it is not used, remove it to avoid further
confusion.
Change-Id: I65d44d24f449436ca6229928eee2c5a021793055

A huge part of the logs is irrelevant bash aliases captured by the
openstack client timing wrapper from the run of "openstack complete",
which is only helpful on interactive systems where you'll interact
with the command line. Call it directly to avoid capturing the logs.
While we're here, turn off tracing inside the oscwrap function, which
is called frequently. It's not useful for debugging.
Change-Id: I1cb5399fe7ee6f0e547a9cfff70396aa2007632e

Remove nova cells v1 support, which also allows/necessitates removing
support for nova networks (which was only supported with cells v1) and
nova-consoleauth (which was required by cells v1 but is unnecessary
otherwise).
The Depends-On isn't really necessary, but it's here to make sure this
doesn't merge until we _really_ have killed cells v1.
I honestly expected this patch would be bigger.
Change-Id: I90316208d1af42c1659d3bee386f95e38aaf2c56
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Depends-On: Ib0e0b708c46e4330e51f8f8fdfbb02d45aaf0f44