openstack/devstack - devstack - OpenDev: Free Software Needs Free Tools

Change-Id: I221a03b2f8314fee66672c23d7ece4eb900c038c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This reverts commit 6990b06cd3. This
should no longer be necessary as packages are requiring simplejson.
Change-Id: I74b0f93457f91e7d53d54737d52f67075088faca
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

Under as-yet-unidentified conditions, we can end up with a version of
packaging that is too old for the version of latest version of
setuptools. This is a known issue and expected behavior and per [1]
$subject is the preferred resolution.
[1] https://github.com/pypa/setuptools/issues/4483#issuecomment-2237219597
Change-Id: I9232f3fae1598297e83c4ea37339896f7dcbd44f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

This partially reverts commit 18b4251bf4.
Support for loading distutils from the standard library is now deprecated
since v70.3.0 [1]. It was needed initially to ease the transition and
can be removed now.
[1] https://setuptools.pypa.io/en/latest/history.html#v70-3-0
Change-Id: Ib929219ae81b802a4632963ef71a258edf4deee5

9-stream jobs failing since [1] merged as these still use
GLOBAL_VENV=False.
egg-info directory is not created in project source
directory when pyproject.toml is used in the project.
pyproject.toml being added across projects[2] to support pip 23.1.
[1] https://review.opendev.org/c/openstack/nova/+/899753
[2] https://review.opendev.org/q/topic:%22pip-23.1-support%22
Change-Id: I53954a37461aee5dd7f487d6bd205caef4408392

Workaround to avoid failure due to missing osc dependency
removed in [1]
[1] https://review.opendev.org/c/openstack/python-openstackclient/+/920001
Change-Id: I3f7541e691717186b7c73f10ffabae6fc0c5c9f9

This change installs setuptools in the requirements
and global venv to ensure that distutils is present
This change also adds new single and two node
nodeset for noble and a devstack platform job as nonvoting.
Change-Id: Ie1f8ebc5db75d6913239c529ee923395a764e19c

In the _setup_package_with_constraints_edit name of the package was
always discovered from the setup.cfg file. But as some projects
implements PEP-621 (see [1] for the SQLAlchemy for example) it is not
enough now.
This patch adds parsing pyproject.toml file also if name is not found in
the setup.cfg file.
[1] a8dbf8763a
Closes-Bug: #2052509
Change-Id: Iee9262079d09a8bd22cd05a8f17950a41a0d1f9d

Since we are python3 only for openstack we create a single python3
virtualenv to install all the packages into. This gives us the benefits
of installing into a virtualenv while still ensuring coinstallability.
This is a major change and will likely break many things.
There are several reasons for this. The change that started this effort
was pip stopped uninstalling packages which used distutils to generate
their package installation. Many distro packages do this which meant
that pip installed packages and distro packages could not coexist in the
global install space. More recently git has made pip installing repos as
root more difficult due to file ownership concerns.
Currently the switch to the global venv is optional, but if we go down
this path we should very quickly remove the old global installation
method as it has only caused us problems.
Major hurdles we have to get over are convincing rootwrap to trust
binaries in the virtualenvs (so you'll notice we update rootwrap
configs).
Some distros still have issues, keep them using the old setup for now.
Depends-On: https://review.opendev.org/c/openstack/grenade/+/880266
Co-Authored-By: Dr. Jens Harbott <frickler@offenerstapel.de>
Change-Id: If9bc7ba45522189d03f19b86cb681bb150ee2f25

We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.
Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc

The previous change, I237f5663b0f8b060f6df130de04e17e2b1695f8a, removed
a SETUPTOOLS flag, but not the comment explaining why that flag was
previously set. Clean up that comment.
Change-Id: I32b0240fd56310d7f10596aaa8ef432679bfd66a

There are two problems with dbcounter installation on Jammy. The first
is straightforward. We have to use `py_modules` instead of `modules` to
specify the source file. I don't know how this works on other distros
but the docs [0] seem to clearly indicate py_modules does this.
The second issue is quite an issue and requires story time. When
pip/setuptools insteall editable installs (as is done for many of the
openstack projects) it creates an easy-install.pth file that tells the
python interpreter to add the source dirs of those repos to the python
path. Normally these paths are appended to your sys.path. Pip's isolated
build env relies on the assumption that these paths are appeneded to the
path when it santizes sys.path to create the isolated environemnt.
However, when SETUPTOOLS_SYS_PATH_TECHNIQUE is set to rewrite the paths
are not appended and are inserted in the middle. This breaks pip's
isolated build env which broke dbcounter installations. We fix this by
not setting SETUPTOOLS_SYS_PATH_TECHNIQUE to rewrite. Upstream indicates
the reason we set this half a decade ago has since been fixed properly.
The reason Jammy and nothing else breaks is that python3.10 is the first
python version to use pip's isolated build envs by default.
I've locally fiddled with a patch to pip [1] to try and fix this
behavior even when rewrite is set. I don't plan to push this upstream
but it helps to illustrate where the problem lies. If someone else would
like to upstream this feel free.
Finally this change makes the jammy platform job voting again and adds
it to the gate to ensure we don't regress again.
[0] https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
[1] https://paste.opendev.org/show/bqVAuhgMtVtfYupZK5J6/
Change-Id: I237f5663b0f8b060f6df130de04e17e2b1695f8a

This reverts commit 7a3a7ce876 and
bcd0acf6c0 and part of
f1ed7c77c5 which all cap our pip
installs.
Given the pip ecosystem can often incorporate major changes, tracking
upstream at least generally gives us one problem at a time to solve
rather than trying to handle version jumps when LTS distros update.
The new dependency resolver included some changes that disallow
setting URL's like "file:///path/to/project#egg=project" in
constraints. Apparently the fact it used to work was an accident of
the requires/constraints mechanism; it does make some sense as the URL
doesn't really have a version-number that the resolver can put in an
ordering graph.
The _setup_package_with_constraints_edit function comment highlights
what this is trying to do
 # Updates the constraints from REQUIREMENTS_DIR to reflect the
 # future installed state of this package. This ensures when we
 # install this package we get the from source version.
In other words; if constraints has "foo==1.2.3" and Zuul has checked
out "foo" for testing, we have to make sure pip doesn't choose version
1.2.3 from pypi.
It seems like removing the entry from upper-requirements.txt is the
important part; adding the URL path to the on-disk version was just
something that seemed to work at the time, but isn't really necessary.
We will install the package in question which will be the latest
version (from Zuul checkout) and without the package in
upper-requirements.txt nothing will try and downgrade it.
Therefore the solution proposed here is to remove the adding of the
URL parts.
This allows us to uncap pip and restore testing with the new
dependency resolver.
Closes-Bug: #1906322
Change-Id: Ib9ba52147199a9d6d0293182d5db50c4a567d677

A new setuptools release has changed the way pip installs are done,
see [0]. With this change we switch back to using the distro
method for global pip installs.
Temporarily make grenade jobs non-voting in order to allow this
patch to be backported.
[0] http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016905.html
Change-Id: I5d8aa0e58e0409c54451b51de5eb70ba9a68d849

Recent regression spotted by Dmitry Tantsur.
DevStack dropped Py2 support but the now-unused-in-devstack
python3_enable got its result nastily inversed.
Change-Id: I4b37cc847a24705c4955cec2e6e45f0514705f1b

python2 is EOL, let's move on and only support python3.
Change-Id: Ieffda4edea9cc19484c04420ed703f7141ef9f15

Focal only provides a pip3 binary, no pip3.8. Instead of working around
that with a symlink, use the module instead.
Add version information output for this variant.
Change-Id: I7af194ecc40e4d43c10ce067a661bb6ab4ca37d4

Recent change to devstack dropped installing test-requirements [1]
However, this caused gate failures due to lack of glance-store
deps for cinder and swift support.
This patch makes devstack install relevant extras depending on
enabled features.
Additionally, relevant functions are added/fixed to make this
possible.
glance-store = glance_store (for gerrit search match)
[1] https://review.opendev.org/715469
Change-Id: I0bf5792a6058b52936115b515ea8360f6264a7c9

This is a test of installing openstack and then seeing if it works.
OpenStack components do not need test-requirements to operate,
that's why they are test-requirements.
Additionally, as we look forward to depsolver pip, this is going
to screw us because we don't apply constraints to linters, which
are expressed in - you guessed it, test-requirements.
Change-Id: I8f24b839bf42e2fb9803dc7df3a30ae20cf264eb

This reverts commit 79b8e79488.
This is breaking things in various jobs, most notably because
we do not put constraints on linters - but we install
test-requirements which then can conflict with each other.
Change-Id: Ibc5603c61b38ce44db58fb27a27352f59123ad09

The new pip depsolver is coming this summer. Until it's ready,
run pip check at the end of devstack to make sure we're not
somehow installing conflicting package versions. We shouldn't
be, because of constraints, but if we are, better to know and
start figuring it out.
Change-Id: Id98f0848ff5a252d93e5f8029df2d069924d603f

After Python 2 is getting unsupported, new distros
like CentOS 8 and RHEL8 have stopped providing 'python'
package forcing user to decide which alternative to
use by installing 'python2' or 'python3.x' package
and then setting python alternative.
This change is intended to make using python3 command as
much as possible and use it as default 'python' alternative
where needed.
The final goals motivating this change are:
 - stop using python2 as much as possible
 - help adding support for CentOS 8 and RHEL8
Change-Id: I1e90db987c0bfa6206c211e066be03ea8738ad3f

Set default python3 version as 3 for cases python3
is not installed before running DevStack.
Implements installation of required python3x package
for RedHat family distros with package name depending on
configurable ${PYTHON3_VERSION}. Examples:
 3 => python3 python3-devel (default one)
 3.6 => python36 python36-devel
 3.7 => python37 python37-devel
This should help in situations where there are more
than one python available for given platform and
DevStack is asked to pick one by its full 3.x version
Change-Id: I49d86bc9193165e0a41e8e8720be37d81a4e7ee0

[1] stopped installing pip for py2 when py3 is being used.
This patch makes sure we check only for py3 pip then.
Also removed some no-longer-relevant comment and
made uninstall behave the same.
Check for pip>=6 removed too.
See also [2].
[1] 279a7589b0
[2] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012182.html
Change-Id: I36ee53e57e468d760b80a7e621b90899867a8efd

Some distros do not install python3/python3-devel with the minimal
install. F29 doesn't install -devel, and neither Centos 7 or 8
install either. This patch ensures that these packages get installed.
Ideally, PYTHON3_VERSION would be set *after* ensuring that python3
was installed, but it gets a little tricky with all of the includes.
This sets it to 3.6 as nothing uses 3.5 anymore.
Change-Id: I7bdfc408b7c18273639ec26eade475856ac43593

Everything in OpenStack *must* be Python 3 supporting now, which means
it's time to remove the functionality that allows us to blacklist
packages that didn't support Python 3.
Change-Id: I7c8cf538ec88bd4056b0109f19671e3d65f5da3a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

This was added in 2012 with I89677fd54635e82b10ab674ddeb9ffb3f1a755f0,
but I can not see it being used anywhere currently.
It's use of virtualenv's has become problematic in a python2
deprecated world, but since it is not used, remove it to avoid further
confusion.
Change-Id: I65d44d24f449436ca6229928eee2c5a021793055

All the OpenStack projects should be able to run under Python 3 now so
the fallback installation of the Python 2 libraries should not be
needed any longer. This also avoids the problem of script files
installed by the libraries sometimes being overwritten by the Python 2
version leading to incorrect execution later, as discussed in
http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009226.html
This reverts commit a2eb89417f.
Change-Id: I1cdb7e4a209872f1620be556b7278879a4b86df5

This updates links going to git.openstack.org and review.openstack.org
to go to their respective opendev locations to avoid redirects.
Change-Id: I78e3bb5303718962f591117f9c0ee11f2314b128
Closes-Bug: #1833256 

We know empirically that some legacy gate jobs pass and appear
to be running with python3 but actually pip was invoked with
PYTHON3_VERSION unset so that they are actually ran with python2
packages.
As a followup to this discussion [1], add a safety check in the
get_pip_command function to ensure that a python version has been
set when it is invoked.
[1] https://review.openstack.org/#/c/622415/4/inc/python@283
Change-Id: I3a08406fb7d68282c6b98abb33a625821510046a

If we are running with python3, just assume that any
package that is not blacklisted is available for py3
and just attempt to install it and let pip sort it out
whether it gets installed from a local or remote package.
Change-Id: Ic05d183e489320f6dfc721575d47e7e4d661f87c
Closes-Bug: #1820892 

I7d16194d6ba1391ca31251d5b50cbb8de033fc38 added wrong behavour
on Fedora > 26 and Centos 7 when python3 disabled
pip should install packages in /usr/bin
Closes-Bug: #1820070
Change-Id: I3a8efbc8eb6e311db9c7347577c5d2047ba523a9

This makes the grep match in check_python3_support_for_package_local
the same as check_python3_support_for_package_remote.
Change I0349de2026c49279ba7f262d5e86d37018d66326 in grenade started
setting the PYTHON3_VERSION variable, and then we recently started
using bionic nodes everywhere which means we're running python 3.6.
The etcd3gw package has a python 3 and 3.5 classifier, but not 3.6:
https://pypi.org/project/etcd3gw/
The pip_install function code that is dealing with installing py3
packages is hitting a problem installing etcd3gw if the package is
local because of the more restrictive grep in the
check_python3_support_for_package_local function, and since
PYTHON3_VERSION=3.6 now, we don't install from py3 and install
etcd3gw on python 2.7 which makes services like cinder-volume and
cinder-backup, which use etcd3gw, fail when they are running under
python 3 (they get module import errors).
This simply removes the $ restriction on the grep. Looking at the
change that added those local/remote functions:
 I243ea4b76f0d5ef57a03b5b0798a05468ee6de9b
There is no explanation for the difference, it just said:
 Also, since not many packages are classified correctly, fallback
 to looking for just "Programming Language :: Python :: 3" and
 log a message for the package to highlight the problem.
So that's what this change does.
Note that alternatives would be:
1. Update the etcd3gw package to add the 3.6 classifier and do
 a release (this should probably happen anyway).
2. Add etcd3gw to ENABLED_PYTHON3_PACKAGES but that would be a
 short-term hack workaround.
Change-Id: Icd3768870ba0f1659bb2e6f002043d975047b73e
Closes-Bug: #1820892 

when installing with python 3.6 on centos7 pip installs
packages to /usr/local/bin as it does on new versions
of fedora.
this change updates the check to include centos
Change-Id: I7d16194d6ba1391ca31251d5b50cbb8de033fc38

This adds a -bindep option to the key development library install
functions. With this option the bindep.txt file will be referenced
and the relevant packages installed.
Change-Id: I856f1f59fca49b6020920d8f859b797f3b904300

Remove the requirement that services explicitly enable python3 support
in order to be tested under python3 when running with python3
enabled. Keep the enable_python3_package() function for backwards
compatibility, for now, since it is called in some devstack plugins.
Explicitly add swift to the set of packages that should not be installed
using python3 by default until full support is available.
Change-Id: I8ab0a7c242bbf5bf3f091f5a85a98e2f4543f856
Signed-off-by: Doug Hellmann <doug@doughellmann.com>

This function is nowhere used as confirmed by codesearch:
http://codesearch.openstack.org/?q=is_in_projects_txt&i=nope&files=&repos=
We can remove the function.
Note that usage of this function was removed in change
I843208e2e982eb04931b76f5cb4bd219fbcd70de in 2015.
This came up in context of https://review.openstack.org/619089 where
requirements/projects.txt is getting removed.
Change-Id: I487d3f9e340bd45e83245b9ca91e8e3b5ac3ae02

Old-style test fails on Ubuntu when python3 enabled, with:
.../devstack/inc/python: line 52: [: 16.04: integer expression expected
Use bash-style test, which doesn't attempt to evaluate the RHS if the
LHS evaluates to false
Change-Id: If18031ab98c9060e5825c3a8d3c647bd3705cd9c
Closes-Bug: #1796174 

If a project shows up in zuul's required-projects list, add it
to LIBS_FROM_GIT automatically. This way, when a user specifies
that a job requires a zuul-project, it gets used in testing, but
otherwise, it doesn't (pypi is used instead).
Also add information about what happens behind the scenes for both
LIBS_FROM_GIT and plugin dependencies.
This moves the check performed in check_libs_from_git to
a helper function which is installed for most kinds of
installations. This means that if someone sets LIBS_FROM_GIT to
"foobar", devstack won't error anymore, as nothing is going to
try to install foobar, therefore the check won't run on that.
However, as we move to automated generation of the local config,
that error is not likely to happen. This check was originally
added due to an error in the upper-constraints file (where a
constraint name did not match a package name). This location of
the check would still catch that type of error.
Change-Id: Ifcf3ad008cf42d3d4762cfb3b6c31c93cfeb40db

Change-Id: Ia77163f330f674146b369dfebea56bd97820057e
Depends-On: https://review.openstack.org/554662 

On Fedora 27, the default Pytho 3 install prefix changed from /usr
to /usr/local:
https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
Closes-Bug: #1741901
Change-Id: Id40620efdf173189df053b5d380a801092933f83

python-openstacksdk does not match its pip name which is openstacksdk.
So setting python-openstacksdk in LIBS_FROM_GIT leads to devstack
thinking there is a problem.
Put in a workaround for now. It would be better to either:
 a) rename python-openstacksdk repo to openstacksdk
 b) rename the pip name for openstacksdk back to python-openstacksdk
 c) add general support in the various GIT hashes for a pip name
Change-Id: I57cf95763d54ad2060a4ce2af91c3ba18ca04db0

The 'pip list' command prints the "safe name" which converts _'s to
-'s amongst other things; e.g. glance_store becomes
 "glance-store 0.21.1.dev22 /opt/stack/glance_store"
Because people may use these more familiar "file system" names in
LIBS_FROM_GIT automatically convert names when checking if libraries
are installed.
Change-Id: I30524f80a341f38dfa794a8f629d859e85a4a448

In commit f0cd9a8b08 we changed to
use column format, but it checks for zero length string and
check_libs_from_git fails.
Change-Id: I97b52b80efb33749647229a55147a08afa112dd2

The old code was strip()ing the version string instead of split()ing the
version string so we always got the first character of the version
string. This worked fine as long as the pip version was single digit but
as soon as it rolls over to '10.stuff' we will compare:
 pip version 1 (instead of 10) > 6
Which fails bceause 1 is less than six. Instaed we really do want to
compare 10 > 6 so use split on '.' instead.
Change-Id: Ic7d0c04d7fa77774ab2d70fb9d11f182becec553