openstack/devstack - devstack - OpenDev: Free Software Needs Free Tools

[1] stopped installing pip for py2 when py3 is being used.
This patch makes sure we check only for py3 pip then.
Also removed some no-longer-relevant comment and
made uninstall behave the same.
Check for pip>=6 removed too.
See also [2].
[1] 279a7589b0
[2] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012182.html
Change-Id: I36ee53e57e468d760b80a7e621b90899867a8efd

Some distros do not install python3/python3-devel with the minimal
install. F29 doesn't install -devel, and neither Centos 7 or 8
install either. This patch ensures that these packages get installed.
Ideally, PYTHON3_VERSION would be set *after* ensuring that python3
was installed, but it gets a little tricky with all of the includes.
This sets it to 3.6 as nothing uses 3.5 anymore.
Change-Id: I7bdfc408b7c18273639ec26eade475856ac43593

Everything in OpenStack *must* be Python 3 supporting now, which means
it's time to remove the functionality that allows us to blacklist
packages that didn't support Python 3.
Change-Id: I7c8cf538ec88bd4056b0109f19671e3d65f5da3a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

This was added in 2012 with I89677fd54635e82b10ab674ddeb9ffb3f1a755f0,
but I can not see it being used anywhere currently.
It's use of virtualenv's has become problematic in a python2
deprecated world, but since it is not used, remove it to avoid further
confusion.
Change-Id: I65d44d24f449436ca6229928eee2c5a021793055

All the OpenStack projects should be able to run under Python 3 now so
the fallback installation of the Python 2 libraries should not be
needed any longer. This also avoids the problem of script files
installed by the libraries sometimes being overwritten by the Python 2
version leading to incorrect execution later, as discussed in
http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009226.html
This reverts commit a2eb89417f.
Change-Id: I1cdb7e4a209872f1620be556b7278879a4b86df5

This updates links going to git.openstack.org and review.openstack.org
to go to their respective opendev locations to avoid redirects.
Change-Id: I78e3bb5303718962f591117f9c0ee11f2314b128
Closes-Bug: #1833256 

We know empirically that some legacy gate jobs pass and appear
to be running with python3 but actually pip was invoked with
PYTHON3_VERSION unset so that they are actually ran with python2
packages.
As a followup to this discussion [1], add a safety check in the
get_pip_command function to ensure that a python version has been
set when it is invoked.
[1] https://review.openstack.org/#/c/622415/4/inc/python@283
Change-Id: I3a08406fb7d68282c6b98abb33a625821510046a

If we are running with python3, just assume that any
package that is not blacklisted is available for py3
and just attempt to install it and let pip sort it out
whether it gets installed from a local or remote package.
Change-Id: Ic05d183e489320f6dfc721575d47e7e4d661f87c
Closes-Bug: #1820892 

I7d16194d6ba1391ca31251d5b50cbb8de033fc38 added wrong behavour
on Fedora > 26 and Centos 7 when python3 disabled
pip should install packages in /usr/bin
Closes-Bug: #1820070
Change-Id: I3a8efbc8eb6e311db9c7347577c5d2047ba523a9

This makes the grep match in check_python3_support_for_package_local
the same as check_python3_support_for_package_remote.
Change I0349de2026c49279ba7f262d5e86d37018d66326 in grenade started
setting the PYTHON3_VERSION variable, and then we recently started
using bionic nodes everywhere which means we're running python 3.6.
The etcd3gw package has a python 3 and 3.5 classifier, but not 3.6:
https://pypi.org/project/etcd3gw/
The pip_install function code that is dealing with installing py3
packages is hitting a problem installing etcd3gw if the package is
local because of the more restrictive grep in the
check_python3_support_for_package_local function, and since
PYTHON3_VERSION=3.6 now, we don't install from py3 and install
etcd3gw on python 2.7 which makes services like cinder-volume and
cinder-backup, which use etcd3gw, fail when they are running under
python 3 (they get module import errors).
This simply removes the $ restriction on the grep. Looking at the
change that added those local/remote functions:
 I243ea4b76f0d5ef57a03b5b0798a05468ee6de9b
There is no explanation for the difference, it just said:
 Also, since not many packages are classified correctly, fallback
 to looking for just "Programming Language :: Python :: 3" and
 log a message for the package to highlight the problem.
So that's what this change does.
Note that alternatives would be:
1. Update the etcd3gw package to add the 3.6 classifier and do
 a release (this should probably happen anyway).
2. Add etcd3gw to ENABLED_PYTHON3_PACKAGES but that would be a
 short-term hack workaround.
Change-Id: Icd3768870ba0f1659bb2e6f002043d975047b73e
Closes-Bug: #1820892 

when installing with python 3.6 on centos7 pip installs
packages to /usr/local/bin as it does on new versions
of fedora.
this change updates the check to include centos
Change-Id: I7d16194d6ba1391ca31251d5b50cbb8de033fc38

This adds a -bindep option to the key development library install
functions. With this option the bindep.txt file will be referenced
and the relevant packages installed.
Change-Id: I856f1f59fca49b6020920d8f859b797f3b904300

Remove the requirement that services explicitly enable python3 support
in order to be tested under python3 when running with python3
enabled. Keep the enable_python3_package() function for backwards
compatibility, for now, since it is called in some devstack plugins.
Explicitly add swift to the set of packages that should not be installed
using python3 by default until full support is available.
Change-Id: I8ab0a7c242bbf5bf3f091f5a85a98e2f4543f856
Signed-off-by: Doug Hellmann <doug@doughellmann.com>

This function is nowhere used as confirmed by codesearch:
http://codesearch.openstack.org/?q=is_in_projects_txt&i=nope&files=&repos=
We can remove the function.
Note that usage of this function was removed in change
I843208e2e982eb04931b76f5cb4bd219fbcd70de in 2015.
This came up in context of https://review.openstack.org/619089 where
requirements/projects.txt is getting removed.
Change-Id: I487d3f9e340bd45e83245b9ca91e8e3b5ac3ae02

Old-style test fails on Ubuntu when python3 enabled, with:
.../devstack/inc/python: line 52: [: 16.04: integer expression expected
Use bash-style test, which doesn't attempt to evaluate the RHS if the
LHS evaluates to false
Change-Id: If18031ab98c9060e5825c3a8d3c647bd3705cd9c
Closes-Bug: #1796174 

If a project shows up in zuul's required-projects list, add it
to LIBS_FROM_GIT automatically. This way, when a user specifies
that a job requires a zuul-project, it gets used in testing, but
otherwise, it doesn't (pypi is used instead).
Also add information about what happens behind the scenes for both
LIBS_FROM_GIT and plugin dependencies.
This moves the check performed in check_libs_from_git to
a helper function which is installed for most kinds of
installations. This means that if someone sets LIBS_FROM_GIT to
"foobar", devstack won't error anymore, as nothing is going to
try to install foobar, therefore the check won't run on that.
However, as we move to automated generation of the local config,
that error is not likely to happen. This check was originally
added due to an error in the upper-constraints file (where a
constraint name did not match a package name). This location of
the check would still catch that type of error.
Change-Id: Ifcf3ad008cf42d3d4762cfb3b6c31c93cfeb40db

Change-Id: Ia77163f330f674146b369dfebea56bd97820057e
Depends-On: https://review.openstack.org/554662 

On Fedora 27, the default Pytho 3 install prefix changed from /usr
to /usr/local:
https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
Closes-Bug: #1741901
Change-Id: Id40620efdf173189df053b5d380a801092933f83

python-openstacksdk does not match its pip name which is openstacksdk.
So setting python-openstacksdk in LIBS_FROM_GIT leads to devstack
thinking there is a problem.
Put in a workaround for now. It would be better to either:
 a) rename python-openstacksdk repo to openstacksdk
 b) rename the pip name for openstacksdk back to python-openstacksdk
 c) add general support in the various GIT hashes for a pip name
Change-Id: I57cf95763d54ad2060a4ce2af91c3ba18ca04db0

The 'pip list' command prints the "safe name" which converts _'s to
-'s amongst other things; e.g. glance_store becomes
 "glance-store 0.21.1.dev22 /opt/stack/glance_store"
Because people may use these more familiar "file system" names in
LIBS_FROM_GIT automatically convert names when checking if libraries
are installed.
Change-Id: I30524f80a341f38dfa794a8f629d859e85a4a448

In commit f0cd9a8b08 we changed to
use column format, but it checks for zero length string and
check_libs_from_git fails.
Change-Id: I97b52b80efb33749647229a55147a08afa112dd2

The old code was strip()ing the version string instead of split()ing the
version string so we always got the first character of the version
string. This worked fine as long as the pip version was single digit but
as soon as it rolls over to '10.stuff' we will compare:
 pip version 1 (instead of 10) > 6
Which fails bceause 1 is less than six. Instaed we really do want to
compare 10 > 6 so use split on '.' instead.
Change-Id: Ic7d0c04d7fa77774ab2d70fb9d11f182becec553

The pip list command supports a --format=columns option which outputs
things in space delimited columns. Switch to using that.
Change-Id: I5140a7d83bf567b1c3c67516112eb4c57074fa53

Strip the [<extras>] string from a <package_dir>[<extras>] argument
when looking for the package directory. Explain what the heck is
going on.
Change-Id: I79beb5c3e9e7c35c91cdd0d5a1d91532bebc4b6d
Closes-Bug: #1721638 

As described in the change, "pip freeze" has issues with the way
zuulv3 clones repos without a remote. This is an attempt to use "pip
list" to check for local install
Change-Id: I33d25f86b6afcadb4b190a0f6c53311111c64521

lib/nova does a pip re-install of libvirt-python to rebuild the python
library incase the underlying libvirt version changed during package
installs. In offline mode, the underlying version of libvirt can't
have changed; so we have the situation that we've removed the libvirt
python bindings but can't reinstall them (because we're offline).
This fixes that particular situation, but skipping uninstalls in
offline mode seems generically OK.
Change-Id: I2b75d45d94d82f87d996c7570c125d46f5f99f6a
Closes-Bug: #1708369 

Some URLs are broken, so fix them.
The others are redirect to new URLs,
so replace them with new ones.
The config options of nova serial console proxy
have been gathered in nova/conf/serial_console.py.
So the description in doc/source/guides/nova.rst
is fixed.
Change-Id: Ifd81cc09969341fbf8f135a913fc6003b94e0acc

To reduce the total number of invocations necessary for pip which isn't
the quickest thing ever (due to needing to evaluate constraints and deps
lists and what is currently installed) combine the main installation of
software with its test-requirements.txt file which should roughly halve
our pip invocations.
Change-Id: Ibcc3264136e66d34a879ad1c90a62e1bb6a84243

'pip uninstall' will hang running stack.sh if it has to
prompt the user for input, use -y.
Change-Id: Ic94639e444b87fd3538463d5a51c01a0208a2ab2
Closes-bug: #1691172 

libvirt-python compiles against the currently installed libvirt. If
you upgrade that, it needs to rebuild, however it won't change
versions, so pip install just noops. Force an uninstall / reinstall of
it every time to handle potential upgrades of libvirt.
Change-Id: If34541b34aa6d55eedaf6c603fd1fe92eb887308

Add packages required to run devstack with USE_PYTHON3=True.
Change-Id: Iee43c9335bd82c10cfaeffb02d1d99290c34bb83

When variables use the 'declare' directive, it is by default a local
variable. While other variables have global scope.
For example:
 declare -A AN_ARRAY # local in scope
 foo=1 # global in scope
This causes errors to occur as some of the variables will be local only
and others will be global.
Update the code, as appropriate, so that variables using the 'declare'
directive also include the '-g' flag to have them also be global. Not
every instance of a declared variable has been updated.
Closes-Bug: #1669509
Co-Authored-By: John L. Villalovos <john.l.villalovos@intel.com>
Change-Id: I2180b68fe861ad19c6d4ec0df0f9f8a528347862

An initial install for devstack-tools, this will need to use all the
fun pip extra variables for installation, however the current
pip_install always prefers python2, and we only want to do python3
here.
Change-Id: I3dcdb35130f76fad81cb7b0d4001b7e96efbbd84

When installing a library from source and python 3 is enabled, first run
the installation process with python 2 enabled to ensure the library is
also installed under python 2 for any services not yet running under
3. The python 3 version is installed second so that command line tools
installed with libraries are installed under python 3 when python 3 is
enabled.
Change-Id: Ibb0f7a68d21081bf7652a0c1515080c0c54888ca
Signed-off-by: Doug Hellmann <doug@doughellmann.com>

Add variables ENABLED_PYTHON3_PACKAGES and DISABLED_PYTHON3_PACKAGES to
work like ENABLED_SERVICES and DISABLED_SERVICES and to manage which
packages are installed using Python 3. Move the list of whitelisted
packages in pip_install to the default for ENABLED_PYTHON3_PACKAGES,
except swift which is not enabled by default for now.
Add enable_python3_package and disable_python3_package functions to make
editing the variables from local.conf easier.
Add python3_enabled_for and python3_disabled_for functions to check the
settings against packages being installed by pip.
Update pip_install to check if python3 is disabled for a service, then
see if it is explicitly enabled, and only then fall back to looking at
the classifiers in the packaging metadata.
Update pip_install messages to give more detail about why the choice
between python 2 and 3 is being made for a given package.
Change-Id: I69857d4e11f4767928614a3b637c894bcd03491f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>

Previous this was set the zake, but that was revert to missing
dependencies issue and because zake is a test fixture and not somthing
to deploy.
This change configures the Cinder dlm with this one is zookeeper.
And it installs tooz and the extra dependencies needed for the
zookeeper driver.
To do it, this commit have to introduce a new method for package
installation: 'pip_install_gr_extras package extra1,extra2'.
Change-Id: Idca310c08e345db59840eb31434c6cb1f849fa70

Really close to getting swift and keystone under uwsgi working, so
let's white list them. Won't affect any existing jobs, so we should
be good.
Change-Id: I51d56d16a5b175bd45dee09edc0b2748d72a5d06

Use trueorfalse to normalize the values for USE_PYTHON3
Install 3.5 instead of 3.4 When USE_PYTHON3 is specified.
Also, since not many packages are classified correctly, fallback
to looking for just "Programming Language :: Python :: 3" and
log a message for the package to highlight the problem.
Also special case some services that are *almost* ready
Depends-On: Id48e1b328230fcdf97ed1cb4b97f4c3f9cf6eb8a
Depends-On: Ib7d9aa0e0b74a936002e0eea0b3af05102b06a62
Change-Id: I243ea4b76f0d5ef57a03b5b0798a05468ee6de9b

On Ubuntu Xenial there is no way to install python3.4
Use value specified in PYTHON3_VERSION
Change-Id: Ibc69b1c8270bdd240c82cf2acfdfd0730ef0f182

In the 25.0.0 release [1] of setuptools during any install
operation the package in not overwritten. If a package is
installed from another requirement via pip and then it is
installed again from git, it is not updated causing
check_libs_from_git to fail.
[1] https://setuptools.readthedocs.io/en/latest/history.html#v25-0-0
Change-Id: Ibaa1d4157816ea649f4452756fbde25951347001
Closes-Bug: #1605998 

For client debugging that invokes multiple libs it can be useful
to have all libs directly in git and not listing all of them
in LIBS_FROM_GIT.
TrivialFix
Change-Id: Ie631cc4045231ebbe8177d2d113e47e4bf83f61c

It seems pip distinguishes paths with .. or extra / for constraints.
For example, the following directories are considered different.
 /path/to/dir
 /path/to//dir
 /path/to/dir/subdir/..
This commit tries to normalize the given directory name to avoid
"Could not satisfy constraints for 'xxxx': installation from path
or url cannot be constrained to a version" error due to directory
name mismatch.
Reference: https://github.com/pypa/pip/pull/3582
Closes-Bug: #1542545
Change-Id: Iae9d58c27d3b10bca16e4a471507c4d5c16439a0