openstack/devstack - devstack - OpenDev: Free Software Needs Free Tools

It was kept for compatibility in renaming which was merged long ago[1],
and is no longer necessary.
[1] 3ed99c0b27
Depends-on: https://review.opendev.org/c/openstack/grenade/+/936095
Change-Id: I6a66359c19d0385beafb4c5e57b6ec3cd6d9cc54

Module lib/neutron was introduced long time ago as new module to deploy
neutron. It was intended to replace old lib/neutron-legacy module. But
since very long time it wasn't really finished and used by anyone and
lib/neutron-legacy is defacto standard module used by everyone to deploy
neutron with devstack.
In [1] unfinished lib/neutron was deprecated and now it's time to remove
it from the devstack code.
This patch also renames old "lib/neutron-legacy" module to be
"lib/neutron" now.
Previously "old" lib/neutron-legacy module was accepting neutron
services names wit "q-" prefix and "new" lib/neutron module was accepting
services with "neutron-" prefix. Now, as there is only one module it
accepts both prefixes.
For historical reasons and to be consistent with old lib/neutron-legacy
which was widely used everywhere, services will be named with "q-"
prefix but both prefixes will be accepted to enable or disable services.
This patch also moves _configure_neutron_service function to be called
at the end of the "configure_neutron" after all agents and service
plugins are already configured.
[1] https://review.opendev.org/c/openstack/devstack/+/823653
Related-bug: #1996748
Change-Id: Ibf1c8b2ee6b6618f77cd8486e9c687993d7cb4a0

We have a *ton* of stuff in devstack that is very linear, specifically
the ten-ish minutes we spend loading osc to run a single API command
against something. We also generate configs, sync databases, and other
things that use one core of our worker and make our runtime longer
than it really needs to be.
The idea in this patch is to make it super simple to run some things
in the background and then wait for them to finish before proceeding
to something that will require them to be done. This avoids the
interleaving you would expect by redirecting the async tasks to a log
file, and then cat'ing that log file synchronously during the wait
operation. The per-task log file remains so it's easier to examine
it in isolation.
Multiple people have reported between 22-30% improvement in the
time it takes to stack with this. More can be done, but what is here
already makes a significant difference.
Change-Id: I270a910b531641b023c13f75dfedca057a1f1031

Right now a system configured with the ceph plugin will not survive
a reboot because the backing disk we create and mount isn't mounted
at startup, preventing ceph from starting and the rest of nova/glance
from working.
This makes create_disk() idempotently write an fstab rule for the
disk we make, and adds a destroy_disk() handler for cleanup.
Change-Id: I50cd4234f51a335af25be756bd2459dca5aa343c

Previously pyc files were only cleaned if clean.sh was run.
with this change a new clean_pyc_files function was introduced
with the logic that was previously in clean.sh but it is now
invoked from unstack.sh
With the previous behavior you could not stack with horizon
enabled then unstack and stack again due to the presence of pyc
files that were owned by root.
By moving the clean to unstack in stead of clean.sh you can
now stack, unstack and stack again without hitting the pyc issue.
since unstack is invoked by clean the existing clean.sh behavior has
not changed in practice except for the fact the pyc files are
removed sooner in the process.
This change also removes support for findutils < 4.2.3
Ubuntu 12.04 and CentOS 6 both have 4.4.2 since they were
released 8 years ago and are now EOL its fair to assume
that all modern distros have 4.2.3+
https://repology.org/project/findutils/versions
Change-Id: I13c9aad9be7e0930a0d875b7d382090caf0b8982

We dropped the ability to override USE_SYSTEMD to False when we deleted
screen support in [0], so we can also clean up any conditionals based
on it.
Also clean up the logging setup functions, dropping local vars for
parameters that we don't actually support anymore.
[0] I8c27182f60b0f5310b3a8bf5feb02beb7ffbb26a
Change-Id: I5cbce9f2c42e111761e8689447b3f8cbb7ea2eb5

Change-Id: Id777576d8876d7ba257f0243f3b4ce5756dd9b58

These appear to have been missed in commit 'cdba1b371'.
Change-Id: Ibb7fd7a0d54750c16360dfceb8e6b024ed8f504e

* Check KEYSTONE_DEPLOY flag and cleanup appropriately
* When we stop process, we should not wipe uwsgi config we should
 remove files only on cleanup
* We should not call cleanup *BEFORE* configure, we are just wiping
 out the uwsgi ini files
* cleanup_placement should be called from clean.sh
Change-Id: I066f5f87ff22d7da2e3814f8c2de75f2af625d2b

Commit 5edae54 introduced the usage of systemd in Devstack. This allowed
the transition away from 'screen'. Systemd needs "user unit files" to
describe the services. Currently, those unit files get only created when
an openstack service (n-cpu, c-sch, g-api, ...) is in the list of enabled
services (`ENABLED_SERVICES`). This means, when Devstack is fully stacked,
there is no way to start the systemd unit of an openstack service which
is *not* in that list.
This commit changes that behavior, and creates the systemd unit files
independently of the list ENABLED_SERVICES. This means, when Devstack
is fully stacked, I can start a systemd unit of an openstack service which
wasn't in the ENABLED_SERVICES list. This allows more flexible lifecycle
management of openstack services in the gate, which is useful for tests
which test components which are not in the "default configuration" (e.g.
the "nova-serialproxy" service).
The `clean.sh` script purges all traces of systemd user unit files created
by devstack.
Change-Id: I0f7e1ee8723f4de47cbc56b727182f90a2b32bfb

There was detection code in clean.sh to only run it if a screen
session was found, but in systemd world, that's obviously not
true. This was causing me (and others) substantial confusion.
Change-Id: I204e94cd86b8c67012aabfca74796e593151c3a4

Findutils added in release 4.2.3 a new --delete action for deleting
matching files. This action performs better than -exec rm {} \;
because it doesn't have to spawn an external process. This change
uses a new action whenever is possible.
Change-Id: Iff16a86b18e924cfe78ac7c6107910940ce51e03

This removes all of the heat code from the devstack tree, in favor of the
devstack plugin in Heat's tree.
Depends-On: I4bed1e5cef5afa7b049b07640086a86a3f881e13
Depends-On: Ic392bcc24bc374ee8511a94f1d8f6ac23131c7e3
Change-Id: I5b60422bf1f5fa78aa8f3383f7a222e0356d9e42

Uses lib/placement, but relies on some functionality from
lib/nova. This leads to some weirdness since the nova has
special status in stack.sh. If/when placement is extracted
it may be good to follow the devstack plugin structure
instead.
Because the placement code is currently a part of nova, there
are dependencies in lib/placement on a some $NOVA_* variable
and, if virtenv is being used, the virtualenv used by nova.
Because placement currently runs using nova's configuration
settings, not a lot actually happens in lib/placement: apache
is configured and keystone accounts and endpoints are created.
If PLACEMENT_DB_ENABLED is true then a separate placement db
will be configured.
When complete the initial version of the placement service will
provide support for managing resource providers, inventories and
allocations.
The placement api only runs under mod-wsgi.
Change-Id: I53dd3e6b41de17387a0e179fc9ac64c143b6a9eb

The horizon cleanup function wasn't being called at all during
cleanup which left the Apache configuration.
Change-Id: Iff5336d0c5e79cfc82f1c648afaabb869d86020e

Currentlly, the *.pyc files could not be removed in any scripts or
functions. But the redundant files would lead stack.sh not to find the
correct script for some versions after branch switched from master to
stable/mitaka in migration_helpers.sync_database_to_version.
So this commit adds the process of cleaning all the *.pyc files in
clean.sh.
It is needed to execute clean.sh before re-stack.sh to prevent the
exception.
Change-Id: I9ba0674d6b20b13c0a26b22cd5d1939daa121a94
Closes-Bug: #1599124 

Background for this work can be read on the mailing list:
http://lists.openstack.org/pipermail/openstack-dev/2016-May/094063.html
Usage of the new Neutron is by setting the following in
ENABLED_SERVICES:
* neutron-api
* neutron-l3
* neutron-agent
* neutron-dhcp
* neutron-metadata-agent
For now, the new neutron library supports just the ML2 plugin, with the
Open vSwitch and Linux Bridge agents supported. All other Neutron
plugins should be creating their own DevStack plugin if they wish for
DevStack to support them. Many of them already do.
Other notable changes compared to neutron-legacy:
* Rely on the Neutron defaults, and force Neutron to make
 sane defaults instead of all kinds of knobs in DevStack.
* Default to rootwrap daemon support
* Use the security group driver by default
* interface_driver can now use NEUTRON_AGENT (linuxbridge, openvswitch), since
 they are entrypoints in neutron's setup.cfg
* Use NEUTRON_AGENT variable to determine which agent to run
 Works with NEUTRON_AGENT set to either "linuxbridge" or "openvswitch"
 Default is openvswitch for the time being.
* Set ML2 configuration for VXLAN support
* Remove Xen hypervisor stuff - it should be a plugin
* Move L3 crud into separate service file:
 There's a lot of L3 configuration that was in the main neutron file, but
 a lot of it is self contained and can be moved into its own file.
 The new l3 service file will contain all the previous L3 plumbing and
 configuration that the OpenStack Gate expects, while also eventually
 moving the whole l3 network creation step into a single hook that can be
 overridden by plugins.
* Introduce a check for a function "neutron_plugin_create_initial_networks" which
 will become the mechanism through which different topologies, and
 networking plugins can create and wire the initial networks that are
 created during a stack.sh run.
The new lib/neutron is considered experimental, and followup patches
will build upon this one. Existing users of lib/neutron-legacy should
remain unharmed.
Co-Authored-By: Hirofumi Ichihara <ichihara.hirofumi@lab.ntt.co.jp>
Co-Authored-By: Dean Troyer <dtroyer@gmail.com>
Change-Id: I31b6362c6d9992f425f2dedbbeff2568390a93da

devstack failed to install because glance:
Could not determine a suitable URL for the plugin
patch I618ea8e27b49af360c905df85af06d9b1eef8407 tries
to fix this problem, but with a wrong way because path is not
correct, the clouds.yaml is not under /path/to/devstack/~/.config/openstack/
but ~/.config/openstack.
patch I8af6bd465f74099c560dddba6b5221dd79cbc965 tries to
fix this problem, but with a worng way to specify the path,
~$STACK_USER/.config/openstack/clouds.yaml will not expand with
a variable, only const string can.
$ whoami
zqfan
$ touch ~/.config/openstack/clouds.yaml
$ export STACK_USER=zqfan
$ rm -rf ~$STACK_USER/.config/openstack/clouds.yaml
$ ls ~/.config/openstack/
clouds.yaml
Change-Id: I549817d2f4638be615991c1726b39d270ba71357
ref: I618ea8e27b49af360c905df85af06d9b1eef8407

After staring at a bunch of logs, try to crisp up the ps4 output for
maximum readability.
This also adds PS4 to all calling scripts by having a common PS4 in
stackrc. It should make understanding when clean fails a bit more
straight forward.
Change-Id: Ia1f8dff5c1102c32c146a020a9f033c65d2c50de

the /etc/openstack directory needs to be removed with sudo privs,
the loop at the end will not suffice.
Change-Id: Icc0ac45f9216d538ca214176d90241f973a4687b

There have been a few reported errors of folks running into
issues with old clouds.yaml files that linger around. Remove
these if the user runs clean.sh.
1. http://lists.openstack.org/pipermail/openstack-dev/2016-January/084283.html
2. https://ask.openstack.org/en/question/84825/glance-upload_image-fails-in-latest-devstack/
3. https://ask.openstack.org/en/question/86071/devstack-installation-failure/
Change-Id: I618ea8e27b49af360c905df85af06d9b1eef8407

The existing GetOSVersion has a lot of unused code which is wrong in
several ways
 - the only path tested in upstream CI is with lsb_release, because
 it's pre-installed on all nodes
 - the /etc/redhat-release checking probably still works, but is
 unnecessary
 - If using lsb_release, os_UPDATE has never actually been set.
 - the /etc/SuSE-release branch checking is broken if the lsb package
 is actually installed. lsb checking does not set os_UPDATE but yet
 the SuSE DISTRO setting relies on this to set a patch level (and so
 does some of the rpm tags). SuSE 11 is up to update 3, but the rpm
 matching is stuck hard-coded to update 2. I'm guessing
 installation is actually broken there.
 - the debian checking branch is broken. The VERSION tags have been
 removed and were not supposed to be relied on anyway (see notes in
 [1])
This simplifies things:
 - remove OSX checking (moved here after discussions in
 I31d0fdd30928ecc8d959a95838b1d3affd28ac6f)
 - only use the output of lsb_release.
 - A small best-effort check to pre-install lsb packages if not
 detected (that avoids chicken-egg-problem of package-install
 wrappers relying on os_* flags).
 - The unset os_UPDATE is removed. It's only previous use was for
 setting separate suse versions in the DISTRO element for matching
 during package installs (since removed)
 - DISTRO setting is modified to use the parts of os_RELEASE it wants.
 Per-above, this is the correct place to parse out specifics.
 - Call out the is_* functions, which are a better way to detect
 platforms
 - Export the variables as read-only, since they shouldn't be reset
[1] http://sources.debian.net/src/base-files/7.5/debian/changelog/
Change-Id: I46a2c36d95327087085df07cb797eb91249a893c

This removes all of the ironic code from the devstack tree, in favor of
the devstack plugin in Ironic's tree.
Depends-On: I659e3de5c64df798441798ff48ba5c9c0506585a
Depends-On: I2c52bc014f1b0dbc6b0ae22a4deb4132b4c28621
Change-Id: I5125fce295c79600781469c2f48bea80e7600081

The interactive password prompt currently saves to .localrc.auto
However, this is removed when you re-run stack; that is required as it
is how we source the localrc bits of local.conf, and we want the
users' changes to be picked up.
The passwords, however, should remain constant, because everything has
already been setup with them. So write them to a separate file. Note
we source before localrc so it can still overwrite them.
Some minor flow-changes too
Change-Id: I9871c8b8c7569626faf552628de69b811ba4dac0
Closes-Bug: #1505872 

The ceilometer project is moving to using a devstack plugin rather
than having ceilometer in the base devstack. This is to allow
greater control and flexibility.
Change-Id: I413ab159474b7d7231ad66d3a482201f74efe8a8

When clean.sh is executed, it shows "command not found" messages.
Commands are defined in lib/lvm, however lib/lvm doesn't include clean.sh.
This pache add lib/lvm to clean.sh.
Change-Id: I56672e949d25f7cdcda879badd992f849d06c749
Closes-Bug: 1486392

os-client-config consumes clouds.yaml files, which is now supported in
python-openstackclient and shade. It also makes for a non-envvar way of
getting config info into functional tests.
Change-Id: I1150b943f52f10d19f8434b27e8dde73a14d7843

Once the trove code is copied into the trove repo and it is used
as a devstack-plugin, we can remove trove-specific code from
devstack.
Change-Id: I8f9f1a015edb7ec1033e2eaf0b29ab15d89384ce
Depends-On: I3506dec0e6097f9c2e9267110fdfb768faa23c85

Change-Id: Id957f585d2aa93075b138d462d6076d2d70d450e

clean.sh picks the parent of LOGFILE and wipes it clean! So if you
set it to a log file in the users root directory, you lose everything
We should delete just the LOGFILE and cleanup LOGDIR and SCREEN_LOGDIR if
they are explicitly set.
Change-Id: I45745427dcaed3dcf0b78cc9ed680833d9d555e8

Preparing to refactor lib/neutron to support Neutron as the default
network config. lib/neutron will be renamed internally and refined
to support a couple of specific configurations.
Change-Id: I0d3773d14c4c636a4b915734784e7241f4d15474

Change-Id: I67b970992479e50dc054f8c4a77a20e724e3e305

Introduce the tooling to build virtual environments.
* tools/build_venv.sh: build a venv
* introduce lib/stack to house functionality extracted from stack.sh that
 is needed in other places, such as Grenade; start with stack_install_service
 to wrap the venv install mechanics
* declare PROJECT_VENV array to track where project venvs should be installed
* create a venv for each project defined in PROJECT_VENV in stack_install_service()
Change-Id: I508588c0e2541b976dd94569d44b61dd2c35c01c

Building a bunch of virtual envs later is going to be tedious if we do not
pre-cache certain annoying-to-build packages.
* tools/build_wheels.sh: pre-build some wheels for annoying package installs
* list distro package dependencies in files/*/venv
* list packages to pre-build as wheels in files/venv-requirements.txt
* install database Python modules when setting up the database
Change-Id: Idff1ea69a5ca12ba56098e664dbf6924fe6a2e47

This adds the test infrastructure for testing that unstack.sh and
clean.sh do the right thing, and actually stop what's expected. This
is designed to be used in upstream testing to make unstack and clean a
bit more certain.
It includes numerous fixes to make these pass in an errexit
environment with the gate config. The scripts still don't run under
errexit because we don't assume we've handled all possible cleanup safely.
Change-Id: I774dfb2cc934367eef2bb7ea5123197f6da7565b

This makes a bunch of variable cleanups that will let -o nounset
function, for the time being we hide nounset behind another setting
variable so that it's not on by default.
Because this is bash, and things are only executed on demand, this
probably only works in the config it was run in. Expect cleaning up
all the paths to be something that takes quite a while.
This also includes a new set of unit tests around the trueorfalse
function, because my change in how it worked, didn't. Tests are good
m'kay.
Change-Id: I71a896623ea9e1f042a73dc0678ce85acf0dc87d

This has been deleted from the nova tree, we should purge it from
devstack, as it will not work any more.
Change-Id: I20501fec140998b91c9ddfd84b7b10168624430a

Icehouse is for long behind our back, so let's remove that hack.
Conflicts:
	lib/oslo
This reverts commit db5fadb5cb.
Change-Id: I06d3b0a8779ba51e05c439832ef3b7dbdc97ded1

The docs/source has been introduced by change-id:
Ide5f7d980a294c7a9f8a3decaed0939f1c239934, but clean.sh try to clean
whole docs direcotry, which is not right.
Meanwhile, there are directories in FILES_TO_CLEAN variable, which means
rm -f will not work. We need to use rm -rf.
Change-Id: I4e10d70d5638ce7e7a9215f5bd848422aa90aad3
Closes-Bug: #1335399 

Add the /etc/neutron directory to the list of directories
that are deleted.
Change-Id: Ia560179a3cf121f8d09a0c10c8b68375fdd13be7

Change-Id: Iefe1a34b09958d21d2516fdce0937e00e32afcb5

oslo_clean is still needed at this point, removing it was
premature, especially for upgrade testing.
Change-Id: Ic845d835f587923423f83ac698bd825f3fa5dd1f

libraries in openstack shouldn't be installed editable, as it
causes all manner of issues (especially complicated by the use
of namespace packages). Install these globally as part of the
devstack installation process.
Change-Id: I11acb169e74069be0618e57496ff342f9e788493

clean.sh was incorrectly looping over the list of files to remove. Because of
this the files were not being removed.
Change-Id: Ie0559e1d396a4d35df6a12dfbceefa7eb261bac5
Closes-Bug:1285924