openstack/devstack - devstack - OpenDev: Free Software Needs Free Tools

We have a *ton* of stuff in devstack that is very linear, specifically
the ten-ish minutes we spend loading osc to run a single API command
against something. We also generate configs, sync databases, and other
things that use one core of our worker and make our runtime longer
than it really needs to be.
The idea in this patch is to make it super simple to run some things
in the background and then wait for them to finish before proceeding
to something that will require them to be done. This avoids the
interleaving you would expect by redirecting the async tasks to a log
file, and then cat'ing that log file synchronously during the wait
operation. The per-task log file remains so it's easier to examine
it in isolation.
Multiple people have reported between 22-30% improvement in the
time it takes to stack with this. More can be done, but what is here
already makes a significant difference.
Change-Id: I270a910b531641b023c13f75dfedca057a1f1031

All jobs using ceph as a storage backend have been moved over
to using the devstack-plugin-ceph repo in project-config so we
should be safe to remove the now unused lib/ceph file.
The files are left in place because the devstack plugin does not
install xfsprogs but it's used by the create_disk function.
And the ceph cinder backend file is left in place since the
devstack-plugin-ceph repo uses that by setting
CINDER_ENABLED_BACKENDS=${CINDER_ENABLED_BACKENDS:-ceph}.
Change-Id: I3fb09fc92bc6ab614e86d701ea46d5741a76b7a8

Before the code in the extra.d plugins was removed from the devstack
tree they could define the order they ran. When this code is decomposed
into a plugin, there is still a need to do some form of ordering. This
caused problems with the Ironic devstack plugin and Tempest because the
code is run in this order:
 1. The tempest configuration is run from extra.d, processing
 DEFAULT_INSTANCE_TYPE, and writing the flavor_ref to tempest.conf
 2. The Ironic devstack plugin is run, creating the flavor needed for
 DEFAULT_INSTANCE_TYPE
This leads to build failures as tempest can not find the required
flavor, so it writes which ever flavor it can find at the time into
flavor_ref. Ironic now has code it its devstack plugin duplicated from
the tempest plugin to work around this problem until this is merged.
This patch fixes this by using the test-config phase to move the tempest
plugin as late as possible in the devstack process.
Change-Id: I3d98692e69d94756e0034c83a247e05d85177f02

For the tempest plugin install inside the tox venv to hold we need to
ensure that it's the last thing run that touches the tox venv before
devstack ends. Otherwise there is a chance we'll recreate the venv in
a later step of installing and configuring tempest. This commit
moves the plugin installation into it's own function and calls that
function as last phase of the tempest setup to make sure it runs last.
Change-Id: Ie253171537e8c5a9887cc30aba1cad4b31e57663

For testing reasons it's typically very useful to have a second non
admin user to cross check that it can't do a thing to the first
user. It was useful enough we always created it with tempest (though
we didn't always use it).
This makes devstack always create an alt_demo user, which is available
in occ as devstack-alt. This will help us unwind some of the keystone
v3 breaks with functional tests using keystone cli to build this
second user.
Change-Id: Iaaf02469180563e2d8c413fee0ee66ada2296cfa

thirdparty.boto tests was removed from tempest and this section
is no needed more
Change-Id: Ibec0534d118423be7a51c2e2b82f1c61531503a0

This removes all of the ironic code from the devstack tree, in favor of
the devstack plugin in Ironic's tree.
Depends-On: I659e3de5c64df798441798ff48ba5c9c0506585a
Depends-On: I2c52bc014f1b0dbc6b0ae22a4deb4132b4c28621
Change-Id: I5125fce295c79600781469c2f48bea80e7600081

This adds a flag to skip ironic code if the ironic devstack plugin is in
use. This flag will be set to true in ironic's devstack plugin to
indicate that the plugin should be in control, rather than devstack.
This is for the transition period only, and will be removed with the
rest of the ironic code in the devstack tree, once the gate
is configured to use the ironic plugin.
Change-Id: Id01d97fd13fa9f866d645ec5077834ddb78b2b89

override_defaults phase was added to devstack in [1]
but documentation was pending. This patch adds the
same. For history around override_defaults, one can
refer to the mail thread [2]
Also fixes a small typo
[1]: https://review.openstack.org/#/c/167933/
[2]: http://lists.openstack.org/pipermail/openstack-dev/2015-March/059621.html
Change-Id: I1b58ca0ce0e4b85a1dbd710b4c426606fd4dcf45

Controller node wouldn't install libvirt package. The package will only
been installed at nodes which had enable nova-compute.
We only need to configure libvirt secret if it had enable nova-compute.
Change-Id: I9cd6baf1820ce9f71c276d7e8b670307833578a5
Closes-Bug: 1515960

Zaqar's devstack code has been moved into a plugin in the Zaqar repo.
This patch removes the remaining code from devstack.
Depends-On: Iceefabb6cd528b23075a91e8039b8264eb3f33f5
Change-Id: Ifcf54fa2d4a5bf49b6757b593bb70cdeda8edb2a

Once the Sahara related code moved to Sahara repo and used, we can
remove Sahara specific code from Devstack.
Partial-Implements: bp sahara-devstack-intree
Change-Id: I34412b5cb2e86944b8555b8fd04b43556eb2bbe6
Depends-on: I2e00b2ebc59dd3be6a0539dea2985f2e801a1bd7
Depends-on: I07c3fede473030e8a110cbf5a08309f890905abf

Once the trove code is copied into the trove repo and it is used
as a devstack-plugin, we can remove trove-specific code from
devstack.
Change-Id: I8f9f1a015edb7ec1033e2eaf0b29ab15d89384ce
Depends-On: I3506dec0e6097f9c2e9267110fdfb768faa23c85

diskimage-builder is a utility rather than a service, and is already
installed in devstack via pip when required.
lib/dib was created to allow an image to be created during a devstack
run for the heat functional tests, however this approach is no longer
being taken and there are no other known uses for lib/dib.
This change removes lib/dib and moves the pip mirror building to
lib/heat so that snapshot pip packages of the heat agent projects can
be made available to servers which the heat functional tests boot.
This also removes tripleo-image-elements, which has never
been utilised, and since images won't be created
during heat functional test runs it is no longer required.
Change-Id: Ic77f841437ea23c0645d3a34d9dd6bfd1ee28714

The gantt source trees are dead projects, we should remove them from
devstack.
Change-Id: I0f271891846d2ad7a7a8edd975cdfb4f08419f0c

Tuskar server support is proposed as an external plugin here
I67dc8420582a40e18de7d5e00094bccc1184a7f9, we should remove Tuskar
from the main devstack tree.
Change-Id: I93e97e7ae8fd03cabc1245a10588c4474d7e3728

The code for creating service users is almost exactly the same. Abstract
this into a function that can be reused and standardized.
Change-Id: I3a4edbff0a928da7ef9b0097a5a8d508fdfab7ff

Now that we have a patch to make this pluggable [1], lets remove this
from the devstack repository.
[1] https://review.openstack.org/#/c/153700/
Change-Id: Ia23f99476ec16df9cc12f63864189835a09eb644

The default project means that a user gains token scoping information
for a project if they don't specify another. This is something we want
to discourage for user creation. User's should specify there own
authentication scope when they authenticate.
Change-Id: I42c3060d59edfcd44d04cd166bad500419dd99bc

This makes a bunch of variable cleanups that will let -o nounset
function, for the time being we hide nounset behind another setting
variable so that it's not on by default.
Because this is bash, and things are only executed on demand, this
probably only works in the config it was run in. Expect cleaning up
all the paths to be something that takes quite a while.
This also includes a new set of unit tests around the trueorfalse
function, because my change in how it worked, didn't. Tests are good
m'kay.
Change-Id: I71a896623ea9e1f042a73dc0678ce85acf0dc87d

Sometimes we want to run some benchmarks on virtual machines that will be
backed by a Ceph cluster. The first idea that comes in our mind is to
use devstack to quickly get an OpenStack up and running but what about
the configuration of Devstack with this remote cluster?
Thanks to this commit it's now possible to use an already existing Ceph
cluster. In this case Devstack just needs two things:
* the location of the Ceph config file (by default devstack will look
for /etc/ceph/ceph.conf
* the admin key of the remote ceph cluster (by default devstack will
look for /etc/ceph/ceph.client.admin.keyring)
Devstack will then create the necessary pools, users, keys and will
connect the OpenStack environment as usual. During the unstack phase
every pools, users and keys will be deleted on the remote cluster while
local files and ceph-common package will be removed from the current
Devstack host.
To enable this mode simply add REMOTE_CEPH=True to your localrc file.
Change-Id: I1a4b6fd676d50b6a41a09e7beba9b11f8d1478f7
Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

Fake plugin can work with heat image. Even more, sahara uses heat
to provision instances. So, registering image in sahara image
registry.
Change-Id: Ie498fc05d0afe7f276b6f29f20a61abb58a5c676
Closes-Bug: #1402856 

We use InnoDB everywhere, so there should be no issues with long unicode
keys. Dropped charset parameter for recreate_database since it's not
needed anymore.
Change-Id: Ib768402a9337c918309030a92ab81da17269f4f6

Adds initial support for configuring and starting the tuskar-api
service.
To enable, add the following to your localrc:
enable_service tuskar
enable_service tuskar-api
The aim of this addition is to provide a more accessible (non devtest)
way for developers to run up tuskar in a familiar devstack environment.
See the official repos for more information:
https://github.com/openstack/tuskar/
https://github.com/openstack/python-tuskarclient
Change-Id: Id0c3c0d3a38100c66dbe6e3adf1f715162f99742

expand the devstack support for libraries from released versions to
support python-* clients and tempest_lib.
Depends-On: I81b0d228e7769758c61e5b0323ecfce8c8886d39
Change-Id: I26fac0ccf8fd4818e24618d56bf04b32306f88f6

Prior to this patch, if we wanted to boot a VM in Ceph using
libvirt_image_type we must had Cinder enabled. This patch allows you to
use libvirt_image_type without having Cinder enabled.
Change-Id: Ia61e6effc5a4ccba69f4fa48f6e9984f15bb8979
Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

sahara-dashboard has been removed in the commit:
 Change-Id: I1f0e93e1dee3d065c4f00d8bf2042bebc8d45a22
However, there is still sahara-dashboard in extras.d/70-sahara.sh. This
causes an error.
/opt/stack/new/devstack/extras.d/70-sahara.sh: line 7:
/opt/stack/new/devstack/lib/sahara-dashboard: No such file or directory
So this commit just removes it.
Change-Id: Iae76dc5f92a1f0a557d43d74b9b5d2c347cc23a9
Closes-Bug: #1363828 

All other projects installs their clients in the same lib file as the
main services.
Remove sahara-dashboard installation
Sahara dashboard is now merged into the Horizon and enabled by default
(when data_processing endpoint is available), so, we should stop
installing it as separated dashboard.
Change-Id: I1f0e93e1dee3d065c4f00d8bf2042bebc8d45a22

Enabling service 'dib' will install the following repos:
* diskimage-builder
* tripleo-image-elements
* os-collect-config
* os-refresh-config
* os-apply-config
These repos are already pre-fetched in devstack-gate.
This will facilitate gating on changes in these projects by building
a custom image then running the heat-slow job against that image.
diskimage_builder is pip installed from the current git checkout
if the dib service is enabled. This allows devstack gating on
diskimage-builder changes while also allowing diskimage-builder to
be installed from releases for other scenarios (for example, ironic).
Change-Id: Ia911cdee86f5b2e2ba1557e5aa8bf392b92ef555

This patch changes all Marconi references to Zaqar.
Change-Id: Iec4aeca54c0f383c995fba564bc59daa550d3747
Implements: blueprint project-rename

The new lib installs a full Ceph cluster. It can be managed
by the service init scripts. Ceph can also be installed in
standalone without any other components.
This implementation adds the auto-configuration for
the following services with Ceph:
* Glance
* Cinder
* Cinder backup
* Nova
To enable Ceph simply add: ENABLED_SERVICES+=,ceph to your localrc.
If you want to play with the Ceph replication, you can use the
CEPH_REPLICAS option and set a replica. This replica will be used for
every pools (Glance, Cinder, Cinder backup and Nova). The size of the
loopback disk used for Ceph can also be managed thanks to the
CEPH_LOOPBACK_DISK_SIZE option.
Going further pools, users and PGs are configurable as well. The
convention is <SERVICE_NAME_IN_CAPITAL>_CEPH_<OPTION> where services are
GLANCE, CINDER, NOVA, CINDER_BAK. Let's take the example of Cinder:
* CINDER_CEPH_POOL
* CINDER_CEPH_USER
* CINDER_CEPH_POOL_PG
* CINDER_CEPH_POOL_PGP
** Only works on Ubuntu Trusty, Fedora 19/20 or later **
Change-Id: Ifec850ba8e1e5263234ef428669150c76cfdb6ad
Implements: blueprint implement-ceph-backend
Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

An additional call hook for the extras.d plugins that is called before
any service installation occurs turns out to be needed by Ceph and
useful for other plugins.
This is called between the installation of the system packages listed
as prerequisites and the installation of the actual services.
Change-Id: Id723bdc7542182d2b2eff40c0dc0127307f094c5

only if nova is enabled.
Change-Id: I4627ab97eccd7fd4a2ffc31b6a0bb939b2343659

OpenDaylight
OpenDaylight needs to know the physical network mapping
used with vlan tenant isolation. This is similar to how
the bridge_mappings cofig is used to map the physical
network to the bridge,
i.e. bridge_mappings=physnet1:br-eth1,physnet2:br-eth2.
OpenDaylight needs the physical network device itself:
provider_mappings=physnet1:eth1,physnet2:eth2.
OpenDaylight needs the mapping because the device port is
used when pushing flows. The neutron API's provide the
physical provider network value for example as physnet1.
DocImpact:
Use of VLANs with ML2 and the OpenDaylight mechanism driver
requires OpenDaylight Helium or newer to be installed.
Closes-Bug: #1315492
Change-Id: If7af9ce5735b01c35315a54c79355ca87b72c4b4

OpenDaylight integration with Openstack requires some extra configuration
information to be set in the ovsdb via the other_config value. Currently
only the local ip address is set. bridge_mappings needs to be added to support
vlan isolation.
This requires changing the form of the existing ovs-vsctl command to set
the other_config value. The current command does a replace so the last
command to run overwrites any other values. The command is changed to
append values instead.
Change-Id: Ia001ecf27c8f5ace05af014ecce0b13f6d490ac1

Now that https://review.openstack.org/#/c/85589/ is merged, the
OpenDaylight ML2 driver requires that url, username and password
parameters are set.
Change-Id: Iecbdd3275387bea145a5d274d359a15eceaee6b7
Closes-Bug: #1304226 

Add ability to create/register qemu vms for Ironic testing purposes
Implements bp:deprecate-baremetal-driver
Change-Id: If452438fcc0ff562531b33a36cd189b235654b48

There are several backward compatibility nits.
Change-Id: I93cac543375896602d158860cc557f86e41bcb63

* Move remaining role creation to create_keystone_accounts()
* Move glance creation to create_glance_accounts()
* Move nova/ec2/s3 creation to create_nova_accounts()
* Move ceilometer creation to create_ceilometer_accounts()
* Move tempest creation to create_tempest_accounts()
* Convert moved code to use OpenStackClient for setup
* files/keystone_data.sh is removed
Note that the SERVICE_TENANT and ADMIN_ROLE lookups in the other service
implementations are not necessary with OSC, all operations can be done
using names rather than requiring IDs.
Change-Id: I4283ca0036ae39fd44ed2eed834b69d78e4f8257

Should source ``lib/opendaylight`` in ``extras.d/80-opendaylight.sh``
only when appropriate services are enabled.
Fix for bug/1290033
Change-Id: Ifa470e1e132029f3c5bf255f27c4e96373b339a8
Signed-off-by: Roey Chen <roeyc@mellanox.com>

This adds support for running OpenDaylight as an OpenStack Neutron plugin
under devstack. This entails downloading the latest version of OpenDaylight,
configuring it, and running it as a service under devstack. This code also
includes pieces which configure Open vSwitch on each devstack node to point
at OpenDaylight as their OpenFlow and OVSDB control interface. This is
required for compute hosts, which will not be running any Neutron software
on them at all. This post-devstack configuration is handled in the extras
directory because of the fact there is no Neutron code running on the compute
hosts themselves.
Closes-bug: #1273917
Change-Id: I696e7c7fe63c835f90c56105775def305a702877

We're doing to use common keystone configuration approach - section
keystone_authtoken with config opts from the python-keystoneclient
auth_token middleware.
Change-Id: Ibbe0c76ee3b00045f5cb5134bd7661e9cef6ccdd

cleanup_ironic should not be started on "unstack" phase,
otherwise API service not restarted because auth cache
directory removed.
Change-Id: I3da86b9fb8c3ce1185aff05df0fa83cf259708f4