openstack/devstack - devstack - OpenDev: Free Software Needs Free Tools

ETCD_DOWNLOAD_URL is set to github url, in our CI, we can point
ETCD_DOWNLOAD_URL to a url in tarballs.openstack.org possibly
in devstack-gate
Download the etcd binaries and drop them into /opt/stack/bin and
use it from there. Cache the tgz for subsequent use (local workflow)
daemon-reload is called twice once from inside the write_user_unit_file
and then when we adjust the entries with additional things recommended
by the etcd team. We need a better way to do this in the future.
Added a TODO to verify the downloaded artifact later. The etcd team
posts gpg signature, we could verify that or run sha256sum and hard
code that in lib/etcd3 file. We would have to update it whenever we
bump the etcd3 version.
We use the public key "CoreOS Application Signing Key <security@coreos.com>"
with ID FC8A365E to verify the integrity of the downloaded file
Any jobs that need to be run on architectures where v3.1.7 is not available
should rey the v3.2.0-rcX release candidates. We can switch to v3.2.0
when it gets released.
Initial version of this code was borrowed from the dragonflow
repo:
http://git.openstack.org/cgit/openstack/dragonflow/tree/devstack
Change-Id: Ibbb430fb1dbf66942168e0cb52d990ab6a2eb8d7

tls-proxy is the way we're now doing a standard install using https
between services. There is a lot more work to make services directly
handle https, and having python daemons do that directly is a bit of
an anti pattern. Nothing currently tests this in project-config from
my recent grepping, so in the interest of long term maintenance,
delete it all.
Change-Id: I910df4ceab6f24f3d9c484e0433c93b06f17d6e1

To avoid using legacy functions accidentially.
Depends-On: Ida1f83b6b3ef9b76be13c063c7e35a8703214078
Change-Id: I3ff136fc8330c92007cdfe91b77d7f9865eabd8d

This removes all of the heat code from the devstack tree, in favor of the
devstack plugin in Heat's tree.
Depends-On: I4bed1e5cef5afa7b049b07640086a86a3f881e13
Depends-On: Ic392bcc24bc374ee8511a94f1d8f6ac23131c7e3
Change-Id: I5b60422bf1f5fa78aa8f3383f7a222e0356d9e42

Previously the usage of neutron debug ports was removed by
5e01c47e4d but there was still call to
teardown_neutron_debug. Recently a change to devstack-gate
1d6cc0771a3399300117f488e9d71e7ea46a4d82 caused that call to be
triggered and breaking the gate-devstack-dsvm-updown job.
This patch deletes the call and comments regarding setup_neutron_debug
and teardown_neutron_debug.
Change-Id: Ifdacb0cec1307db469bd66f551474539184cf2cd

In some initialization conditions (having never ran stack.sh) the
result of unstack.sh is dependent on if the user had previously
installed lvm2 or disabled the cinder service.
This change makes all results the same with a bit of LBYL.
There's also a drive-by to put a comment back where it belongs after
being accidentally moved in the related change.
Related-Change: I09b1a7bee0785e5e1bb7dc96158a654bd3f15c83
Change-Id: I9a7e052677d60cbbbdd582877f3c6c48c387f668
Closes-Bug: #1619195 

Uses lib/placement, but relies on some functionality from
lib/nova. This leads to some weirdness since the nova has
special status in stack.sh. If/when placement is extracted
it may be good to follow the devstack plugin structure
instead.
Because the placement code is currently a part of nova, there
are dependencies in lib/placement on a some $NOVA_* variable
and, if virtenv is being used, the virtualenv used by nova.
Because placement currently runs using nova's configuration
settings, not a lot actually happens in lib/placement: apache
is configured and keystone accounts and endpoints are created.
If PLACEMENT_DB_ENABLED is true then a separate placement db
will be configured.
When complete the initial version of the placement service will
provide support for managing resource providers, inventories and
allocations.
The placement api only runs under mod-wsgi.
Change-Id: I53dd3e6b41de17387a0e179fc9ac64c143b6a9eb

This was used solely by bigswitch, and everyone else has moved over to
devstack plugins. Cleaning this out makes the core logic much simpler.
Depends-On: I8fd2ec6e651f858d0ce109fc335189796c3264b8
(grenade removal)
Change-Id: I47769fc7faae22d263ffd923165abd48f0791a2c

Background for this work can be read on the mailing list:
http://lists.openstack.org/pipermail/openstack-dev/2016-May/094063.html
Usage of the new Neutron is by setting the following in
ENABLED_SERVICES:
* neutron-api
* neutron-l3
* neutron-agent
* neutron-dhcp
* neutron-metadata-agent
For now, the new neutron library supports just the ML2 plugin, with the
Open vSwitch and Linux Bridge agents supported. All other Neutron
plugins should be creating their own DevStack plugin if they wish for
DevStack to support them. Many of them already do.
Other notable changes compared to neutron-legacy:
* Rely on the Neutron defaults, and force Neutron to make
 sane defaults instead of all kinds of knobs in DevStack.
* Default to rootwrap daemon support
* Use the security group driver by default
* interface_driver can now use NEUTRON_AGENT (linuxbridge, openvswitch), since
 they are entrypoints in neutron's setup.cfg
* Use NEUTRON_AGENT variable to determine which agent to run
 Works with NEUTRON_AGENT set to either "linuxbridge" or "openvswitch"
 Default is openvswitch for the time being.
* Set ML2 configuration for VXLAN support
* Remove Xen hypervisor stuff - it should be a plugin
* Move L3 crud into separate service file:
 There's a lot of L3 configuration that was in the main neutron file, but
 a lot of it is self contained and can be moved into its own file.
 The new l3 service file will contain all the previous L3 plumbing and
 configuration that the OpenStack Gate expects, while also eventually
 moving the whole l3 network creation step into a single hook that can be
 overridden by plugins.
* Introduce a check for a function "neutron_plugin_create_initial_networks" which
 will become the mechanism through which different topologies, and
 networking plugins can create and wire the initial networks that are
 created during a stack.sh run.
The new lib/neutron is considered experimental, and followup patches
will build upon this one. Existing users of lib/neutron-legacy should
remain unharmed.
Co-Authored-By: Hirofumi Ichihara <ichihara.hirofumi@lab.ntt.co.jp>
Co-Authored-By: Dean Troyer <dtroyer@gmail.com>
Change-Id: I31b6362c6d9992f425f2dedbbeff2568390a93da

In I33525e2b83a4497a57ec95f62880e0308c88b34f, we switched
from zookeeper to dlm. Somehow this got left behind.
Change-Id: I41d13d33c9a81271d4a9752cbe98c0028a17ab1e

Change-Id: I7382eccf96d14df223777cd43344d554b8452a77

These flags were not functioning as described. Check if UNSTACK_ALL is
set in env or -a flag is set when calling script.
Change-Id: I128d32b1e74ee46e24a9eb2e81560e58137b1553
Closes-Bug: #1546687 

After staring at a bunch of logs, try to crisp up the ps4 output for
maximum readability.
This also adds PS4 to all calling scripts by having a common PS4 in
stackrc. It should make understanding when clean fails a bit more
straight forward.
Change-Id: Ia1f8dff5c1102c32c146a020a9f033c65d2c50de

The existing GetOSVersion has a lot of unused code which is wrong in
several ways
 - the only path tested in upstream CI is with lsb_release, because
 it's pre-installed on all nodes
 - the /etc/redhat-release checking probably still works, but is
 unnecessary
 - If using lsb_release, os_UPDATE has never actually been set.
 - the /etc/SuSE-release branch checking is broken if the lsb package
 is actually installed. lsb checking does not set os_UPDATE but yet
 the SuSE DISTRO setting relies on this to set a patch level (and so
 does some of the rpm tags). SuSE 11 is up to update 3, but the rpm
 matching is stuck hard-coded to update 2. I'm guessing
 installation is actually broken there.
 - the debian checking branch is broken. The VERSION tags have been
 removed and were not supposed to be relied on anyway (see notes in
 [1])
This simplifies things:
 - remove OSX checking (moved here after discussions in
 I31d0fdd30928ecc8d959a95838b1d3affd28ac6f)
 - only use the output of lsb_release.
 - A small best-effort check to pre-install lsb packages if not
 detected (that avoids chicken-egg-problem of package-install
 wrappers relying on os_* flags).
 - The unset os_UPDATE is removed. It's only previous use was for
 setting separate suse versions in the DISTRO element for matching
 during package installs (since removed)
 - DISTRO setting is modified to use the parts of os_RELEASE it wants.
 Per-above, this is the correct place to parse out specifics.
 - Call out the is_* functions, which are a better way to detect
 platforms
 - Export the variables as read-only, since they shouldn't be reset
[1] http://sources.debian.net/src/base-files/7.5/debian/changelog/
Change-Id: I46a2c36d95327087085df07cb797eb91249a893c

This attempts to make the zookeeper installation a bit more modular
(assuming that other folks will want to add other dlms as plugins),
and addresses the service start issues with zookeeper under
ubuntu/upstart.
Zookeeper is not going to be installed by default. Services need to
ask for it with use_dlm.
Change-Id: I33525e2b83a4497a57ec95f62880e0308c88b34f

In Tokyo, there was a cross project session on distributed
key locking:
https://etherpad.openstack.org/p/mitaka-cross-project-dlm
In support of the discussion there, we'll need support for
a zookeeper service in Devstack and ability to use libraries
like Tooz for DLM functionality.
In this review, we pick up some configuration files from
monasca-api and copy the lib/template to implement the
zookeeper lifecycle. Those services that need zookeeper
need to add "zookeeper" in ENABLED_SERVICES.
Change-Id: Icef26e5cdaa930a581e27d330e47706776a7f98f

Componets => Components
pluggins => plugins
Change-Id: I82634a55fd5895599099c94817af7d8d2f602859

The ceilometer project is moving to using a devstack plugin rather
than having ceilometer in the base devstack. This is to allow
greater control and flexibility.
Change-Id: I413ab159474b7d7231ad66d3a482201f74efe8a8

On some system Cinder is not enabled so we can't assume LVM is installed. So
do not try to `sed` /etc/lvm/lvm.conf or clean LVM VG if cinder is not enabled
Change-Id: I09b1a7bee0785e5e1bb7dc96158a654bd3f15c83

Once the trove code is copied into the trove repo and it is used
as a devstack-plugin, we can remove trove-specific code from
devstack.
Change-Id: I8f9f1a015edb7ec1033e2eaf0b29ab15d89384ce
Depends-On: I3506dec0e6097f9c2e9267110fdfb768faa23c85

This avoids us leaving the filter in the global lvm config. Without cleaning
this up, we can hit some failures to run stack.sh because devices are excluded
that we need to be able to see. This resets it to what it was before when
we do a cleanup.
Also, do this before we add the line, so we don't add multiple lines on
successive runs.
Closes-bug: #1437998
Change-Id: Idbf8a06b723f79ef16a7c175ee77a8c25f813244

Fix documentation build errors and RST formatting
Change-Id: Id93153400c5b069dd9d772381558c7085f64c207

during the glusterfs integration it was seen that plugins might need
to set new defaults on projects before the project files load. Create
a new override-defaults phase for that.
Intentionally not adding to the documentation yet until we're sure
this works right in the glusterfs case.
Reported-By: Deepak C Shetty <deepakcs@redhat.com>
Change-Id: I13c961b19bdcc1a99e9a7068fe91bbaac787e948

Preparing to refactor lib/neutron to support Neutron as the default
network config. lib/neutron will be renamed internally and refined
to support a couple of specific configurations.
Change-Id: I0d3773d14c4c636a4b915734784e7241f4d15474

unstack.sh always stop dstat process, however unstack.sh show 'dstat: no process
found' when dstat is disabled.
This patch stop function of stop_dstat, when dstat is disabled.
Change-Id: If9054826bed8a7fedd4f77ef4efef2c0ccd7f16e
Closes-Bug: #1435148 

is_keystone_enabled() was calling is_service_enabled(), which is what called
is_keystone_enabled() in the first place. Make it work as designed and
also change calls to use the full service name. Note that this is all
still comptible with the prior usage of 'is_service_enabled key'.
Change-Id: I9c28377ecf074b7996461d2a4ca12d88dfc4d47e

This adds the test infrastructure for testing that unstack.sh and
clean.sh do the right thing, and actually stop what's expected. This
is designed to be used in upstream testing to make unstack and clean a
bit more certain.
It includes numerous fixes to make these pass in an errexit
environment with the gate config. The scripts still don't run under
errexit because we don't assume we've handled all possible cleanup safely.
Change-Id: I774dfb2cc934367eef2bb7ea5123197f6da7565b

Change-Id: I17cb5e5d93bfe6fc6746ee530f639e2ed42da85a

Now that we have a working external plugin mechanism stackforge
projects definitely don't need to be directly in devstack. These were
largely unused previously anyway.
Change-Id: I300686b2ac976d9b454404842b3f210fd7c239d9

DevStack currently lacks support for LVM ephemeral storage in Nova.
This support is important for testing of Nova's LVM backend. The
proposed change adds a default volume group, to be shared by Cinder
and Nova. It also adds a configuration option NOVA_BACKEND, which
must be LVM if it is set, that determines whether Nova should be
configured to use LVM ephemeral storage.
Change-Id: I4eb9afff3536fbcd563939f2d325efbb845081bb

This makes a bunch of variable cleanups that will let -o nounset
function, for the time being we hide nounset behind another setting
variable so that it's not on by default.
Because this is bash, and things are only executed on demand, this
probably only works in the config it was run in. Expect cleaning up
all the paths to be something that takes quite a while.
This also includes a new set of unit tests around the trueorfalse
function, because my change in how it worked, didn't. Tests are good
m'kay.
Change-Id: I71a896623ea9e1f042a73dc0678ce85acf0dc87d

This is an initial pass at plugin infrastructure for devstack which
allows specifying an external repository via:
enable_plugin <name> <giturl> [branch]
It implements the devstack specification for this at
I173dee3d57967b1d2ffd30e4868a2832aeac97ce
Change-Id: I8e4175313b3cf0b12e981122358b1288a7eb0746

This has been deleted from the nova tree, we should purge it from
devstack, as it will not work any more.
Change-Id: I20501fec140998b91c9ddfd84b7b10168624430a

Now that we are on pip 1.5.6 lets drop the workaround to make pip 1.4
work. As this is a development/testing tool requiring a newer pip
shouldn't be an issue. Also stack.sh installs pip by default.
Work around introduced in https://github.com/pypa/pip/issues/709
Change-Id: I0e7aad1d21f4fce4c020ce36685bb56893c66bdc

`stop_dstat` function was extracted into $TOP_DIR/lib/dstat in commit
e0b08d04ab but the new lib was not sourced in
unstack.sh.
Change-Id: I6878552d9f384fdd7882e52000b37ac6a60844c3
Closes-Bug: 1366713

In order to use dstat on the new side of grenade, pull dstat code into
its own lib with the function start_dstat
Change-Id: I5c908d594a6f3a90ed4b3f744002bf606841cf07

Certificates and the index were left in the data directory after
running unstack. That would break devstack on the next run.
partial blueprint devstack-https
Change-Id: I6eb552a76fb29addf4d02254c027e473d6805df1

The bm_poseur git repository link has been broken
for over 11 months. The virtualized/fake baremetal
environment is not working and has not worked in a
long time. Now, on the tail of enabling 'enable -o errexit',
this functionality now has a hard break.
Change-Id: I3cbd8db58c422bc5273d2433278aaa5e449ecfd9
Closes-Bug: 1285954

This converts the special cases in the is_service_enabled() function to call
individual functions declared by the projects. This allows projects that
are not in the DevStack repo and called via the extras.d plugin to handle
an equivalent service alias.
* Ceilometer
* Cinder
* Glance
* Neutron
* Nova
* Swift
TODO: remove the tests from is_service_enabled() after a transition period
Patch Set 2: Rebased
Change-Id: Ic78be433f93a9dd5f46be548bdbd4c984e0da6e7

Also adds an is_ironic_enabled() function to prepare for an upcoming
change in is_service_enabled().
Change-Id: I6e6e0e8b70221e231785ab27e9b5d4836933ac4c

Also adds an is_trove_enabled() function to match
https://review.openstack.org/69497 changes for is_service_enabled().
Change-Id: Ic0408ff6d9816aec8a3506931470470342a5dcd7

unstack.sh and stack.sh both have to "configure projects", but the
code was different. This change makes it so the 2 sections of the
files are the same.
Change-Id: Ia06f8bbfbe2a6e87fb406e34e13a39bd7fa9e5af

- add lib/ceilometer and lib/heat to source list for when stop_heat
and stop_ceilometer functions are called.
- add lib/tls source to lib/keystone for when is_ssl_enabled_service
function called.
Change-Id: Ief05766e9cfda71fb6392c8a757d04751283414e
Closes-Bug: #1269047 

* Save PID when using screen in screen_it()
* Add screen_stop()
* Call out service stop_*() in unstack.sh functions so screen_stop()
 can do its thing
Closes-bug: 1183449
Change-Id: Iac84231cfda960c4197de5b6e8ba6eb19225169a