openstack/openstack-ansible-os_trove - openstack-ansible-os_trove - OpenDev: Free Software Needs Free Tools

Somehow the previous patch merged without this required
argument.
Change-Id: I18a99443d457a7b50f4c1beb7cc123f716f1ad20

There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement two new variables:
- trove_oslomsg_rpc_setup_host
- trove_oslomsg_notify_setup_host
These are used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.
We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.
Change-Id: I88514f00c4b4c0452b573d765bb895cb562f952e

In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
Change-Id: I9ce106569ced891c551b36506d360e4b8718c7e3

With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: I27326b56cfec69444dae0ca1c7abf0b78ad2299b

There are times when a deployer will need to reconfigure parts of
an environment and having a general purpose tag to run said operation
will be important especicially should the deployer be needing to
reconfigure systemd unit files in a downtime event. This change adds
a general purpose systemd tag where include_role and systemd is found
which will assit operators with day2 operational tasks.
Change-Id: Iab6c46d082e0218dcbd37ac4ac7a48fac7080a47
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Change-Id: Idfa2ba78862912aea856c1e54b22e45de857a9c4
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement a new variable called 'trove_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable 'trove_galera_address' which
has been used for a long time, but never documented. A bunch of unused
variables have also been removed.
The extras folder is removed given that trove's playbooks have been
merged into the integrated repository.
Change-Id: I0bf801416bd0349534588fc7272dffd4b86b7bb9

This prevents data to be leaked into the callback plugin.
Change-Id: I52c62a5a3267023087343adeffc6301443c5f703

We reinitialize the venv to ensure that the right version of
python is in the venv, but we do not want virtualenv to also
replace pip, setuptools and wheel so we tell it not to. If we
don't do this then virtualenv will install the latest available
version, which is not what we want.
Change-Id: I59a70585a2bfe4916b3df8eb627f6f6a4b1ef19b
Partial-Bug: #1764470 

This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.
Change-Id: Id833d4ecf9d17bb74c1c0702c00ace241ac48873
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: Iad3a44f35c57d92dcbbfde4acbd86a3a4eb1879d
Partial-Bug: #1741634 

The Trove service has split it's tests for Tempest in a seperate
repo so this patch updates it.
This patch also fixes the lint issue to use the systemd module
reload feature instead of a command.
Closes-Bug: #1747608
Co-Authored-By: Major Hayden <major@mhtx.net>
Depends-On: I5f1ecb71c6125c29d6487bef6c2be215c1b561c1
Change-Id: I26d384a5c9aadade94052cc859f9405335635ed4

Fix circular python issues, same fix in Keystone and Barbican
Change-Id: I77e60ed42ab43e3f5984b8a52dc5898325df1c8e

Remove the unnecessary trove_regular_user. The documenation that was
referenced [1] when this was added to the role is intended to configure
Trove for development purposes. The trove_regular_user is not used by the
Trove service and is only being created to give the developer a non-admin
user to use for testing.
[1] https://docs.openstack.org/trove/latest/install/manual_install.html
Change-Id: I7c9289e191255e92530ea62568b212320f705ff8

Change-Id: Ib423aac8a0d5394b5f6803352c3f606cce18f0e1

Change-Id: I735a5e2e6d07813ad34c5203a4ef8cf29b17939c

In order to do a developer mode that allows installation of packages
from local file, which take precedence over the developer_mode
constraints, we need to allow the order of the constraints to be
changed.
This patch adds a "pip_install_developer_constraints" var which is used
to set the developer mode constraints. By default this will leave the
same behaviour but will allow additional constraints to be added, or the
developermode constraints file to be overriden altogether.
Change-Id: I64affcd1b3d5580e1cee5047df465e2ab8144211

In order to make it easier to detect the currently deployed
venv for a service, and therefore allow smarter decisions
for things like upgrading, we implement the venv tag as a
local fact.
The file used to store facts will be the same for all
OpenStack services, with each service using its own section.
Example:
"ansible_local": {
 "openstack_ansible": {
 "trove": {
 "venv_tag": "14.2.1"
 }
 }
}
Change-Id: Id2545ad38d78c5c28310b67a8a3eb74b8b51b026

When openstack_networks is empty list, an error is seen while
setting trove service network id. The error seen was "list object
has no element 0". Added a validation task to fail if trove network
is not created yet.
Change-Id: I9acfc8e06f461073777469712fd6ce50e3187b30

We use an SSH bastion host which we do our deployment through. The
deployment host doesn't have direct access to the same network as the
host. As a result the venv local checksum lookup fails.
I have described this here:
https://bugs.launchpad.net/openstack-ansible/+bug/1689283
This is a simple fix for this problem, assuming everything is good it
will need repeating in multiple places in the code base.
Change-Id: I681491539aa4aa3b3c88ce059ae6d1f7e56f651d

The update of the apt cache and the package installation
can all be handled in a single task by providing the
package action plugin with the right parameters. This
removes an extra task to optimise execution.
The minimum Ansible version is raised to 2.2 due to a
known bug [1] in Ansible's apt module which does not
update the cache properly if the cache update and the
install are combined in a single task.
[1] https://github.com/ansible/ansible-modules-core/issues/1497
Change-Id: Ieaf67c9a776c43e9fd39c63a92642b55a49bb9bc

To operate properly the trove guest agent needs access to
rabbitmq and also the neutron network for trove to use must
be created and defined in the trove.conf file.
This changeset adds documentation, tasks and
configuration defaults to setup the networking for
trove.
Change-Id: Idcf87c2eef0af475c02412f03433d22d7b08643f

In the Ocata release, trove added support for encrypting the rpc
communication between the guest instances and the control plane.
These settings allow the user to specify installation specific
keys versus using the default keys.
Change-Id: Ie42d754d58e983a15b553ad8a399813c9a700344

This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: I6dccb6eeabceea30922cbcf3a60e32e841612fd0
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Updating the trove-post-install to generate the
trove-guestagent.conf during the os_trove deployment.
Closes-Bug: #1658694
Change-Id: I4d8ac1b40d36b188718a02b89a1ea70fb89e7111

Fixes the ability to deploy a venv in cases where:
1) developer_mode is not enabled
2) A cached venv is not downloaded from the repo server
Additional cleanup to the developer_mode venv deployment
logic is implemented by adding a *_venv_download var
which is used to decouple developer_mode from the
cached venv extraction process so that a deployer
can force venv builds in-place (disable cached
venv usage) without enabling developer mode
constraints.
Change-Id: Id4e911dd28bf3e22fe5927b06464ee15c57f010e

This patch ensures that all directories under `/openstack/venvs`
are created with `0755` permissions by default. This prevents
permission denied errors when running certain commands from the
virtual environment.
Change-Id: I6a5fbdd16dd2558341c7e7d647a18b5028471315

Change-Id: I5f9d920cc99bcc62859663434b2615c317d8fbdc
Implements: blueprint trusty-removal

Change-Id: I2586f36e23d5decec524babf8ef8de2cb6be6468

From Newton onwards we're able to take advantage of
the Ansible package module, instead of conditionally
executing the apt or yum module.
Implementing this is an optimisation which we can do
in master and backport to stable/newton,
reducing the execution time.
Reference: http://docs.ansible.com/ansible/package_module.html
Change-Id: Id2f6ff7e1bd016e07f0b7a10d9ef2844070eb1c5
Related-Bug: #1642654 

In the case of trove, we follow service configuration as per the
official manual installation documentation [1].
It uses trove_for_trove_usage as a service tenant unlike other openstack
services which uses default service tenant. For any project service to
work, the configuration [keystone_authtoken] section should be updated
with the correct user, project, tenant and password.
Proposed change fixes this issue so Trove service configurations are
aligned with the trove admin user settings for tenant and password.
[1]: http://docs.openstack.org/developer/trove/dev/manual_install.html
Closes-bug: #1644079
Change-Id: I92a426132e20669731bc65526b20f60a3f34b348

Reinitializes (copies python, etc binaries) into the venv when
dropping a new venv into place. This is needed because the Python
binary packaged with the venv may not match the Python running on
the host it is being installed to. (ie. in the case of a Xenial
repo container and a Trusty target host.)
Change-Id: If20462b0d57ee8ec11a479dbaf463e1cd309ac97
Partial-Bug: #1637509 

This patch removes some extra tasks for detecting systemd and uses
the fact instead.
Partial-Bug: #1640125
Change-Id: Iabcd8ecbf6e10ba8e65d3ac01ae635ce9294aec2

Ansible 2.2 now treats the 'name' argument for the pip module
as a list, removing the need for us to implement the join
filter to optimise the install execution.
Change-Id: I630ceadba486c10e5b8b913a0d18e1af0e1ccbc3

Starting in Ansible 2.0, the get_url [1] module provides the
ability for a checksum to be provided to the get_url module
which will be verified against the local destination file
and the task skipped if it matches.
[1] http://docs.ansible.com/ansible/get_url_module.html
This patch implements the use of this functionality.
The ability to ignore a venv download failure is also removed
as this is not necessary or desirable. It is better for the
download to fail and the playbook execution to stop immediately
so that the failure point is exposed.
Change-Id: Iea1e6428d5c2e22eb6bce71d3ad5350864b536e1

Now ansible apt module correctly behaves, so it's time
to deprecate these cruft tasks for apt.
Change-Id: I70ee091de9b970558a0b8b94856d09c8579f11c0

The current constraints generation for the
installation involves multiple tasks and multiple
variables.
Using multiple tasks extends the installation time
unnecessarily and the additional variables are
unnecessary.
This patch aims to simplify the mechanism and
hopes to speed it up a little.
Change-Id: Ie37ab6be65cfb9f368cec7da6b9de78ca5385075

Preparing this role for the ansible-lint version bump
Change-Id: I37ed27a28c0551af99a1f730b2b61d0ad1db4a3b

This change removes the use of 'ignore_errors: true' because it causes deployers
to see red output and a stacktrace, which traditionally means something is broken,
even when the failure is known to have a fall back option or be intentional. This
conversion will provide a generally cleaner interface.
It should be noted that the 'failed' filter will still function normally. Tasks
with the 'failed_when: false' option will still be marked as 'failed' in any
registered variable. This change simply makes the output look cleaner.
Change-Id: Ic24c9ee327c7293753dd7b7fba4cde1987722f5f
Closes-Bug: #1633438
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Change-Id: I06ae08392e66229b9120c41309f1ba5e7301b217

Forcing dynamic includes for inventory-based conditionals
results in incomplete execution.
Change-Id: I0a2e28928f36aa8a7497eb2f3a190ad5d7a2fe62

As per: http://docs.openstack.org/developer/trove/dev/manual_install.html
We need the following:
keystone --os-username <OpenStackAdminUsername> --os-password <OpenStackAdminPassword> --os-tenant-name <OpenStackAdminTenant> --os-auth-url http://<KeystoneIP>:<KeystonePort>/v2.0 tenant-create --user trove_for_trove_usage
keystone --os-username <OpenStackAdminUsername> --os-password <OpenStackAdminPassword> --os-tenant-name <OpenStackAdminTenant> --os-auth-url http://<KeystoneIP>:<KeystonePort>/v2.0 user-create --user regular_trove_user --pass trove --tenant trove_for_trove_usage
keystone --os-username <OpenStackAdminUsername> --os-password <OpenStackAdminPassword> --os-tenant-name <OpenStackAdminTenant> --os-auth-url http://<KeystoneIP>:<KeystonePort>/v2.0 user-create --user admin_trove_user --pass trove --tenant trove_for_trove_usage
keystone --os-username <OpenStackAdminUsername> --os-password <OpenStackAdminPassword> --os-tenant-name <OpenStackAdminTenant> --os-auth-url http://<KeystoneIP>:<KeystonePort>/v2.0 user-role-add --user admin_trove_user --tenant trove_for_trove_usage --role admin
keystone --os-username <OpenStackAdminUsername> --os-password <OpenStackAdminPassword> --os-tenant-name <OpenStackAdminTenant> --os-auth-url http://<KeystoneIP>:<KeystonePort>/v2.0 service-create --user trove --type database
keystone --os-username <OpenStackAdminUsername> --os-password <OpenStackAdminPassword> --os-tenant-name <OpenStackAdminTenant> --os-auth-url http://<KeystoneIP>:<KeystonePort>/v2.0 endpoint-create --service trove --region RegionOne --publicurl 'http://<EnvironmentPublicIP>:<EnvironmentPort>/v1.0/$(tenant_id)s' --adminurl 'http://<EnvironmentPublicIP>:<EnvironmentPort>/v1.0/$(tenant_id)s' --internalurl 'http://<EnvironmentPublicIP>:<EnvironmentPort>/v1.0/$(tenant_id)s'
Closes-Bug: #1626726
Change-Id: I30a5bf3b3a7f369527b87f8925b167cd47c52d5a

Ansible 2.1.1 introduces a regression in the way conditional
includes are handled which results in every task in the
included file being evaluated even if the condition for the
include is not met. This extends the run time significantly
for a deployment.
This patch forces all conditional includes to be dynamic.
Change-Id: Ifab57201c962b084a1d531d788b25526cd899ce4
Related-Bug: https://github.com/ansible/ansible/issues/17687 

While doing some preliminary testing using a prototype AIO, the
following issues where observed and fixed.
The trove CLI is expecting the service name to be 'database' in
keystone. Update from 'dbaas' to 'database'.
Add the tenant id to the trove service URLs, they are needed.
Ignore failures when restarting services since all trove services
are attempted to be restarted in all trove containers, which
produces invalid combinations.
When calling the trove-manage CLI to create the DB, provide the
trove conductor conf file so the CLI has the DB connection
information.
Add a blank line after the transport_url specification, otherwise
the following line is added to the URL and forms an invalid value.
Add Nova and Keystone configuration values to the trove api conf
file since they are needed by the trove api service.
Add Nova configuration values for the trove task manager service.
Default to using the internal URL to for nova client.
Change-Id: If70077ea5d66151999b8965c218e4cb853e6f81a

This commit allows the deployer to create 3 containers to run
Trove-API, Trove-Conductor and Trove-Taskmanager.
Change-Id: If93330d48f53745d45af351b9de9a4a733af943a