openstack/openstack-ansible-os_cloudkitty - openstack-ansible-os_cloudkitty - OpenDev: Free Software Needs Free Tools

There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement a new variable called 'cloudkitty_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable cloudkitty_galera_address which
has been used for a long time, but never documented.
Change-Id: I7bb3f38556558ef14723361d3672a1d06f839b52

Change-Id: I585f57c69f5d0c3e415f784152f8f900ac0c56ca
Closes-Bug: 1769624

We reinitialize the venv to ensure that the right version of
python is in the venv, but we do not want virtualenv to also
replace pip, setuptools and wheel so we tell it not to. If we
don't do this then virtualenv will install the latest available
version, which is not what we want.
Change-Id: Id29497b36cd2b9cca3f3b08474e1001d528931b6
Partial-Bug: #1764470 

With the implementation of [1] the MySQL-python package is
no longer present on the service container. Given that all
other services have the playbook execute these tasks with
delegation to the first galera_server container, we can do
that here too.
[1] https://review.openstack.org/562276
Change-Id: I2c70ac1324ea027ae4a8ed1e70c5706b47fcd57f

This prevents data to be leaked into the callback plugin.
Change-Id: Iff9971b5508425e809b0a1ccfe40af04252aa014

This prevents data to be leaked into the callback plugin.
Change-Id: Ia490c2a62bb53b9bdc191787820e1fc12b8aaa58

In the previous patch removing the virtualenv-tools, we've updated
paths to the proper binaries, but we don't reset the virtualenv
like what we used to do.
Change-Id: I8bd55222c6e058c7c26075aab219043c5fb076ed
Partial-Bug: #1741634 

virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: I7a363fa0890101fd2f4007a866ca73b2f0e80d60
Partial-Bug: #1741634 

In order to do a developer mode that allows installation of packages
from local file, which take precedence over the developer_mode
constraints, we need to allow the order of the constraints to be changed.
This patch adds a "pip_install_developer_constraints" var which is used
to set the developer mode constraints. By default this will leave the
same behaviour but will allow additional constraints to be added, or the
developermode constraints file to be overriden altogether.
Additionally, fix bug where incorrect "keystone_developer_mode" variable
was used in cloudkitty role.
Change-Id: Ic28a42d4475e9334567d155ba182e15c7a829d4d

We use an SSH bastion host which we do our deployment through. The
deployment host doesn't have direct access to the same network as the
host. As a result the venv local checksum lookup fails.
I have described this here:
https://bugs.launchpad.net/openstack-ansible/+bug/1689283
This is a simple fix for this problem, assuming everything is good it
will need repeating in multiple places in the code base.
Change-Id: Iae75d5fbaa3f8f74f69574b51dbae84948b8dcb6

Consolidate distro package install tasks into a
single task using the package module and pass
the package list into the name instead of using
a with_items loop.
The minimum Ansible version is raised to 2.2 due to a
known bug [1] in Ansible's apt module which does not
update the cache properly if the cache update and the
install are combined in a single task.
[1] https://github.com/ansible/ansible-modules-core/issues/1497
Change-Id: Id00e704fd89f5dc40a143a9567c859ba78b76668

The policy.json file is currently read continually by the
services and is not only read on service start. We therefore
cannot template directly to the file read by the service
(if the service is already running) because the new policies
may not be valid until the service restarts. This is
particularly important during a major upgrade. We therefore
only put the policy file in place after the service restart.
This patch also tidies up the handlers and some of the install
tasks to simplify them and reduce the tasks/code a little.
Change-Id: I8ece1f91db9177f30623f09036e75794d607e17e

This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Idac3056145b92adcd0b5b328296c59b71bf28441
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

We don't support trusty in master anymore, so this makes sures the
support of upstart is phased out.
Whenever possible, we use the systemd module instead, that reduces the
amount of tasks.
Change-Id: I50d377ce6c5e80386954fd551e566c55f57c5a3a
Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@rackspace.co.uk>

Fixes the ability to deploy a venv in cases where:
1) developer_mode is not enabled
2) A cached venv is not downloaded from the repo server
Additional cleanup to the developer_mode venv deployment
logic is implemented by adding a *_venv_download var
which is used to decouple developer_mode from the
cached venv extraction process so that a deployer
can force venv builds in-place (disable cached
venv usage) without enabling developer mode
constraints.
Change-Id: I8bb1ff09d676ce3d2faa2baee385695bffb2db59

Change-Id: Ib579885209f447b501fcb338da4d9321c0aa4498
Implements: blueprint remove-pki

Change-Id: I523b5f1ae56c1ff19ca6bd95e3e0e7f00e282c71

Reinitializes (copies python, etc binaries) into the venv when
dropping a new venv into place. This is needed because the Python
binary packaged with the venv may not match the Python running on
the host it is being installed to. (ie. in the case of a Xenial
repo container and a Trusty target host.)
Change-Id: Ie87ebd7aefaf7117e8fbd9711e887233eec8ff1c
Partial-Bug: #1637509 

Ansible 2.2 now treats the 'name' argument for the pip module
as a list, removing the need for us to implement the join
filter to optimise the install execution.
Change-Id: I76bdcc75bf5bbe0c766409d43648dcf567a9de6f

Starting in Ansible 2.0, the get_url [1] module provides the
ability for a checksum to be provided to the get_url module
which will be verified against the local destination file
and the task skipped if it matches.
[1] http://docs.ansible.com/ansible/get_url_module.html
This patch implements the use of this functionality.
The ability to ignore a venv download failure is also removed
as this is not necessary or desirable. It is better for the
download to fail and the playbook execution to stop immediately
so that the failure point is exposed.
Change-Id: Ib457a9c68f4cf627b0496574cda628ed304692c1

Preparing this role for the ansible-lint version bump
Change-Id: I36f5b951b5dcaa52e35079f32e98f2aae94e2df0

Now ansible apt module correctly behaves, so it's time
to deprecate these cruft tasks for apt.
Change-Id: Ia32c1496ca818582a287ec70569a63ae78e1e2b6

The current constraints generation for the
installation involves multiple tasks and multiple
variables.
Using multiple tasks extends the installation time
unnecessarily and the additional variables are
unnecessary.
This patch aims to simplify the mechanism and
hopes to speed it up a little.
Change-Id: I018e2d345912722a6c5e6fbbbb5afd424807f3a4

This change removes the use of 'ignore_errors: true' because it causes deployers
to see red output and a stacktrace, which traditionally means something is broken,
even when the failure is known to have a fall back option or be intentional. This
conversion will provide a generally cleaner interface.
It should be noted that the 'failed' filter will still function normally. Tasks
with the 'failed_when: false' option will still be marked as 'failed' in any
registered variable. This change simply makes the output look cleaner.
Change-Id: I9b2a7c6ca8ea0f69ca7defc7266c17be224b4b9e
Closes-Bug: #1633438
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

As the service setup for upstart/systemd makes use of the inventory,
the dynamic inclusion being forced causes unpredictable behaviour.
This patch removed the forced dynamic inclusion.
Change-Id: Ideefeadb75ba5de9c23f5ad1b57588d7eb03e887

Change the 'cloudkitty_service_names' from a list to a dictionary mapping
of services, groups that install those services. This brings the
method into line with that used in the os_neutron role in order to
implement a more standardised method.
The init tasks have been updated to run once and loop through this
mapping rather than being included multiple times and re-run against
each host. This may potentially reduce role run times.
Currently the reload of upstart/systemd scripts may not happen if
only one script changes as the task uses a loop with only one result
register. This patch implements handlers to reload upstart/systemd
scripts to ensure that this happens when any one of the scripts
change.
The handler to reload the services now only tries to restart the
service if the host is in the group for the service according to the
service group mapping. This allows us to ensure that handler
failures are no longer ignored and that no execution time is wasted
trying to restart services which do not exist on the host.
Finally:
- Common variables shared by each service's template files have
 been updated to use the service namespaced variables.
- Unused handlers have been removed.
- Unused variables have been removed.
Change-Id: Ieb96fac62906ac9eb387edb450df0c0ca9c0ccb2

Forcing dynamic includes for inventory-based conditionals
results in incomplete execution.
Change-Id: Ib35bcc230041c87336dfaff6a740282e3055f110

Ansible 2.1.1 introduces a regression in the way conditional
includes are handled which results in every task in the
included file being evaluated even if the condition for the
include is not met. This extends the run time significantly
for a deployment.
This patch forces all conditional includes to be dynamic.
Change-Id: Ie732af6a11815d4ee8efa27233d49a173372bb97
Related-Bug: https://github.com/ansible/ansible/issues/17687 

In order to make it easier to differentiate between the lists of
python packages, distribution packages, downloaded packages,
package pins and other similar variables the variable names are
being changed to ensure that they have a more explicit suffix
that defines the purpose and makes the naming more consistent.
This is to facilitate a lookup plugin which will be able to look
up all the package lists and present them as a consolidated piece
of data which may be used for artifact preparation.
Change-Id: Ie416ef490b4e50b57bc3db870e1cb5540df6446e

The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: I20c3e2c1c63c2ad90835ac60c84d032197ea527c

This change implements support for Xenial and CentOS 7 within the
os_cloudkitty role.
Change-Id: Id2abe9099171080bd3d3ceb3a54222a5dfdefe2f
Implements: blueprint multi-platform-host

The keystone library has been updated to support adding a user to a
domain without specifying a project. Use it instead of shelling out
to the openstackclient.
Change-Id: I8e6e68957af02626e3c3e8c68189f04ea3dd0816

Unlike the Ansible apt module, the Ansible pip module does not
recognise a with_items list and process all the items at once.
To optimise the pip install tasks, this patch replaces the use
of with_items with a join filter so that the pip install task
does an install with all the packages in a list, ensuring that
the execution is one action instead of many.
Change-Id: Ibde294cb8f58463f63ecde8877ee58374270e9ec

Remove all tasks and variables related to toggling between installation
of cloudkitty inside or outside of a Python virtual environment.
Installing within a venv is now the only supported deployment.
Additionally, a few changes have been made to make the creation of the
venv more resistant to interruptions during a run of the role.
* unarchiving a pre-built venv will now also occur when the venv
 directory is created, not only after being downloaded
* virtualenv-tools is run against both pre-built and non pre-built venvs
 to account for interruptions during or prior to unarchiving
Change-Id: I84dd86e4e8244de5a6ac55a302df0c9737d6388e
Implements: blueprint only-install-venvs

This patch includes the following changes to enable OpenStack-CI gate
testing and to allow the role to pass a basic convergence test:
 - Base configuration for OpenStack's CI Infrastructure.
 - Base documentation build configuration.
 - A Vagrantfile for local testing.
 - A base test configuration in order to execute the standard tests.
 - Added a CONTRIBUTING file.
 - Add reno scaffolding for release notes management.
 - Correcting name of handler to restart services in install tasks.
 - Define ``cloudkitty_galera_password`` and
 ``cloudkitty_user_password`` in test variables.
 - Define default ``debug`` and ``cloudkitty_service_region`` variables.
Change-Id: I7bcc8c589d506d8b71eb08fc87861a4c31a6e947

Kitty cli client and horizon dashboard were not
working. The endpoints had a v1 and tenant in them
which was not working correctly. This change removed
the v1 and tenant bits from the end point which fixed
the UI and CLI.
Implements: blueprint role-cloudkitty

Adds a task to fix the signing dir for cloudkitty

Adds the upstart scripts for kitty api
and kitty processor. It also enables the tasks
to restart the services.

The changes bring the role inline with others to make it possible
to download a pre-built venv.
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

The template for cloudkitty.conf was mostly hard
coded. This change adds the variables to many of
the hard coded things.

Removed the service setup from the task

Removed the service file because the logic
was already in the domain file.