openstack/openstack-ansible-os_cloudkitty - openstack-ansible-os_cloudkitty - OpenDev: Free Software Needs Free Tools

In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
With the dependent patches, the openstack_openrc role is now executed once
on the designated host, so it is no longer required as a meta-dependency for
the role.
Depends-On: https://review.openstack.org/579233
Depends-On: https://review.openstack.org/579959
Change-Id: Ib46638d1cb35e677ecfe789a32bb69a024042d6f

Use MySQL as coordination backend for cloudkitty-processors.
Otherwise prices may get calculated multiple times.
Change-Id: I661ca3b7ffb4f64242422297bafc938ca5082a12
Closes-Bug: 1777432

There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement a new variable called 'cloudkitty_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable cloudkitty_galera_address which
has been used for a long time, but never documented.
Change-Id: I7bb3f38556558ef14723361d3672a1d06f839b52

The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.
Change-Id: Ib058bc3dc6e97871f9c3a177c7f7dec48709ec3f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Use one variable for collected openstack services and another for
systemd services.
Change-Id: I1946243df319c00638e4ec4a4f1f144fd2131956
Closes-Bug: 1769625

The upstream requierements do not fit openstack-ansible venvs.
So explicitly loosen up until fixed upstream.
Change-Id: I307c9eefac4ab25a88bee7621461e41320bc4a70
Closes-Bug: 1769621

virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: I7a363fa0890101fd2f4007a866ca73b2f0e80d60
Partial-Bug: #1741634 

The keystonemiddleware library recently switched to using the
cryptography library over pycrypto, which was unmaintained. See
Iced7f5115e49ccf4f7f5bf6813cb5988b95c248b
Change-Id: I08e9ca9ab28c9f9a81305d5a9865e19f6d0012de
Co-Authored-By: Nolan Brubaker <nolan.brubaker@rackspace.com>

Option "rpc_backend" from group "DEFAULT" is deprecated for removal
(Replaced by [DEFAULT]/transport_url). Its value may be silently
ignored in the future.
Change-Id: Ia352ef7377e6e1f51fffb3181ad5ab422275f39a
Implements: blueprint deprecate-rpc-backend

This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Idac3056145b92adcd0b5b328296c59b71bf28441
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Fixes the ability to deploy a venv in cases where:
1) developer_mode is not enabled
2) A cached venv is not downloaded from the repo server
Additional cleanup to the developer_mode venv deployment
logic is implemented by adding a *_venv_download var
which is used to decouple developer_mode from the
cached venv extraction process so that a deployer
can force venv builds in-place (disable cached
venv usage) without enabling developer mode
constraints.
Change-Id: I8bb1ff09d676ce3d2faa2baee385695bffb2db59

Change-Id: Ib579885209f447b501fcb338da4d9321c0aa4498
Implements: blueprint remove-pki

Change-Id: I72b791bdad0ffb7c928eb4c627094968d179173d

Change-Id: I523b5f1ae56c1ff19ca6bd95e3e0e7f00e282c71

Change-Id: I0dc824ccd4e46d9ab76d0908a1f28f3e9b1fe927
Closes-Bug: #1641695 

Preparing this role for the ansible-lint version bump
Change-Id: I36f5b951b5dcaa52e35079f32e98f2aae94e2df0

The current constraints generation for the
installation involves multiple tasks and multiple
variables.
Using multiple tasks extends the installation time
unnecessarily and the additional variables are
unnecessary.
This patch aims to simplify the mechanism and
hopes to speed it up a little.
Change-Id: I018e2d345912722a6c5e6fbbbb5afd424807f3a4

Change the 'cloudkitty_service_names' from a list to a dictionary mapping
of services, groups that install those services. This brings the
method into line with that used in the os_neutron role in order to
implement a more standardised method.
The init tasks have been updated to run once and loop through this
mapping rather than being included multiple times and re-run against
each host. This may potentially reduce role run times.
Currently the reload of upstart/systemd scripts may not happen if
only one script changes as the task uses a loop with only one result
register. This patch implements handlers to reload upstart/systemd
scripts to ensure that this happens when any one of the scripts
change.
The handler to reload the services now only tries to restart the
service if the host is in the group for the service according to the
service group mapping. This allows us to ensure that handler
failures are no longer ignored and that no execution time is wasted
trying to restart services which do not exist on the host.
Finally:
- Common variables shared by each service's template files have
 been updated to use the service namespaced variables.
- Unused handlers have been removed.
- Unused variables have been removed.
Change-Id: Ieb96fac62906ac9eb387edb450df0c0ca9c0ccb2

This patch adds a var used by the repo_build process to determine
which host group to check for members of before building the pip
packages required by this role. This provides the ability to
optimise the repo build process to only building the packages
that are required for an environment.
Depends-On: Idda16b4f382eee57c7469af898859d6d81d4eb30
Change-Id: Iedb4c0c91daa291f881a9fc00c5b8c40bb312ce3

The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: I20c3e2c1c63c2ad90835ac60c84d032197ea527c

Update default variables with a few corrections:
 * Fix name of 'cloudkitty_output_pipeline' variable
 * Change 'cloudkitty_service_type' to 'rating'
 * Remove unnecessary and redundant 'cloudkitty_pip_packages'
Change-Id: I9768471d1f67d074282e63033d9bd626fb4175af

Remove all tasks and variables related to toggling between installation
of cloudkitty inside or outside of a Python virtual environment.
Installing within a venv is now the only supported deployment.
Additionally, a few changes have been made to make the creation of the
venv more resistant to interruptions during a run of the role.
* unarchiving a pre-built venv will now also occur when the venv
 directory is created, not only after being downloaded
* virtualenv-tools is run against both pre-built and non pre-built venvs
 to account for interruptions during or prior to unarchiving
Change-Id: I84dd86e4e8244de5a6ac55a302df0c9737d6388e
Implements: blueprint only-install-venvs

This patch includes the following changes to enable OpenStack-CI gate
testing and to allow the role to pass a basic convergence test:
 - Base configuration for OpenStack's CI Infrastructure.
 - Base documentation build configuration.
 - A Vagrantfile for local testing.
 - A base test configuration in order to execute the standard tests.
 - Added a CONTRIBUTING file.
 - Add reno scaffolding for release notes management.
 - Correcting name of handler to restart services in install tasks.
 - Define ``cloudkitty_galera_password`` and
 ``cloudkitty_user_password`` in test variables.
 - Define default ``debug`` and ``cloudkitty_service_region`` variables.
Change-Id: I7bcc8c589d506d8b71eb08fc87861a4c31a6e947

Kitty cli client and horizon dashboard were not
working. The endpoints had a v1 and tenant in them
which was not working correctly. This change removed
the v1 and tenant bits from the end point which fixed
the UI and CLI.
Implements: blueprint role-cloudkitty

Adds a task to fix the signing dir for cloudkitty

Adds the upstart scripts for kitty api
and kitty processor. It also enables the tasks
to restart the services.

The cloudkitty.conf and defaults/main had some
incorrect info in it keeping kitty from working
correctly with keystone v3

The changes bring the role inline with others to make it possible
to download a pre-built venv.
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

The template for cloudkitty.conf was mostly hard
coded. This change adds the variables to many of
the hard coded things.

Removed the service setup from the task