openstack/openstack-ansible-os_cloudkitty - openstack-ansible-os_cloudkitty - OpenDev: Free Software Needs Free Tools

This patch aims to migrate service from usage of regular syslog files
to journald. We also disable uwsgi logging, since it dublicates
requests that are logged by service itself.
Change-Id: I4cd3270ef873b6f595b151014e82c8b4b35f12aa

The variables cloudkitty_developer_mode and cloudkitty_venv_download
no longer carry any meaning. This review changes cloudkitty to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.
As part of this, we move the installation out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.
We also change include_tasks to import_tasks and include_role
to import_role so that the tags in the python_venv_build role
will work.
Change-Id: Ieaec0a12a1ed6ea1f9a98a615fd8c6bff67a9d0c

Accoring to the cloudkitty docs it is strongly recommended to
install the API server in a WSGI host such as mod_wsgi.
The tasks are a combination of os_cinder steps and the cloudkitty
docs for wsgi:
https://docs.openstack.org/cloudkitty/latest/install/mod_wsgi.html#mod-wsgi
Change-Id: I4173a32f4989f7e23205083738ccd81f7ea6f9f6
Closes-Bug: 1769632

This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed.
The systemd journal would normally be populated with the standard out of
a service however with the use of uwsgi this is not actually happening
resulting in us only capturing the logs from the uwsgi process instead
of the service itself. This change implements journal logging in the
service config, which is part of OSLO logging.
OSLO logging docs found here: <https://docs.openstack.org/oslo.log/3.28.1/journal.html>
Change-Id: Ieff0119cb09e8971fab5b0393f023010a554edff
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

This introduces oslo.messaging variables that define the RPC and
Notify transports for the OpenStack services. These parameters replace
the rabbitmq values and are used to generate the messaging
transport_url for the service.
This patch:
* Add oslo.messaging variables for RPC and Notify to defaults
* Update transport_url generation in conf
* Add oslo.messaging to tests inventory and update tests
* Install extra packages for optional drivers
* Add release note
Change-Id: I0612252927f81b3584054f6cd77a2826c5d3fc3b

Use MySQL as coordination backend for cloudkitty-processors.
Otherwise prices may get calculated multiple times.
Change-Id: I661ca3b7ffb4f64242422297bafc938ca5082a12
Closes-Bug: 1777432

The sections are part of the cloudkitty repo:
https://github.com/openstack/cloudkitty/blob/master/etc/cloudkitty/api_paste.ini
But weren't inlcuded in osa.
Change-Id: I2579c8b88bcf2ec162002a8a8978c7720bfc8112
Closes-Bug: 1769630

Each collector needs its own config section, so use cloudkitty_collector
to create these sections.
Change-Id: I6bc4c395693265ea08067bb293a99a72b48fe9a6
Closes-Bug: 1769629

Use one variable for collected openstack services and another for
systemd services.
Change-Id: I1946243df319c00638e4ec4a4f1f144fd2131956
Closes-Bug: 1769625

Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.
[1]https://review.openstack.org/#/c/508522/
Change-Id: I5e2ac3a6b044515c84695b441602b254f4859bd8
Implements: blueprint deprecate-auth-uri-option

Option "rpc_backend" from group "DEFAULT" is deprecated for removal
(Replaced by [DEFAULT]/transport_url). Its value may be silently
ignored in the future.
Change-Id: Ia352ef7377e6e1f51fffb3181ad5ab422275f39a
Implements: blueprint deprecate-rpc-backend

Option "rabbit_use_ssl" from group "oslo_messaging_rabbit" is deprecated.
Use option "ssl" from group "oslo_messaging_rabbit".
Change-Id: Ic9157cf1eb4d56843dcc4141e49c7a3ec3bcba89
Implements: blueprint deprecate-rabbit-use-ssl

The systemd unit 'TimeoutSec' value which controls the time
between sending a SIGTERM signal and a SIGKILL signal when
stopping or restarting the service has been reduced from 300
seconds to 120 seconds. This provides 2 minutes for long-lived
sessions to drain while preventing new ones from starting
before a restart or a stop.
The 'RestartSec' value which controls the time between the
service stop and start when restarting has been reduced from
150 seconds to 2 seconds to make the restart happen faster.
These values can be adjusted by using the *_init_config_overrides
variables which use the config_template task to change template
defaults.
Change-Id: Ia10703ca1cfa5afd1dec09a52f1b981711796e33

This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Idac3056145b92adcd0b5b328296c59b71bf28441
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

We don't support trusty in master anymore, so this makes sures the
support of upstart is phased out.
Whenever possible, we use the systemd module instead, that reduces the
amount of tasks.
Change-Id: I50d377ce6c5e80386954fd551e566c55f57c5a3a
Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@rackspace.co.uk>

Change-Id: Ib579885209f447b501fcb338da4d9321c0aa4498
Implements: blueprint remove-pki

Change-Id: I523b5f1ae56c1ff19ca6bd95e3e0e7f00e282c71

Change-Id: I0dc824ccd4e46d9ab76d0908a1f28f3e9b1fe927
Closes-Bug: #1641695 

Change the 'cloudkitty_service_names' from a list to a dictionary mapping
of services, groups that install those services. This brings the
method into line with that used in the os_neutron role in order to
implement a more standardised method.
The init tasks have been updated to run once and loop through this
mapping rather than being included multiple times and re-run against
each host. This may potentially reduce role run times.
Currently the reload of upstart/systemd scripts may not happen if
only one script changes as the task uses a loop with only one result
register. This patch implements handlers to reload upstart/systemd
scripts to ensure that this happens when any one of the scripts
change.
The handler to reload the services now only tries to restart the
service if the host is in the group for the service according to the
service group mapping. This allows us to ensure that handler
failures are no longer ignored and that no execution time is wasted
trying to restart services which do not exist on the host.
Finally:
- Common variables shared by each service's template files have
 been updated to use the service namespaced variables.
- Unused handlers have been removed.
- Unused variables have been removed.
Change-Id: Ieb96fac62906ac9eb387edb450df0c0ca9c0ccb2

OSLO logging currently defaults the 'use_stderr' option to True
which results duplicate logs in service daemon logs for both
upstart and systemd. To correct this issue the use_stderr
option has been set to false.
Change-Id: I177bcd724433543cb39bce7bce19f43a5675d492
Closes-Bug: 1588051
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Update default variables with a few corrections:
 * Fix name of 'cloudkitty_output_pipeline' variable
 * Change 'cloudkitty_service_type' to 'rating'
 * Remove unnecessary and redundant 'cloudkitty_pip_packages'
Change-Id: I9768471d1f67d074282e63033d9bd626fb4175af

This change implements support for Xenial and CentOS 7 within the
os_cloudkitty role.
Change-Id: Id2abe9099171080bd3d3ceb3a54222a5dfdefe2f
Implements: blueprint multi-platform-host

Remove all tasks and variables related to toggling between installation
of cloudkitty inside or outside of a Python virtual environment.
Installing within a venv is now the only supported deployment.
Additionally, a few changes have been made to make the creation of the
venv more resistant to interruptions during a run of the role.
* unarchiving a pre-built venv will now also occur when the venv
 directory is created, not only after being downloaded
* virtualenv-tools is run against both pre-built and non pre-built venvs
 to account for interruptions during or prior to unarchiving
Change-Id: I84dd86e4e8244de5a6ac55a302df0c9737d6388e
Implements: blueprint only-install-venvs

Adds a task to fix the signing dir for cloudkitty

Adds the upstart scripts for kitty api
and kitty processor. It also enables the tasks
to restart the services.

The cloudkitty.conf and defaults/main had some
incorrect info in it keeping kitty from working
correctly with keystone v3

The template for cloudkitty.conf was mostly hard
coded. This change adds the variables to many of
the hard coded things.

Removed the service setup from the task