openstack/openstack-ansible-os_cloudkitty - openstack-ansible-os_cloudkitty - OpenDev: Free Software Needs Free Tools

When rabbit_stream_fanout is set to true, rabbit_qos_prefetch_count
should be set to a positive value.
This also fixes defenition of transient_quorum_queue as if
rabbit_stream_fanout is enabled - transient_quorum_queue should be also
enabled. [1]
[1] https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L726-L736
Change-Id: I2c25783d4c65aa04ad9305adf2cbb6028e1dab57

During last release cycle oslo.messaging has landed [1] series of extremely
useful changes that are designed to implement modern messaging
techniques for rabbitmq quorum queues.
Since these changes are breaking and require queues being re-created,
it makes total sense to align these with migration to quorum queues by default.
[1] https://review.opendev.org/q/topic:%22bug-2031497%22
Change-Id: I520ae8cf928404a9ed077d6e6812c4f13d0102bb

In order to be able to globally enable notification reporting for all services,
without an need to have ceilometer deployed or bunch of overrides for each
service, we add `oslomsg_notify_enabled` variable that aims to control
behaviour of enabled notifications.
Presence of ceilometer is still respected by default and being referenced.
Potential usecase are various billing panels that do rely on notifications
but do not require presence of Ceilometer.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/914144
Change-Id: I60746cedeec323bcede3bb5aa4968f0388e262bb

Cloudkitty has released it's v2 API couple of cycles ago with [1]. High time to
enable it as well.
[1] https://review.opendev.org/c/openstack/cloudkitty/+/832923
Change-Id: I11838780e84d6e721033b5c7aa77c8a0ca8ea121

This change implements and enables by default quorum support
for rabbitmq as well as providing default variables to globally tune
it's behaviour.
In order to ensure upgrade path and ability to switch back to HA queues
we change vhost names with removing leading `/`, as enabling quorum
requires to remove exchange which is tricky thing to do with running
services.
Change-Id: I722aefb5290a68311faf1d44c279151e8492466a

Implement support for service_tokens. For that we convert
role_name to be a list along with renaming corresponding variable.
Additionally service_type is defined now for keystone_authtoken which
enables to validate tokens with restricted access rules
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845690
Change-Id: Icb1de8c7e0a5196a4df457a5d4a3ca524d4622d0

- Implemented new variable ``connection_recycle_time`` responsible for SQLAlchemy's connection recycling
- Set new default values for db pooling variables which are inherited from the global ones.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/819424
Change-Id: I4ae28b356a404c1e76e5cff986d4fe0172b564f2

The Cloudkitty role has been lacking some attention for a while
causing some of the configuration and examples outdated.
Now that Cloudkitty playbook and dashboard can be installed using
OSA without running any additional playbooks from this repository
i have removed those from here.
Change-Id: I0fe96c318273f0016d93cf043bda74feb11c63f8

Change-Id: If8362185261aefa741f9c21c57b3dfe258d8543d

As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to replace policy.json to
policy.yaml and remove deprecated policy.json.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: I5714f88ab0df41adbac52ff65f16f37e69bafb79

This patch aims to add a prefix for memcached_server
on each role to give the ability for deployers to
override the location of memcached cluster. I.e users
wants to create a single memcached cluster with k8s
for each service.
We also add pymemcache based on [1]
[1] https://review.opendev.org/711429
Change-Id: I64614a02bd3a0d99a69a9c7ac59552ba7cb1fc88

Move service to use uWSGI role instead of iternal task for uwsgi
deployment. This aims to ease the maintenance of uWSGI and speedup
metal deployments as the same uwsgi environment will be used
across all services.
Change-Id: I5b0bcd6a51f237fcd772368f51a406421ffb52f4

As cloudkitty requires metrics.yml to be present in etc directory
to be fully operational, we're distributing this file by default[1]
cloudkitty_metrics_overrides variable added in order to override default
[1] http://logs.openstack.org/30/673530/2/check/openstack-ansible-functional-ubuntu-bionic/c6eee8a/logs/openstack/openstack1/cloudkitty-processor.service.journal.log.txt.gz#_Jul_30_14_44_00
Change-Id: Ie08cfd0d8dc4835712df0b937170b631aebd10c8

This patch aims to migrate service from usage of regular syslog files
to journald. We also disable uwsgi logging, since it dublicates
requests that are logged by service itself.
Change-Id: I4cd3270ef873b6f595b151014e82c8b4b35f12aa

The variables cloudkitty_developer_mode and cloudkitty_venv_download
no longer carry any meaning. This review changes cloudkitty to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.
As part of this, we move the installation out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.
We also change include_tasks to import_tasks and include_role
to import_role so that the tags in the python_venv_build role
will work.
Change-Id: Ieaec0a12a1ed6ea1f9a98a615fd8c6bff67a9d0c

Accoring to the cloudkitty docs it is strongly recommended to
install the API server in a WSGI host such as mod_wsgi.
The tasks are a combination of os_cinder steps and the cloudkitty
docs for wsgi:
https://docs.openstack.org/cloudkitty/latest/install/mod_wsgi.html#mod-wsgi
Change-Id: I4173a32f4989f7e23205083738ccd81f7ea6f9f6
Closes-Bug: 1769632

This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed.
The systemd journal would normally be populated with the standard out of
a service however with the use of uwsgi this is not actually happening
resulting in us only capturing the logs from the uwsgi process instead
of the service itself. This change implements journal logging in the
service config, which is part of OSLO logging.
OSLO logging docs found here: <https://docs.openstack.org/oslo.log/3.28.1/journal.html>
Change-Id: Ieff0119cb09e8971fab5b0393f023010a554edff
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

This introduces oslo.messaging variables that define the RPC and
Notify transports for the OpenStack services. These parameters replace
the rabbitmq values and are used to generate the messaging
transport_url for the service.
This patch:
* Add oslo.messaging variables for RPC and Notify to defaults
* Update transport_url generation in conf
* Add oslo.messaging to tests inventory and update tests
* Install extra packages for optional drivers
* Add release note
Change-Id: I0612252927f81b3584054f6cd77a2826c5d3fc3b

Use MySQL as coordination backend for cloudkitty-processors.
Otherwise prices may get calculated multiple times.
Change-Id: I661ca3b7ffb4f64242422297bafc938ca5082a12
Closes-Bug: 1777432

The sections are part of the cloudkitty repo:
https://github.com/openstack/cloudkitty/blob/master/etc/cloudkitty/api_paste.ini
But weren't inlcuded in osa.
Change-Id: I2579c8b88bcf2ec162002a8a8978c7720bfc8112
Closes-Bug: 1769630

Each collector needs its own config section, so use cloudkitty_collector
to create these sections.
Change-Id: I6bc4c395693265ea08067bb293a99a72b48fe9a6
Closes-Bug: 1769629

Use one variable for collected openstack services and another for
systemd services.
Change-Id: I1946243df319c00638e4ec4a4f1f144fd2131956
Closes-Bug: 1769625

Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.
[1]https://review.openstack.org/#/c/508522/
Change-Id: I5e2ac3a6b044515c84695b441602b254f4859bd8
Implements: blueprint deprecate-auth-uri-option

Option "rpc_backend" from group "DEFAULT" is deprecated for removal
(Replaced by [DEFAULT]/transport_url). Its value may be silently
ignored in the future.
Change-Id: Ia352ef7377e6e1f51fffb3181ad5ab422275f39a
Implements: blueprint deprecate-rpc-backend

Option "rabbit_use_ssl" from group "oslo_messaging_rabbit" is deprecated.
Use option "ssl" from group "oslo_messaging_rabbit".
Change-Id: Ic9157cf1eb4d56843dcc4141e49c7a3ec3bcba89
Implements: blueprint deprecate-rabbit-use-ssl

The systemd unit 'TimeoutSec' value which controls the time
between sending a SIGTERM signal and a SIGKILL signal when
stopping or restarting the service has been reduced from 300
seconds to 120 seconds. This provides 2 minutes for long-lived
sessions to drain while preventing new ones from starting
before a restart or a stop.
The 'RestartSec' value which controls the time between the
service stop and start when restarting has been reduced from
150 seconds to 2 seconds to make the restart happen faster.
These values can be adjusted by using the *_init_config_overrides
variables which use the config_template task to change template
defaults.
Change-Id: Ia10703ca1cfa5afd1dec09a52f1b981711796e33

This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Idac3056145b92adcd0b5b328296c59b71bf28441
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

We don't support trusty in master anymore, so this makes sures the
support of upstart is phased out.
Whenever possible, we use the systemd module instead, that reduces the
amount of tasks.
Change-Id: I50d377ce6c5e80386954fd551e566c55f57c5a3a
Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@rackspace.co.uk>

Change-Id: Ib579885209f447b501fcb338da4d9321c0aa4498
Implements: blueprint remove-pki

Change-Id: I523b5f1ae56c1ff19ca6bd95e3e0e7f00e282c71

Change-Id: I0dc824ccd4e46d9ab76d0908a1f28f3e9b1fe927
Closes-Bug: #1641695 

Change the 'cloudkitty_service_names' from a list to a dictionary mapping
of services, groups that install those services. This brings the
method into line with that used in the os_neutron role in order to
implement a more standardised method.
The init tasks have been updated to run once and loop through this
mapping rather than being included multiple times and re-run against
each host. This may potentially reduce role run times.
Currently the reload of upstart/systemd scripts may not happen if
only one script changes as the task uses a loop with only one result
register. This patch implements handlers to reload upstart/systemd
scripts to ensure that this happens when any one of the scripts
change.
The handler to reload the services now only tries to restart the
service if the host is in the group for the service according to the
service group mapping. This allows us to ensure that handler
failures are no longer ignored and that no execution time is wasted
trying to restart services which do not exist on the host.
Finally:
- Common variables shared by each service's template files have
 been updated to use the service namespaced variables.
- Unused handlers have been removed.
- Unused variables have been removed.
Change-Id: Ieb96fac62906ac9eb387edb450df0c0ca9c0ccb2

OSLO logging currently defaults the 'use_stderr' option to True
which results duplicate logs in service daemon logs for both
upstart and systemd. To correct this issue the use_stderr
option has been set to false.
Change-Id: I177bcd724433543cb39bce7bce19f43a5675d492
Closes-Bug: 1588051
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Update default variables with a few corrections:
 * Fix name of 'cloudkitty_output_pipeline' variable
 * Change 'cloudkitty_service_type' to 'rating'
 * Remove unnecessary and redundant 'cloudkitty_pip_packages'
Change-Id: I9768471d1f67d074282e63033d9bd626fb4175af

This change implements support for Xenial and CentOS 7 within the
os_cloudkitty role.
Change-Id: Id2abe9099171080bd3d3ceb3a54222a5dfdefe2f
Implements: blueprint multi-platform-host

Remove all tasks and variables related to toggling between installation
of cloudkitty inside or outside of a Python virtual environment.
Installing within a venv is now the only supported deployment.
Additionally, a few changes have been made to make the creation of the
venv more resistant to interruptions during a run of the role.
* unarchiving a pre-built venv will now also occur when the venv
 directory is created, not only after being downloaded
* virtualenv-tools is run against both pre-built and non pre-built venvs
 to account for interruptions during or prior to unarchiving
Change-Id: I84dd86e4e8244de5a6ac55a302df0c9737d6388e
Implements: blueprint only-install-venvs

Adds a task to fix the signing dir for cloudkitty

Adds the upstart scripts for kitty api
and kitty processor. It also enables the tasks
to restart the services.

The cloudkitty.conf and defaults/main had some
incorrect info in it keeping kitty from working
correctly with keystone v3

The template for cloudkitty.conf was mostly hard
coded. This change adds the variables to many of
the hard coded things.

Removed the service setup from the task