openstack/openstack-ansible-os_cloudkitty - openstack-ansible-os_cloudkitty - OpenDev: Free Software Needs Free Tools

This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.
Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.
Because this patch is generated by a script, there may be some
cosmetic changes to the layout of the YAML file(s) as the contents are
normalized.
See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html
Change-Id: I77fadf17c4e08f837af3e375477a38bb0e4867f0
Story: #2002586
Task: #24319 

Change-Id: I4f28e1832e16f23fb0f283e7bcd50e95d120ef6c

The TESTING_BRANCH environment variable is provided by the
run_tests.sh script and is derived from the .gitreview file.
This ensures that once the master branch becomes a stable
branch, the constraints from the stable branch in the
integrated repository will automatically get used once the
.gitreview file is updated.
To ensure that the required environment variables are present
we export them appropriately in run_tests.sh and modify the
tox configuration to pass them into the tox test.
Change-Id: I21e185ac2127146b33cbd4e67b15134eaa6bf69f
Needed-By: https://review.openstack.org/579371 

Change-Id: If2f0c08eaa97c4efd19b67ea79ef4eb8b424857c

include is marked as deprecated since ansible 2.4[0]
Switch to include_tasks or import_playbook as necessary
[0] https://docs.ansible.com/ansible/2.4/include_module.html#deprecated
Change-Id: I0297ca2a66b3344cfc20a1adf81b71339174a205

Change-Id: Iafa86669b41be5641a9c5eb45a1e321beb928e0b

The use of 'include_tasks' and a loop of variables creates
a situation where a user is unable to use tags to scope the
inclusion of only the MQ tasks when running the playbooks.
The use-case this is important for is when the rabbitmq
containers are destroyed and rebuilt in order to resolve
an issue with them, and the user wishes to quickly recreate
all the vhosts/users.
Ansible's 'include_tasks' is a dynamic inclusion, and dynamic
inclusions are not included when using tags. The nice thing
about dynamic inclusions is that they completely skip all
tasks when the condition does not apply, cutting down deploy
time. However, given the use-case, we should rather take on
the extra deployment time.
This patch changes the dynamic inclusion to a static one,
adds a 'common-mq' tag to cover all MQ implementations,
and re-implements the 'common-rabbitmq' tag for the tasks
that relate to RabbitMQ specifically.
It also implements conditionals for each task set so that
the rpc/notify tasks can be skipped if a vhost/user is not
required for that purpose (eg: swift does not use RPC, and
most roles will not use notifications by default).
Depends-On: https://review.openstack.org/588191
Change-Id: Ied4df931d0464173649c39ac82253c541b195cb1

The yaml should start with "---"
Change-Id: Iaf43682075718e85c1e22f8d4ac28fd65dea0d17

This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed.
The systemd journal would normally be populated with the standard out of
a service however with the use of uwsgi this is not actually happening
resulting in us only capturing the logs from the uwsgi process instead
of the service itself. This change implements journal logging in the
service config, which is part of OSLO logging.
OSLO logging docs found here: <https://docs.openstack.org/oslo.log/3.28.1/journal.html>
Change-Id: Ieff0119cb09e8971fab5b0393f023010a554edff
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

1. The venv tag override is unnecessary, given there is a default.
2. The install branch override is unnecessary, given there is a
 default. The constraint in defaults was set to use the
 'cloudkitty_requirements_git_install_branch' instead of this
 variable anyway. This has been corrected to the right var.
3. The galera address for role tests has a specific var for it.
4. The 'cloudkitty_requirements_git_install_branch' is a relic
 of the past, and has been removed.
5. The playbook is very dated and is using a lot of old mechanisms
 and unnecessary vars. It is updated to be normalised with the
 playbooks from the integrated build.
Change-Id: I6a8e8b3ec8b00790416648beb1c4ce0257406409

There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement two new variables:
- cloudkitty_oslomsg_rpc_setup_host
- cloudkitty_oslomsg_notify_setup_host
These are used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.
We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.
Finally, we remove the test mq setup tasks and clean up any unused
or unnecessary variables configured in tests.
Change-Id: Iac217b80df3da79a944185a90ba4c1ff5409945b

This introduces oslo.messaging variables that define the RPC and
Notify transports for the OpenStack services. These parameters replace
the rabbitmq values and are used to generate the messaging
transport_url for the service.
This patch:
* Add oslo.messaging variables for RPC and Notify to defaults
* Update transport_url generation in conf
* Add oslo.messaging to tests inventory and update tests
* Install extra packages for optional drivers
* Add release note
Change-Id: I0612252927f81b3584054f6cd77a2826c5d3fc3b

Change-Id: I5dd365ec21959c60df5f83ad98705f94633e3d8e

With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: I77c037ec01e192ec0723050c05e296d4ccbfb79e

In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
With the dependent patches, the openstack_openrc role is now executed once
on the designated host, so it is no longer required as a meta-dependency for
the role.
Depends-On: https://review.openstack.org/579233
Depends-On: https://review.openstack.org/579959
Change-Id: Ib46638d1cb35e677ecfe789a32bb69a024042d6f

Using tox for requirements management requires in-repo
requirements files for all our repositories. Rather than
do that, we make use of the tests repo to capture our
common requirements and use this to install them.
This reduces our review requirement rate and simplifies
maintenance for us for the tox config. It also makes it
usable with 'Depends-On', which is marvellous!
The tox requirements definitions for docs/releasenotes
builds are left in-place as those are standard entries
across the community. If that changes at some point, we
can re-assess those entries too.
Depends-On: https://review.openstack.org/579208
Change-Id: Ib9a986352cf525d7d78ff3cfe1c63d74e30bff8d

Change-Id: I735c45cc492a56b42b2a16bbf41cb4cc85d04d28

Change-Id: Ie051d256da7d1776fa045412ba5e64ee9591686e

In order to have more centralized configuration, switch the role
to use the project-template instead.
Depends-On: I3df2c670beeb78baaa1515bcd27e8f2b0d95b3a9
Change-Id: I4a18fc449bad3395547baa0f4ba1e4f95e8900e0

Change-Id: I4bd223a3f5fbd36839e8b79103994ba4a7122fd8

1) Add documentation url link.
2) Add release notes url link.
3) Add the project home url link.
Change-Id: I9d47b013fbcea1fa765754b19999babf38dc4a3e

Use MySQL as coordination backend for cloudkitty-processors.
Otherwise prices may get calculated multiple times.
Change-Id: I661ca3b7ffb4f64242422297bafc938ca5082a12
Closes-Bug: 1777432

There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement a new variable called 'cloudkitty_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable cloudkitty_galera_address which
has been used for a long time, but never documented.
Change-Id: I7bb3f38556558ef14723361d3672a1d06f839b52

The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.
Change-Id: Ib058bc3dc6e97871f9c3a177c7f7dec48709ec3f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Change-Id: If2f0eef5da0a2347b575ec59d1f8efcf85c9ebc9

Change-Id: Ia168ab1ab8a5f868c3686b4f68b5c25bcb659c90

As the ceilometer collector has been removed from cloudkitty repo[1],
we should stop to use it.
[1]https://review.openstack.org/#/c/548630/
Change-Id: I9c244e6a763c2fd82a1f32915e033c4c1b0df280

Change-Id: Ic32d0ad2a8d323a95645263c0e970e387152483b

Change-Id: I997b79594e3848efa1ae43bdffc75fb41c580293

The sections are part of the cloudkitty repo:
https://github.com/openstack/cloudkitty/blob/master/etc/cloudkitty/api_paste.ini
But weren't inlcuded in osa.
Change-Id: I2579c8b88bcf2ec162002a8a8978c7720bfc8112
Closes-Bug: 1769630

Each collector needs its own config section, so use cloudkitty_collector
to create these sections.
Change-Id: I6bc4c395693265ea08067bb293a99a72b48fe9a6
Closes-Bug: 1769629

Use one variable for collected openstack services and another for
systemd services.
Change-Id: I1946243df319c00638e4ec4a4f1f144fd2131956
Closes-Bug: 1769625

Change-Id: I585f57c69f5d0c3e415f784152f8f900ac0c56ca
Closes-Bug: 1769624

The upstream requierements do not fit openstack-ansible venvs.
So explicitly loosen up until fixed upstream.
Change-Id: I307c9eefac4ab25a88bee7621461e41320bc4a70
Closes-Bug: 1769621

Change-Id: I26835fa198ffd6ecfdbddeea75a17014eb0b5ad6

We reinitialize the venv to ensure that the right version of
python is in the venv, but we do not want virtualenv to also
replace pip, setuptools and wheel so we tell it not to. If we
don't do this then virtualenv will install the latest available
version, which is not what we want.
Change-Id: Id29497b36cd2b9cca3f3b08474e1001d528931b6
Partial-Bug: #1764470