openstack/openstack-ansible-rabbitmq_server - openstack-ansible-rabbitmq_server - OpenDev: Free Software Needs Free Tools

This implements a new variable rabbitmq_custom_config to
be able to place custom configuration to the rabbitmq.conf
e.g. to configure installed plugins
Change-Id: I952eefe646b00f60184f8d353f6f055bbdc4ac90

This implements new variable rabbitmq_erlang_extra_args that enable
deployers to define extra flags for erlang [1]
[1] https://www.rabbitmq.com/runtime.html#cpu-reduce-idle-usage
Change-Id: Ibb0963962d6c7483d54fac69960910c0b13802a5

With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.
With that we also update metdata to reflect current state.
Change-Id: I63f6a2c803370736e969aa3a4dea3ea959316def

With this patch we ensure that duplicated records are not
created with rabbitmq role if hosts file already contain
OSA managed block. Managing hosts still might be required for
role usage outside of the OSA so we workaround this usecase.
Change-Id: Ia20902f0ffe21ce563966fee4d233e5ec3afe3d9
Related-Bug: #1960587 

Replace placing templates for systemd overrides with
systemd_service role, that will handle overrides for us in more
convenient and unified way.
Change-Id: I2759b1949e9ecc98953f414c6f9838aed7dd8499

Currently Debian Bullseye doesn't have erlang provided by external repos
So in order to update rabbitmq version used we need to implement
rabbitmq_erlang_install_method, so that we could use external rabbitmq
with distro provided erlang.
Hpefully that is temporary solution and erlang packages for bullseye
will be built one day.
Change-Id: I32256271759d26522c17fe14c75b41da4c86c31a

Starting from RabbitMQ 3.7.0 it's recommended to use new-style
config which is simply an ini file.
It's easier to read and maintain config file in ini fromat rather then
in classic erlang.
At the same time we still keep old-style config as it might have settings
that are not supported in new-style config.
There're no evidences that used there options are still supported,
but it's worth deprecating them in follow-up patch anyway.
Change-Id: I239366ad4aa2bc7a02d826b6c2f94631f4b0e622

It might be desired by deployer to remove already applied policy.
For that policy state should be explicitly passeda as absent
for the module.
Change-Id: I24bb110998eef978daf618964c1ee3713eb6b339

Supports two scenarios:
1) variables defined in defaults/main.yml are sufficient to create
a root/intermediate CA certificate for rabbitmq when this role
is used outside openstack-ansible.
2) when:
openstack_pki_dir
openstack_pki_setup_host
openstack_pki_authorities
openstack_pki_service_intermediate_cert_name
are defined, an external CA already created on the deploy host
with a previous run of ansible-role-pki will be used as the CA.
Server certificates for the rabbitmq instances are created from the
data in rabbitmq_pki_certificates in both situations:
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/788031
Change-Id: I4cb7c48a74a307217b645cb8528fdbb0f7b9f596

Set up TLS1.2 as recommended in https://www.rabbitmq.com/ssl.html
Change-Id: I634ffe7ed47b8670bd2aad029dac1313cebd2961

See https://github.com/ansible/ansible/issues/73654
Change-Id: Ia6dac13c7e2206f4a86ef8e21c5b1cd80eb16e0e

Depends-On: https://review.opendev.org/734672
Depends-On: https://review.opendev.org/735289
Change-Id: I35690f9a54337d6a268406ddcf6726f7040ac966

Change-Id: Id186abad9f7a99879e041176670f4c6c3ed85aff

This unifies the way of rabbitmq installation across all distros.
Also includes reno for deprecation of file installation method.
Change-Id: Idcf2d298e2808ef7b1a2160fc94cd6c1b5929182

These default to 0.0.0.0 and can be overriden in a real deployment
where the correct management network address is known.
Change-Id: If989ccee6449578316e2e8dbe502b6b17c7af9c5

With this configuration, the database Mnesia reduces the activity to the
disk. It is useful when a huge of queues/exchanges/bindings are created
and destroyed.
With the default value (100), RabbitMQ could log {Mnesia is overloaded}.
Moved to 300. The range should be between 100 and 1000.
Read [1] for more info.
[1] http://erlang.org/doc/man/mnesia.html#dump_log_write_threshold
Change-Id: I6dcfc9db02bcd8c8f0a1ebf58d9c3ceb84cae10a

The rabbitmq bind address can now be set using the
`rabbitmq_port_bindings` option which creates a hash of rabbitmq
connection options allowing multiple bind addresses and port
configurations to be present.
This change also organizes the config so that its now far more human
readable.
The option `rabbitmq_disable_non_tls_listeners` has been removed. This
was done because it is now irrelevant given its possible to set the
specific bind addresses and port configurations.
Change-Id: I103e406f5393a4ce3f7d6cd7f7e25e2058b0e796
Signed-off-by: Kevin Carter <kevin@cloudnull.com>

The override rabbitmq_memory_high_watermark can be used to
alter the overall memory consumption of rabbitMQ and more
importantly define when the garbage collection (gc) is happening.
The old default value of 0.4 can lead up to 80% memory usage
during gc which can lead to OOM scenarios.
The new default value is set to 0.2, lowering the maximum memory
usage to 40%
Change-Id: Iedbb459a5d17f16bbb204b0b8e989ae84c77f8a6

We have in the past provided "safe defaults" in the role, because
this role is focused on a rabbitmq server for openstack.
Nowadays, this role is used outside OpenStack-Ansible, and should be
made more independant of it.
Having default policies is a problem because it forces users to define
an empty policy as an override, or their own policy, overriding the existing
"safe default".
This provides a boolean, defaulting to false, to conditionally add the
openstack queues policies. If set to true, we'll apply the "previous behaviour"
to automatically deploy the "safe defaults", which is adding
`rabbitmq_openstack_policies` to the user defined ``rabbitmq_policies``.
Depends-On: https://review.openstack.org/640300
Change-Id: I0bf6e1829ade63052c0c7efe81afb0b765857687

Without this policy, the queues reply_*, *_fanout_* are mirrored, across
all the RabbitMQ nodes.
It is not necessary to mirror the queues above, because they are usually
bound to one specific consumer, so the mirroring is not essential.
The new policy reduces the number of mirror queues and increases, in
general, the performances because it reduces the number of copies across
the cluster.
More info about RabbitMQ HA [1]
[1]: https://www.rabbitmq.com/ha.html
ref:
http://eavesdrop.openstack.org/meetings/oslo/2019/oslo.2019年02月25日-15.22.txt
Change-Id: I783d314aaa68b09abb4fed90818165b9e61e9758

We make remote network hits to get the GPG keys which are quite
unreliable, and apt_key does not support using a proxy properly [1]
so let's store them inside the role and use them.
The implementation here matches that which was done in the
galera_client role in I520ccbadf3320b0d07fc83e3dbec9ea2bd16ec83
[1] https://github.com/ansible/ansible/issues/31691
Change-Id: I2715c904975b7940af72bd422904e748d3bae953

Previously the 'distro' install method was implemented as the
default for CentOS and SUSE, however, some remnants were left
behind, making the configuration confusing and causing some
issues due to the gpg verification on RedHat trying to use the
RabbitMQ gpg check.
Rather than try and keep the many confusing installation methods
for these distributions, we simply remove the other options and
all the config entries related to it.
As part of this, we need to clean up the functional tests which
previously implemented checks based on the 'file' install type
which is no longer used by any distribution.
Change-Id: I28199ce149f6893d688d11177ec950b17dbf0886

In the past, ansible_hostname could contain a FQDN or a hostname.
This is not the case anymore as fqdns are now stored in facts as
ansible_fqdn while short hostnames are stored in ansible_hostname.
This also simplifies the reading by using json_query.
Change-Id: I0755684846e42b4a17c4e46d70940535daaf73e7

The version of erlang we're using, while functional, is not recommended.
This change updates our version of erlang for rabbitmq to be the
recommended version by changing the pinned version of erlang.
The apt package pinning meta dependency has been removed and put into
the apt task file as an include role. To ensure a more uniform process
for setting the version of erlang used with rabbitmq the new variable
`rabbitmq_erlang_version_spec` has been added. This option has a default
of "null" and is set by distro specific variables when applicable.
Erlang version support matrix: https://www.rabbitmq.com/which-erlang.html
Change-Id: Iced12d4533eec068bd11d8bd235d81308ef40427
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

This adds a var similar to what we have in the galera_server role [1],
making the local node's incoming address a var. This allows the
role's consumer to define a different node address than the one used
by ansible in ansible_host.
[1] https://github.com/openstack/openstack-ansible-galera_server/blob/master/defaults/main.yml#L92
Change-Id: I30e51668545095c8f6d91902516767d7bf158202

This patch optimizes the Erlang/RabbitMQ installation on CentOS. It
includes:
 * Installing RabbitMQ from RabbitMQ's repository
 * Installing Erlang all-in-one from RabbitMQ's repository
 * Remove old versionlocks before applying new ones (fixes bug)
The erlang-solutions repository is hosted in eastern Europe and has
high latency to the USA and western Europe. Installing from that
repository brings in over 80 individual packages, which causes
additional delays in the role.
The Erlang all-in-one package from RabbitMQ's repository contains
only the Erlang bits that RabbitMQ needs. Also, it has HiPE enabled
by default, which is recommended by RabbitMQ developers for
better performance.
Closes-Bug: 1712596
Change-Id: I1bfcc96f353bd27b0004d93e250bb041eee48bdb

This patch increases the default ulimit for RabbitMQ to 65536. This
lines up well with other deployments (see bug report).
Closes-Bug: 1714511
Change-Id: I55ec4590c2c382bf26e9178ce9500fddcd6128dc

It may be required to override the repo information
in its entirety, rather than just the URL. This
patch allows that to be done.
Change-Id: Ib49c057465a2a7e394a830a33a03ad9de3150899

The RabbitMQ community and project recommend erlang 19.3.x for the best,
most stable, results. This change downloads the recommended version of
erlang [ https://www.rabbitmq.com/which-erlang.html ] from ESL and moves
the erlang packages for distros into a specific list installed only when
the ``rabbitmq_install_method`` variable is not set to "file".
Change-Id: Ief57b1fdcb791f0f8ed20c1cbb54ea19d4373f81
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Deployers can override the `rabbitmq_disable_non_tls_listeners`
variable, setting a value of `True` if they wish to enable this feature.
Change-Id: I4fe39099dbe8973d2655845c19882c404d4f20b1

In some cases, deployers may want to use a repo instead of a deb
file (limited connectivity, repo mgmt tools already in place, ...)
This commit introduces this behavior in an optional manner: The
default behavior is still using a deb file. If using an external
repository, the repo used will be RabbitMQ recommended one hosted
on packagecloud.io.
Change-Id: If366d7411cffd59e4988ceba6fc47eac2b384ddf

This commit removes the container_name var reference from
defaults/main.yml, and replaces it with ansible_hostname instead. This
helps when installing this role on a non-OpenStack-Ansible deployment
which does not have container_name set.
Change-Id: I7da72c45b50bff5722e1e5c1900c8aa56759eb7f

Previously the role assumed that all deployers wanted
all queues to be HA. Performance testing by Mirantis and
Intel has suggested that throughput increases can be dramatic
when disabling HA queues, see:
 http://docs.openstack.org/developer/performance-docs/test_results/mq/rabbitmq/index.html
Therefore, we should allow deployers to set their own
policies when desired.
Change-Id: I156b8618e78d74823ec568053157afc853baad84
Closes-Bug: #1607830 

The ``rabbitmq_server`` role now supports configuring HiPE compilation
of the RabbitMQ server Erlang code. This configuration option may improve
server performance for some workloads and hardware. Deployers can
override the ``rabbitmq_hipe_compile`` variable, setting a value of
``True`` if they wish to enable this feature.
Change-Id: I433d94eff00ac82a9069f9092faa87d449190442

The rabbitmq_plugins variable was mentioned as a loop item but was not
actually being used within the task to enable the management plugin.
It can now be overridden to enable or disable any RabbitMQ plugins.
Change-Id: I35f82427be464c811ee098ef676825aaa1e05d8d
Related-Bug: 1617431

1. Add collect_statistics_interval and rates_mode configurations in
rabbitmq.config template which enables to define custom value for
those fields.
2. Add collect_statistics_interval and rates_mode defaults in
defaults/main.yml
Change-Id: Ie349528929398f53a9d87e7fc02e0c95c9d6d4f1
Closes-bug: 1617516

The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: I3227dc6f15e0307926e65427bc635c34e2baa87f

The pycrypto package no longer needs to be installed. It was
previously necessary due to the use of memcache for moving
SSL certificates around. As we're no longer using this
method the package installation and its binary deps may be
removed.
Change-Id: I805edeb47dc888411a785aacdb7d93d0cb05c393

The default value of "rabbitmq_all" remains, but this
change should allow deployers to target the role to multiple
groups now.
Change-Id: Idc1bc44b005f09f9dc7eb9778719a8ba92756c06
Implements: blueprint multi-rabbitmq-clusters

Separate files have been created for vars and tasks related to a
specific package manager. The 'vars_files_var' variable has been created
to store a list of files to search for distro specific variables.
The 'rabbitmq_apt_packages' variable has been deprecated and renamed to
the more generalized 'rabbitmq_dependencies' to better describe its
purpose and to simplify reuse of existing install tasks between multiple
distros.
Change-Id: I1940593978b733501daf5fe25edd393f2f6bee0c

This change ensures that the RabbitMQ ServerName is being set to
the **ansible_hostname** fact. This ensures the RabbitMQ server
has an appropriate server name which is compliant to the RFC.
Change-Id: I1f6a161b3af8fd0edd677ac1c3f396e6e8b5d02e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Change-Id: I52be3d82e11d3c74f5ec974e0dcc7a6bb4b628b8

Change the default install version of RabbitMQ Server to 3.6.1-1.
Additionally, the functional test playbook file has been split into
multiple, more focused, files. Tests to ensure that the expected version
of RabbitMQ Server is running and to upgrade from 3.5.7 to 3.6.1 have
been added.
Change-Id: I8907a3a71334be7b89ad743c3f2031e0a42a00f2

This fix will add ERLANG VM tuning parameters with increased
default settings to support larger installations.
Following overrides have been added:
 rabbitmq_async_threads defaults to 128 (from 32)
 rabbitmq_process_limit defaults to 1048576 (from 256k)
Closes-Bug: #1549940
DocImpact
Change-Id: Ia0fab288db8aa287e667dfc843f02d7ec318a816

Workarounding the upstream ansible apt module bug
documented here:
https://github.com/ansible/ansible-modules-core/pull/1517
For the next versions of ansible we'll be using, we should
check if the apt bug is fixed. When it's fixed, we could
abandon this change and use the standard apt module
with correct cache handling.
Change-Id: I4b27e630757ca228ed3543bb1d30187d2ee4e73b