openstack/openstack-ansible-rabbitmq_server - openstack-ansible-rabbitmq_server - OpenDev: Free Software Needs Free Tools

RabbitMQ team has reverted their support policy, and made
4.1 series supported for 18 month again.
Change-Id: Ifb2336b8b2d1b3b3118934a8a0727059e017df59
Signed-off-by: Dmitriy Rabotyagov <dmitriy.rabotyagov@cleura.com>

RabbitMQ has changed their docs to suggest using deb1.rabbitmq.com
instead of ppa1.rabbitmq.com.
For Debian Trixie ppa1.rabbitmq.com contains only RabbitMQ 3.12,
while deb1.rabbitmq.com contains all of 3.13, 4.0 and 4.1
We leave erlang as is for now, as it's not available for Trixie yet.
Change-Id: I98dfeb1706701787e7a7e912e37910108cec3a10
Signed-off-by: Dmitriy Rabotyagov <dmitriy.rabotyagov@cleura.com>

RabbitMQ supports Erlang 27 since release 4.0.4 [1]
With that Debian Trixie package is available only for Erlang 27
[1] https://www.rabbitmq.com/docs/which-erlang
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/965798
Change-Id: I055941dd6829c4055377fd46ec6cec8f9351546e
Signed-off-by: Dmitriy Rabotyagov <dmitriy.rabotyagov@cleura.com>

In case of defining different IP address for SSH and Management
network in metal scenario, as described in [1], you get SSH address
defined as `ansible_host`, which results in HAProxy configuring
backends on SSH address rather then management one, where
service is actually listening.
We use `ansible_host` as a fallback for non-OSA usage of the role.
[1] https://docs.openstack.org/openstack-ansible/latest/reference/inventory/configure-inventory.html#having-ssh-network-different-from-openstack-management-network
Change-Id: I85a164b181b41025f8f872dbdf5426bf58f6f36e
Signed-off-by: Dmitriy Rabotyagov <dmitriy.rabotyagov@cleura.com>

During migration to Quorum queues we have renamed vhosts
to not contain leading slash for them. However, some services
at the same time were configured not to have RPC messaging,
but only notifications whenever needed - like keystone, glance, etc.
This left such vhosts still containing HA policy on them, leading
to RabbitMQ upgrade failures with:
```
rabbitmq-server[2934794]: Application rabbit exited with reason:
{{failed_to_deny_deprecated_features,[classic_queue_mirroring]},{rabbit,start,[normal,[]]}}
```
This patch implements a check of deprecated feature flags and
disables all HA policies across the board before upgrade happens.
As we don't want to force removal when running RabbitMQ 3.13,
the removal is enforced only during upgrade to 4.0.
Conditions are maintained on a task level rather then a block
to make it extendable with other deprecated flags in the future.
Change-Id: I372421e0347687a1d861b548fd20ee92b150c301
Signed-off-by: Dmitriy Rabotyagov <dmitriy.rabotyagov@cleura.com>

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/934042
Change-Id: I231ea65b5340c43bffb541ce23d3a8e600015c2d

At the moment rabbitmq_upgrade will temporary stop all nodes in the
cluster, shortly disrupting operations.
This behavior is now changed to perform a regular rolling upgrade [1]
Community right now relying on the rabbitmq_upgrade variable to
"recover" the cluster state by basically re-bootstrapping the cluster.
Such behaviour should be implemented separately in a follow-up patch.
[1] https://www.rabbitmq.com/docs/upgrade#rolling-upgrade
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/931801
Change-Id: Ibb3fae0d85dc31237ce169bb4cde9551dd5cabb6

We do have a variable `rabbitmq_erlang_version_spec` which is applicable
only for Debian, while for EL the variable is only private.
This patch does some alignment on things:
* spec is replaced with just a version, as we don't need ternary logic
anymore.
* defenition of erlang version is unified across distros
Change-Id: Ic72212fa2d6df55f2f60d1d38c48310765442e51

Rabbitmq provide a new set of ppa fronted by Cloudflare, use those
instead of the previous one which is no longer mentioned in the
rabbitmq installation guide.
Change-Id: I6eff083b95b392e9f8a05f79ee2ec49705257a64

The apt-key module is deprecated so the code is refactored to allow
any of the deb822_repository features to used instead.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-apt_package_pinning/+/927903
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/907886
Change-Id: I96f1f23d6a1cca223ad4fc48a1caa21833a98b22

Stable feature flags must be enabled before upgrades to ensure
the service can still start. These additional tasks check for
any disabled feature flags and enable them. This is done
immediately after upgrades so that systems can run with them
enabled striaght away.
Closes-Bug: #2038818
Change-Id: I5211a30899f544a9f7e609e85551a92b245d25bf

Update the zed release notes configuration to build from
unmaintained/zed.
Change-Id: I220491d548f4fe1031e572de645962238a039207

Policies were always applied to target `all` aspects and
there was no way to have them target only e.g. exchanges.
This can be important though, see [1].
This change enables the use of the apply_to parmeter via the existing
variables while maintaining `all` to be the default.
[1] https://www.rabbitmq.com/docs/parameters#how-policies-work
Change-Id: I61d68c630599b8ef2382663e8a37fdc456435c11

Add the ability to configure the logging options for the rabbitmq-server
via the rabbitmq_log variable and key-value pairs
Change-Id: If9f28f29311150ed7b8ba61d2ddf7e12ca07d3c4

Add ability to set the environment variable RABBITMQ_USE_LONGNAMES
via the rabbitmq-env.conf to be able to use the FQDN for a node
Change-Id: I766bdbe847eb24010bfe9912f07e89d78bbc1bed

This implements a new variable rabbitmq_custom_config to
be able to place custom configuration to the rabbitmq.conf
e.g. to configure installed plugins
Change-Id: I952eefe646b00f60184f8d353f6f055bbdc4ac90

This implements new variable rabbitmq_erlang_extra_args that enable
deployers to define extra flags for erlang [1]
[1] https://www.rabbitmq.com/runtime.html#cpu-reduce-idle-usage
Change-Id: Ibb0963962d6c7483d54fac69960910c0b13802a5

Upstream has notified us[1] about the immediate deprecation and removal
of the PackageCloud repositories we are using. Thus, we must move to the
CloudSmith repos already being used for the erlang dependency for
RabbitMQ.
[1] https://github.com/rabbitmq/rabbitmq-server/discussions/8386
Closes-Bug: 2021410
Change-Id: I361757b8f76a6c2d087989a36d8f7543aba69e16

Change-Id: If8c83682093cb08e8a092550c44010134cd210d3

Add file to the reno documentation build to show release notes for
stable/zed.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/zed.
Sem-Ver: feature
Change-Id: Ic392f8ee059787620d67be181e998b0484ef443b

With this patch we ensure that duplicated records are not
created with rabbitmq role if hosts file already contain
OSA managed block. Managing hosts still might be required for
role usage outside of the OSA so we workaround this usecase.
Change-Id: Ia20902f0ffe21ce563966fee4d233e5ec3afe3d9
Related-Bug: #1960587 

Replace placing templates for systemd overrides with
systemd_service role, that will handle overrides for us in more
convenient and unified way.
Change-Id: I2759b1949e9ecc98953f414c6f9838aed7dd8499

Switch to using cloudsmith repo as a source for rabbitmq and erlang.
This allows us to be consistent about repository that is
used across supported distributions along with switching from
unreliable erlang-solutions that tend to fail periodically.
Change-Id: Ia438ee3b0aa1ba95aff014776e936516b83181ec

Starting from RabbitMQ 3.7.0 it's recommended to use new-style
config which is simply an ini file.
It's easier to read and maintain config file in ini fromat rather then
in classic erlang.
At the same time we still keep old-style config as it might have settings
that are not supported in new-style config.
There're no evidences that used there options are still supported,
but it's worth deprecating them in follow-up patch anyway.
Change-Id: I239366ad4aa2bc7a02d826b6c2f94631f4b0e622

The sync from https://review.opendev.org/733244 updated to
openstackdocstheme 2.2.1 and reno 3.1.0 versions.
Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.
openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
Change-Id: I5c6c99763cffd75824c31efd37caec6e4828ac43

Add file to the reno documentation build to show release notes for
stable/ussuri.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/ussuri.
Change-Id: I80561fc623a89b89d6c2bc1143c751d543863d4e
Sem-Ver: feature

New version of openstackdocstheme (Victoria+) respects pygments_style.
Since this repo is using now Victoria (master) requirements but has
not branched for Ussuri yet, it uses the new version.
Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.
Change-Id: Iad5aa6453e61bf08555805072e9b6e433e656e93

This repo is now testing only with Python 3, so let's make
a few cleanups:
- Remove setup.* files, those are not needed for this repo.
- Cleanup */source/conf.py to remove now obsolete content.
- Remove install_command from tox.ini, the default is fine,
 move constraints into deps
- Enable warnings for docs build
Change-Id: I07da37d6f15aefc7b2fa558a6ead7028d6c9c556

This unifies the way of rabbitmq installation across all distros.
Also includes reno for deprecation of file installation method.
Change-Id: Idcf2d298e2808ef7b1a2160fc94cd6c1b5929182

This patch replaces git.openstack.org with opendev.org as redirection
from old path was enabled.
Change-Id: Iae8d1f21e85f73efb56230cffc3ad39793ec3f3b

Add file to the reno documentation build to show release notes for
stable/train.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.
Change-Id: I56250e12e8eb596af69c69ad847a4c068e9abf68
Sem-Ver: feature

Add file to the reno documentation build to show release notes for
stable/stein.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/stein.
Change-Id: I2962c47e99404ed6ea0edea96744f6119c988a52
Sem-Ver: feature

With this configuration, the database Mnesia reduces the activity to the
disk. It is useful when a huge of queues/exchanges/bindings are created
and destroyed.
With the default value (100), RabbitMQ could log {Mnesia is overloaded}.
Moved to 300. The range should be between 100 and 1000.
Read [1] for more info.
[1] http://erlang.org/doc/man/mnesia.html#dump_log_write_threshold
Change-Id: I6dcfc9db02bcd8c8f0a1ebf58d9c3ceb84cae10a

The rabbitmq bind address can now be set using the
`rabbitmq_port_bindings` option which creates a hash of rabbitmq
connection options allowing multiple bind addresses and port
configurations to be present.
This change also organizes the config so that its now far more human
readable.
The option `rabbitmq_disable_non_tls_listeners` has been removed. This
was done because it is now irrelevant given its possible to set the
specific bind addresses and port configurations.
Change-Id: I103e406f5393a4ce3f7d6cd7f7e25e2058b0e796
Signed-off-by: Kevin Carter <kevin@cloudnull.com>

The override rabbitmq_memory_high_watermark can be used to
alter the overall memory consumption of rabbitMQ and more
importantly define when the garbage collection (gc) is happening.
The old default value of 0.4 can lead up to 80% memory usage
during gc which can lead to OOM scenarios.
The new default value is set to 0.2, lowering the maximum memory
usage to 40%
Change-Id: Iedbb459a5d17f16bbb204b0b8e989ae84c77f8a6

Without this policy, the queues reply_*, *_fanout_* are mirrored, across
all the RabbitMQ nodes.
It is not necessary to mirror the queues above, because they are usually
bound to one specific consumer, so the mirroring is not essential.
The new policy reduces the number of mirror queues and increases, in
general, the performances because it reduces the number of copies across
the cluster.
More info about RabbitMQ HA [1]
[1]: https://www.rabbitmq.com/ha.html
ref:
http://eavesdrop.openstack.org/meetings/oslo/2019/oslo.2019年02月25日-15.22.txt
Change-Id: I783d314aaa68b09abb4fed90818165b9e61e9758

We make remote network hits to get the GPG keys which are quite
unreliable, and apt_key does not support using a proxy properly [1]
so let's store them inside the role and use them.
The implementation here matches that which was done in the
galera_client role in I520ccbadf3320b0d07fc83e3dbec9ea2bd16ec83
[1] https://github.com/ansible/ansible/issues/31691
Change-Id: I2715c904975b7940af72bd422904e748d3bae953

Change-Id: Ib0a55b2907baf5fd8f37c02b3538edb02729ab1d

The version of erlang we're using, while functional, is not recommended.
This change updates our version of erlang for rabbitmq to be the
recommended version by changing the pinned version of erlang.
The apt package pinning meta dependency has been removed and put into
the apt task file as an include role. To ensure a more uniform process
for setting the version of erlang used with rabbitmq the new variable
`rabbitmq_erlang_version_spec` has been added. This option has a default
of "null" and is set by distro specific variables when applicable.
Erlang version support matrix: https://www.rabbitmq.com/which-erlang.html
Change-Id: Iced12d4533eec068bd11d8bd235d81308ef40427
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Change-Id: I957251be9dce419a6db8a272986fc5d12bfe3e9b

Release notes are version independent, so remove version/release
values. We've found that projects now require the service package
to be installed in order to build release notes, and this is entirely
due to the current convention of pulling in the version information.
Release notes should not need installation in order to build, so this
unnecessary version setting needs to be removed.
This is needed for new release notes publishing, see
I56909152975f731a9d2c21b2825b972195e48ee8 and the discussion starting
at
http://lists.openstack.org/pipermail/openstack-dev/2017-November/124480.html
.
Change-Id: I467712d9122270788f19d2fc8aa9481b1379bcf0

This patch implements an initial set of jobs intended to match
the current job execution method. It does not intend to improve
how the jobs are executed - only to replicate what is currently
in openstack-infra/openstack-zuul-jobs and provide the platform
to iterate on.
Change-Id: I7a1c312268b7219cb3e04058e9563a2fe0c469f0

This patch optimizes the Erlang/RabbitMQ installation on CentOS. It
includes:
 * Installing RabbitMQ from RabbitMQ's repository
 * Installing Erlang all-in-one from RabbitMQ's repository
 * Remove old versionlocks before applying new ones (fixes bug)
The erlang-solutions repository is hosted in eastern Europe and has
high latency to the USA and western Europe. Installing from that
repository brings in over 80 individual packages, which causes
additional delays in the role.
The Erlang all-in-one package from RabbitMQ's repository contains
only the Erlang bits that RabbitMQ needs. Also, it has HiPE enabled
by default, which is recommended by RabbitMQ developers for
better performance.
Closes-Bug: 1712596
Change-Id: I1bfcc96f353bd27b0004d93e250bb041eee48bdb

The release note didn't make it in with the RabbitMQ ulimit
adjustment patch. This patch adds the note.
Closes-Bug: 1714511
Change-Id: I858f459fad829571c66f3baac4432c3d0318820d