openstack/openstack-ansible-rabbitmq_server - openstack-ansible-rabbitmq_server - OpenDev: Free Software Needs Free Tools

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/939824
Change-Id: Iba67d3e321c6ebc256bb63de27ce61c26db10ab0

Update ansible meta
Treat backslash in strings as a literal when required
See https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
Refactor ansible line length > 160 chars
Depends-On: https://review.opendev.org/755484
Change-Id: Iaf26e6e943d0b45a7b96657bbb9047a2d919c9aa

ansible_python_interpreter is now set to auto in the tests repo.
it doesn't need to be overwritten in the test anymore.
Depends-On: https://review.opendev.org/734672
Depends-On: https://review.opendev.org/735289
Depends-On: https://review.opendev.org/734676
Change-Id: I27da99b0ec5fe309b1bb19e073f4ab89502b000e

Older ansible versions used search as a filter but this is no longer
valid, and search should be used as a test
Change-Id: I4d846b7704e22d1ae58163d413c86dd01ff3563c

Change-Id: Id294850bce5c6485cd704c595812630b98a9c572

This patch changes the default deployment back to using the
RabbitMQ external repositories. This restores the behaviour
to the same as the old and ships the 3.7.x series.
We deploy without HiPE by default however our testing is doing
it with HiPE. This patch drops it so we can actually test
behaviour that we deploy by default.
Change-Id: I49747e104f118ef1fb2bdeb329c92d2f5fb66c56

We have in the past provided "safe defaults" in the role, because
this role is focused on a rabbitmq server for openstack.
Nowadays, this role is used outside OpenStack-Ansible, and should be
made more independant of it.
Having default policies is a problem because it forces users to define
an empty policy as an override, or their own policy, overriding the existing
"safe default".
This provides a boolean, defaulting to false, to conditionally add the
openstack queues policies. If set to true, we'll apply the "previous behaviour"
to automatically deploy the "safe defaults", which is adding
`rabbitmq_openstack_policies` to the user defined ``rabbitmq_policies``.
Depends-On: https://review.openstack.org/640300
Change-Id: I0bf6e1829ade63052c0c7efe81afb0b765857687

The yaml should start with "---"
Change-Id: I2b60f713ef94af587fee4c32a1e45e016a1209f5

Previously the 'distro' install method was implemented as the
default for CentOS and SUSE, however, some remnants were left
behind, making the configuration confusing and causing some
issues due to the gpg verification on RedHat trying to use the
RabbitMQ gpg check.
Rather than try and keep the many confusing installation methods
for these distributions, we simply remove the other options and
all the config entries related to it.
As part of this, we need to clean up the functional tests which
previously implemented checks based on the 'file' install type
which is no longer used by any distribution.
Change-Id: I28199ce149f6893d688d11177ec950b17dbf0886

In I473b4162e141e05692e89f0420ea74c745dfc9e5 we forgot
to remove the ansible-role-requirements.yml entry for
previous_rabbitmq_server, so we do it here.
Change-Id: I1e844b0bfa41296075221992472d9369242e22ee

The connection plugin no longer falls back on using the
inventory_hostname as the container_name. Set container_name as a host
var for each container in the test inventory.
Change-Id: I29025dea82ff8a0bcd03cbcf39ae7b34470f0a89

Now that we're using the general templates, we can slim down
the role test definitions. We can also remove tests which are
not being watched, or which are fundamentally broken. With
this we can also remove unused scripts/plays.
We do the following:
1. We remove the 'upgrade' job, given that it's been broken
 all cycle due to the way the job executes and we need to
 regroup to figure out another way to do it.
Change-Id: I473b4162e141e05692e89f0420ea74c745dfc9e5

include is marked as deprecated since ansible 2.4[0]
Switch to include_tasks or import_playbook as necessary
[0] https://docs.ansible.com/ansible/2.4/include_module.html#deprecated
Change-Id: I0fcc921ab5382d5f5a303be0bb989f135fa5cf9b

The version of erlang we're using, while functional, is not recommended.
This change updates our version of erlang for rabbitmq to be the
recommended version by changing the pinned version of erlang.
The apt package pinning meta dependency has been removed and put into
the apt task file as an include role. To ensure a more uniform process
for setting the version of erlang used with rabbitmq the new variable
`rabbitmq_erlang_version_spec` has been added. This option has a default
of "null" and is set by distro specific variables when applicable.
Erlang version support matrix: https://www.rabbitmq.com/which-erlang.html
Change-Id: Iced12d4533eec068bd11d8bd235d81308ef40427
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Now that run_tests.sh handles the tests repo clone, we can
remove the use of the older tests-repo-clone.sh script.
Change-Id: I04b2c0614b3a328b8d805f64f8ed517435bf0ddc

Change-Id: Ia9c4eb9657246e6669310b45cd735390e2f7b285

The meta-play test-setup-host.yml already includes
test-prepare-containers.yml so there's no need to
re-run it.
Change-Id: I91b63aeebc4afe67be160245eb8872c16c2da2ad

The current functional test does a full upgrade process,
but uses the same current role for both the previous
version and the new version.
Ideally we want to test the initial deploy using the
previous series' role, so that we're actually testing
a series upgrade, not just a rabbitmq upgrade.
This patch separates the upgrade test out and makes the
default test just be a deployment. This allows us to
more clearly see where something breaks.
Depends-On: https://review.openstack.org/543245
Change-Id: I43e1e224cd4237ac37b40fde5ce7c279331a012d

Change-Id: I36e140347a60d57e65c803248260af23c7d19df1

With the merge of https://review.openstack.org/520177 in the
tests repo some ansible-lint failures which previously were
not being picked up are now detected.
Change-Id: I8b72bf83996451166fca5b20990c549fc0129306

This patch implements an initial set of jobs intended to match
the current job execution method. It does not intend to improve
how the jobs are executed - only to replicate what is currently
in openstack-infra/openstack-zuul-jobs and provide the platform
to iterate on.
Change-Id: I7a1c312268b7219cb3e04058e9563a2fe0c469f0

Change-Id: Iae23dfc74bba6a5df9dd5d51d7f08aaa6cadc940

The openSUSE OBS OpenStack repositories provide rabbitmq-server and
erlang packages tested in an openSUSE OpenStack cloud environment so
it's best to use these instead of upstream ones. As such we change the
default option to be 'distro' instead of 'file'. However, for the tests
we use 'file' so we can install an older version and test the upgrade
path. Moreover, we drop the 'zypper lock' code since the OBS repositories
only provide one version of the needed packages. Finally, we fail if the
installation method is set to external since there is no upstream
repositoiry for openSUSE.
Change-Id: I79b8f0af97b6fcbf904f4b497a8ab0e070d20542

Depends-On: I3641da0cf67809ca0b343a9692fd5c7592fadb78
Change-Id: Ie37982fcdf5f9e0a290f733635680a19e9531d59

This patch optimizes the Erlang/RabbitMQ installation on CentOS. It
includes:
 * Installing RabbitMQ from RabbitMQ's repository
 * Installing Erlang all-in-one from RabbitMQ's repository
 * Remove old versionlocks before applying new ones (fixes bug)
The erlang-solutions repository is hosted in eastern Europe and has
high latency to the USA and western Europe. Installing from that
repository brings in over 80 individual packages, which causes
additional delays in the role.
The Erlang all-in-one package from RabbitMQ's repository contains
only the Erlang bits that RabbitMQ needs. Also, it has HiPE enabled
by default, which is recommended by RabbitMQ developers for
better performance.
Closes-Bug: 1712596
Change-Id: I1bfcc96f353bd27b0004d93e250bb041eee48bdb

The distribution repositories may provide different RabbitMQ versions
and even change during time. As such, in order to be able to reliably
test the upgrade path, we need to install the rpm/deb packages directly
instead of replying to the distribution repositories. This is mostly a
problem for openSUSE which uses the OBS repositories since there is not
an upstream one for it.
Change-Id: I3641da0cf67809ca0b343a9692fd5c7592fadb78

The upstream project provides a rabbitmq package for SUSE so we default to
rabbitmq_install_method == 'file' like the rest of the distributions.
This also updates the rabbitmq test version to 3.6.9.
Change-Id: Iaf6918262e10b7b9129a86fa10090e34602b8f6f

Add support for the openSUSE Leap distributions. On openSUSE we also
need to provide our own rabbitmq-server.service systemd file which
doesn't depend on epmd.socket and epmd.service unit files.
Change-Id: I7907da612654d6593cfa40470eaec0be633ba4e9

Make the old and new versions for RabbitMQ configurable to allow
distributions to test different upgrade paths.
Change-Id: I32f9348405f3b6bedd0529bf027536af56d024d7

When executing the tests repo clone in OpenStack-CI,
use zuul-cloner instead of git to enable cross-repo
testing. This ensures that if a dependent patch from
the tests repo is noted using 'Depends-On: <change-id>'
in the commit message, that patch will be included.
Depends-On: Idce7abebf32f24c356a27e099fbca954d917402b
Depends-On: I5da7802d61d2ab6b03908138e3a3ed2db22e3d29
Change-Id: I469bef1a5aad2b1c31972ec8f8ac3d7c0cdf4cc7

Add 'become' and root 'user' options to the playbook instead of using
'ansible_become' through an inventory variable.
Change-Id: Ic89c3786b51d6b3416e4d3903fde4d46dbe7ade2

Change-Id: I921084fd063e2e4a10d0818b508d01e439c357a3

Change-Id: I4847382d614f582944e8711ab7b5b37740f9b3ce

This enables the CentOS functional upgrade test, which was originally
disabled due to a bug in Ansible 2.1.
Change-Id: I748889a1c74acf4f8d67fa9e5b250816d209611b

This commit does the following three things:
1. Updates test-install-rabbitmq-server.yml to use the correct
 rabbitmq host group
2. Copies test-upgrade.yml to test.yml, which should test both
 installation and upgrade of RabbitMQ
3. Updates test.yml to only perform upgrade testing when the package
 manager is != 'yum', this is due to [1]
4. Deletes test-upgrade.yml
[1] https://github.com/ansible/ansible-modules-core/pull/4547
Change-Id: I8f6c1b8d0de4666ab5f892162caa66f5c5c966d9

RabbitMQ nodes must all be stopped prior to a major/minor version
upgrade [1]. The role does this by distinguishing between the upgrader
node and the rest in separate stop and start tasks.
Upgrades can fail when more than one member of rabbitmq_all are not
members of the cluster. This is due to a bug fixed for greenfield
deployments by 5dc67955f0. The same fix
was not applied to upgrades because major/minor upgrades require all
RabbitMQ nodes to be stopped which is incompatible with serialising the
role in isolation.
This change uses a play to stop all but one of the nodes, prior to
running the rabbitmq_server role, and then serialises the running of
the role so that one node is upgraded at a time. This minimises the
downtime as much as possible while allowing the role to be applied to
one node at a time.
[1] http://www.rabbitmq.com/clustering.html#upgrading
Change-Id: Icca5cb1a96f83063223b6ddbeb02eeb562b0931b

For any openstack git sources, git.openstack.org must be used.
Change-Id: I98214b7da0644292508da84f459a65b3f1ea67fb

This patch consumes the test scripts implemented by
https://review.openstack.org/375061 to ensure that
the tests and test preparation is consistent and
more maintainable.
Change-Id: Iea6a5793509df959e877a8974e3322a0b92ba360

It is now included in test-setup-host
Change-Id: I2e205d5a839761efb70a395d6a38f37dcdaae6b7

This commit also moves the upgrade test to its own
scenario using tox environment func_upgrade
Closes-Bug: #1623592
Change-Id: Ib79d9c261c3bf5e018806867ebe804b5853908e9
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Previously the role assumed that all deployers wanted
all queues to be HA. Performance testing by Mirantis and
Intel has suggested that throughput increases can be dramatic
when disabling HA queues, see:
 http://docs.openstack.org/developer/performance-docs/test_results/mq/rabbitmq/index.html
Therefore, we should allow deployers to set their own
policies when desired.
Change-Id: I156b8618e78d74823ec568053157afc853baad84
Closes-Bug: #1607830 

The ``rabbitmq_server`` role now supports configuring HiPE compilation
of the RabbitMQ server Erlang code. This configuration option may improve
server performance for some workloads and hardware. Deployers can
override the ``rabbitmq_hipe_compile`` variable, setting a value of
``True`` if they wish to enable this feature.
Change-Id: I433d94eff00ac82a9069f9092faa87d449190442

Change the default install version of RabbitMQ Server to 3.6.5.
Change-Id: I35768c51e133dbf4f680a10d79b289beaaca71f9

The default value of "rabbitmq_all" remains, but this
change should allow deployers to target the role to multiple
groups now.
Change-Id: Idc1bc44b005f09f9dc7eb9778719a8ba92756c06
Implements: blueprint multi-rabbitmq-clusters

Change the default install version of RabbitMQ Server to 3.6.3.
The 3.6.3 Debian package adds support for systemd and no longer leaves a
rabbitmq-server process running for signal handling to the erlang VM.
The `pgrep` command checking that rabbitmq is running has been updated
to search entire process names to account for this change.
Change-Id: I742b03f64cd0232835b7143507e3d3cd14d79f23

This patch cleans up the container cache preparation in the tests
due to the following patches:
- https://review.openstack.org/307856 (new image build process)
- https://review.openstack.org/315114 (new apt config process)
- https://review.openstack.org/322188 (new resolver config process)
Change-Id: I6732c158197a686c36a02075595111e00183d934