openstack/puppet-openstack-integration - puppet-openstack-integration - OpenDev: Free Software Needs Free Tools

I98fc088cd87c1412544f9590ce7a925b413297e2 broke unit 5.5 for
puppet-tripleo and probably puppet-pacemaker, since it can't find the
Puppetfile by default anymore.
Change-Id: I43718a849d09653e6c352b581aef6bbbf372fc41

- scenario001 and 004 will be running without ceph
 until ceph repos get available for CentOS8
- scenario003 will run not run with linuxbridge until next CentOS
 minor version as it has issues:-
 - https://bugzilla.redhat.com/show_bug.cgi?id=1720637
- Add Puppetfile_centos7 to have different puppet modules
 as compared to other distros. Currently only puppet-mysql
 needs to be kept pinned for centos7 as newere mariadb version
 is required to remove the pin, CentOS8 and ubuntu have required
 mariadb version so pin is removed. So automatic updates of puppet
 modules will not be applied for CentOS7 as those jobs are going to
 be removed in coming months.
- check for rdo_dlrn url existence only for RedHat distros.
Change-Id: I98fc088cd87c1412544f9590ce7a925b413297e2

Depends-On: https://review.opendev.org/#/c/667439/
Change-Id: I6e11c12ad73c89466aefbb926c111ed1d00784b8

Change-Id: I467e86b132314d6bf5331ff44a254d38a84dfab0

Adds support to set the jobs to use Ceph
Nautilus when testing.
Change-Id: Ibf2b82435a0f9c29d363c67a973399228bfb90a3

This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: I34c2be045d56d99792fcc1533b67c2c8de1aedac

Use hiera version 5 instead of version 3 which is
deprecated and outputs a warning.
Change-Id: I26ee3329aa75109625476d6c355e82fed2c050e6

Disable SSL as currently there are known issues in services
to work with python3 + SSL on Fedora.
Also fixed following:-
- watcher to work without SSL
- Handle source /etc/bashrc for users don't have .bashrc created
- Install python3-rbd in Fedora and RedHat > 7
Change-Id: I5b67f40ec3c687f5282c65e10b4a1ee1fe5528f3

In puppet 6 they have moved out a lot of builtin
resources to their owns modules since they didn't
belong in the Puppet core however in deb/rpm packaging
they are still bundled in but not when installing from
gem.
So we must add them in here so that they are available
in our unit tests.
Change-Id: I7060191395eff9c1e3146cb10594a38bef113117

Add puppet6 testing and run all those on Ubuntu Bionic.
Change from Xenial to Bionic for all other default jobs.
Run libraries unit for 6.0 on Ubuntu Bionic.
Change-Id: I4d9f1d1b62acece9bc0b4ce7bd49d2093af8e5ee

Don't fail on selalert notices with scenario-py3,
Fedora jobs are failing on that, so remove the
Fedora check.
Change-Id: I46dc06d3e8e005ff0e18a10d17a75ef9e1c9e23d

Puppet 5 for Bionic is now released and we don't need
to run the nightly puppet5 builds anymore.
Change-Id: If91bd851f3f67813cc44064be01290b9bd617cbb

We now have support to Mimic and can select that version
and this patch fixes Puppet 5 for Bionic we can add it to CI.
When this is fixed the only thing left is to add beaker acceptance
testing for Bionic.
Bionic only has Puppet 5 and can currently only use Ceph Mimic.
Change-Id: I6f94e214a70f92278f1f3806471ef2f622a938c3

Also modify provision manifest to skip provisioning
some services.
The new scenario will be used for testing py3 services
starting with keystone and glance.
Also update puppet-apache reference to the required commit,
this can be updated once a new tag is released with the commit.
Change-Id: I55558f35068457175e2ef508f9e5989151db4aa2

Change-Id: Ie4bd3c46882a1cd4ec2be8c6bb4850e17fe79905

We are closing in on adding Ceph Mimic support and
want to ensure we can select either Luminous or Mimic
for running our tests.
For example Ubuntu 18.04 must run Ceph Mimic because
Luminous is not available (yet), Debian doesn't have
Ceph Mimic packages so we must run Luminous there.
Depends-On: https://review.openstack.org/#/c/576247/
Change-Id: If84ecbf86f3ebde8e69a1a26759ad967d061725a

Change-Id: I409c11f31d4b82ef9a3d6ba20e2dd045156d66a9

This change makes sure the latest puppet5 version
is used when running integration jobs for puppet5.
The puppetlabs yum mirror is ready to be used and
has the proper packages and is mirrored, example:
* http://mirror.mtl01.inap.openstack.org/yum-puppetlabs/puppet5/
For the puppetlabs apt mirror the puppet5 pool is
needed before this will work. A patch has been fixed
for that here [1]. Therefore this change cannot be
merged until [1] is merged.
[1] https://review.openstack.org/#/c/565867/
Change-Id: I1a2716fdd4b960b12ff24303038bec5ea8fb80a3

Change-Id: Iab59f0e818f8c5be1beba200f6c475669ce6ba0b

At first, I thought this would go in puppet-ceph, but in fact,
every compute node needs a librados dependency which is in the
Ceph repo. So at then end, all nodes need GPG-KEY-ceph, so it
is a good idea to install the repository key there.
Change-Id: Ia33718a8350a60c69ce9632eddb6156f9b43c745

Nodepool images no longer have Puppet in them anymore so there
is no point in removing files and packages which are not there
anymore in the first place.
Change-Id: Idb89512251ca9125fcb3f44768873d2cb76eb879

It appears that the puppetlabs Debian repository is set to always
install a repository for Xenial (instead of using lsb_release to
find the OS), and that the Puppetlabs repo GPG key is outdated. This
patch fixes that.
Change-Id: I35231cdf7503d129bb408bc5d29a2cfe95c4e08e

Error[1] when executing grep command. add sudo for the grep commands
when it is currently enabled.
[1]:http://logs.openstack.org/63/554163/1/check/puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/dd454e9/job-output.txt.gz#_2018年03月19日_08_44_16_398090
Change-Id: I5e5e89a4d691fc3890fbc863caa1fc87e42e7d87
Closes-Bug: #1756832 

It broke all unit tests jobs:
http://logs.openstack.org/98/535698/16/gate/puppet-openstack-unit-4.8-centos-7/bd7dedf/job-output.txt.gz#_2018年02月05日_23_19_05_369251
This reverts commit 9f14f75ec6.
Change-Id: Ibcb2bb60a668d232371e4d87febad6d67b3302d8

Change-Id: Ia832c1f8dd27bd1ec919f9156df38a67b4d51cc1

The PUPPETFILE and PUPPETFILE_DIR environment
variables are deprecated since several releases
and are superseeded by the --puppetfile and
--moduledir arguments. [0] [1]
This patch replaces all those occurences to future
proof it when the env vars are removed.
[0] http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/job-output.txt.gz#_2017年12月21日_22_52_53_499924
[1] https://github.com/puppetlabs/r10k/pull/479
Change-Id: I216fe01a13e46c90bbb1455df72b25daf899c9f0

This patch moves in the Puppet OpenStack integration
jobs as well as the module tests in-repo.
Depends-On: Ib96ee354cbf9c88f9265f5743a97b2d013addc70
Change-Id: Idd846c569a6e067125e130f189be9ae8ffba1646

The command "r10k puppetfile install -v" fails with:
Unrecognized options: verbose.
Rework the command to be compatible with the r10k 1.1.4.
Closes-bug: #1707629
Change-Id: I05d888125284d2808fb248e52afc95ab2b948353
Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>

We can remove workarounds, we don't support Mitaka & Hammer anymore
(EOL).
Change-Id: I9e4bee87bac23a2d28937249620cb9c8208f4368

Allow to deploy Puppet 5 (experimental) when setting
PUPPET_MAJ_VERSION=5 (currently done by experimental jobs).
It will pull packages from nightly builds, so we can use it and see if
things are working.
Change-Id: I3d0f2f4309c425435370d7f7d4d367c62396c8fc

Currently, p-o-i allways tries to install puppet package
from puppetlabs repo. In some cases, as RDO-CI we use p-o-i
to test puppet modules from RDO distro and it's recommended
to test it using puppet version provided in RDO repos which
uses different paths.
This patch provides a way to override this behavior by using
env variables as:
MANAGE_REPOS=false
PUPPET_PKG=puppet
PUPPETFILE_DIR=/etc/puppet/modules
Change-Id: I680504ab65d780deba53387e9b6604d3102de47b

Puppet 3 was EOL as of December 31, 2016. Since we don't use it in the
gate for Pike, let's drop the configuration bits in p-o-i for it. This
change leaves the PUPPET_MAJ_VERSION support on the off chance we need
it in the future but basically 4 is currently the only supported
version.
Change-Id: I0f04a3cb194ec2225010fb4e062703195b79baf1

I think warning messages doesn't add much value in console
and sometimes we miss messages with (err).
Change-Id: Ic06ad59f71393dcc2f0f861e45d367fb835a14fc

If you run something like all-in-one.sh as a non-root user, it will fail
when it goes to move the puppet log file. We need to run it as sudo if
the user is not root.
Change-Id: I8b9d7e587f31a82fb4b59de770882139fd168667

To improve troubleshooting and readability, let's not have the puppet
logs print to the console. This change updates the puppet apply to
output the logs to puppet.log which will still be captured and provided
in the CI results. When the puppet apply fails, we will grep for
warnings and errors from the puppet log and only print those out which
should simplify troubleshooting failures.
Change-Id: I1084223f805f1960aebb9c5868fdef84ff3667e4

The use of -r for cp command is redundant with -a being used as
documentation on Linux has -R and -r as the same parameter. The
issue on Mac OS X is that it will error on about using -R and -r
with cp. Change removes redundant -r from cp command.
Change-Id: If7122efc8b84f8aa8a07fe4f8ee7568a048c9461

puppet-openstack-integration currently takes care of installing itself.
This is done through the Puppetfile which references the p-o-i repository.
This works with zuul-cloner because Zuul with ensure the current reference
is cloned.
However, when running outside of the gate, this means that p-o-i will
completely disregard any local code and instead install from the git source
specified in the Puppetfile.
Change-Id: I665347dbede2c53bd5537d4a29d07bfe679c50e5

Puppetlabs release package was fixed so we can unpin it in our CI.
Change-Id: Ie18ff211b919b8ae4bc5a6d28ab8afe948991ed4

1) Add 2GB swap by default
Add some swap when running CI jobs, so we can reduce the number of
failures in CI related to missing memory.
By default 2GB, can be disabled or increased/decreased with parameters.
2) Promote RDO repository to have Oslo DB from master
python-oslo-db 4.13.x releases are known to break Gnocchi.
https://review.openstack.org/367221 will fix it, but it is not yet
in any release, so RDO unpinned oslo.db until that happens.
3) Disable Ironic testing on Ubuntu. Packages are broken in recent
 Newton upgrade. They are working on it.
4) Enable br_netfilter kernel module on Ubuntu.
See https://bugs.launchpad.net/cloud-archive/+bug/1621651
Even if it's not critical, it's a nice-to-have because it removed the
ERROR that we had in neutron logs. We'll see how the bug report evolve
and maybe remove this workaround.
5) Export lsmod in logs, easier for us to debug later.
6) Disable linuxbridge on scenario003 for Ubuntu
Also see https://bugs.launchpad.net/cloud-archive/+bug/1621651
7) Pin Puppetlabs release package.
Context: https://tickets.puppetlabs.com/browse/CPR-378
Change-Id: I732ef375a681f03af751ebb6ebd5b12df55a0ecc

When submitting a patch in puppet-ceph stable/hammer, we want to clone
puppet-openstack-integration from stable/mitaka.
This patch makes sure of that.
Change-Id: I7a7190f0c9e7257f74672d8a8d63678706c63243

Currently, puppet-ceph stable/hammer CI tries to clone Puppet OpenStack
modules using stable/hammer branch but they don't have it, only
puppet-ceph does.
We need to override ZUUL_BRANCH with a fake 'stable/mitaka' for Puppet
OpenStack modules and then checkout 'stable/hammer' for puppet-ceph.
This workaround will allow us to:
* test puppet-ceph/hammer with mitaka modules
* test puppet-ceph/master with newton and beyond modules
Note: for our r10k users, no problem, they use git to checkout the right
branch, so they don't have the bug.
Change-Id: I004ee95e6bd97992aeebd08447396e3b8a29e09f

This change adds hiera support into the scenarios so we can override
items via the common.yaml heira file if need be. This change adds
support for two environment variables with run_tests.sh.
HIERA_CONFIG - full path to the hiera yaml configuration file
MANAGE_HIERA - used to enable/disable writing out a hiera.yaml config
file
Change-Id: Iad0050e09e0cc610e0eadc00e328151e88eee10e

This makes gem binary usage consistent (i.e, r10k) and prevents
encoutering a command not found error.
Change-Id: I3ef01274417cd88403d71af0e6216850d77cc81c
Closes-Bug: #1592113 

Change-Id: I78324c8787b129e91f1ac32694085bde5b6043fe

Change-Id: Iefed86a09eaf4e571d38c801f79e9fc989bc57d6

When running periodic jobs, ZUUL_BRANCH & ZUUL_REF are empty because we
don't have any reference or branch set for the jobs.
To not make zuul failing when cloning Puppet modules, let's set defaults
values:
ZUUL_BRANCH default to master (we are running tests against master now)
If we need to test stable branches, we'll revisit later but it would be
a bigger change (job names, etc).
ZUUL_REF default to None.
Change-Id: I34b298230bc174a3fdd431095043543c4c223296

In order to help be able to determine what part of the script a failure
occurs in, this change adds a print header function and updates the
shell scripts to print some header information for various actions.
Change-Id: I8e622c4d1c9c03aa3dbcb27712a346ce3284003f

When testing rspec locally the r10k binary is never found because the
GEM_BIN_DIR is wrongly set to GEM_HOME/bin. GEM_HOME is set by bundler
and GEM_HOME/bin do not point where bundler put the binary. The
binaries are handled by bundler by changing the PATH. rbenv, use
similar tricks.
install_all is only used outside of the CI, so this modification
wouldn't change anything as far as the official CI is concerned.
Change-Id: I07668d64f9adf6713f5c267cf45650ba92ee0b9c
Closes-Bug: 1550331

Change-Id: Ibf7d3aa962e2e0727f0593f70186521602c48ca6