openstack/puppet-openstack-integration - puppet-openstack-integration - OpenDev: Free Software Needs Free Tools

This splits the existing function to repository set up and actual
package installation and adds an additional knob to skip setting up
puppetlabs repo (and install puppet from distributions).
Change-Id: I3568883537e8cc29f234754c78fc2425ad6a8f4b

The bundle install commands supports the --minimal-deps option instead
of the --minimal-dep option. (The last 's' is missing)
This fixes the wrong option used.
Change-Id: Ic72c97f6230d6b730b2688f0f87bb5de8b35f4f1

In the past we had to workaround some issues related to r10k
installation and dependencies. However, apparently some of those issues
have been fixed and we don't need to pin to specific gem versions
anymore.
Change-Id: I16436d23a7853afe029920b3ea2f61058c133010

Change-Id: Ie9efc39070db608185a358f817f1074fd4f5810f

The --no-ri and --no-rdoc option is removed in later versions.
Change-Id: I199e980fe13d4932e7d3f9f484ea3dc25e58f01c

The default behavior of `gem install` with regard to the location of the
installed gems and binaries is different in CentOS7, CentOS8 and Ubuntu
so let's make options explicit to get the desired behavior in all cases
and configurations, what means getting gems installed under GEM_HOME to
avoid depending on system wise PATH declarations and installed gems.
Change-Id: I992a930491f65ba9871c3f94806afce54bef9c7b

gettext-3.3.0 needs Ruby >= 2.5.0, which is not
available in CentOS7, so need to pin it.
Needed-By: https://review.opendev.org/#/c/701557/
Change-Id: Id3cf104767cd4d2f1662b2c4ca63cd7feee41bff

Change-Id: I4875b8bc8b2333046fc3a08b4669774fd26c89cb

It broke all unit tests jobs:
http://logs.openstack.org/98/535698/16/gate/puppet-openstack-unit-4.8-centos-7/bd7dedf/job-output.txt.gz#_2018年02月05日_23_19_05_369251
This reverts commit 9f14f75ec6.
Change-Id: Ibcb2bb60a668d232371e4d87febad6d67b3302d8

The PUPPETFILE and PUPPETFILE_DIR environment
variables are deprecated since several releases
and are superseeded by the --puppetfile and
--moduledir arguments. [0] [1]
This patch replaces all those occurences to future
proof it when the env vars are removed.
[0] http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/job-output.txt.gz#_2017年12月21日_22_52_53_499924
[1] https://github.com/puppetlabs/r10k/pull/479
Change-Id: I216fe01a13e46c90bbb1455df72b25daf899c9f0

verbose option will help us to see why gem install can fail -- and also
make sure we're deploying from the mirrors.
Change-Id: Ia2b4955468ac1189f6f81798c684649390621155

Version 2.2.7 depends on semantic_puppet ~> 1.0 which
is incompatible with latest r10k which depends on
semantic_puppet ~> 0.1.0
Change-Id: I7055c17358a9b72d7b665bf5eec36eb1642922db

Currently, p-o-i allways tries to install puppet package
from puppetlabs repo. In some cases, as RDO-CI we use p-o-i
to test puppet modules from RDO distro and it's recommended
to test it using puppet version provided in RDO repos which
uses different paths.
This patch provides a way to override this behavior by using
env variables as:
MANAGE_REPOS=false
PUPPET_PKG=puppet
PUPPETFILE_DIR=/etc/puppet/modules
Change-Id: I680504ab65d780deba53387e9b6604d3102de47b

This is not longer required as git-review is not installed in last image in nodepool nodes [1] and can break p-o-i in other scenarios
[1] http://logs.openstack.org/88/459488/9/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/b95b566/console.html#_2017年05月29日_19_34_35_426653
This reverts commit f02d8c3d4d.
Change-Id: I267638d38d32e688ad33052e3f476c286be6fa45

This patch is a workaround to bring back integration & beaker jobs.
The real fix is here but not merged: https://review.openstack.org/#/c/468823/
Change-Id: I4745bf5e02013095f727ce70a4fed1fcb2e673b7

Puppet 3 was EOL as of December 31, 2016. Since we don't use it in the
gate for Pike, let's drop the configuration bits in p-o-i for it. This
change leaves the PUPPET_MAJ_VERSION support on the off chance we need
it in the future but basically 4 is currently the only supported
version.
Change-Id: I0f04a3cb194ec2225010fb4e062703195b79baf1

In newer versions of the puppetlabs provided puppet-agent, some of the
tooling has been moved to /opt/puppetlabs/puppet/bin. We need to include
this so that things like gem contianue to work.
Change-Id: I4296ed611830af581d64777377c87a51230241dc

beaker-puppet_install_helper 0.6.0 has made the PUPPET_INSTALL_TYPE now
default to agent which was the previous behavior we had when running
under puppet 4. Since puppet 3 is now officially EOL, this switches the
defaut to assume puppet agent packaging rather than the foss version.
Change-Id: I3243f9d7bdb19840f6462c5b6b237c2314a71fbe

1) Pin fast_gettext
The fast_gettext gem 1.2.0 and greater require ruby 2.1 which we don't
have.
2) Pass a generic cpu to qemu instead of trying to pass host-model
We don't need to pass any CPU extensions in the gate, given the
context where we don't use KVM or (proper) nested virtualization.
Truthfully, this also becomes required as there are upcoming changes
in qemu-kvm 2.6.0 which breaks us when trying to pass extensions
while using qemu.
Worth noting that devstack has been setting 'none' [1] as well.
[1]: 1c442eebc8/lib/nova_plugins/hypervisor-libvirt (L42)
Change-Id: I881bd712266ae63b0ac9074ec18bd6a41b946c3a

Second run of run_tests.sh script will fail with error:
ln: failed to create symbolic link '/usr/sbin/puppet': File exists
Force symlink creation fixes this issue.
Change-Id: I82622859b096e07c374b532b1fa69b5a335a6ef4

This makes gem binary usage consistent (i.e, r10k) and prevents
encoutering a command not found error.
Change-Id: I3ef01274417cd88403d71af0e6216850d77cc81c
Closes-Bug: #1592113 

When running Puppet4, the bin not in $PATH so beaker jobs fail to run
Puppet. Also we use SUDO in OpenStack Infra to run Beaker so we can use
PATH.
This patch is a workaround to make it possible.
Change-Id: Ie25a414f86d9d4b9d60e28d2680a13a503abad8b

PUPPET_VERSION is already used by puppetlabs/beaker and we don't want to
overlap with this variable. So let's rename it to PUPPET_MAJ_VERSION to
avoid strange behaviors when running Beaker jobs.
Change-Id: I74be2eec2be53f59af196f37ee348ff6856af8bc

In order to help be able to determine what part of the script a failure
occurs in, this change adds a print header function and updates the
shell scripts to print some header information for various actions.
Change-Id: I8e622c4d1c9c03aa3dbcb27712a346ce3284003f

This commit just exports PATH when testing against Puppet 4.
 Tests pass without this in some environments but is dependent on sudo
 configurations, to get tests passing in OpenStack Infra, this is
 required.
Change-Id: I04b486fb398aaad1a3333477995ec00a76b9da0b

Those small typos and non-idiom can lead to unexpected behavior, better
fix them now.
Change-Id: I218ba0f3bba832d0929fb52c21f7e66774da494e
Closes-Bug: #1561539 

We need to export PUPPET_BASE_PATH otherwise beaker & unit CI jobs will
fail to copy modules in the right path:
mv /modules/openstack-integration /modules/openstack_integration
Change-Id: I1c99de3f99367fb7677977fee4c5cf08afda5977

This commit puts in place the ability to run
 puppet-openstack-integation against Pupept 4 by making release files
 and paths dynamic based off an environment variable, PUPPET_VERSION.
 Configurability is required because of the way Puppet 4 is installed,
 as an all-in-one package known as "PC1" or "puppet-agent" which
 includes Ruby, Facter 3, and other dependencies.
Change-Id: Id3af52f4938378a5a620765321cecd547e079660

readlink on OSX does not support the -f option. As an alternative, we
can use pwd -P to resolve the path.
Change-Id: Ibaeadea9685f7a9603198ed1a509176cf70f9b21
Closes-Bug: #1506504 

* Move out some code from install_modules.sh to have a functions script.
* Update install_modules.sh script to use functions.
Change-Id: Ie87b8eac71d5edb555e9a1637bea84bc9e268497

* Stop setting SElinux as permissive and bring it to 'enforced' again,
 since our blocker is fixed [1].
* Run `apt-get update` only in run_tests.sh script. It's the only place
 we actually need to run it since beaker jobs already manage that task.
* in run_tests.sh, use $SUDO instead of sudo, to be consistent.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1249685
Change-Id: Ic3101a0a080b798b334cd2f212ec726f243880b5

This reverts commit 9da87065f5.
Packaging [1] is fixed now.
[1] http://www.rabbitmq.com/releases/rabbitmq-server/v3.5.5/
Change-Id: Ibb3b108432b8bcf76a54bf8dd6f7f61616d8ced1

RabbitMQ 3.5.5 has a bug [1] that prevent puppetlabs-rabbitmq module to
create permissions:
Could not evaluate: cannot parse line from
list_user_permissions:/usr/sbin/rabbitmqctl: 19: [: Linux: unexpected
operator]
This patch is a workaround to make sure we install the previous stable
version of RabbitMQ (3.5.4). Since puppetlabs-rabbitmq module will only
check if the package is 'installed', it won't try to update it.
[1] https://github.com/rabbitmq/rabbitmq-server/issues/321
Change-Id: I8e0fa8e62e93b62317011fd7c8a2730cd293d4c7

Allow to change PUPPETFILE_DIR but keep '/etc/puppet/modules' by
default.
Change-Id: Ie32d18d477c271b2724290edbbfd4ae7ea110480

Recently we decided to ensure SELinux was in permissive mode to avoid
package issues. If a user has its node already with SELinux disabld the
call to setenforce 0 would fail with the following issue :
usr/sbin/setenforce: SELinux is disabled
This commit fixes that
Change-Id: Ie42f62325d0f3e863feb7e412e073341e0f04a36

python-cffi is preventing Nova & Keystone to run when SElinux is
enforced.
The bug is reported [1] and RDO team is working on it.
In the meantime, we need to disable SElinux so we can run our CI again.
This patch might be reverted once a fix is released in packaging.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1249685
Change-Id: I4720ec701cb36f56a9556a124204db32fa8e8434

Since our recent discussion about Ubuntu packaging, we decided to bump
Ubuntu packaging to trusty-proposed/liberty so we can install Liberty in
a better shape than current situation (Staging repo).
This patch installs UCA liberty repository.
It also makes sure `apt-get update` ran before so we are able to install
ubuntu-cloud-keyring package in the gate.
Change-Id: I57e2ddc5d0e228142f8045623ffbde362c0e7f7b

Note: this is a temporary fix and will be reverted once packaging is
fixed upstream.
python-pymysql is a new dependency in Liberty and current OpenStack
packages do not depend on it right now. It's work in progress by
tu
team.
Change-Id: I5d997e27abce98c6edc551350cec27c4611e736f

This reverts commit 8b9db6da86 and applies
some changes to not break Puppet beaker jobs.
* First init of run_tests.sh script that will just run
 install_modules.sh script with sudo. The script also makes sure GEM_HOME is set
 so we can install r10k from gems and run it later.
* install_modules: make sure to find r10k binary
Change-Id: I24c7494e7c9201915d9aa2bf314a9534ddd27dd5

Make sure there is no puppet modules pre-installed in the image where we
run integration testing by cleaning up the /etc/puppet/modules/*
directory.
Change-Id: I489717356b75b95c1a881b6f3e554cfec66ee6f1

This reverts commit 2921e36c79.
Change-Id: I88055528d143325457cdc2461dedb37f703374b9

This is a temp fix: currently Puppet CI is broken because it can't find
the path of the script. This patch aims to fix it but it will be changed
again later, we need to change project-config repo.
Change-Id: I1bf3ea6a26be9b79794e048aac125e80a1d858a6

When launching run_tests.sh we need to make sure to call
install_modules.sh.
I've moved it into the tools directory, and installed the modules
local vs global. This is mostly personal preference but avoids puppet
modules conflicts with global /etc/puppet folder.
Change-Id: I77c445a71720e0c8dd4744777c2bc6a0a6663590
Depends-On: I87c12faffab953ad2039d2b9ed9ab420e97ab2b7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

Change-Id: I48a26531ee996585f11fbce294baa722fe048692
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

This repo will be cloned by the puppet modules and used to install
modules in functional tests. The script uses r10k to deploy modules
from the Puppetfile because this is the Puppet Labs recommended way to
manage puppet environments, and it is a popular method among our users.
We install all modules from git, not the forge, because r10k does not
give us dependency resolution so installing from the forge does not buy
us anything. If the modules have stable branches in git, r10k uses the
stable branch as a ref, otherwise it uses a tag (pinning to a specific
release).
If zuul-cloner is available, the script first installs the external
modules with r10k, then extracts the OpenStack modules from the
Puppetfile to install with zuul-cloner. In the future, this part may be
split out into a JJB job.
Note: because r10k purges the modules directory before installing
modules, and this feature cannot be turned off[1], we must run r10k to
install the external modules before running zuul-cloner to install the
OpenStack modules.
[1] https://github.com/puppetlabs/r10k/issues/172
Change-Id: Ide59ef4bdf53a04957e3593815b7514a0e744e0e