openstack/openstack-ansible-haproxy_server - openstack-ansible-haproxy_server - OpenDev: Free Software Needs Free Tools

This change enables log collection within the gate so that further analysis
on gate tasks can be performed post build. This is very useful when
debugging problems.
Change-Id: I76f771e8ca3a434fc3fd710a9ff035cb2bfad182
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

In order to make it easier to differentiate between the lists of
python packages, distribution packages, downloaded packages,
package pins and other similar variables the variable names are
being changed to ensure that they have a more explicit suffix
that defines the purpose and makes the naming more consistent.
This is to facilitate a lookup plugin which will be able to look
up all the package lists and present them as a consolidated piece
of data which may be used for artifact preparation.
Change-Id: Id9a356f78162a77edc27209be215f04380a631dc

google storage url changed so this URL needs to be updated.
Change-Id: I99b577badc8db3ce0cb6f683c233ef6fee18022e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Change-Id: Ie67f6b3c300522da2a1b6bd06800b0bdbc6ccf58
Implements: blueprint osa-install-guide-overhaul

Allow endpoints to be disabled in the haproxy_default_services
dict.
This is needed specifically for https://review.openstack.org/#/c/340175/
which requires that the nova-metadata-api service move to the
compute nodes when the networking-calico neutron plugin is used.
In such an environment, the metadata nodes serve requests only for
the local hypervisor and LB endpoints for the metadata service
are not needed.
The corresponding flag disabling haproxy_service_enabled will be
added to the Calico change.
Change-Id: I584fe2647ba0d6a70908e55168360752a08261c5

The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: I3732efabfa4fc7e80a8f172abd1415fd54489763

'with_' clauses are evaluated before 'when' clauses. In Ansible 1.9
the task is silently skipped when a variable within a 'with_' clause is
undefined, Ansible 2 provides a deprecation warning.
Separate the task deploying a user provided ssl cert and key into
two and check them individually for 'haproxy_user_ssl_cert' or
'haproxy_user_ssl_key' being defined.
Change-Id: I75367fe25d15d35ff60203b7c1d78437d613404d

Horizon should listen on port 80 and does not need a scheme
redirect when ssl is disabled.
We should always configure the HAProxy PPA and install the latest
version regardless of whether or not SSL is enabled or disabled.
Change-Id: I7a7d574251fe1139da0826d64958529fdf2f5b4c
Closes-Bug: #1596548 

When executing the bootstrap-host role with Ansible 2.1, the following
deprecation warning is issued in the output for some tasks.
[DEPRECATION WARNING]: Using bare variables is deprecated.
This patch addresses the tasks to fix the behaviour appropriately.
Change-Id: I5e5dba540256935ffb03ace27c6c43abc15f52da

This change implements CentOS7 and Ubuntu 16.04 support for the HAProxy
role. Because RHEL does not package HATop the installation of HATop has
been moved to a source installation so that it can be used universally.
Implements: blueprint multi-platform-host
Change-Id: Ib4f33185202b694b9611cc5fd6323c30a1c8d489
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Use the previous tasks changed state to create haproxy log
directories. Both use the same "log_dir.rc != 0" logic.
This allows the play to run in check mode.
Change-Id: I731ea1ec7822266cd9a433b328e5dc5cdd0a205a

This change makes it so that all services are expecting SSL termination
at the load balancer by default. This is more indicative of how a real
world deployment will be setup and is being added such that we can test
a more production like deployment system by default.
The AIO will now terminate SSL in HAProxy using a self-signed cert.
Depends-On: I63cfecd6793ba2b28c294d939c9b1c466940cbd1
Depends-On: Iba63636d733fa1eb095564b8bf33a8159d9c2a00
Depends-On: Ib31a48dd480ecb376a6a8c5b35b09dfa5d2e58f6
Depends-On: Ibdeb8b981ca770ce4f56beeae05afd3379964859
Change-Id: Id87fab39c929e0860abbc3755ad386aa6893b151
Co-Authored-By: Logan V <logan2211@gmail.com>
Signed-off-by: Logan V <logan2211@gmail.com>
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Use user and group configuration option, uid and gid need a number
instead of a name. Specifying a name for uid/gid attributes will make
haproxy run under the root user.
Change-Id: I1d6db661b7d0958e1ba888770e3ce789d7cb4a76

Workarounding the upstream ansible apt module bug
documented here:
https://github.com/ansible/ansible-modules-core/pull/1517
For the next versions of ansible we'll be using, we should
check if the apt bug is fixed. When it's fixed, we could
abandon this change and use the standard apt module
with correct cache handling.
Change-Id: I2aaf00da175f31d0157bbc4ae30a4e176b055078

The haproxy check script that is installed with keepalived expects to use
'killall', however this package is not installed in the container templates
by default and therefore the haproxy role must install it in order for
keepalived to leave FAULT state.
Change-Id: I8048aaa16b163acfe3da6863aef26adbe18bd73e

Instead, reference the 'haproxy' service group. This allows the plays
to be container agnostic and move the haproxy deployment closer to
being container-compatible.
Change-Id: Ibfe0e3a0cb742c8d5c3943922da0a30a56016266

Allow deployers to specificy a remote URL to download apt signing
keys from.
Example:
galera_client_gpg_keys:
 - key_name: 'mariadb'
 url: "https://some.webserver.com/mariadb.gpg"
 fallback_url: "https://other.webserver.com/mariadb.gpg"
 hash_id: '0xcbcb082a1bb943db'
 - key_name: 'percona-xtrabackup'
 keyserver: 'hkp://keyserver.ubuntu.com:80'
 fallback_keyserver: 'hkp://p80.pool.sks-keyservers.net:80'
 hash_id: '0x1c4cbdcdcd2efd2a'
Change-Id: I781cb8f5744c4e1e8e728a8ad308d135d2e5922c

This change allows deployers to specify locally sourced apt keys
on the ansible host rather than requiring all apt keys to be
downloaded from upstream keyservers.
The current implementation requires that all containers and hosts
we deploy to have an internet connection to download apt keys for
the various repos. This change allows the user to source apt keys
from the deployment host like for example:
ceph_gpg_keys:
 - key_name: 'ceph'
 data: "{{ lookup('file', '/etc/openstack_deploy/keys/ceph.gpg') }}"
 hash_id: '0xe84ac2c0460f3994'
Note: Deployers can already set the repo URLs to use local sources,
so this enables fully offline package installation.
Change-Id: I1607c7a5c9bb4d5e06dedbc76c84a77014305df2

The change adds logging for haproxy on localhost through the use
of rsyslog which is now a dependency. The logs will be stored in
/var/log/haproxy which will later be indexed and shipped to the
logging server. The change makes it possible to debug issues with
haproxy using specific log files instead of having to go digging
through syslog.
Change-Id: Id942ce159ea45703259f7aff0e5a85780a83370b
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Change-Id: I31486f7f8de5a410de2847ee1ecfc44ac75bce28

Change-Id: If1717f539167d4a6b67e14616bd1de7fbd31dd46
Closes-Bug: #1499097 

For increased security against possible SSL attacks, configure HAProxy
SSL frontends with a cipher suite. Default to the existing
ssl_cipher_suite variable defined in user_variables.
Change-Id: Ida64765bb4ebec0bbfa118e2eeedfb36ad2bd3f8
Closes-Bug: #1498726 

This introduces the variable haproxy_ssl_dh_param
It sets the maximum size of the DH parameters used
for generating key in DHE key exchange.
Higher values increase CPU load but is more secure.
This value is ignored if static DH params are given
in cert file.
Change-Id: Idca02a8337fa3790ddfb849d9e2e87d60076c399

pem generation should always start from closer certificate
to the top of the chain. This commit fixes that.
Change-Id: I315bf4f818cc8eb606823a48843f1931e1779223
Closes-Bug: #1493421 

This change brings similar changes as this one targeting horizon:
i.e.:
* The server key/certificate (and optionally a CA cert) are
 distributed to all haproxy containers.
* Two new variables have been implemented for a user-provided
 server key and certificate:
 - haproxy_user_ssl_cert: <path to cert on deployment host>
 - haproxy_user_ssl_key: <path to cert on deployment host>
 If either of these is not defined, then the missing cert/key
 will be self generated on each container. No distribution
 of the self generated certificates accross all the hosts
 is planned.
* A new variable has been implemented for a user-provided CA
 certificate:
 - haproxy_user_ssl_ca_cert: <path to cert on deployment host>
* The 'haproxy_cert_regen' variable has been renamed
 to 'haproxy_ssl_self_signed_regen' to have the same
 naming convention as horizon.
* A change of certificates, whether user dropped
 or role generated, triggers pem generation and server restart
DocImpact
Closes-Bug: #1487380
Change-Id: I0c88d197d8ede820ac4e0388e67a2da06b003c2b

This bug is triggered when haproxy is deployed on multiple hosts
and external_lb_vip is different than the internal one.
As all host receive the same configuration, and are expected
to restart the haproxy service more than once
(once during role and once post_tasks), the playbook will fail,
because the restart of the service fails. The restart of the
service fails on some hosts because haproxy tries to
start/bind to an ip the host doesn't have (avoiding ip conflicts)
This allows haproxy to bind on non_local addresses by
addng a sysctl change in the playbook: net.ipv4.ip_nonlocal_bind = 1
The sysctl is changed for the containers/systems
when external_lb_vip is different than internal address and
the number of haproxy hosts is more than one thanks to a group_var.
Side-effect: other services are able to bind on non-local addresses
if the sysctl is changed.
This could be overriden by setting the variable haproxy_bind_on_non_local
in your user_* variables. If set to false, then the ip_non_local_bind
sysctl won't be changed.
Closes-Bug: #1487409
Change-Id: I41b3a5a4ba2d48192b505e3720456a77484aa92b

This patch enables the HAProxy webstats for all the configures
backends and frontends.
A password entry is added to user_secrets.yml for the webstats
password.
It also adds variables for port number, username and password
which can be overridden in user_variables.yml appropriately.
Change-Id: Iec866ad124bec6fb0b8524a966adf64e22422035
Closes-Bug: #1446432 

This makes possible to use:
`hatop -s /var/run/haproxy.stat`
with F9/F10 buttons to enable/disable backends.
Closes-Bug: #1487378
Change-Id: I6711cc7aa31701466e5a624589588b1b53e24f02

This patch adds the option to provide an SSL certificate for the
Keystone service (either self-signed or user provided) and to
configure the endpoints and Keystone service appropriately.
* A new boolean variable called 'keystone_ssl' enables/disables
 the configuration of SSL for the Keystone service.
* The server key/certificate (and optionally a CA cert) are
 distributed to all keystone containers and used for the setup
 of SSL endpoints if the appropriate protocol is set.
* The internal/public and the admin endpoints can be set to be
 served via http or https seperately via the
 'keystone_service_*_proto' variables.
* The logic to determine the appropriate load balancing
 configuration based on the Keystone endpoint protocol has
 been implemented in the haproxy vars.
* Two new variables have been implemented for a user-provided
 server key and certificate:
 - keystone_user_ssl_cert: <path to cert on deployment host>
 - keystone_user_ssl_key: <path to cert on deployment host>
 If either of these is not defined, but a Keystone endpoint
 has been configured for SSL, then the missing cert/key
 will be self generated on the first Keystone container and
 distributed to the other containers.
* A new variable has been implemented for a user-provided CA
 certificate:
 - keystone_user_ssl_ca_cert: <path to cert on deployment host>
* A new variable called 'keystone_ssl_self_signed_subject' has
 been implemented to allow the user to override the certificate
 properties, such as the CN and subjectAltName.
Upgrade notes:
* The SSL-based client authentication configuration in Apache
 has been removed as it appears to be unused.
* The minimum Ansible version for the os_keystone and
 haproxy_server roles have been increased to v1.9.0 as it's
 the minimum version that supports ternary filters.
* The boolean 'keystone_ssl_enabled' has been renamed to
 'keystone_ssl'. This maintains a pattern set in the haproxy
 role for enablement of ssl offloading in the load balancer.
* The Apache configuration appropriately implements the
 'SSLCACertificateFile' instead of the 'SSLCACertificatePath'
 directive in order to ensure that the appropriate signing
 certificate is provided to the browser.
* The 'keystone_self_signed_regen' variable has been renamed
 to 'keystone_ssl_self_signed_regen'.
* The default names for the deployed keys/certificates have been
 changed:
 - /etc/ssl/certs/apache.cert > /etc/ssl/certs/keystone.pem
 - /etc/ssl/private/apache.key > /etc/ssl/private/keystone.key
DocImpact
Partial-Bug: #1466827
Implements: blueprint keystone-federation
Change-Id: I4c5ea7b6bfc3d7d7230a7440fa501241826c9dee
Co-Authored-By: Miguel Grinberg <miguelgrinberg50@gmail.com>

This patch changes the apt task for installing haproxy packages
from only checking for presence to always checking for the
latest package version.
This is essential to allow a deployer to switch from a
configuration that does not implement SSL to one that does.
Change-Id: Iaf6eaedba835a332920336b1cb66190924537301
Closes-Bug: #1475597 

This patch fixes the formatting for an haproxy service which has
ssl enabled.
Without this patch the 'reqadd X-Forwarded-Proto' line ends up
on the same line as the 'set request_option' line.
Change-Id: I0567797304646d7da9badfd193a1368b8a97b0db
Closes-Bug: #1475242 

This change adds support for SSL to the haproxy role. When
enabled, this implements/upgrades haproxy to v1.5.x from a PPA.
* A new boolean variable called 'haproxy_ssl' enables/disables
 the configuration of SSL for the haproxy service.
* A new variable called 'haproxy_ssl_self_signed_subject' has
 been implemented to allow the user to override the certificate
 properties, such as the CN and subjectAltName.
* A new variable called 'haproxy_cert_regen' has been
 implemented to allow the user to regenerate the self-signed
 certificate used for the SSL endpoint.
* SSL will only be enabled for a load balanced service if
 haproxy_ssl is true in the service vars. This has only been
 implemented for the Keystone service endpoints in this patch.
* The keystone admin service endpoint will only have SSL enabled
 if keystone_service_adminuri_proto == 'https'.
* The keystone internal/public service endpoint will only have
 SSL enabled if keystone_service_publicuri_proto == 'https'.
Implements: blueprint keystone-federation
Change-Id: I069f1a0f928feb754816b7d450929fb62df66244

This change adds a specific update task to all tasks that all the
apt ansible module. This change was done to ensure that the cache
is updated as expected when instructed to do so. The reason that
the cache update is being removed from the grouping is because
there is an upstream bug that is effecting the process by which
the apt cache is updated when there is a package list to process
within the same task. The work around to make this function as
expected is to move the update into its own task without a package
list.
Upstream Ansible bug:
 - https://github.com/ansible/ansible-modules-core/issues/1497
Change-Id: Ic06d89a76d772c12888b4bc4bbf147be58b0c150
Related-Bug: 1464771

This new role is now providing the ability for a user to pin apt
packages as they see fit. The idea is to allow someone to implement
pinning in a generic way that can be represented as a global variable
or as a hostvar. The new role has been added to all install roles as
a dependency which will allow it to ensure that packages are pinned
everywhere as would be expected.
Change-Id: I354e8515570fa7174366ba57d57aece3c304568e

This change implements the blueprint to convert all roles and plays into
a more generic setup, following upstream ansible best practices.
Items Changed:
* All tasks have tags.
* All roles use namespaced variables.
* All redundant tasks within a given play and role have been removed.
* All of the repetitive plays have been removed in-favor of a more
 simplistic approach. This change duplicates code within the roles but
 ensures that the roles only ever run within their own scope.
* All roles have been built using an ansible galaxy syntax.
* The `*requirement.txt` files have been reformatted follow upstream
 Openstack practices.
* Dynamically generated inventory is now more organized, this should assist
 anyone who may want or need to dive into the JSON blob that is created.
 In the inventory a properties field is used for items that customize containers
 within the inventory.
* The environment map has been modified to support additional host groups to
 enable the seperation of infrastructure pieces. While the old infra_hosts group
 will still work this change allows for groups to be divided up into seperate
 chunks; eg: deployment of a swift only stack.
* The LXC logic now exists within the plays.
* etc/openstack_deploy/user_variables.yml has all password/token
 variables extracted into the separate file
 etc/openstack_deploy/user_secrets.yml in order to allow seperate
 security settings on that file.
Items Excised:
* All of the roles have had the LXC logic removed from within them which
 should allow roles to be consumed outside of the `os-ansible-deployment`
 reference architecture.
Note:
* the directory rpc_deployment still exists and is presently pointed at plays
 containing a deprecation warning instructing the user to move to the standard
 playbooks directory.
* While all of the rackspace specific components and variables have been removed
 and or were refactored the repository still relies on an upstream mirror of
 Openstack built python files and container images. This upstream mirror is hosted
 at rackspace at "http://rpc-repo.rackspace.com" though this is
 not locked to and or tied to rackspace specific installations. This repository
 contains all of the needed code to create and/or clone your own mirror.
DocImpact
Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
Closes-Bug: #1403676
Implements: blueprint galaxy-roles
Change-Id: I03df3328b7655f0cc9e43ba83b02623d038d214e