openstack/openstack-ansible-haproxy_server - openstack-ansible-haproxy_server - OpenDev: Free Software Needs Free Tools

Instead of hardcoding specific supported tunable options, we
just pass key as an option to haproxy config.
This change might break deployments during upgrades, since format of
values in variable has changed, but appropriate release note was written
We also increase maxrewrite by default, as otherwise usage of CSP leads
to 500 error.
Change-Id: I949960420ed5dbd6d58f0de7dae0ac629a85b7fc
Related-Bug: https://github.com/haproxy/haproxy/issues/1597
Needed-By: https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/844815 

This adds TLS v1.3 support to the HAProxy role by default, along
with a new variable to manage cipher suites.
The old variable for TLS v1.2 and below ciphers is renamed for
consistency, but is still supported as a default where overridden
by deployments.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/823943
Change-Id: Iaf9709ac5f5ac8db281a9ec7278cef274186ba15

This could be achieved using the
haproxy_ssl_letsencrypt_setup_extra_params variable, but this
makes it a bit neater.
Change-Id: Iee2d5a10e1762b23fcb3f3140950c76a754743b7

HAProxy added native Prometheus support from v2.0. This can be
enabled using the existing stats endpoint via an additional
/metrics path.
Change-Id: If9528969c7915db06138c0746dc419d8302f0e7c

We're providing an option to have an IP address per VIP
address. Currently it's used only for creating self-signed
SSLs signed with internal CA per each VIP. With follow-up
patches that will also allow to provide user certificates
per VIP, making possible to cover internal and external
endpoints with different non-wildcard certs.
Change-Id: I0a9eb7689eb42b50daf5c94c874bb7429b271efe

Change-Id: I568273d0ef1d5ee391a42981e66cc9895b9d71b6

The external PKI role can generate a self signed CA and Intermediate
certificate, and then create a server certificate for haproxy if
no defaults are overridden.
The new openstack_pki_* settings allow an external self signed CA
to be used, but still create valid haproxy server certificates from
that external CA in an openstack-ansible deployment.
The original beheviour providing user supplied certificates in the
haproxy_user_ssl_* variables will still work, disabling the generation
of certificates but using the external PKI role to just install the
supplied certs and keys.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/788031
Change-Id: I7482f55e991bacd9dccd2748c236dcd9d01124f3

When HAProxy is run in multi-process mode, the single stats page
shows metrics for one of the processes at a time, with a random
selection made on page reload.
Whilst a more complete solution may be to enable a stats page for
each process, this is a little cumbersome. This addition allows
the stats page to be pinned to one process, providing a partial
snapshot of the state of the instance.
Change-Id: Id9314e5b267aafeaf34c82874eb8bfe0713dfac3

Readjust hatop installtion method, removed haproxy_hatop_downloader and
deployment-host variables. added "haproxy_hatop_install | bool" condition.
Change-Id: I51423fff67e6e427f6c7d163d8d1aac6bcd82ca9

you can add prometheus metric exposed directly via haproxy if your
version is recent enough.
https://www.haproxy.com/blog/haproxy-exposes-a-prometheus-metrics-endpoint/
Change-Id: I10e7220071290301a85409a1f74fcbad2743d19d

New hatop package has python3 support.
Change-Id: I69c01f330feb67d92b6b01fea589a35969879da2

There's no real need in asking user to manually provide http-01 port and
address when we already have corresponding variables we rely on.
Change-Id: Id0d2a73c863d9bbb8b6280ce42f918127baea354

This variable will allow to globally control if SSL should be also used
for internal/admin endpoints, or for public only
Change-Id: I1fa990bab5801a6e6fde7176b2011ab1977b30ae

If this role is used outside the context of openstack-ansible then the
self signed certificate distribution tasks will fail if the haproxy_all
group is not defined, even if self signed certificates are not being used.
Change-Id: Iebc4a293fa8e3566bc910de305e6519a25f2884f

We use the built in python3 http server to bring up a temporary backend
on the node which wants to renew a certificate. The timeout set so that
the haproxy health check has noticed the backend come up before certbot
runs.
There is otherwise a race condition between the haproxy healthcheck and
the certbot challenge request arriving at the acme-challenge endpoint.
Change-Id: I2f5f9457c43c68f2881bf9d44f43434ca7b43859

Currently the only method is by downloading the certbot-auto script
and executing that. Some distros supply a so this patch sets up
an option for a future patch to add distro package support
Change-Id: Ie32e6f577c9aa898906ee76199fd0ebe75d5ae95

When setting up certbot for the first time, many extra parameters
are available. This new variable allows these to be passed. A typical
example is passing --staging in order to use the letsencrypt staging
endpoint rather than the production one.
Change-Id: I42f9e1f68c3a3533a3377f37063f4924cdf77bd6

This patch adds two new variables for a service:
* haproxy_redirect_scheme
This variable allows a custom string to be specified to override
the default condition used to redirect http to https.
* haproxy_frontend_acls
This variable works in the same way as haproxy_acls except it applies
the acl to the frontend rather than the backend configuration. This
can be required when some paths are not redirected to https but must
instead be handled by a specific backend.
Change-Id: I6b13375ba738d7659681ca773297d0b6b0fd7efb

Adding options to be able to override the default behaviour of
haproxy binding to external_lb_vip_address and internal_lb_vip_address.
The default behaviour stays the same after this change.
Change-Id: I76044aea498d73e97087719279ba0a37a9eb28e9

As extra_lb_tls_vip_addresses required a default due to how it's used
I'm also adding a default for extra_lb_vip_addresses and removing the if
defined for it for clarity.
Change-Id: If217f811dab9cfa2f459f5f50bc67bcf31ddbaaa

The existing extra_lb_vip_addresses parameter will add extra haproxy
VIPs without TLS. This patch adds a new extra_lb_tls_vip_addresses
parameter for adding VIPs with TLS enabled.
Change-Id: Ib6f38200775d31633d57a680fae475dbf7abc6c9

Had an issue where HAProxy logs could not be found.
The /dev/log socket was not available to the chrooted filesystem.
We need to mount the socket and persist it.
Change-Id: I2a1ce48f90c5f85b1238842f17ad2c9708333629

The HTTP keepalive mode is currently hardcoded to "http-server-close"
for all HTTP services. This disables keepalive for HAProxy to backend
connections, but leaves it enabled for client connections to HAProxy.
This is problematic especially for service to service calls (e.g.
nova-api to neutron). If a request is made at the same time the HAProxy
keepalive timeout expires, the result of the request is undefined. This
leads to code 500 error responses from the nova-api because the request
from nova-api to neutron failed. "Connection aborted" error messages in
the logs are an indication of this issue.
There is also a bug report[1] about the same issue in devstack which was
solved by disabling keepalive and a script[2] to reproduce the issue in
devstack.
This adds a default and per service variables to set the HTTP keepalive
mode used by HAProxy. The default value is changed to "forceclose" to
disable HTTP keepalive on the server and client side. With HTTP
keepalive disabled the issue can no longer be reproduced.
[1] https://bugs.launchpad.net/devstack/+bug/1630664
[2] https://github.com/JordanP/openstack-snippets/blob/master/keepalive-race/keep-alive-race.py
Change-Id: If819912873270f0568974925490023310f9cbd66

certbot-auto wants to install depedencies which are possibly
not fulfilled by the repo-server - so bypass installation here.
Use the venv bin later for the renew script.
Minor errors are also fixed.
Change-Id: I4087bbcb4fe6182cb090a5b6b85bea36768b4f4f

- installs certbot-auto
- generates and validates ssl cert
- installs cert in haproxy settings
- renew cert with cron
Change-Id: Iea59ec2893a988b184ca8bc70e1d273ac071551e

Secure by default
Change-Id: I70007af94bfd5e482662ab72d25bf090cf5d0834

There are many possible options that can be set for haproxy backend
servers but the current template does not provide a means for these
to be supplied. This patch follows the pattern already used with
haproxy_backend_options and creates a new haproxy_backend_server_options.
Change-Id: Ic312e5915a5df07121ffadca643ca6e4013e00ee

Allow deprecation of haproxy endpoints by setting the state of the
service to 'absent'. It will also now clean up any config files
when there are no backends, or the service is disabled.
Change-Id: I1db5932c559b5e04d330c114164869dd43c1cbb2

space between variable and value also the variable name was incorrect
corrected the name throughout playbook
This closes-bug: 1775938
Change-Id: I3920d7d89b74f12a6a7633d5c5c54a27ee029d31

The commented out section in the defaults file referenced wrong
parameter names. The parameters are called haproxy_* instead of hap_*.
Change-Id: Ieb6afece65d8872e8be83c8a848577cc052117f0

Convert the extra package download tasks to use the same pattern as
other roles, including validating the checksum of the download.
Change-Id: I72891a3321eda65ef802bcbc5073251fc2fb9a03

This patch adds a new variable, ``haproxy_stats_refresh_interval``,
which allows a deployer to configure their preferred refresh
interval for the haproxy stats page.
Release notes are included.
Closes-Bug: 1742526
Change-Id: I3979299478a8e9b479a4c4e821f2a45e1b2679cb

The repo/keys are left over from Newton/Trusty and
were not removed when Trusty support was removed.
The required packages were only necessary in order
to facilitate the repo addition.
The var haproxy_distro_packages is defined in all
distro-specific vars files, so its presence in
defaults is unnecessary.
The apt pinning meta dependency is no longer
required - it's another leftover from Ubuntu Trusty.
A task is included to remove the old config files.
Change-Id: I912cd170d05c4a9befe3420971ddf68ff2ddde2b

Currently the timeout values for client/connect/server are hard coded
in templates/haproxy.cfg.j2. In order to give users flexibility to
override these values, for example increase timeout or reduce timeout,
I replaced hard coded timeout values with variables.
The original default values are still kept in defaults/main.yml.
Closes-Bug: #1696703
Change-Id: I72e9691a074df04e9fb7c4ddc0fe610c8d13feff

When using a TLS proxy, the certificate validation may fail.
This patch allows the validation to be optionally disabled.
Change-Id: I4bc854486a5ef694c4bc8a29546586a068f55af7

Max number of connection may be set up, as well as major tunables
may be added as the variables.
Change-Id: I5b333b79680d81b030810a7e94e3cc4bfe724649

This fix supports to configure timeout for http-request which in turn
helps in avoiding DDOS Attack. This ensures HAProxy to let five seconds
to a client to send its whole HTTP request, otherwise HAProxy would
shut the connection with an error.
Change-Id: I2e6b886e6c1e4a63ecb6513cd2d2672ac738a909

This fix will help in overriding the haproxy ssl bind options.
Change-Id: I45ebf6d532988bd5f842e553de24330863f3989f

Provide a default 'haproxy_ssl_cipher_suite' var for when this role is
run outside of an integrated OpenStack-Ansible deployment.
Change-Id: Ib6ec9c95e166eb78173c9d3a642494cd2f04f0f8

Added an option to configure acl in haproxy_server role by modifying
service.conf.j2 file. It makes developer easy to specify multiple
acl rules to front end that maps to a single backend server.
Change-Id: I528d9f276b4e1f680dd35d77999836f5a87c7c87

In order to make it easier to differentiate between the lists of
python packages, distribution packages, downloaded packages,
package pins and other similar variables the variable names are
being changed to ensure that they have a more explicit suffix
that defines the purpose and makes the naming more consistent.
This is to facilitate a lookup plugin which will be able to look
up all the package lists and present them as a consolidated piece
of data which may be used for artifact preparation.
Change-Id: Id9a356f78162a77edc27209be215f04380a631dc

google storage url changed so this URL needs to be updated.
Change-Id: I99b577badc8db3ce0cb6f683c233ef6fee18022e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: I3732efabfa4fc7e80a8f172abd1415fd54489763

This change implements CentOS7 and Ubuntu 16.04 support for the HAProxy
role. Because RHEL does not package HATop the installation of HATop has
been moved to a source installation so that it can be used universally.
Implements: blueprint multi-platform-host
Change-Id: Ib4f33185202b694b9611cc5fd6323c30a1c8d489
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

This change makes it so that all services are expecting SSL termination
at the load balancer by default. This is more indicative of how a real
world deployment will be setup and is being added such that we can test
a more production like deployment system by default.
The AIO will now terminate SSL in HAProxy using a self-signed cert.
Depends-On: I63cfecd6793ba2b28c294d939c9b1c466940cbd1
Depends-On: Iba63636d733fa1eb095564b8bf33a8159d9c2a00
Depends-On: Ib31a48dd480ecb376a6a8c5b35b09dfa5d2e58f6
Depends-On: Ibdeb8b981ca770ce4f56beeae05afd3379964859
Change-Id: Id87fab39c929e0860abbc3755ad386aa6893b151
Co-Authored-By: Logan V <logan2211@gmail.com>
Signed-off-by: Logan V <logan2211@gmail.com>
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

Workarounding the upstream ansible apt module bug
documented here:
https://github.com/ansible/ansible-modules-core/pull/1517
For the next versions of ansible we'll be using, we should
check if the apt bug is fixed. When it's fixed, we could
abandon this change and use the standard apt module
with correct cache handling.
Change-Id: I2aaf00da175f31d0157bbc4ae30a4e176b055078

The haproxy check script that is installed with keepalived expects to use
'killall', however this package is not installed in the container templates
by default and therefore the haproxy role must install it in order for
keepalived to leave FAULT state.
Change-Id: I8048aaa16b163acfe3da6863aef26adbe18bd73e