Help:Project puppetserver

This page contains information about project puppetservers, helps you decide why you would use one, and provides basic trouble shooting information.

Prior to Debian Bookworm, similar workflows were accomplished via the role::puppetmaster::standalone puppet role. This process is now deprecated, but the documentation can be found at Obsolete:Standalone_puppetmaster.

A project puppetserver can be used to deliver either a custom or experimental Puppet configuration to one or more hosts in a Cloud VPS project. All Cloud VPS instances run puppet via cron about every 30 minutes. The default configuration pulls from the shared Cloud VPS puppetserver which has a pristine checkout of operations/puppet.git and labs/private.git. Sometimes you want to have a puppetserver that you control as a Cloud VPS user. You can use role::puppetserver::cloud_vps_project to accomplish this.

These are the popular reasons for wanting to use one:

1. Testing puppet patches before they have been merged in operations/puppet.git
2. Keep project local secrets in a labs/private.git repo

1. Create a Debian Bookworm instance with at least 2 GB of RAM (g4.cores1.ram2.disk20 to support a handful of VMs, or a larger size for a bigger project)
2. (optional) create and mount a 5GB cinder volume on /srv. That will allow you to transfer certs and puppet code to future upgraded puppetservers. (See Help:Adding_disk_space_to_Cloud_VPS_instances for detailed instructions)
3. Apply the role role::puppetserver::cloud_vps_project to the new VM
4. Apply the following hiera settings:

   profile::puppet::agent::force_puppet7: true

   puppetmaster: puppet

5. Force two consecutive puppet runs on the instance using sudo -i run-puppet-agent
6. Generate the server certs: sudo puppetserver ca generate --certname $(hostname -f)
7. Make sure puppetserver can restart: sudo systemctl restart puppetserver

You will now find a checkout of operations/puppet.git in /srv/git/operations/puppet and a checkout of labs/private.git in /srv/git/labs/private.

If you are not planning on keeping any secrets in your puppetserver, you can turn on autosigning. This will automatically accept new clients as they contact the server, and save you a step later on when adding clients. You can do that by adding the following to the hiera configuration for the puppetserver instance:

profile::puppetserver::autosign: true

This works for any number of instances, including the puppetserver itself.

1. Using the Horizon interface:
   1. Set the hiera variable puppetmaster: <your puppetserver's FQDN>. (Scroll all the way to the bottom of the Puppet Configuration tab, to the Hiera Config button, and add the variable name/value in YAML format.) Use the fully qualified domain name of the puppetserver (i.e. hostname.projectname.eqiad1.wikimedia.cloud which can be found by running hostname -f on the puppetserver host).
      • This variable can be set either per-instance or project wide.
2. On the client instance:
   1. Run puppet to apply the modified hiera configuration: sudo -i run-puppet-agent
   2. Remove existing SSL certificates created with the prior puppetmaster: sudo rm -rf /var/lib/puppet/ssl (NOTE: you will see certificate verify failed: [self signed certificate in certificate chain ...] when you run puppet if you don't do this.
   3. Only if the client is the puppetserver itself, run also:

      sudorm/srv/puppet/server/ssl/ca/signed/$(hostname-f).pem
      sudomkdir-p/var/lib/puppet/ssl/private_keys
      sudocp-v/srv/puppet/{server/,}ssl/private_keys/$(hostname-f).pem
      sudomkdir/var/lib/puppet/ssl/certs/
      sudocp-v/srv/puppet/{server/,}ssl/certs/$(hostname-f).pem
      

   4. Force a puppet run to generate a new local SSL certificate: sudo -i run-puppet-agent
3. On the puppetserver (If you enabled autosigning in the master, you can skip this step):
   1. Check that the fingerprint of the client's key matches the one shown when it was generated running sudo -i puppet cert list --all (Puppet 5) or sudo -i puppetserver ca list --all (Puppet 7)
   2. Sign the key of the client: sudo -i puppet cert sign <client-fqdn> (Puppet 5) or sudo puppetserver ca sign --certname <client-fqdn> (Puppet 7). (if you need to clean an old certificate it's sudo puppetserver ca clean --certname <client-fqdn> for Puppet 7)
4. On the client:
   1. Run puppet twice to check that is connecting correctly to its new server and that the second run does not produce any new configuration changes: sudo -i run-puppet-agent

Changes made in /srv/git/operations/puppet or /srv/git/labs/private will not immediately be applied by the puppetserver; first these changes must be deployed. Deploy by running sudo puppetserver-deploy-code.

puppetserver-deploy-code will only deploy the production branch in /operations/puppet and only deploy the master branch in /labs/private.

In most cases, git operations in /srv/git will trigger a hook that automatically deploys code. But, when in doubt, manually redeploy to be sure that you're applying the latest production branch.

Failing to run puppet agent as root can result in the surprising error:

Debug: Creating new connection for https://puppet:8140
Error: Could not request certificate: getaddrinfo: Name or service not known
Exiting; failed to retrieve certificate and waitforcert is disabled

The clones of the Puppet repositories in /srv/git must be owned by and interacted via the the gitpuppet user. A helper script pgit is made available in $PATH.

Hiera in Cloud VPS has different settings than in Wikimedia Production puppet, and it can be tricky to get your values into the right place in labs/private.git or puppet in general. First, check /etc/puppet/hiera.yaml on your standalone puppetserver for guidance. You may just need to add things to labs/private/hieradata/labs/<VPS project name>/common.yaml.

You can verify that a node can "see" your hiera settings with the command sudo puppet lookup --node <fqdn of puppet client> --compile --explain <your::hiera::key> on the project puppetserver. The compile is needed to set all the right facts for hiera in Cloud VPS, and there will be a lot of warnings you can ignore. The answer will be at the end of the command's output.

To add a new secret to your puppetserver to use in your manifests, you have to add a new local commit to the repository under /srv/git/labs/private in the puppetserver node.

Note that there might be a leftover git hook preventing you to add a new commit with the message:

Localcommitsarenotallowedinthisrepository.Pleasegotofrontendpuppetmasterandcommit

This is a leftover from the puppet migration to puppet 7 and can be ignored, to do so just rm /srv/git/labs/private/.git/hooks/pre-commit, and it will not bother you again.

WARNING: This is is difficult and complicated. Proceed with caution.

Puppetdb can be enabled on standalone puppetservers on Cloud VPS by designating a puppetdb server with the role::puppetdb role and a lot of hiera values on both the DB server and the puppetmaster. This requires significant effort and is not a recommended configuration unless you have significant experience with puppet, need it and are able to maintain the setup, including your own postgresql database. It is guaranteed to break your puppet setup if you just enable it without following a particular order. Notes on that can be found on the project puppetserver puppetdb notes page.

Typically development is done on non-Cloud VPS machines, this applies to Puppet as well, so a common use case is to develop locally and then push to your Cloud VPS project's puppetserver for testing.

If you want your Cloud VPS instance to act as yet another remote Git repository, you have to configure Git to use sudo as well. To do that, in your local computer's git clone of the Puppet repository

$ gitremoteaddmy-instancemy-instance.my-project.eqiad1.wikimedia.cloud:/srv/git/operations/puppet
$ gitconfigremote.my-instance.receivepack"sudo git-receive-pack"
$ gitconfigremote.my-instance.uploadpack"sudo git-upload-pack"

After that, you can push and pull branches from and to your Cloud VPS instance. If you pull commits from your Cloud VPS instance, remember to reset authorship information with git commit --amend --reset-author before pushing to Gerrit.

You cannot push to the branch that is currently checked out. To avoid having to log into your Cloud VPS puppetserver and checking out branches, etc., you can create a post-receive hook by copying:

#!/bin/bash
#
# Copyright 2015, 2016 Tim Landscheidt
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# When a commit is pushed to a branch with a name starting with
# "development/", this post-update script will replace all "local
# hacks" that have a matching Gerrit "Change-Id:" footer with the ones
# in this branch and add new "local hacks" on top. When a branch with
# a name starting with "development/" is deleted, it will remove all
# "local hacks" that are referenced in that branch.
# Hooks are executed in the ".git" subdirectory, so move upwards to
# allow rebases, etc. to work. As hooks have the environment variable
# "GIT_DIR" set to ".", this means we need to amend it.
cd..
exportGIT_DIR=.git
sedcommands=$(mktemp)
# Iterate over all branches that were changed.
whilereadoldrefnewrefbranch;do
# Only process pushes to branches with a name starting with
# "development/".
if["$branch"="${branch#refs/heads/development/}"];then
continue
fi
# Iterate over all changes in the branch that are not part of
# "production".
if["$newref"="0000000000000000000000000000000000000000"];then
range=production..$oldref
else
range=production..$newref
fi
forcommitin$(gitlog--format=format:%H--reverse$range);do
# Check whether there is already a "local hack" with the same
# Gerrit "Change-Id:" footer and remove it.
changeid="$(gitlog-1--format=format:%B$commit|sed-ne's/^Change-Id: //p;')"
localhack="$(gitlogorigin/production..production--format=format:%H--grep"^Change-Id: $changeid\$")"
# Was the branch deleted?
if["$newref"="0000000000000000000000000000000000000000"];then
# Then delete the local hack if it exists.
if[-n"$localhack"];then
echo"/^pick ${localhack:0:7} .*\$/d">>"$sedcommands"
fi
else
# Otherwise update or add the local hack.
if[-n"$localhack"];then
echo"s/^pick ${localhack:0:7} .*\$/pick $commit/">>"$sedcommands"
else
echo"\$apick $commit">>"$sedcommands"
fi
fi
done
done
if!GIT_SEQUENCE_EDITOR="sed -i -f $sedcommands"gitrebase-iorigin/production;then
gitrebase--abort
echo"0ドル: Could not rebase changes.">&2
fi
rm-f$sedcommands

to /srv/git/operations/puppet/.git/hooks/post-receive and making it executable (chmod +x /srv/git/operations/puppet/.git/hooks/post-receive).

With this hook in place, when you push commits to a branch with a name that starts with development/:

• if a "local hack" (a commit that is in the production branch, but not in origin/production) with the same Change-Id already exists, it will replace that commit with the one in your pushed branch, or
• if no local hack with the same Change-Id exists, it will apply it on top of other local hacks.

If you delete a branch (git push -f my-instance :development/branch), the local hacks in that branch are removed.

You can use the script git-puppet-test to combine pushing your local development branch to the Cloud VPS puppet master with running Puppet on several hosts and executing a test script:

#!/usr/bin/python3
#
# Copyright 2014, 2015, 2016 Tim Landscheidt
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
importargparse
importre
importsubprocess
importsys
parser = argparse.ArgumentParser()
parser.add_argument('--set-pre-command',
 dest='pre_command',
 help='set the pre-command')
parser.add_argument('--set-post-command',
 dest='post_command',
 help='set the post-command')
parser.add_argument('--set-hosts',
 dest='hosts',
 help='set the hosts')
args = parser.parse_args()
# Ensure there are no uncommitted changes.
if subprocess.check_output(['git', 'status', '--porcelain']):
 print('Uncommitted changes.')
 sys.exit(1)
# Determine change-ID.
commit_message = subprocess.check_output(['git', 'cat-file', '-p', 'HEAD'],
 universal_newlines=True)
change_id = None
for line in commit_message.split('\n'):
 if line.startswith('Change-Id: '):
 change_id = line[len('Change-Id: '):]
if change_id is None:
 print("Couldn't find Change-Id.")
 sys.exit(1)
# Determine remote repository for puppetserver.
puppetmaster_remote = None
for line in subprocess.check_output(['git', 'remote', '-v'],
 universal_newlines=True).split('\n'):
 m = re.fullmatch(r'([^\t]+)\t'
 r'.*\.wmflabs:/srv/git/operations/puppet \(push\)',
 line)
 if m:
 puppetmaster_remote = m.group(1)
if puppetmaster_remote is None:
 print("Couldn't determine remote repository for puppetmaster.")
 sys.exit(1)
# Process --set-* options and terminate if they were given.
if args.pre_command is not None:
 subprocess.check_call(['git',
 'config',
 'puppet-test.%s.pre-command' % change_id,
 args.pre_command])
if args.post_command is not None:
 subprocess.check_call(['git',
 'config',
 'puppet-test.%s.post-command' % change_id,
 args.post_command])
if args.hosts is not None:
 subprocess.check_call(['git',
 'config',
 'puppet-test.%s.hosts' % change_id,
 args.hosts])
if any((args.pre_command is not None,
 args.post_command is not None,
 args.hosts is not None)):
 sys.exit(0)
# Run pre-command.
try:
 pre_command = subprocess.check_output(['git',
 'config',
 '--get',
 'puppet-test.%s.pre-command' %
 change_id],
 universal_newlines=True)
 subprocess.call(pre_command, shell=True)
except subprocess.CalledProcessError:
 pass
# Push change to puppetserver.
subprocess.check_call(['git',
 'push',
 '-f',
 puppetmaster_remote,
 'HEAD:development/' + change_id])
# Run Puppet on configured hosts.
try:
 hosts = subprocess.check_output(['git',
 'config',
 '--get',
 'puppet-test.%s.hosts' % change_id],
 universal_newlines=True)
 subprocess.call(['clush',
 '-w',
 hosts,
 'tmplog="$(mktemp)" && '
 'if ! sudo puppet agent -tv > "$tmplog" 2>&1; then '
 'cat "$tmplog"; '
 'fi && '
 'rm -f "$tmplog"'])
except subprocess.CalledProcessError:
 pass
# Run post-command.
try:
 post_command = subprocess.check_output(['git',
 'config',
 '--get',
 'puppet-test.%s.post-command' %
 change_id],
 universal_newlines=True)
 subprocess.call(post_command, shell=True)
except subprocess.CalledProcessError:
 pass

This is especially useful if you work on a patch over a longer period of time, or the tests that you need to run are complex or boring. The shell command to run before and after pushing a branch and the hosts to run Puppet on are specified by the entries puppet-test.$changeid.pre-command, puppet-test.$changeid.post-command and puppet-test.$changeid.hosts in .git/config with $changeid being the Change-Id of the commit at HEAD. You can set those also by git puppet-test --set-post-command $command, etc.

When a patch is merged that you previously pushed as a local hack, if it is identical, i. e. the changes are the same, it will just "disappear" from the list of local hacks. If it has been amended, the automatic pull with git-sync-upstream will fail, and you will have to log in and remove the local hack with git rebase -i origin/production production and deleting the line of the local hack in question. You need to apply the same steps if you want to abandon a local hack that you no longer want to work on.

NOTE: If the puppet master is shared among more than one person, pushing will overwrite or mess in other ways with each other's changes!

In case a puppetserver is used to test a single branch/patchset at a time, a simple solution is to to push to an intermediate bare repo and then update the non-bare authoritative repo.

The git hook added in 159720 achieves this. So assuming you have a role::puppetserver::cloud_vps_project configured, you can setup a local bare repo e.g. in your Cloud VPS home (as your user):

cd ~
[ -d ~/puppet.git ] || git clone --bare --no-hardlinks --branch production /srv/git/operations/puppet/ ~/puppet.git
install -m755 /srv/git/operations/puppet/modules/puppetmaster/files/self-master-post-receive ~/puppet.git/hooks/post-receive

and on the development host add the relevant remote:

git remote add project_puppetmaster ssh://PUPPET_MASTER_HOSTNAME/~/puppet.git

then to test a puppet change in your Cloud VPS puppet master from your local host you can force-push to that remote in the "production" branch, e.g.

git push -f project_puppetmaster HEAD:production

Most new puppetserver have a CA cert that expires in 5 years. If your CA is expiring, it may be time to build a new puppetserver! Regenerating the CA cert is possible but require acting on every managed instance so is non-trivial.

These instructions were originally paraphrased and adapted from here and have since been updated to function with Puppet 7.

On the puppetserver:

$ FQDN="$(hostname-f)"# change this to the fqdn the clients use if it's not the same as the host fqdn
$ SERIAL="$(opensslx509-in/srv/puppet/server/ssl/ca/ca_crt.pem-noout-serial|cut-f2-d'=')"
$ disable-puppet"CA renewal"
$ systemctlstoppuppetserver.service
$ opensslx509-x509toreq-in/srv/puppet/server/ssl/ca/ca_crt.pem-signkey/srv/puppet/server/ssl/ca/ca_key.pem-out/srv/puppet/server/ssl/ca/ca_csr.pem
$ cat>/root/puppet-ca-extension.cnf<<EOF
[CA_extensions]
basicConstraints = critical,CA:TRUE
nsComment = "Puppet Ruby/OpenSSL Internal Certificate"
keyUsage = critical,keyCertSign,cRLSign
authorityKeyIdentifier=keyid,issuer
subjectKeyIdentifier = hash
EOF
$ opensslx509-req-days3650-in/srv/puppet/server/ssl/ca/ca_csr.pem-signkey/srv/puppet/server/ssl/ca/ca_key.pem-out/srv/puppet/server/ssl/ca/ca_crt.pem-extfile/root/puppet-ca-extension.cnf-extensionsCA_extensions-set_serial"$SERIAL"
$ [[-f/srv/puppet/server/ssl/certs/ca.pem]]&&mv/srv/puppet/server/ssl/certs/ca.pem/srv/puppet/server/ssl/certs/ca.pem.old
$ ln-s/srv/puppet/server/ssl/ca/ca_crt.pem/srv/puppet/server/ssl/certs/ca.pem
$ rm-v"/srv/puppet/server/ssl/certs/$FQDN.pem""/srv/puppet/server/ssl/private_keys/$FQDN.pem""/srv/puppet/server/ssl/public_keys/$FQDN.pem""/srv/puppet/server/ssl/ca/signed/$FQDN.pem"
$ puppetservercagenerate--certname"$FQDN"--subject-alt-names"$FQDN"--ca-client
$ systemctlstoppuppetserver.service
$ enable-puppet"CA renewal"

For the above steps, an experimental script is also available: phab:P76199

root@cloud-puppetmaster-03:~# run-puppet-agent
...
Notice: /Stage[main]/Profile::Puppetmaster::Frontend/Puppetmaster::Web_frontend[puppetmaster.cloudinfra.wmflabs.org]/File[/var/lib/puppet/ssl/certs/puppetmaster.cloudinfra.wmflabs.org.pem]/ensure: defined content as '{md5}dc365c2c170e357710dd00754f35de70' (corrective)
Notice: /Stage[main]/Profile::Puppetmaster::Frontend/Puppetmaster::Web_frontend[puppetmaster.cloudinfra.wmflabs.org]/File[/var/lib/puppet/ssl/private_keys/puppetmaster.cloudinfra.wmflabs.org.pem]/ensure: defined content as '{md5}0ec67b1df9f79a1b3fd33c52c475a220' (corrective)
Notice: Applied catalog in 10.33 seconds

/srv/puppet/server/ssl/private_keys/puppet.pem
/srv/puppet/server/ssl/certs/puppet.pem
/srv/puppet/server/ssl/private_keys/puppetmaster.cloudinfra.wmflabs.org.pem
/srv/puppet/server/ssl/certs/puppetmaster.cloudinfra.wmflabs.org.pem
# tothepuppetmasterprivaterepo(cloud-puppetmaster-03rightnow):
modules/secret/secrets/puppetmaster/puppet_privkey.pem
modules/secret/secrets/puppetmaster/puppet_pubkey.pem
modules/secret/secrets/puppetmaster/puppetmaster.cloudinfra.wmflabs.org_privkey.pem
modules/secret/secrets/puppetmaster/puppetmaster.cloudinfra.wmflabs.org_pubkey.pem

Finally, remove the locally cached certificate on each client of that puppetserver:

$ # It is probably safe to run this tenant-wide; clearing the ca cert
$ # on a host unaffected by a cert rotation appears to be harmless.
$ rm/var/lib/puppet/ssl/certs/ca.pem

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

• Testing catalog with puppet-compiler

• Cloud VPS
• Documentation
• Puppet