Setting up Apache with Certbot/Let’s Encrypt but keep it offline until fully configured

Question

I am setting up an Apache web server and running into a chicken-and-egg problem: I don’t want to expose the system to the Internet until it is fully configured (which includes certain security settings). Configuring includes setting an SSL certificate. However, I can’t perform HTTP-based validation as described in https://letsencrypt.org/how-it-works/#domain-validation without the server being reachable from outside.

So my workaround would be to configure Apache with a self-signed certificate, then open it up and migrate to Let’s Encrypt.

Does certbot --apache work for replacing a self-signed cert with one from Lets Encrypt?

The other alternative would be to use DNS-based validation for Let’s Encrypt enrollment, but I am not sure about automatic renewal:

• Can I renew the certificate automatically without having to update the DNS record?
• Or can I initially create the certificate using DNS validation and then use HTTP validation (with automatic setup) for the renewals?

Answer 1

I went through the process of switching from self-signed certs to Let's Encrypt a few weeks ago...self-signed have been good enough for my needs for decades because I really only care about encryption rather than authentication and identification, but browsers are becoming more and more obstructive about accepting self-signed and the "unknown" CA chain. gatekeeping scumbags gotta gatekeep for profit.

I ended up using acmetool as all the other cert bot scripts seemed like half-arsed hacks written in python. It works just fine for initial certificates with both DNS and Web validation, and automatic renewal of those certificates.

I have it configured to reload apache, postfix, dovecot, prosody, gitlab, koreader sync server, and several other ssl-enabled services when the certificate gets updated. Some of these (like gitlab which is running in docker) required some custom handling (i.e. docker exec -it gitlab pkill nginx), fortunately that's easy to do with a script in /etc/acme/hooks/.

Anyway:

1. What security problems do you envision if you open up non-encrypted port 80 http for Let's Encrypt to validate against?

   Just temporarily delete or comment out any redirection rules you have to redirect all traffic from http:// to https:// and confgure the web server to respond to the validation requests (as documented by the certbot you end up using). Or, at least, add rules to prevent the GET /.well-known/acme-challenge/ HTTP/1.1 Let's Encrypt validation requests being redirected to https.

   BTW, you don't need a certificate, self-signed or otherwise, for Let's Encrypt to validate that you own or control the web server it's signing the cert for - lots of people and organisations needing certs don't have one and have no idea how to even begin creating a self-signed cert (and probably don't even know that such a thing is possible).

2. DNS validation requires the ability to create/update and delete DNS Records. It's mostly useful if you want a wildcard cert for your entire domain - I wouldn't have bothered with it if I didn't need that.

   If you do end up using DNS, I strongly suggest making a separate zone file for the _acme-challenge RR. Otherwise when your main zone is dynamically updated, your carefully hand-crafted and well-commented zone file will be stripped of all order and comments (at least, it does with bind9 - I don't know how nsd or other authoritative domain name servers handle it).

   First, you'll need to generate a key for dynamic DNS updates. All of the documentation I found online mentioned using dnssec-keygen to generate a HMAC-SHA512 key. That documentation is old and obsolete, dnssec-keygen just responds with fatal: unknown algorithm HMAC-SHA512.

   I ended up using tsig-keygen -a HMAC-SHA512 certbot instead.

   Then you need to set up your .acme_challenge zonefile. e.g if your domain is example.net, create a zone file for _acme-challenge.example.net:

; /var/cache/bind/db._acme-challenge.example.net
$TTL 300 ; 5 minutes
_acme-challenge.example.net. IN SOA example.net. root.example.net. (
 2025081710 ; serial
 10800 ; refresh (3 hours)
 3600 ; retry (1 hour)
 604800 ; expire (1 week)
 86400 ; minimum (1 day)
 )
 NS example.net.
$TTL 60 ; 1 minute
 TXT "acme"

Then generate create /etc/bind/named.conf.certbot and make sure you have a line like include "/etc/bind/named.conf.certbot"; in /etc/bind/named.conf:

; /etc/bind/named.conf.certbot
; This bit is copy-pasted from `tsig-keygen -a HMAC-SHA512 certbot`
; This key will also need to be given to your cert bot
key "certbot" {
 algorithm hmac-sha512;
 secret "secret key here";
};
zone "_acme-challenge.example.net" {
 type master;
 file "/var/cache/bind/db._acme-challenge.example.net";
 check-names warn;
 update-policy {
 grant certbot. name _acme-challenge.example.net. txt;
 };
};

DO NOT create an NS record in your main example.net domain to delegate the acme-challenge subdomain to yourself. It's not needed and it won't work. Underscores are allowed in some Resource Record (RR) types, e.g. TXT records, but are not valid characters for NS RRs, and bind will refuse to load the faulty zone file.

• user149408: Regarding security problems, I don’t want to expose the server to the public until it is fully hardened. If the server is open while you’re still installing it and not fully hardened, you open yourself up to whatever vulnerabilities your hardening configuration is meant to close. That depends on the web server and whatever you are running over it.
• cas: So disable (i.e. deny access to) everything but /.well-known/acme-challenge/ until you've got a key, and then shut down apache again until you've hardened it and reconfigured it. and/or add temporary firewall to block port 80 from everywhere except lets encrypt (check to see if they publish a list of source IPs for their validation requests). Anyway, there's not much an attacker can do if the web server 404s or 5xx's every request except the acme challenge for lets encrypt.
• cas: Also, web-server attacks focus on known vulnerabilities in specific web apps, which aren't a problem if you aren't running those things. And mostly they're looking for very low-hanging fruit, servers that haven't been updated or maintained for ages. When i look in my apache logs, I still see bots looking for vulnerabilities that were fixed literally, not figuratively, decades ago.
• user149408: Yet the process is so easy, see my answer below: Install and harden everything, with a self-signed cert (which Apache installs by default), then open up the FW and install the cert. Easy as pie, not even worth the discussion why I need to do this or not.
• cas: Yes, but you seem to be claiming that https is somehow magically more secure against vulnerabilities than http. Which it isn't. Encryption protects against snooping & interception, but it doesn't fix any vulnerabilities or protect you from them. If you're running a vulnerable web app then you're just as vulnerable with https as you are with http. Or more so if you falsely believe it makes you safe.

Answer 2

Apache comes with a preconfigured SSL site (which still needs to be enabled) and a self-signed SSL certificate – at least on Ubuntu, YMMV with other distros.

I tried running certbot --apache, and it replaced the self-signed certificate flawlessly.

So if you want to fully set up the server (except for the certificate) before you open it up to the Internet, this should be the easiest approach.