Server reboot loop with failed renewal #6

syndic-will commented 2023年01月05日 19:54:03 +01:00

I found https://github.com/immers-space/guppe unable to start, each attempt erroring with:

AcmeRequest.requestError: (429 urn:ietf:params:acme:error:rateLimited Error creating new order :: too many failed authorizations recently)

Unfortunately, docker swarm had cycled through so many containers overnight that even the oldest ones in docker ps -a had failed with the 429, by the time I looked at is, so I was unable to observe the initial issue.

For now, I have it running again using the last provisioned cert by using a patched version of auto-encrypt that skips the renewal check (I was not able to simply use the certs with https.createServer as the certs were refused by browsers with an error indicating OCSP stapling was missing).

After a week passes for my LetsEncrypt rate limits to reset, I will attempt another renewal and see what happens.

Probably what's broken it is my load-balancing setup. I have pre-SSL-termination load balancing via Docker Swarm leading to 8 replicas of the Guppe server that use a shared volume for the certs. This works perfectly for loading and using existing certs, but I hadn't considered what would happen with renewal. Perhaps just hitting the rate limit immediately with 8 requests, or issues with 8 processes racing to write to the same disk location.

I found https://github.com/immers-space/guppe unable to start, each attempt erroring with: ``` AcmeRequest.requestError: (429 urn:ietf:params:acme:error:rateLimited Error creating new order :: too many failed authorizations recently) ``` Unfortunately, docker swarm had cycled through so many containers overnight that even the oldest ones in `docker ps -a` had failed with the 429, by the time I looked at is, so I was unable to observe the initial issue. For now, I have it running again using the last provisioned cert by using a patched version of auto-encrypt that skips the renewal check (I was not able to simply use the certs with `https.createServer` as the certs were refused by browsers with an error indicating OCSP stapling was missing). After a week passes for my LetsEncrypt rate limits to reset, I will attempt another renewal and see what happens. Probably what's broken it is my load-balancing setup. I have pre-SSL-termination load balancing via Docker Swarm leading to 8 replicas of the Guppe server that use a shared volume for the certs. This works perfectly for loading and using existing certs, but I hadn't considered what would happen with renewal. Perhaps just hitting the rate limit immediately with 8 requests, or issues with 8 processes racing to write to the same disk location.

syndic-will commented 2023年01月15日 05:20:15 +01:00

Went back in after rate limit expired and limited to 1 replica, but I still couldn't renew. It just sat for a few minutes waiting for the ACME challenge until it was eventually killed by docker swarm for failing healthcheck.

| 🎀 ❨auto-encrypt❩ Certificate ready:
| ❨auto-encrypt❩ ╭───────────────────────────────────────────────────────────────╮
| ❨auto-encrypt❩ │ Serial number : 352740470232785528137855324598517209806516 │
| ❨auto-encrypt❩ │ Issuer : US │
| ❨auto-encrypt❩ │ Subject : a.gup.pe │
| ❨auto-encrypt❩ │ Alternative names: a.gup.pe │
| ❨auto-encrypt❩ │ Issued on : 11/04/2022 (2 months ago) │
| ❨auto-encrypt❩ │ Expires on : 02/02/2023 (in 19 days) │
| ❨auto-encrypt❩ ╰───────────────────────────────────────────────────────────────╯
| 👤 ❨auto-encrypt❩ Creating identity (/root/.small-tech.org/auto-encrypt/production/a.gup.pe/certificate-identity.pem)
| 📃 ❨auto-encrypt❩ Certificate exists, loaded it (and the corresponding private key) from disk.
| 🧐 ❨auto-encrypt❩ Checking if we need to renew the certificate... 
| 🌱 ❨auto-encrypt❩ Certificate expires in 30 days or less. Renewing certificate...
| 🤖 ❨auto-encrypt❩ Renewing Let’s Encrypt certificate for a.gup.pe.
| 📕 ❨auto-encrypt❩ Directory is using endpoint https://acme-v02.api.letsencrypt.org/directory
| ⏰ ❨auto-encrypt❩ Set up timer to check for certificate renewal once a day.
| 🔒 ❨auto-encrypt❩ HTTP server is listening on port 80.
| Guppe server listening on port 443
| 👤 ❨auto-encrypt❩ Creating identity (/root/.small-tech.org/auto-encrypt/production/account-identity.pem)
| 🤖 ❨auto-encrypt❩ Provisioning Let’s Encrypt certificates for a.gup.pe.
| 📈 ❨auto-encrypt❩ Number of authorisations to validate: 1
| 🔒 ❨auto-encrypt❩ HTTP server is now only responding to Let’s Encrypt challenges.
| ⚠ ❨auto-encrypt❩ Received non-ACME HTTP request for /, not responding.
| ⚠ ❨auto-encrypt❩ Received non-ACME HTTP request for /, not responding.
| ⚠ ❨auto-encrypt❩ Received non-ACME HTTP request for /, not responding.
| 🚮 ❨auto-encrypt❩ Destroying HTTP Server...
| 🚮 ❨auto-encrypt❩ HTTP Server is destroyed.
| Guppe server closed
| ⚙️ ❨auto-encrypt❩ Configuration initialised.
| 🚑 ❨auto-encrypt❩ Warning: Failed renewal attempt detected. Old certificate files found. Attempting to recover...
| 🚑 ❨auto-encrypt❩ Cleaning up previous state and restoring old certificate...
| 🚑 ❨auto-encrypt❩ Recovery attempt complete.
| 🎀 ❨auto-encrypt❩ Certificate ready:
| ❨auto-encrypt❩ ╭───────────────────────────────────────────────────────────────╮
| ❨auto-encrypt❩ │ Serial number : 352740470232785528137855324598517209806516 │
| ❨auto-encrypt❩ │ Issuer : US │
| ❨auto-encrypt❩ │ Subject : a.gup.pe │
| ❨auto-encrypt❩ │ Alternative names: a.gup.pe │
| ❨auto-encrypt❩ │ Issued on : 11/04/2022 (2 months ago) │
| ❨auto-encrypt❩ │ Expires on : 02/02/2023 (in 19 days) │
| ❨auto-encrypt❩ ╰───────────────────────────────────────────────────────────────╯
| 👤 ❨auto-encrypt❩ Creating identity (/root/.small-tech.org/auto-encrypt/production/a.gup.pe/certificate-identity.pem)
| 📃 ❨auto-encrypt❩ Certificate exists, loaded it (and the corresponding private key) from disk.
| 🧐 ❨auto-encrypt❩ Checking if we need to renew the certificate... 
| 🌱 ❨auto-encrypt❩ Certificate expires in 30 days or less. Renewing certificate...
| 🤖 ❨auto-encrypt❩ Renewing Let’s Encrypt certificate for a.gup.pe.
| 📕 ❨auto-encrypt❩ Directory is using endpoint https://acme-v02.api.letsencrypt.org/directory
| ⏰ ❨auto-encrypt❩ Set up timer to check for certificate renewal once a day.
| 🔒 ❨auto-encrypt❩ HTTP server is listening on port 80.
| Guppe server listening on port 443
| 👤 ❨auto-encrypt❩ Creating identity (/root/.small-tech.org/auto-encrypt/production/account-identity.pem)
| 🤖 ❨auto-encrypt❩ Provisioning Let’s Encrypt certificates for a.gup.pe.
| 📈 ❨auto-encrypt❩ Number of authorisations to validate: 1
| 🔒 ❨auto-encrypt❩ HTTP server is now only responding to Let’s Encrypt challenges.

node v16.19.0
@small-tech/auto-encrypt v3.1.0

The only difference between my current setup and past ones that did renew without issue is the switch from docker-compose to docker swarm, but I'm using essentially the same compose file and you can see in the logs above that I am receiving traffic on http/80

Went back in after rate limit expired and limited to 1 replica, but I still couldn't renew. It just sat for a few minutes waiting for the ACME challenge until it was eventually killed by docker swarm for failing healthcheck. ``` | 🎀 ❨auto-encrypt❩ Certificate ready: | ❨auto-encrypt❩ ╭───────────────────────────────────────────────────────────────╮ | ❨auto-encrypt❩ │ Serial number : 352740470232785528137855324598517209806516 │ | ❨auto-encrypt❩ │ Issuer : US │ | ❨auto-encrypt❩ │ Subject : a.gup.pe │ | ❨auto-encrypt❩ │ Alternative names: a.gup.pe │ | ❨auto-encrypt❩ │ Issued on : 11/04/2022 (2 months ago) │ | ❨auto-encrypt❩ │ Expires on : 02/02/2023 (in 19 days) │ | ❨auto-encrypt❩ ╰───────────────────────────────────────────────────────────────╯ | 👤 ❨auto-encrypt❩ Creating identity (/root/.small-tech.org/auto-encrypt/production/a.gup.pe/certificate-identity.pem) | 📃 ❨auto-encrypt❩ Certificate exists, loaded it (and the corresponding private key) from disk. | 🧐 ❨auto-encrypt❩ Checking if we need to renew the certificate... | 🌱 ❨auto-encrypt❩ Certificate expires in 30 days or less. Renewing certificate... | 🤖 ❨auto-encrypt❩ Renewing Let’s Encrypt certificate for a.gup.pe. | 📕 ❨auto-encrypt❩ Directory is using endpoint https://acme-v02.api.letsencrypt.org/directory | ⏰ ❨auto-encrypt❩ Set up timer to check for certificate renewal once a day. | 🔒 ❨auto-encrypt❩ HTTP server is listening on port 80. | Guppe server listening on port 443 | 👤 ❨auto-encrypt❩ Creating identity (/root/.small-tech.org/auto-encrypt/production/account-identity.pem) | 🤖 ❨auto-encrypt❩ Provisioning Let’s Encrypt certificates for a.gup.pe. | 📈 ❨auto-encrypt❩ Number of authorisations to validate: 1 | 🔒 ❨auto-encrypt❩ HTTP server is now only responding to Let’s Encrypt challenges. | ⚠ ❨auto-encrypt❩ Received non-ACME HTTP request for /, not responding. | ⚠ ❨auto-encrypt❩ Received non-ACME HTTP request for /, not responding. | ⚠ ❨auto-encrypt❩ Received non-ACME HTTP request for /, not responding. | 🚮 ❨auto-encrypt❩ Destroying HTTP Server... | 🚮 ❨auto-encrypt❩ HTTP Server is destroyed. | Guppe server closed | ⚙️ ❨auto-encrypt❩ Configuration initialised. | 🚑 ❨auto-encrypt❩ Warning: Failed renewal attempt detected. Old certificate files found. Attempting to recover... | 🚑 ❨auto-encrypt❩ Cleaning up previous state and restoring old certificate... | 🚑 ❨auto-encrypt❩ Recovery attempt complete. | 🎀 ❨auto-encrypt❩ Certificate ready: | ❨auto-encrypt❩ ╭───────────────────────────────────────────────────────────────╮ | ❨auto-encrypt❩ │ Serial number : 352740470232785528137855324598517209806516 │ | ❨auto-encrypt❩ │ Issuer : US │ | ❨auto-encrypt❩ │ Subject : a.gup.pe │ | ❨auto-encrypt❩ │ Alternative names: a.gup.pe │ | ❨auto-encrypt❩ │ Issued on : 11/04/2022 (2 months ago) │ | ❨auto-encrypt❩ │ Expires on : 02/02/2023 (in 19 days) │ | ❨auto-encrypt❩ ╰───────────────────────────────────────────────────────────────╯ | 👤 ❨auto-encrypt❩ Creating identity (/root/.small-tech.org/auto-encrypt/production/a.gup.pe/certificate-identity.pem) | 📃 ❨auto-encrypt❩ Certificate exists, loaded it (and the corresponding private key) from disk. | 🧐 ❨auto-encrypt❩ Checking if we need to renew the certificate... | 🌱 ❨auto-encrypt❩ Certificate expires in 30 days or less. Renewing certificate... | 🤖 ❨auto-encrypt❩ Renewing Let’s Encrypt certificate for a.gup.pe. | 📕 ❨auto-encrypt❩ Directory is using endpoint https://acme-v02.api.letsencrypt.org/directory | ⏰ ❨auto-encrypt❩ Set up timer to check for certificate renewal once a day. | 🔒 ❨auto-encrypt❩ HTTP server is listening on port 80. | Guppe server listening on port 443 | 👤 ❨auto-encrypt❩ Creating identity (/root/.small-tech.org/auto-encrypt/production/account-identity.pem) | 🤖 ❨auto-encrypt❩ Provisioning Let’s Encrypt certificates for a.gup.pe. | 📈 ❨auto-encrypt❩ Number of authorisations to validate: 1 | 🔒 ❨auto-encrypt❩ HTTP server is now only responding to Let’s Encrypt challenges. ``` node v16.19.0 @small-tech/auto-encrypt v3.1.0 The only difference between my current setup and past ones that did renew without issue is the switch from docker-compose to docker swarm, but I'm using essentially the same compose file and you can see in the logs above that I am receiving traffic on http/80

aral commented 2025年02月07日 11:42:49 +01:00

As load balancing is not a supported use case for Small Web, I’m closing this.

As load balancing is not a supported use case for Small Web, I’m closing this.