My previous setup to maintain Let's Encrypt certificates, which worked, was:
WAN -> pfSense -> Ubuntu 18.04 -> haproxy -> certbot v0.27.0 (for renewal)
pfSense forwarded all TCP traffic on port 80 and 443 to Ubuntu. haproxy knew about the LE certs, and all backend services were http. (All https traffic terminated before it got to an haproxy backend server.)
The working command to renew certs was:
sudo certbot renew --tls-sni-01-port=8888
A while back, I built a shiny new Ubuntu 22.04 server to replace the old one. Of course, I installed the latest version of software, like certbot v1.21. I copied configs, like haproxy.cfg, and scripts, like my certbot renewal, and I thought everything worked, but I didn't realize that cert renewal was going to start failing when it came time to renew them. Apparently, the --tls-sni-01-port switch is no longer supported, but I can’t certbot renewal to work with the --http-01-port switch.
My command is now:
sudo certbot renew --http-01-port=8888
The output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.ca.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for mydomain.ca
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: mydomain.ca
Type: connection
Detail: During secondary validation: xxx.xxx.xxx.xxx: Fetching http://mydomain.ca/.well-known/acme-challenge/nAD9FV5-ZLrfSs_VlK5IXsHItqxk32mEcj0HyWPhFII: Timeout during connect (likely firewall problem)
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 8888. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.
Failed to renew certificate mydomain.ca with error: Some challenges have failed.
(xxx.xxx.xxx.xxx is my public IP, and mydomain.ca is not actually the domain, of course.)
I can confirm that:
- pfSense forwards TCP port 80 to haproxy.
- When I run certbot, the haproxy log shows activity on my frontend acl to my backend server on localhost:8888.
- Ubuntu’s firewall is disabled.
In haproxy.cfg, I did have:
redirect scheme https code 301 if !{ ssl_fc }
to redirect http traffic to https, but that is currently commented out. Still, I get the error above.
I found the original website that helped me originally set up certbot renewal from behind haproxy, but it now seems outdated because it mentions using the now non-existent certbot switch for renewal.
So I am stumped. I must be missing something simple.
I am hoping I don’t have to gut haproxy.cfg to diagnose and to get this working because I have services that I’d like to keep working, and I am hoping I don’t have to post my haproxy.cfg because it has the domain names in it that would be nice not to publicize, but let me know if the above is not enough to go on, or if you need to see something else.
