When I woke up this morning, I decided to hack away at a blog post for a little bit. As soon as I opened the site, my heart sank. This is the PITA thing about running your own site. Like owning your own house instead of renting an apartment, or buying a car instead of taking a bus, when something breaks it's on you to fix it. And mechanics ain't cheap. Happy days.
I wouldn't click through this even if it let me, but even better I have HSTS enabled. That's a way for a website to ask a browser to only access it via https (although it's not required to - more on that below), so Chrome honored the request by not providing a way to click through at all. Ouch.
It only took a moment to realize what had happened. See where it says ERR_CERT_DATE_INVALID? Certificate date invalid... as in my SSL certificate. Clicking on that text expanded another block that made it abundantly clear by showing the expiration date - yesterday. My SSL certificate had expired so that's no bueno, but I had told the browser to only access via https, so... doubly no bueno.
Your connection is not private
Attackers might be trying to steal your information from grantwinney.com
(for example, passwords, messages, or credit cards). NET::ERR_CERT_DATE_INVALID
Subject: grantwinney.com
Issuer: Let's Encrypt Authority X3
Expires on: Jun 13, 2017
Current date: Jun 14, 2017
LetsEncrypt certificates expire every 90 days, which is entirely reasonable considering it's a free service and a certain percentage of websites using them will vanish from the face of the interwebz eventually. They need to be cleaned up. But I had set it up to use CRON to automatically renew, so my thought was why didn't that work? (Side note: if you're setting up Ghost on DigitalOcean - I did and I think it's great - here is an excellent writeup for configuring Let's Encrypt.)
First, I SSH'd into my server and tried running the script manually to do a renew with /opt/certbot-auto renew. That gave me a little more insight into what was going wrong. One of the things the script does is create a couple temporary files (long random names in an "acme-challenge" directory), which it then attempts to access using http (sort of... get to that in a minute). That didn't work - you can see where it gets a 404 when it tries to access those files - so the CRON job was working okay but the job it was executing was failing.
root@ghost-1gb-nyc3-01-my-blog:/opt# ./certbot-auto renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log
-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/grantwinney.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for grantwinney.com
http-01 challenge for www.grantwinney.com
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/ghost/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/grantwinney.com.conf produced an unexpected error: Failed authorization procedure. grantwinney.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://grantwinney.com/.well-known/acme-challenge/bzfZnECf_zXY1TA9TK2ZqVZrzOOjaHnpN4yclEtFhgA: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>", www.grantwinney.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.grantwinney.com/.well-known/acme-challenge/SkkeIZHYF3F9kelPA3Lcl5GAgcwONo3plSpSYTOXXRw: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>". Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/grantwinney.com/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: grantwinney.com
Type: unauthorized
Detail: Invalid response from
http://grantwinney.com/.well-known/acme-challenge/bzfZnECf_zXY1TA9TK2ZqVZrzOOjaHnpN4yclEtFhgA:
"<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>"
Domain: www.grantwinney.com
Type: unauthorized
Detail: Invalid response from
http://www.grantwinney.com/.well-known/acme-challenge/SkkeIZHYF3F9kelPA3Lcl5GAgcwONo3plSpSYTOXXRw:
"<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address.
I initially assumed that HSTS somehow was blocking access to those files, but after opening a support ticket on the Let's Encrypt community forum, I learned some new things from one of the moderators:
Based on the output shared above it looks like Certbot tells Boulder (the Let's Encrypt server-side component of the CA) to perform the validation requests against your domain for the /.well-known/acme-challenge/ file, but Boulder gets a 404 response for the requests! Your webserver doesn't give back the validation files to prove to Let's Encrypt you control the domain.
I think it isn't an HSTS problem for a few reasons:
- The HTTP-01 challenge is an HTTP request, not HTTPS
- If it were HSTS based, the connection would fail before a 404 was returned by the server.
- Boulder doesn't honour HSTS at all! We specifically ignore it to avoid this problem
The important one that really clarified some of my understanding of HSTS was number 3 above. It seemed like a chicken and egg problem to me, where the script instructs Boulder to access the challenge files via http, but due to my setup and expired cert that access is not possible. But without the access, it wouldn't be able to renew. Armed with my new HSTS knowledge, I found a post that shows how to bypass HSTS and view a site anyway in chrome:
How To Bypass Chrome's HSTS Warnings - (just type "badidea" while the page is displayed)
After that, I decided to see if the directory it was complaining about in the above output was indeed present but just inaccessible. I created the .well-known/acme-challenge directory myself and threw an empty "test.txt" file in there; tried to access it from the browser and got a 404 too.
Getting near the end now! It seemed to have something to do with routing the request to the correct location. My server uses nginx, and that's configured with the /etc/nginx/sites-enabled/default file. The last section of my nginx config file, for listening to 443, had a small blurb about where to direct requests for .well-known:
location ~ ^/.well-known {
root /var/www;
}
That's the wrong location. My blog is hosted one directory lower. I made the change and restarted nginx with service nginx restart.
location ~ ^/.well-known {
root /var/www/ghost;
}
Then the certificate renewal worked and, after the following output, I restarted nginx again.
root@ghost-1gb-nyc3-01-my-blog:/opt# ./certbot-auto renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log
-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/grantwinney.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for grantwinney.com
http-01 challenge for www.grantwinney.com
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/ghost/.well-known/acme-challenge
-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/grantwinney.com/fullchain.pem
-------------------------------------------------------------------------------
Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/grantwinney.com/fullchain.pem (success)
I'll have to figure out why it didn't email me a warning, which I think it's supposed to do. It's possible I set it up with a now-defunct email address. But at least the site's back up and running. Hopefully if someone else runs into the same problem, it'll be a simple nginx config file fix too!
This is post #13 in my personal challenge to complete 30 Days of Blogging. My goal is to become more comfortable with blogging in a more frequent and informal manner.