How we identified clients with SSL certificates affected by Let's Encrypt mass-revocation

Yesterday, we sent out notifications to all our clients that are affected by the Let's Encrypt mass revocation of SSL certificates. In this post, we'll share the details how we found those certificates.

Now, the morning after, we're well rested and in good shape to do a proper write-up on the matter.

Getting a list of all domains to check

As part of our uptime monitoring, users can add a site to Oh Dear with specific URL parameters. So in order to get a list of domains we needed to verify, it wasn't as simple as:

SELECT domain FROM sites;

Instead, we used Laravel's lazy collections to quickly filter all teams with active subscriptions and extract the relevant domain name.

Team::cursor()
    ->filter(fn (Team $team) => $team->hasActiveSubscriptionOrIsOnGenericTrial())
    ->flatMap(fn (Team $team) => $team->sites)
    ->map(fn (Site $site) => $site->domain())
    ->each(fn (string $output) => echo $output . PHP_EOL);

This produced a new-line separated list of domains that we need to check.

Let's save those in domains.txt, since we're moving to some CLI tricks now.

Retrieving the serial for each certificate

Now we find the active Serial Number for each of those certificates. It involves connecting to each site over SSL/TLS, getting the certificate and saving the Serial Number.

The original idea came from a Hacker News comment, we modified it to get some better error handling and control of the output.

# Create a directory to hold all serial numbers
mkdir -p serials

# Loop all domains, connect and fetch the serial
for i in $(cat domains.txt); do
  echo "Connecting to $i ... "

  (
    openssl s_client -connect $i:443 -servername $i -showcerts < /dev/null 2> /dev/null |
    openssl x509 -text -noout |
    grep -A 1 "Serial Number" |
    tr -d : |
    tail -n 1
  ) | tee serials/$i;

done

The openssl s_client connects to the domain (using Server Name Indication (SNI) with the -servername option) and lists all certificates.

Now, in serials/*, we have a directory full of domain names and their corresponding certificate serial.

Combining all serials

We'll make a single list with all the serials we need to check. This way, we can optimize our grep commands for later.

$ cat serials/* | tr -d " " | sort | uniq > serial-numbers.txt

The file serial-numbers.txt is now a gigantic list of serial numbers.

Finding the serials in the 1.2GB text file

Let's Encrypt has released a text-file with all affected certificates. This file includes the Serial Number (which we now have) together with all domains/SANs on the certificate.

Our first attempt was to simply grep our way through the file for each serial found. But grep is single-threaded, so we could only utilize a single CPU core for searching through a pretty big file.

This was taking too long, so we quickly adapted our method and started to search through the log in parallel.

Lucky for us, we started preparing a new set of servers for our crawlers that check for broken links last week. Those servers were still idling as they aren't in production yet. This was the perfect time to use that spare capacity.

First, we split the big file of serials (called serial-numbers.txt) in many equal pieces.

$ split -l 1000 serial-numbers.txt

This gives us a list of many files, all with 1000 serial numbers in it. The file naming is predictable:

$ ls -l x*
-rw-rw-r-- 1 immutable immutable 2672 Mar  3 20:27 xaa
-rw-rw-r-- 1 immutable immutable 2948 Mar  3 20:27 xab
-rw-rw-r-- 1 immutable immutable 2948 Mar  3 20:27 xac
-rw-rw-r-- 1 immutable immutable 2960 Mar  3 20:27 xad

In order to utilize all our cores, we used each file as the pattern input to grep and sent the job to the background for processing.

for file in $(ls x*); do
	\grep -P "$(cat $file | tr "\n" "|" | sed -e 's/|/\|/g' | sed -e 's/|$//' )" ../ssl-cert/caa-rechecking-incident-affected-serials.txt >> results.txt &
done

That rather ugly-looking tr & sed pipeline in there transforms the input file from a new-line separated list of serials, to a |-separated list. This is used in grep to indicate the "or" statement, any line may match.

In the long form, it turns our input of this:

$ cat xaa
01009ba...
0111839...
011539e...
0135d43...

... into this:

01009ba...|0111839...|011539e...|0135d43...

Because we sent each grep command to the background in our for-loop, using the & at the end of the command, we now have many grep's running in parallel.

What followed was, to me as a sysadmin, a thing of beauty. ?

The crunching continued for a while, and we now had a list of affected serials stored in results.txt.

At this point, things were getting a bit late, so we resorted to even weirder Bashness to match these serials back to the domain names.

Matching the serials back to domains

We loop each affected serial and match it back to the domain:

for line in $(cat results.txt ); do
	\grep $line serials/*;
done |
	awk '{print $1}' |
	sed 's/\// /' |
	awk '{print $2}' |
	sed 's/:/ /' |
	awk '{print $1}'

Looking at it now, awake, it could've been much cleaner. But, it got the job done! We now have a list of domain names of clients we need to notify.

Sending the mails to clients

To inform our clients, we resorted back to PHP. This allows us to send the notification e-mails in our own style/branding.

Mail::to($users)->send(new LetsencryptRevokedMail($domain));

It uses the power of Laravel Mailables to make this really easy.

A rush job because time was against us

We didn't do as a clean a job as we'd normally do. There were no tests, no clean integrations, and most of it was hacked together on very short-notice.

But, there wasn't much choice. The list of affected certificates was released yesterday, and within 48 hours the revocation was to take place. It was up to us to notify our users asap. After all, those affected still needed to renew their certificates!

We're happy to see the list of affected domains beforehand though. The normal procedure is that the revoced certificates end up in Certificate Revocation Lists (CRL), but at that point the revocation has already happened.

This allowed us to be proactive and inform clients ahead of time!