Resolving the AddTrust External CA Root certificate expiration

Some of our users have received reports about their AddTrust External CA Root or USERTrust RSA Certification Authority certificate. The problem occurs because the remote server sends a root certificate in the chain that will expire in less than 14 days.

Here are the steps to verify this and a few tips on how to resolve it.

What are the AddTrust External CA Root expiration notifications?

Oh Dear checks all the certificates your server sends back to us whenever we connect to it.

Sometimes we just get 1 certificate back, sometimes we receive an entire chain of certificates (this is usually the correct thing to do, minus the root certificate).

Sometimes, we receive certificates where - in the middle of the chain - an expired certificate is present. We alert on these, as clients might block connections when one certificate in the chain is expired.

Sometimes, and it's rare, a server sends a root certificate along that is close to expiry, but actually isn't needed.

For some of our users, they've received these reports for the AddTrust External CA Root and USERTrust RSA Certification Authority root certificates.

Verify that the SSL certificates are indeed about to expire

It's a bit technical, so if this doesn't make a whole lot of sense, we suggest you reach out to your hosting provider or your SSL Certificate provider - they'll be able to help out!

Forward them this post, and they'll be able to fix things for you.

In this example, we'll connect to a random Tumblr blog and request the certificates. Tumblr appears to be one of the larger providers worldwide that's sending a soon-to-expire root certificate along in their chain.

Update: they since removed the old expiring root from their chain.

$ openssl s_client -showcerts -connect world-of-cats.tumblr.com:443

CONNECTED(00000006)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = *.tumblr.com
verify return:1
---
Certificate chain
 0 s:/CN=*.tumblr.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
-----BEGIN CERTIFICATE-----
MIIGpzCCBY+gAwIBAgIRAOsw1/2DvyzYHRF0zq8c9xQwDQYJKoZIhvcNAQELBQAw
gY8xCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE3MDUGA1UE
AxMuU2VjdGlnbyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBD
QTAeFw0yMDAzMjYwMDAwMDBaFw0yMjA2MjgwMDAwMDBaMBcxFTATBgNVBAMMDCou
dHVtYmxyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPe+MXhf
4lZYnrsY5Ch1L0MqHOc44hIHkmawVVyTA8CynfbbFJWW+3Uoy0tHTcRgXyaV9xU3
oFyaIyYsLEUzfQLLmCuUjs8zSYvDH0jioCAz2HnkBaSuRAeOL2Iuaa9RoUVPrm8H
TwMwNQEjTJXM9SdSDzK8fS78jglTpzy+1CvWXjo9ij4+hiz2UjRkntA1oKXtOgnm
7W63CwD+fs9cW4VJaehjIAXf8AM/vd/WTrsDmr2ne17D05Lg7UgIJJaAo7JkF7bt
v3Be/BEYQTo/Eo0Ao9mqbt0DotNRn3lyh1y6MMhU7Hbr/qQzq1+cyEr+d0yd/ugp
P4c3HKfxD21+6PUCAwEAAaOCA3MwggNvMB8GA1UdIwQYMBaAFI2MXsRUrYrhd+mb
+ZsF4bgBjWHhMB0GA1UdDgQWBBR7eUPnbT0TNZdZJ9sYGcIV+LV+zzAOBgNVHQ8B
Af8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB
BQUHAwIwSQYDVR0gBEIwQDA0BgsrBgEEAbIxAQICBzAlMCMGCCsGAQUFBwIBFhdo
dHRwczovL3NlY3RpZ28uY29tL0NQUzAIBgZngQwBAgEwgYQGCCsGAQUFBwEBBHgw
djBPBggrBgEFBQcwAoZDaHR0cDovL2NydC5zZWN0aWdvLmNvbS9TZWN0aWdvUlNB
RG9tYWluVmFsaWRhdGlvblNlY3VyZVNlcnZlckNBLmNydDAjBggrBgEFBQcwAYYX
aHR0cDovL29jc3Auc2VjdGlnby5jb20wIwYDVR0RBBwwGoIMKi50dW1ibHIuY29t
ggp0dW1ibHIuY29tMIIB9wYKKwYBBAHWeQIEAgSCAecEggHjAeEAdgBGpVXrdfqR
IDC1oolp9PN9ESxBdL79SbiFq/L8cP5tRwAAAXEYg7bxAAAEAwBHMEUCIC/2w8Js
j5l7v6HXiXF3xmZtlnP24wVQyCbuZog5CR4LAiEA94tol2Wv9CfY5+oZOZbguyby
+2GjsTF/Kt6VtkYfu7EAdgDfpV6raIJPH2yt7rhfTj5a6s2iEqRqXo47EsAgRFwq
cwAAAXEYg7a6AAAEAwBHMEUCIEz+QPJzXAcJ+DO/vY35zaJyFO79tb0YGqxIdMK8
QmUxAiEAuMC/Pb9ASdjVA+1V9XRze3+FOuzYgDIukcloJkFTBQIAdwBByMqx3yJG
ShDGoToJQodeTjGLGwPr60vHaPCQYpYG9gAAAXEYg7beAAAEAwBIMEYCIQCRQNow
KwOkT83uWKunFDLxYKvelZx3iDVN4XnT7QSyQAIhAPqkCy4vgInpPdm6vlIVU5w5
HhBlRa5yypT2soaqe3MHAHYAb1N2rDHwMRnYmQCkURX/dxUcEdkCwQApBo2yCJo3
2RMAAAFxGIO2jAAABAMARzBFAiEA02T++BnEL2AifEhiThLu+o9o1rL2gHVRu4qv
gtaVNTkCIEuIiz0vVU4rWmvm2qOqlMer4UzeUq0FDmn6L2ib4uMEMA0GCSqGSIb3
DQEBCwUAA4IBAQC1S8CwUg1o7Nek9AteJHycWe54Yk/kSRN8VFax8AqbWWFF9528
20VpJ/XFFacju1mj7cdEPGLHVJ80Ia3D+1YT7c2OaFa/SI3BVs05BRKmjvxHR7ZM
W7RNZK+8qmMzh2mEjj1LPWezioxec5KR90LjSGIaG3KnWHpQhGSwC3AFzkmrF6Pj
S8z3gfSbuWDicMwEWmiW6Gjy1Xe6jg/DJn8NKwLw5ju+17oyKR7BxQ3nfFltAWSL
qHzYRzzARUalZIQNvYEYNvqcA3rsBnpRLG5N4OCsV/VE/cDDQX4XrOGVdvj4fQlf
QWOOZkkAxvC6iU63He3vFdE1HrmuTqQYTLoS
-----END CERTIFICATE-----
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
   i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
-----BEGIN CERTIFICATE-----
MIIGEzCCA/ugAwIBAgIQfVtRJrR2uhHbdBYLvFMNpzANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTgx
MTAyMDAwMDAwWhcNMzAxMjMxMjM1OTU5WjCBjzELMAkGA1UEBhMCR0IxGzAZBgNV
BAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEYMBYGA1UE
ChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQDEy5TZWN0aWdvIFJTQSBEb21haW4g
VmFsaWRhdGlvbiBTZWN1cmUgU2VydmVyIENBMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA1nMz1tc8INAA0hdFuNY+B6I/x0HuMjDJsGz99J/LEpgPLT+N
TQEMgg8Xf2Iu6bhIefsWg06t1zIlk7cHv7lQP6lMw0Aq6Tn/2YHKHxYyQdqAJrkj
eocgHuP/IJo8lURvh3UGkEC0MpMWCRAIIz7S3YcPb11RFGoKacVPAXJpz9OTTG0E
oKMbgn6xmrntxZ7FN3ifmgg0+1YuWMQJDgZkW7w33PGfKGioVrCSo1yfu4iYCBsk
Haswha6vsC6eep3BwEIc4gLw6uBK0u+QDrTBQBbwb4VCSmT3pDCg/r8uoydajotY
uK3DGReEY+1vVv2Dy2A0xHS+5p3b4eTlygxfFQIDAQABo4IBbjCCAWowHwYDVR0j
BBgwFoAUU3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFI2MXsRUrYrhd+mb
+ZsF4bgBjWHhMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAbBgNVHSAEFDASMAYGBFUdIAAw
CAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNlcnRydXN0
LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNybDB2Bggr
BgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKGM2h0dHA6Ly9jcnQudXNlcnRydXN0LmNv
bS9VU0VSVHJ1c3RSU0FBZGRUcnVzdENBLmNydDAlBggrBgEFBQcwAYYZaHR0cDov
L29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0BAQwFAAOCAgEAMr9hvQ5Iw0/H
ukdN+Jx4GQHcEx2Ab/zDcLRSmjEzmldS+zGea6TvVKqJjUAXaPgREHzSyrHxVYbH
7rM2kYb2OVG/Rr8PoLq0935JxCo2F57kaDl6r5ROVm+yezu/Coa9zcV3HAO4OLGi
H19+24rcRki2aArPsrW04jTkZ6k4Zgle0rj8nSg6F0AnwnJOKf0hPHzPE/uWLMUx
RP0T7dWbqWlod3zu4f+k+TY4CFM5ooQ0nBnzvg6s1SQ36yOoeNDT5++SR2RiOSLv
xvcRviKFxmZEJCaOEDKNyJOuB56DPi/Z+fVGjmO+wea03KbNIaiGCpXZLoUmGv38
sbZXQm2V0TP2ORQGgkE49Y9Y3IBbpNV9lXj9p5v//cWoaasm56ekBYdbqbe4oyAL
l6lFhd2zi+WJN44pDfwGF/Y4QA5C5BIG+3vzxhFoYt/jmPQT2BVPi7Fp2RBgvGQq
6jG35LWjOhSbJuMLe/0CjraZwTiXWTb2qHSihrZe68Zk6s+go/lunrotEbaGmAhY
LcmsJWTyXnW0OMGuf1pGg+pRyrbxmRE1a6Vqe8YAsOf4vmSyrcjC8azjUeqkk+B5
yOGBQMkKW+ESPMFgKuOXwIlCypTPRpgSabuY0MLTDXJLR27lk8QyKGOHQ+SwMj4K
00u/I5sUKUErmgQfky3xxzlIPK1aEn8=
-----END CERTIFICATE-----
 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
-----BEGIN CERTIFICATE-----
MIIFdzCCBF+gAwIBAgIQE+oocFv07O0MNmMJgGFDNjANBgkqhkiG9w0BAQwFADBv
MQswCQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFk
ZFRydXN0IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBF
eHRlcm5hbCBDQSBSb290MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFow
gYgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtK
ZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMS4wLAYD
VQQDEyVVU0VSVHJ1c3QgUlNBIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIICIjAN
BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgBJlFzYOw9sIs9CsVw127c0n00yt
UINh4qogTQktZAnczomfzD2p7PbPwdzx07HWezcoEStH2jnGvDoZtF+mvX2do2NC
tnbyqTsrkfjib9DsFiCQCT7i6HTJGLSR1GJk23+jBvGIGGqQIjy8/hPwhxR79uQf
jtTkUcYRZ0YIUcuGFFQ/vDP+fmyc/xadGL1RjjWmp2bIcmfbIWax1Jt4A8BQOujM
8Ny8nkz+rwWWNR9XWrf/zvk9tyy29lTdyOcSOk2uTIq3XJq0tyA9yn8iNK5+O2hm
AUTnAU5GU5szYPeUvlM3kHND8zLDU+/bqv50TmnHa4xgk97Exwzf4TKuzJM7UXiV
Z4vuPVb+DNBpDxsP8yUmazNt925H+nND5X4OpWaxKXwyhGNVicQNwZNUMBkTrNN9
N6frXTpsNVzbQdcS2qlJC9/YgIoJk2KOtWbPJYjNhLixP6Q5D9kCnusSTJV882sF
qV4Wg8y4Z+LoE53MW4LTTLPtW//e5XOsIzstAL81VXQJSdhJWBp/kjbmUZIO8yZ9
HE0XvMnsQybQv0FfQKlERPSZ51eHnlAfV1SoPv10Yy+xUGUJ5lhCLkMaTLTwJUdZ
+gQek9QmRkpQgbLevni3/GcV4clXhB4PY9bpYrrWX1Uu6lzGKAgEJTm4Diup8kyX
HAc/DVL17e8vgg8CAwEAAaOB9DCB8TAfBgNVHSMEGDAWgBStvZh6NLQm9/rEJlTv
A73gJMtUGjAdBgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/
BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wEQYDVR0gBAowCDAGBgRVHSAAMEQGA1Ud
HwQ9MDswOaA3oDWGM2h0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9BZGRUcnVzdEV4
dGVybmFsQ0FSb290LmNybDA1BggrBgEFBQcBAQQpMCcwJQYIKwYBBQUHMAGGGWh0
dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEMBQADggEBAJNl9jeD
lQ9ew4IcH9Z35zyKwKoJ8OkLJvHgwmp1ocd5yblSYMgpEg7wrQPWCcR23+WmgZWn
RtqCV6mVksW2jwMibDN3wXsyF24HzloUQToFJBv2FAY7qCUkDrvMKnXduXBBP3zQ
YzYhBx9G/2CkkeFnvN4ffhkUyWNnkepnB2u0j4vAbkN9w6GAbLIevFOFfdyQoaS8
Le9Gclc1Bb+7RrtubTeZtv8jkpHGbkD4jylW6l/VXxRTrPBPYer3IsynVgviuDQf
Jtl7GQVoP7o81DgGotPmjw7jtHFtQELFhLRAlSv0ZaBIefYdgWOWnU914Ph85I6p
0fKtirOMxyHNwu8=
-----END CERTIFICATE-----

That's a lot of text right there!

The very last certificate is the AddTrust External CA Root certificate. This is the one that's causing a bit of problems at the moment. If we decode that blob of text, we can see why.

To decode a certificate, copy/paste the certificate between the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- (including those lines) and save it to a text file. It should look a little something like this:

We named our text file certificate.crt.

$ openssl x509 -in certificate.crt -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            13:ea:28:70:5b:f4:ec:ed:0c:36:63:09:80:61:43:36
    Signature Algorithm: sha384WithRSAEncryption
        Issuer: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
        Subject: C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

[...]

That particular certificate expires on May 30 10:48:38 2020 GMT. In other words, in just about 14 days.

Validating the SSL Certificate Path

There are several paths possible to validate the certificate of .tumblr.com. One of them doesn't even require the AddTrust External CA Root certificate:

1	Sent by server	*.tumblr.com
Fingerprint SHA256: 3b46c48112e902c99d6f6ece3dd4877b190936e51289c90c874e219cf0494cd2
Pin SHA256: uSU/pyBXHivUNGcwZD+1TTSBYu6Q4n3GlvZTctoDmdQ=
RSA 2048 bits (e 65537) / SHA256withRSA

2	Sent by server	Sectigo RSA Domain Validation Secure Server CA
Fingerprint SHA256: 7fa4ff68ec04a99d7528d5085f94907f4d1dd1c5381bacdc832ed5c960214676
Pin SHA256: 4a6cPehI7OG6cuDZka5NDZ7FR8a60d3auda+sKfg4Ng=
RSA 2048 bits (e 65537) / SHA384withRSA

3	In trust store	USERTrust RSA Certification Authority   Self-signed
Fingerprint SHA256: e793c9b02fd8aa13e21c31228accb08119643b749c898964b1746d46c3d4cbd2
Pin SHA256: x4QzPSC810K5/cMjb05Qm4k3Bw5zBn4lTdO/nEW/Td4=
RSA 4096 bits (e 65537) / SHA384withRSA

Since that soon-to-expire root certificate that is being sent along isn't actually needed, it should be safe to remove it from your intermediate certificate list.

Or, perhaps even better, replace it with an up-to-date one that is valid for your certificate chain.

Replace or remove the old root-certificate in your chain

It's best to doublecheck this with your SSL Provider, to verify the best course of action here.

If you are in control of your own webserver/proxy/SSL setups, you should be able to find the following certificate somewhere in your intermediate certificate list, and remove it.

This file is usually referenced in your webserver configs, it might look like this:

In Nginx:

	[...]
	 ssl_certificate             /path/to/fullchain.pem;

In Apache:

	[...]
	SSLCertificateChainFile      /path/to/fullchain.pem;

Open that file, take a back-up, and remove the certificate referenced above here.

Restart your webserver to load the new certificate configurations, and doublecheck if everything still works properly.

Why does Oh Dear report on these certificates?

We verify every certificate that gets sent by the server. In this case, the final root certificate that was being sent isn't technically needed to validate the certificate chain, as there's a local root certificate present (on your own device) that perfectly does that already.

It's unclear how every device in the wild would react if a server sends along an expired, but ultimately unneeded, root certificate.

How would an old Android phone react? Or an embedded device, running old firmware? We can't know, so we prefer to err on the side of caution and alert you that the server is sending along expiring certificates.

Hopefully this post can help you identify the problem and roll out a solution!

Update: we will modify our alerting settings

After internal debates, we've decided to make the behaviour of these alerts configurable.

To be clear: the server should not send an expired root certificate back to the client. It's impossible to predict how old devices might respond, and it'll surely break some embedded devices or devices with older SSL validation logic.

However, modern browsers treat this as a non-issue, since they can find a different path to validate the certificate and tie it to a valid root certificate.

In one of our next releases, you will be able to select if we should validate all certificates a server sends, or just the domain certificate. The default will be to validate all certificates, as we've always done.

In some scenario's, it's difficult or near impossible to change the certificate chain (ie: shared hosting setups that offer little to no control of the certificates). For those scenario's, you might want to disable the validation of all certificates (even though it might cause issues for some clients).

Resolving the AddTrust External CA Root certificate expiration

What are the AddTrust External CA Root expiration notifications?

Verify that the SSL certificates are indeed about to expire

Validating the SSL Certificate Path

Replace or remove the old root-certificate in your chain

Why does Oh Dear report on these certificates?

Update: we will modify our alerting settings

More updates

Every second counts in our UI

Improving our Multi-Factor Authentication

Reducing PHP's CPU usage by nearly 40% by upgrading from Ubuntu 20.04 to 22.04

Want to get started? We offer a no-strings-attached 10 day trial. No credit card required.