Carnegie Mellon University

AddTrust External CA Root Expired May 30, 2020

DAY: Saturday
DATE: May 30, 2020 6:48 AM EDT

WHOM DOES THIS AFFECT?

  • Anyone who administers legacy clients that have not received security updates since before mid 2015 and connect to Secure Sockets Layer (SSL) or Transport Layer Security (TLS) encrypted services such as web and email.
  • Anyone who administers OpenSSL based client software.
  • Anyone who administers Linux or macOS OpenLDAP clients that connect to ldap.cmu.edu or ldap.andrew.cmu.edu.
  • Anyone who administers clients configured to explicitly trust the AddTrust External CA Root instead of relying on an operating system or vendor managed truststore. Common examples include:
    • Dell Remote Access Controller (DRAC) connecting to Andrew Active Directory
    • Java applications that do not use the default truststore
    • Lightweight Directory Access Protocol over SSL (LDAPS)
    • radmind
    • Simple Mail Transfer Protocol Secure (SMTPS) and SMTP with STARTTLS sending email to relay.andrew.cmu.edu or smtp.andrew.cmu.edu
  • Anyone who administers SSL or TLS encrypted services connected to by the above clients.

SUMMARY

Sectigo's legacy AddTrust External CA Root certificate expired on May 30, 2020 at 6:48 AM EDT. Modern clients should largely be unaffected. However, legacy clients, OpenSSL based clients, OpenLDAP clients, and clients configured to explicitly trust the AddTrust root instead of relying on an operating system or vendor managed truststore may need client or server reconfiguration to avoid loss of service.

Devices that received security updates after mid 2015 should have the modern USERTrust RSA Certification Authority root certificate (valid until Jan 2038) in their operating system or browser truststores and largely be unaffected.

Legacy devices that have not received updates to support newer roots will inevitably be missing other essential security updates and support for standards required by the modern Internet. We strongly encourage decommissioning these devices if their software cannot be upgraded. Non-upgraded, legacy devices should never be exposed to the Internet and special mitigations should be applied to isolate them from neighbor systems.

Client software based on OpenSSL prior to version 1.1.1 appear to have broken certificate path validation logic and require workarounds.

OpenLDAP clients on some platforms appear to have broken certificate path validation logic and require workarounds.

Clients configured to explicitly trust the AddTrust root may need client reconfiguration to either: 1) rely on operating system or vendor managed truststores or 2) explicitly trust the USERTrust RSA Certification Authority root or the alternative legacy AAA Certificate Services root (valid until Dec 2028).

TECHNICAL DETAILS

Certificate authorities (CAs) often control multiple root certificates, and generally the older the root the more widely distributed it is on older platforms. Taking advantage of this fact, CAs generate cross certificates to ensure that their certificates are as widely supported as possible. A cross certificate is where one root certificate is used to sign another. The cross certificate uses the same public key and Subject as the root being signed.

Certificate path validation is done client-side from leaf to root. Modern clients that receive Trust Chain A with the cross signed intermediate (see below) from servers should ignore it and instead follow Trust Chain B. This applies even after the root of Trust Chain A expires on May 30, 2020. However, some clients may have problems if one or more of the following conditions is true:

  1. The client is too old and does not have the USERTrust RSA Certification Authority root in its operating system or vendor managed truststore.
  2. The client does not properly process Trust Chain B and instead always tries to follow Trust Chain A.
  3. The client is configured to explicitly trust the AddTrust External CA Root and ignores its operating system or vendor managed truststore.

Conditions 1 and 2 may be addressed by configuring the server to send Trust Chain C.

Condition 3 requires the client to be reconfigured to either: 1) use the operating system or vendor managed truststore or 2) explicitly trust the USERTrust RSA Certification Authority root or the alternative legacy AAA Certificate Services root.

Certificates issued by the Carnegie Mellon Certificate Authority from April 30, 2020 through InCommon (powered by Sectigo) will include a link to download Trust Chain C. Again, we encourage you to use Trust Chain B unless you specifically need Trust Chain C for legacy device compatibility or to work around broken client issues.

Trust Chain A
AddTrust External CA Root [Root] (Exp: May 30, 2020)
USERTrust RSA Certification Authority [Intermediate 2] (Exp: May 30, 2020)
InCommon RSA Server CA [Intermediate 1] (Exp: Oct 5, 2024)
End Entity [Leaf Certificate]

Trust Chain B
USERTrust RSA Certification Authority [Root] (Exp: Jan 18, 2038)
InCommon RSA Server CA [Intermediate 1] (Exp: Oct 5, 2024)
End Entity [Leaf Certificate]

Trust Chain C
AAA Certificate Services [Root] (Exp: Dec 31, 2028)
USERTrust RSA Certification Authority [Intermediate 2] (Exp: Dec 31, 2028)
InCommon RSA Server CA [Intermediate 1] (Exp: Oct 5, 2024)
End Entity [Leaf Certificate]

WHAT YOU NEED TO DO

  1. Evaluate whether you have devices that meet Conditions 1, 2, or 3 by setting a client device clock to after June 1, 2020 and testing connectivity.
  2. Apply the recommended fix for clients and/or servers meeting one or more of the Conditions below.

Condition 1 - Legacy Devices - Known Examples

We strongly encourage decommissioning these legacy devices if their software cannot be upgraded. Non-upgraded, legacy devices should never be exposed to the Internet and special mitigations should be applied to isolate them from neighbor systems. Legacy compatibility may be extended by reconfiguring servers to send Trust Chain C (see above). Contact your server admins to discuss whether that is possible.

  • Apple Mac OS X 10.11 (El Capitan) or earlier
  • Apple iOS 9 or earlier.
  • Google Android 5.0 or earlier.
  • Microsoft Windows Vista & 7 if the Update Root Certificates Feature has been disabled since before June 2010.
  • Microsoft Windows XP if an Automatic Root Update has not been received since before June 2010.
  • Mozilla Firefox 35 or earlier.
  • Oracle Java 8u50 or earlier.
  • Embedded devices (especially copy machines) that have not installed a firmware update since before mid 2015.

Condition 2 - Broken Clients - Known Examples

Reconfigure the server to send Trust Chain B or Trust Chain C and reconfigure the client to use the operating system managed truststore. Click the link for additional configuration information.

  • OpenSSL based client software
Client software that use OpenSSL libraries prior to version 1.1.1 for certificate path validation appear to always validate the full Trust Chain A sent from the server even though modern roots were configured to validate Trust Chain B.

This unfortunate behavior was observed on Red Hat Enterprise Linux 6 (OpenSSL 1.0.1e-fips) and 7 (OpenSSL 1.0.2k-fips). On our test devices, advancing the clock past June 1, 2020 and attempting connections to servers that sent Trust Chain A resulted in failed connections. In these tests, OpenSSL returned expired certificate errors even though Trust Chain B's root was available in the truststores. This behavior appears to be fixed in Red Hat Enterprise Linux 8 (OpenSSL 1.1.1c FIPS) and Ubuntu 14.04 (OpenSSL 1.1.1) as the same clock advancing tests resulted in successful connections and OpenSSL validating properly to Trust Chain B's root.

To workaround this unfortunate behavior, configure the server to send Trust Chain B or Trust Chain C and configure the clients to use the operating system managed truststore or explicitly trust either the USERTrust RSA Certification Authority root or AAA Certificate Services root (depending on what the server sends).

Examples of client programs that use OpenSSL for certificate path validation and were tested include OpenLDAP and postfix.

OpenLDAP clients on Red Hat Enterprise Linux 6 and 7 appear to always validate the full Trust Chain A sent from the server even though modern roots were configured to validate Trust Chain B. Once the client device clock was advanced to June 1, 2020, LDAPS connections failed validation on the expired AddTrust root until our test server was reconfigured to send either Trust Chain B or Trust Chain C.

OpenLDAP clients only enable TLS when either TLS_CACERT or TLS_CACERTDIR is set in ldap.conf. Either set TLS_CACERT to be the operating system managed PEM file or set TLS_CACERTDIR to an empty directory to force fallback to the operating system managed truststore.

Test whether a successful LDAPS connection can be made with:

ldapsearch -H ldaps://ldap.cmu.edu:636 -x '(uid=advisor)'

A successful connection will show query results while failing to negotiate TLS will show:

ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Or display debug info including the names of the chain certificates with:

ldapsearch -d 1 -H ldaps://ldap.cmu.edu:636 -x '(uid=advisor)'

ldap.cmu.edu and ldap.andrew.cmu.edu were reconfigured to send Trust Chain C on Wednesday, May 27, 2020 at 11:00 AM. Please reconfigure your OpenLDAP clients to use the operating system managed truststore if you are experiencing query failures.

Configure the server to send Trust Chain B or Trust Chain C and configure the clients to use the operating system managed truststore or explicitly trust either the USERTrust RSA Certification Authority root or AAA Certificate Services root (depending on what the server sends).

See Condition 3 below for client configuration details.

Condition 3 - Explicit CA Trust - Known Examples

Reconfigure the client to use the operating system or vendor managed truststore if possible. If not, reconfigure the client to explicitly trust either the USERTrust RSA Certification Authority root or AAA Certificate Services root (depending on what the server sends). Click the links for additional configuration information.

Upload the USERTrust RSA Certification Authority root (see link below) to DRACs that connect to Andrew Active Directory.
Reconfigure the Java application to use the default truststore that is included with the JDK/JRE for client SSL/TLS connections. See vendor documentation for more information.
  • Lightweight Directory Access Protocol over SSL (LDAPS) clients
See vendor documentation for more information.

ldap.cmu.edu and ldap.andrew.cmu.edu were reconfigured to send Trust Chain C on Wednesday, May 27, 2020 at 11:00 AM. Please reconfigure your OpenLDAP clients to use the operating system managed truststore if you are experiencing query failures.
Update the CA's certificate on radmind clients to replace the AddTrust External CA Root with the USERTrust RSA Certification Authority root
  • Simple Mail Transfer Protocol Secure (SMTPS) and SMTP with STARTTLS sending email to relay.andrew.cmu.edu or smtp.andrew.cmu.edu
    • Postfix
      Set smtp_tls_CAfile to the operating system managed truststore (e.g. for RHEL7 /etc/pki/tls/certs/ca-bundle.crt)
    • Sendmail
      Set confCACERT_PATH to the operating system managed truststore (e.g. for RHEL7 /etc/pki/tls/certs)

Downloads

Trust Chain B (Modern, Preferred) - PEM encoded
Root - USERTrust RSA Certification Authority (Exp: Jan 18, 2038)
Intermediate - InCommon RSA Server CA (Exp: Oct 5, 2024)*
Intermediate and Root

Trust Chain C (Legacy compatibility) - PEM encoded
Root - AAA Certificate Services (Exp: Dec 31, 2028)
Intermediate 2 - USERTrust RSA Certification Authority (Exp: Dec 31, 2028)
Intermediate 1 - InCommon RSA Server CA (Exp: Oct 5, 2024)
Intermediates*
Intermediates and Root

* When configuring the trust chain sent by a server, only the intermediate(s) and end entity certificates are required. Sending the root certificate is optional as the client must already have the root in its truststore. Please use the "Intermediate" or "Intermediates" downloads above instead of the "Root/Intermediate(s) only" or "Intermediate(s)/Root only" links in the InCommon (Sectigo) Enrollment Successful emails.

Testing

  • Qualys SSL Labs SSL Server Test
    Cloud based tool to view the full certificate trust chain sent by an Internet facing web server (TCP 443 only) and test it for TLS/SSL vulnerabilities.
  • SSL Shopper SSL Checker
    Cloud based tool to view the full certificate trust chain sent by an Internet facing server (any TCP port) that doesn't require STARTTLS.
  • Openssl s_client
    Test any SSL, TLS, or STARTTLS enabled service you can access.

    SSL/TLS Connection:


    openssl s_client -connect www.andrew.cmu.edu:443


    STARTTLS Connection:


    openssl s_client -connect relay.andrew.cmu.edu:25 -starttls smtp


    Print the PEM encoded trust chain certificates


    openssl s_client -showcerts -connect www.andrew.cmu.edu:443

MORE INFORMATION

Sectigo AddTrust External CA Root Expiring May 30, 2020

CONTACT

Please direct any questions or feedback to the Carnegie Mellon Certificate Authority (certificate-authority@andrew.cmu.edu)