Changes to Managed TLS certificate chains in response to Let's Encrypt root certificate expiration

πŸ“˜

Note

This change only affects Endpoints using Managed TLS.

On September 30th, 2021 Let's Encrypt's old root certificate expired prompting the need for a new root certificate. Some legacy devices do not trust the new root certificate so Let's Encrypt decided to implement multiple certificate chains using both the old and new root certificates in an attempt to prolong support for these devices. However, this created issues for some older SSL clients even though they trusted the new root certificate.

In response to this development, the default certificate chain used by Managed TLS Endpoints was changed from Let's Encrypt's default chain to their alternate chain. This ensures that any client that trusts the new root certificate will trust the Endpoint's certificate without having to make any changes on the client's side.

If you have a need for the default chain you can configure your App with APTIBLE_DISABLE_ALTERNATE_CHAIN=true and the Managed TLS Endpoints for the App's Services will be issued certificates using the default chain when provisioned or renewed. The aptible endpoints:renew CLI command can be used to issue a new certificate for your Endpoint immediately but be aware of the rate limiting that Let's Encrypt applies to issuing certificates. Specifically, a certificate can only be renewed 5 times in a week.

Renewals are treated specially: they don’t count against your Certificates per Registered Domain limit, but they are subject to a Duplicate Certificate limit of 5 per week.

We have created two Endpoints, one for each certificate chain, that you can use to test against both certificate chains without having to update your Endpoints:

Background

This image describes the changes to the certificate chain that Let's Encrypt uses to issue certificates.

The new default chain works with old clients that continue to trust the old root certificate despite the fact that it has expired. This mainly consists of devices running Android < 7.1.1. The default chain also works with newer clients that don't trust the expired root certificate but do trust the new root certificate and can use the alternate trust chain to verify the Endpoint's certificate. This includes modern web browsers and openssl > 1.0.2.

The main issue that many users have encountered with Let's Encrypt's new default chain is that they, or their users, are connecting to their applications using older SSL clients, such as openssl <= 1.0.2, that cannot handle multiple trust chains. So, despite the fact that the new root certificate is trusted, the client will only check the default chain containing the expired, untrusted root certificate and, therefore, won't trust the Endpoint's certificate.

OpenSSL published a blog post that describes the issue and provides a few workarounds for older versions of openssl. Using only the alternate chain, the new default for Managed TLS Endpoints, is workaround 3 in that list. It guarantees that any client that trusts the new certificate will trust the server's certificate without having to make any changes on the client's side. The tradeoff is that legacy Android devices won't trust the server's certificate, though some modern browsers that run on them will.

Conclusion

The response we received from many Aptible users is that they preferred the alternate chain as more of their users were using older SSL clients than were using legacy Android devices. This combined with the fact that debugging and addressing issues with Let's Encrypt's alternate chain is much more straightforward than that of their default chain ultimately led to the decision to change the default chain used by Managed TLS Endpoints.