Background

Behind every secure Internet connection is a TLS certificate. But despite their ubiquity, certificates present challenges. They can cost thousands of dollars, they require proof of domain ownership, and they expire. And when certificates expire, services go down.

The Centers for Medicare and Medicaid Services (CMS) faced all of these challenges. With dozens of services maintained by various contractors, CMS struggled to stay on top of certificate expirations and suffered several service outages. The price of certificates and manual work of renewing them were secondary but important issues.

Working with CMS staff, Corbalt designed and implemented an integrated set of systems for certificate management. The outcome for CMS was a dramatic reduction in costs and the elimination of a class of outages.

Let's Encrypt

To address both cost and reliability, Corbalt recommended that CMS switch from traditional certificate authorities to Let's Encrypt, a widely used, non-profit authority that offers free certificates and tools for automated certificate management.

While Let's Encrypt solved the cost problem, it presented an issue with renewals. Each time a certificate is renewed, Let's Encrypt, like many certificate authorities, requires the user to prove domain ownership by writing to the DNS. But at CMS, DNS is managed centrally—contractors can't create DNS records. In addition, CMS's DNS provider, Akamai, doesn't support the fine-grained permissions that would allow central IT to delegate management of certificate-related DNS records.

Let's Encrypt provided the cost savings and automation tools that CMS needed, but CMS would need a way to enhance Akamai's offering.

Akamai DNS Proxy

Corbalt designed and implemented a solution—a fine-grained authorization layer in front of Akamai. This system, Akamai DNS Proxy (ADP), receives Akamai DNS requests, enforces its own permissions, and forwards the requests to the real Akamai back-end. Because it presents the standard Akamai API, ADP maintains compatibility with official Akamai tools and libraries, enabling users to renew certificates without corrupting the DNS.

Partial Automation with manage-certs

Let's Encrypt offers a utility—certbot—that helps with automatic renewals. However, when certificates are installed on AWS load balancers, handles only part of the process. A human must periodically check for expiring certificates, run certbot, and upload the generated files.

Because CMS extensively uses AWS, Corbalt created a tool, manage-certs, to simplify this process. Using certbot, manage-certs scans AWS load balacers and renews certificates with a single command, all but eliminating manual work.

Full Automation

To support full automation, manage-certs was designed as a command-line app that fits into continuous integration (CI) workflows. A CI system can periodically run manage-certs, bringing certificates on certain load balancers up to date without human intervention. Credentials are stored securely in the CI system.

Corbalt produced a sample pipeline for CMS's CI system. Teams responsible for managing certificates copy the sample, set application-specific variables, and schedule the pipeline to run periodically. From then on, their certificates are renewed automatically and free of charge.

Conclusion

In the months after CMS adopted Let's Encrypt, Akamai DNS Proxy, and manage-certs, numerous teams switched from traditional certificate authorities and workflows. Certificate costs were dramatically reduced, and certificate expiration ceased to be a risk for many of CMS's critical services.

Postscript: AWS Certificate Manager

After a year of using Let's Encrypt and ADP, CMS approved AWS Certificate Manager (ACM), an official AWS service that automatically renews certificates without DNS edits. Corbalt piloted ACM for CMS and determined that, compared with Let's Encrypt and ADP, ACM is simpler and just as cost-effective. We are now helping teams move to ACM from both traditional certificate authorities and Let's Encrypt.