{
    "version" : "https://jsonfeed.org/version/1",
    "content" : "news",
    "type" : "single",
    "title" : "Let’s Encrypt Those CNAMES, Shall We? |Digital.gov",
    "description": "Let’s Encrypt Those CNAMES, Shall We?",
    "home_page_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/","feed_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/07/lets-encrypt-those-cnames-shall-we/index.json","item" : [
    {"title" :"Let’s Encrypt Those CNAMES, Shall We?","summary" : "This is post 4 in the 5-part series, The Right Tools for the Job: Re-Hosting DigitalGov Search to a Dynamic Infrastructure Environment. This post references the previous posts frequently, so please read those before reading this one if you haven’t done so already. In addition to the DNS challenges created by offering “masked” domains such","date" : "2016-09-07T11:00:56-04:00","date_modified" : "2025-01-27T19:42:55-05:00","authors" : {"nathan-smith" : "Nathan Smith","dmccleskey" : "Dawn Pointer McCleskey"},"topics" : {
        
            "cloud-and-infrastructure" : "Cloud and infrastructure",
            "product-and-project-management" : "Product and project management",
            "search" : "Search"
            },"branch" : "bc-archive-content-3",
      "filename" :"2016-09-07-lets-encrypt-those-cnames-shall-we.md",
      
      "filepath" :"news/2016/09/2016-09-07-lets-encrypt-those-cnames-shall-we.md",
      "filepathURL" :"https://github.com/GSA/digitalgov.gov/blob/bc-archive-content-3/content/news/2016/09/2016-09-07-lets-encrypt-those-cnames-shall-we.md",
      "editpathURL" :"https://github.com/GSA/digitalgov.gov/edit/bc-archive-content-3/content/news/2016/09/2016-09-07-lets-encrypt-those-cnames-shall-we.md","slug" : "lets-encrypt-those-cnames-shall-we","url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/07/lets-encrypt-those-cnames-shall-we/","content" :"\u003cp\u003e\u003cem\u003eThis is post 4 in the 5-part series, \u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/08/18/the-right-tools-for-the-job-re-hosting-digitalgov-search-to-a-dynamic-infrastructure-environment/\"\u003eThe Right Tools for the Job: Re-Hosting DigitalGov Search to a Dynamic Infrastructure Environment\u003c/a\u003e. This post references the previous posts frequently, so please read those before reading this one if you haven’t done so already.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eIn addition to the DNS challenges created by offering “masked” domains such as \u003ctt\u003enasasearch.nasa.gov\u003c/tt\u003e, we also had to solve the problem of how to maintain SSL certificates for the main \u003ctt\u003esearch.usa.gov\u003c/tt\u003e domain along with the “masked” domains of all customers that wanted HTTPS support for their own domains. As also noted in an \u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/08/18/the-right-tools-for-the-job-re-hosting-digitalgov-search-to-a-dynamic-infrastructure-environment/\"\u003eearlier post\u003c/a\u003e, this all needed to be done in a multi-app-server environment with no interruption of service. \u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2016/08/600-x-400-Digital-Encryption-Lock-peterscode-iStock-Thinkstock-465159645.jpg\"\n    alt=\"Digital Encryption Lock\"/\u003e\u003c/div\u003e\n\n\u003c/p\u003e\n\u003ch2 id=\"san-ssl-certificates-and-let8217s-encrypt\"\u003eSAN SSL Certificates and Let’s Encrypt\u003c/h2\u003e\n\u003cp\u003eWe knew we wanted to make use of a \u003ca href=\"https://tools.ietf.org/html/rfc5280#section-4.2.1.6\"\u003emulti-domain Subject Alternative Name\u003c/a\u003e (SAN) SSL certificate, but the prospect of wrangling all the permissions and documentation required for authorizing such a certificate seemed quite daunting. The previous process for adding a new domain to our SSL certificate often took weeks and involved coordinating activities between many parties: the owner of the domain, the hosting provider, the certificate authority, and us:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDomain owner asks us to add their domain to our certificate\u003c/li\u003e\n\u003cli\u003eWe send the request to the hosting provider\u003c/li\u003e\n\u003cli\u003eThe hosting provider sends the request to the certificate authority\u003c/li\u003e\n\u003cli\u003eThe certificate authority performs a domain validation process which is opaque to us\u003c/li\u003e\n\u003cli\u003eThe certificate authority gives the green light to the hosting provider\u003c/li\u003e\n\u003cli\u003eThe hosting provider adds the new domain to our SAN certificate and notifies us\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2016/08/527-x-380-ssl%5c_certificate%5c_process_before.jpg\"\n    alt=\"Diagram of the previous SSL certificate process.\"/\u003e\u003c/div\u003e\n\n\n\u003cp\u003e\u003ca href=\"https://letsencrypt.org/\"\u003eLet’s Encrypt\u003c/a\u003e offers an automated process for obtaining and renewing SSL certificates — including SAN SSL certificates for up to 100 domains — as long as you can prove to them that you control any domain that’s included in a requested certificate. Given that we wanted to be able to quickly offer HTTPS support to new or existing “masked” domain customers, this automation seemed very promising. The fact that Let’s Encrypt is a free service made it all the more compelling. So we set ourselves about the task of making sure we could always prove our ownership of the \u003ctt\u003esearch.usa.gov\u003c/tt\u003e domain as well as “masked” customer domains while keeping the site available for customer use at all times.\u003c/p\u003e\n\u003ch2 id=\"the-acme-protocol-and-dns-vs-http-domain-validation\"\u003eThe ACME Protocol and DNS vs. HTTP Domain Validation\u003c/h2\u003e\n\u003cp\u003eThe process of requesting an SSL certificate from Let’s Encrypt involves running a Let’s Encrypt agent locally which communicates with the Let’s Encrypt Certificate Authority (CA) using the \u003ca href=\"https://github.com/letsencrypt/acme-spec\"\u003eACME Protocol\u003c/a\u003e. The CA issues challenges that the agent attempts to respond to, and the challenges come in the user’s choice of two forms:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDNS challenge: the agent creates a DNS record for a particular sub-domain requested by the CA in order to prove that it has control over DNS for that domain\u003c/li\u003e\n\u003cli\u003eHTTP challenge: the agent creates a text file at a particular location with particular content that the CA can then request in order to prove that the agent has control over web page content for that domain\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSince we cannot create DNS records for our customers’ “masked” domains and it’s not easy even to publish \u003ctt\u003esearch.usa.gov\u003c/tt\u003e DNS records, that left HTTP as the only Domain Validation option for us. Our goal then was to make sure we could have the Let’s Encrypt agent create web resources as needed to respond to any HTTP challenge issued by the CA.\u003c/p\u003e\n\u003ch2 id=\"certbot\"\u003eCertbot\u003c/h2\u003e\n\u003cp\u003eWe chose EFF’s Certbot and followed \u003ca href=\"https://certbot.eff.org/all-instructions/\"\u003etheir straightforward installation instructions\u003c/a\u003e. Certbot has options that will instruct it to attempt to install newly acquired SSL certificates in an Apache or Nginx setup, but we chose to use the \u003ctt\u003ecertonly\u003c/tt\u003e mode of operation which tells Certbot simply to store the new certificates in an output directory. We use this in conjunction with the \u003ctt\u003e\u0026ndash;webroot\u003c/tt\u003e option which tells Certbot that it will be writing files somewhere underneath the webserver’s document root so that they can be publicly available via HTTP requests.\u003c/p\u003e\n\u003cp\u003eTo ask Let’s Encrypt for a SAN SSL certificate for our primary domain \u003ctt\u003esearch.usa.gov\u003c/tt\u003e and, say, two of our customer domains: \u003ctt\u003efind.irs.gov\u003c/tt\u003e, and \u003ctt\u003enasasearch.nasa.gov\u003c/tt\u003e, first we would make sure that our webserver is responding to requests for all three of those domains. Then we would use the following Certbot command from the directory where Certbot is installed:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e\u003cbr /\u003e ./certbot-auto certonly \u0026ndash;webroot \u0026ndash;webroot-path /var/www/http \u0026ndash;domain search.usa.gov \u0026ndash;domain find.irs.gov \u0026ndash;domain nasasearch.nasa.gov\u003cbr /\u003e \u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eRoughly following the \u003ca href=\"https://letsencrypt.org/how-it-works/\"\u003eDomain Validation example\u003c/a\u003e in the Let’s Encrypt documentation, the process of fulfilling this certificate request would complete the two steps illustrated in these diagrams for each domain.\u003c/p\u003e\n\u003ch3 id=\"challenge-hahahugoshortcode2980s4hbhb\"\u003eChallenge \u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2016/08/600-x-143-Lets-Encrypt-howitworks_challenge.jpg\"\n    alt=\"challenge\"/\u003e\u003c/div\u003e\n\n\u003c/h3\u003e\n\u003ch3 id=\"authorization-hahahugoshortcode2980s5hbhb\"\u003eAuthorization \u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2016/08/600-x-380-Lets-encrypt-howitworks_authorization.jpg\"\n    alt=\"authorization\"/\u003e\u003c/div\u003e\n\n\u003c/h3\u003e\n\u003ch3 id=\"challenge--authorization-for-each-domain\"\u003eChallenge / Authorization for each Domain\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eCertbot communicates with the Let’s Encrypt CA indicating we want a SAN SSL certificate for domains \u003ctt\u003esearch.usa.gov\u003c/tt\u003e, \u003ctt\u003efind.irs.gov\u003c/tt\u003e, and \u003ctt\u003enasasearch.nasa.gov\u003c/tt\u003e\u003c/li\u003e\n\u003cli\u003eThe CA instructs the agent to create a file with particular content that will be visible at \u003ctt\u003e\u003ca href=\"http://search.usa.gov/.well-known/acme-challenge/XXXX\"\u003ehttp://search.usa.gov/.well-known/acme-challenge/XXXX\u003c/a\u003e\u003c/tt\u003e\u003c/li\u003e\n\u003cli\u003eThe agent writes the desired content to \u003ctt\u003e/var/www/http/.well-known/acme-challenge/XXXX\u003c/tt\u003e, and the CA’s HTTP request for that file succeeds\u003c/li\u003e\n\u003cli\u003eThe CA instructs the agent to create a file with particular content that will be visible at \u003ctt\u003e\u003ca href=\"http://find.irs.gov/.well-known/acme-challenge/YYYY\"\u003ehttp://find.irs.gov/.well-known/acme-challenge/YYYY\u003c/a\u003e\u003c/tt\u003e\u003c/li\u003e\n\u003cli\u003eThe agent writes the desired content to \u003ctt\u003e/var/www/http/.well-known/acme-challenge/YYYY\u003c/tt\u003e, and the CA’s HTTP request for that file succeeds\u003c/li\u003e\n\u003cli\u003eThe CA instructs the agent to create a file with particular content that will be visible at \u003ctt\u003e\u003ca href=\"http://nasasearch.nasa.gov/.well-known/acme-challenge/ZZZZ\"\u003ehttp://nasasearch.nasa.gov/.well-known/acme-challenge/ZZZZ\u003c/a\u003e\u003c/tt\u003e\u003c/li\u003e\n\u003cli\u003eThe agent writes the desired content to \u003ctt\u003e/var/www/http/.well-known/acme-challenge/ZZZZ\u003c/tt\u003e, and the CA’s HTTP request for that file succeeds\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis reduces the convoluted many-party, multi-step process to one that involves only three parties. Since one of these parties, Let’s Encrypt, is automated, this reduces the entire process time from weeks to minutes.\u003c/p\u003e\n\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2016/08/416-x-387-ssl%5c_certificate%5c_process_after.jpg\"\n    alt=\"New certificate process.\"/\u003e\u003c/div\u003e\n\n\n\u003ch2 id=\"http-domain-validation-in-a-multi-app-server-environment\"\u003eHTTP Domain Validation in a Multi-app-server Environment\u003c/h2\u003e\n\u003cp\u003eThis process works great in a single-server environment because all HTTP requests issued by the CA go to the one host where Certbot is running. But as our service operates in a multi-server environment with various hosts sitting behind an Elastic Load Balancer, you’ll immediately see a problem. There’s no guarantee that an HTTP request issued by the CA will be served by the same host that’s running Certbot. Also, since our site needs to remain up and running at all times, we can’t simply send all traffic to one Certbot-controlled server for the duration of the domain validation process. We need to be able to serve content to our users and serve domain validation responses to Let’s Encrypt at the same time.\u003c/p\u003e\n\u003cp\u003eThe solution to running Certbot in a multi-server environment stems from the fact that all HTTP Domain Validation requests made by the Let’s Encrypt CA have paths that begin with \u003ctt\u003e/.well-known/acme-challenge/\u003c/tt\u003e. We set up a single Certbot EC2 instance with an \u003ca href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html\"\u003eElastic IP address\u003c/a\u003e, assigned the domain name \u003ctt\u003elets-encrypt.infr.search.usa.gov\u003c/tt\u003e to that address, and set up Certbot on the instance. As we are using Apache to direct requests to our application, we set up special \u003ctt\u003e\u003cLocation\u003e\u003c/tt\u003e blocks in the Apache config files for all our application servers to proxy HTTP Domain Validation requests to our “Let’s Encrypt” instance:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e\u003cLocation\u003e /.well-known/acme-challenge/\u003cbr /\u003e ProxyPass \u003ca href=\"http://lets-encrypt.infr.search.usa.gov/.well-known/acme-challenge/\"\u003ehttp://lets-encrypt.infr.search.usa.gov/.well-known/acme-challenge/\u003c/a\u003e\u003cbr /\u003e Require all granted\u003cbr /\u003e\u003c/tt\u003e\u003c/p\u003e\n\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2016/08/600-x-523-lets%5c_encrypt%5c_and%5c_customer%5c_requests.jpg\"\n    alt=\"Let\u0026#39;s encrypt and customer requests.\"/\u003e\u003c/div\u003e\n\n\n\u003cp\u003eThis guarantees that Domain Validation requests go to our Certbot host and all other traffic goes to our pool of application servers. We set up all our proxy EC2 application servers with these special \u003ctt\u003e\u003cLocation\u003e\u003c/tt\u003e blocks as well as all of the application servers in the old hosting environment (not shown in the diagram). So whether the Let’s Encrypt CA issues a Domain Validation request for a domain that was still being handled by the old hosting environment or a domain that was being handled by the new AWS hosting environment, we were guaranteed that the request would eventually get proxied to the host that’s running Certbot.\u003c/p\u003e\n\u003ch2 id=\"tips--tricks\"\u003eTips \u0026amp; Tricks\u003c/h2\u003e\n\u003cp\u003eAt the time of this writing, Certbot is still in its infancy, and installing it varies across environments. The first time you run Certbot, be prepared for the wrapper script to install dependencies on your system.\u003c/p\u003e\n\u003cp\u003eSAN SSL certificates are valid for any domain listed within their Subject Alternative Name field, but they also have a primary domain name listed in their Common Name field. Let’s Encrypt will use the first domain listed in your request as the Common Name field. Our convention is to always list \u003ctt\u003esearch.usa.gov\u003c/tt\u003e as the first domain name in the requested list and then provide the rest of the domain names in alphabetical order. That way the Common Name on our certificate is always \u003ctt\u003esearch.usa.gov\u003c/tt\u003e instead of whichever customer domain happens to be first alphabetically.\u003c/p\u003e\n\u003cp\u003eIf you issue too many certificate requests to Let’s Encrypt, you’ll find yourself receiving rate-limiting error messages. To avoid being rate limited, use Certbot’s \u003ctt\u003e\u0026ndash;staging\u003c/tt\u003e option, which instructs it to use the Let’s Encrypt staging CA.\u003c/p\u003e\n\u003cp\u003eTyping out a bunch of domain names in a Certbot command is tedious and error-prone, and while our example above lists three different domain names, in reality we currently include 36 domain names in our SAN SSL certificate, with many more to come in the near future as federal sites complete their transitions to HTTPS. To save ourselves time and to prevent mistakes, we make use of the fact that Certbot looks for a config file in \u003ctt\u003e/etc/letsencrypt/cli.ini\u003c/tt\u003e for its options. For the above certbot command example, we would end up using the following \u003ctt\u003ecli.ini\u003c/tt\u003e config file:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003edomains = search.usa.gov, find.irs.gov, nasasearch.nasa.gov\u003cbr /\u003e authenticator = webroot\u003cbr /\u003e webroot-path = /var/www/html\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eThis allows us to use the following Certbot command no matter how many customer domains we want to include:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e./certbot-auto certonly\u003c/tt\u003e\u003c/p\u003e\n\u003ch2 id=\"conclusion\"\u003eConclusion\u003c/h2\u003e\n\u003cp\u003eOur HTTP Domain Validation request proxy design ensures we can quickly and easily generate a new SAN SSL certificate that is valid for all of our customers’ domains. It currently takes us only a few minutes to respond to customer requests for HTTPS support as long as they have their DNS set up properly. This combined with the free cost of the Let’s Encrypt service makes it a great solution for us.\u003c/p\u003e\n\u003ch3 id=\"series\"\u003e\n  \u003cem\u003eRead more of this 5-part series:\u003c/em\u003e\n\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/08/18/the-right-tools-for-the-job-re-hosting-digitalgov-search-to-a-dynamic-infrastructure-environment/\"\u003eThe Right Tools for the Job: Re-Hosting DigitalGov Search to a Dynamic Infrastructure Environment\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/02/quality-speed-and-lower-costs-yes-you-can-have-it-all/\"\u003eQuality, Speed, and Lower Costs: Yes, You Can Have It All\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/06/a-domain-by-any-other-name-cnames-wildcard-records-and-another-level-of-indirection/\"\u003eA Domain by Any Other Name: CNAMES, Wildcard Records and Another Level of Indirection\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/12/dnssec-vs-elastic-load-balancers-the-zone-apex-problem/\"\u003eDNSSEC vs. Elastic Load Balancers: the Zone Apex Problem\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n"}
  ]
}