{
    "version" : "https://jsonfeed.org/version/1",
    "content" : "news",
    "type" : "single",
    "title" : "DNSSEC vs. Elastic Load Balancers: the Zone Apex Problem |Digital.gov",
    "description": "DNSSEC vs. Elastic Load Balancers: the Zone Apex Problem",
    "home_page_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/","feed_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/12/dnssec-vs-elastic-load-balancers-the-zone-apex-problem/index.json","item" : [
    {"title" :"DNSSEC vs. Elastic Load Balancers: the Zone Apex Problem","summary" : "This is the final post in the 5-part series, The Right Tools for the Job: Re-Hosting DigitalGov Search to a Dynamic Infrastructure Environment. Federal websites are required to implement DNSSEC, which relies on knowing exactly what server is responding to a request. In Amazon Web Services (AWS), the problem of unreliable servers is solved by","date" : "2016-09-12T13:00:09-04:00","date_modified" : "2025-01-27T19:42:55-05:00","authors" : {"nick-marden" : "Nick Marden"},"topics" : {
        
            "cloud-and-infrastructure" : "Cloud and infrastructure",
            "content-strategy" : "Content strategy",
            "product-and-project-management" : "Product and project management",
            "search" : "Search",
            "security" : "Security"
            },"branch" : "bc-archive-content-3",
      "filename" :"2016-09-12-dnssec-vs-elastic-load-balancers-the-zone-apex-problem.md",
      
      "filepath" :"news/2016/09/2016-09-12-dnssec-vs-elastic-load-balancers-the-zone-apex-problem.md",
      "filepathURL" :"https://github.com/GSA/digitalgov.gov/blob/bc-archive-content-3/content/news/2016/09/2016-09-12-dnssec-vs-elastic-load-balancers-the-zone-apex-problem.md",
      "editpathURL" :"https://github.com/GSA/digitalgov.gov/edit/bc-archive-content-3/content/news/2016/09/2016-09-12-dnssec-vs-elastic-load-balancers-the-zone-apex-problem.md","slug" : "dnssec-vs-elastic-load-balancers-the-zone-apex-problem","url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/12/dnssec-vs-elastic-load-balancers-the-zone-apex-problem/","content" :"\u003cp\u003e\u003cem\u003eThis is the final post in the 5-part series, \u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/08/18/the-right-tools-for-the-job-re-hosting-digitalgov-search-to-a-dynamic-infrastructure-environment/\"\u003eThe Right Tools for the Job: Re-Hosting DigitalGov Search to a Dynamic Infrastructure Environment\u003c/a\u003e.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eFederal websites are \u003ca href=\"https://www.whitehouse.gov/sites/default/files/omb/memoranda/fy2008/m08-23.pdf\"\u003erequired to implement DNSSEC\u003c/a\u003e, which relies on knowing exactly what server is responding to a request. In Amazon Web Services (AWS), the problem of unreliable servers is solved by \u003ca href=\"https://aws.amazon.com/elasticloadbalancing/\"\u003eElastic Load Balancing\u003c/a\u003e (ELB). An ELB containing one or more servers is presented to the world as a single hostname — say, \u003ccode\u003eusasearch-elb.ec2.aws.com\u003c/code\u003e — and requests are routed to individual servers in the ELB pool based on health and capacity. Hosts change without notice, at odds with standard DNSSEC implementations.\u003c/p\u003e\n\u003cp\u003eFor practical reasons, ELBs for human-visible web sites are almost always hidden behind CNAME records in the DNS:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e~ host search.usa.gov\u003cbr /\u003e search.usa.gov is an alias for usasearch-elb.ec2.aws.com.\u003cbr /\u003e usasearch-elb.ec2.aws.com has address 209.85.232.121\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eThis is a fantastic and invaluable abstraction: what appears to be a single hostname called \u003ccode\u003esearch.usa.gov\u003c/code\u003e is actually a multi-datacenter, self-healing, auto-scaling pool of servers.\u003c/p\u003e\n\u003cp\u003eHowever, it runs afoul of one critical restriction of the DNS: the fact the top-most entry in a DNS zone (known as the “zone apex”) \u003ca href=\"http://serverfault.com/questions/613829/why-cant-a-cname-record-be-used-at-the-apex-aka-root-of-a-domain\"\u003ecannot be a CNAME\u003c/a\u003e. So if you want to add this ELB CNAME to the \u003ccode\u003eservice.gov\u003c/code\u003e zone, you’ll have no problem:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e$ORIGIN service.gov.\u003cbr /\u003e ; search.usa.gov is a CNAME to the ELB hostname\u003cbr /\u003e search IN CNAME usasearch-elb.ec2.aws.com.\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003ebut if you only control the \u003ccode\u003esearch.usa.gov\u003c/code\u003e zone (which is the situation we are in), you’re out of luck:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e$ORIGIN search.usa.gov\u003cbr /\u003e ; This following line is not a valid DNS configuration\u003cbr /\u003e @ IN CNAME usasearch-elb.ec2.aws.com.\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eThere are numerous vendor-specific solutions to this problem, typically called \u003ca href=\"https://www.dnsmadeeasy.com/services/anamerecords/\"\u003eANAME\u003c/a\u003e or \u003ca href=\"http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html\"\u003eALIAS\u003c/a\u003e records. They work around the zone apex problem by allowing you to \u003cem\u003econfigure\u003c/em\u003e the zone apex entry as though it were a CNAME, but \u003cem\u003epresent\u003c/em\u003e the answer to the caller as though it were an A record:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e$ORIGIN search.usa.gov\u003cbr /\u003e ; \u0026lsquo;@\u0026rsquo; means \u0026ldquo;the zone apex\u0026rdquo;, i.e. search.usa.gov\u003cbr /\u003e @ 60 IN ALIAS usasearch-elb.ec2.aws.com.\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eHere are the current IP addresses for our ELB\u003cbr /\u003e\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e~ host usasearch-elb.ec2.aws.com\u003cbr /\u003e usasearch-elb.ec2.aws.com has address 52.86.186.226\u003cbr /\u003e usasearch-elb.ec2.aws.com has address 52.0.72.176\u003cbr /\u003e usasearch-elb.ec2.aws.com has address 52.23.22.193\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eNotice that there are no mentions of CNAMEs in this answer\u003cbr /\u003e\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e~ host search.usa.gov\u003cbr /\u003e search.usa.gov has address 52.86.186.226\u003cbr /\u003e search.usa.gov has address 52.0.72.176\u003cbr /\u003e search.usa.gov has address 52.23.22.193\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eThis sleight-of-hand is accomplished by dynamically resolving the A record request at the authoritative nameserver by looking at the underlying ALIAS and getting its current value. By design, the answer issued by the authoritative nameserver has a short (60-second) TTL, because the “correct” answer for \u003ccode\u003eusasearch-elb.ec2.aws.com\u003c/code\u003e could change suddenly in an environment where servers pop in and out of existence without warning. At the cost of additional traffic to the authoritative nameservers, incorrect results are quickly flushed out of the global DNS cache when the ALIAS lookup results change.\u003c/p\u003e\n\u003ch2 id=\"so-what-about-dnssec\"\u003eSo, What About DNSSEC?\u003c/h2\u003e\n\u003cp\u003eWith ANAME or ALIAS records (called ALIAS records for the rest of this post) at our disposal, we could easily satisfy our first two requirements. However, doing so required picking a DNS provider with ALIAS support – and no government-approved hosted DNS providers who support ALIAS also support DNSSEC. We needed a tool that didn’t exist yet.\u003c/p\u003e\n\u003cp\u003eTo find the solution, it was valuable to look at why DNS providers don’t provide DNSSEC support along with ALIAS records. The design of DNSSEC — and the cryptographic assurance it provides about DNS record values — requires taking all of the records in a zone file and computing a cryptographic hash of them, called an \u003ca href=\"https://www.ietf.org/rfc/rfc4034.txt\"\u003eRRSIG\u003c/a\u003e. Computationally, this is a very expensive operation, especially compared to the cost of answering a single DNS request. Therefore, RRSIG calculations are done when the contents of the zone change, not on-the-fly while answering requests.\u003c/p\u003e\n\u003cp\u003eBehind-the-scenes ALIAS expansion throws a wrench into the works here. If the result of a lookup can be different at different times (such as when the list of IP addresses for the \u003ccode\u003eusasearch-elb.ec2.aws.com\u003c/code\u003e ALIAS changes), then the RRSIG itself might need to be recomputed at any moment. This isn’t practical for DNS services that may be serving hundreds of thousands or millions of requests per minute.\u003c/p\u003e\n\u003cp\u003eSo we had to rummage around the Internet for help. A lot. \u003ca href=\"https://www.ietf.org/mail-archive/web/dnsop/current/msg12352.html\"\u003eWe learned\u003c/a\u003e this issue was at the forefront of the minds of some Very Smart People working on IETF-related projects, and so we went to the people who were proposing a solution: \u003ca href=\"https://www.powerdns.com/\"\u003ePowerDNS\u003c/a\u003e, an open-source DNS software provider based in The Netherlands.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/Habbie\"\u003ePeter van Dijk\u003c/a\u003e, one of PowerDNS’ software developers, confirmed the realizations that we’d had:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eALIAS resolution is a useful feature that works by dynamically changing the record values in a zone\u003c/li\u003e\n\u003cli\u003eDNSSEC doesn’t work for zones whose contents change dynamically\u003c/li\u003e\n\u003cli\u003eMost of the time, ALIAS expansion continues to return the same results\u003c/li\u003e\n\u003cli\u003eMost of the time, therefore, correct DNSSEC signatures will continue to be correct\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003ePeter then suggested a very \u003ca href=\"https://github.com/PowerDNS/pdns/pull/3733\"\u003esimple improvement to the PowerDNS software\u003c/a\u003e that could solve our problem: adding the ability for a DNS server to expand ALIAS records (which it already supported) into A records during a zone transfer, or \u003ca href=\"https://cr.yp.to/djbdns/axfr-notes.html\"\u003eAXFR\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThis small change, called \u003ca href=\"https://doc.powerdns.com/md/authoritative/howtos/#using-alias-records\"\u003eoutgoing-axfr-expand-alias\u003c/a\u003e and available beginning in \u003ca href=\"https://www.powerdns.com/downloads.html\"\u003ePowerDNS Authoritative Server 4.0.0\u003c/a\u003e, allows one server (the so-called “DNS master”) to be the only one that knew about our ALIAS records. Then, every minute, “DNS secondaries\u0026quot; acting as the authoritative nameservers for \u003ccode\u003esearch.usa.gov\u003c/code\u003e would initiate an AXFR of the zone from the DNS primary, and would receive a copy of the zone file containing the most up-to-date values for those ALIAS records, expressed as A records. The result of this AXFR would then be compared with the current contents of the zone on the secondary server. If the contents had not changed — almost always the case — then no action would be taken. If the contents had changed, then the zone would be reloaded entirely and re-signed using existing DNSSEC signing features, with notification sent to sysadmins.\u003c/p\u003e\n\u003cp\u003eOverall, the process looks like this:\u003c/p\u003e\n\u003cdiv class=\"image\"\u003e\n  \u003cimg\n    src=\"https://s3.amazonaws.com/digitalgov/_legacy-img/2016/09/600-x-400-multi-tier-alias-and-dnssec-architecture.jpg\"\n    alt=\"A multi-tier alias and DNSSEC architecture flowchart.\"/\u003e\u003c/div\u003e\n\n\n\u003cp\u003eSide note: You can see the script that we wrote to request the AXFR from the master server and compare its contents to the current secondary server zone file on \u003ca href=\"https://gist.github.com/nickmarden/9092c99cf3e201510ca83455fc2d2dab\"\u003eGithub\u003c/a\u003e. It’s quite simple, and relies on straightforward zone management features already built into the PowerDNS software. This script is no longer necessary, but was developed as a precaution against the possibility of a failed AXFR “emptying” the zone on the secondary server. In the released version of PowerDNS 4.0.0, a standard zone slaving configuration has this protection enabled automatically.\u003c/p\u003e\n\u003ch2 id=\"but-what-about-wildcards\"\u003eBut What About Wildcards?\u003c/h2\u003e\n\u003cp\u003eEarlier we talked about using wildcard DNS records to simplify the process of creating \u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/06/a-domain-by-any-other-name-cnames-wildcard-records-and-another-level-of-indirection/\"\u003ecustomer-specific CNAMEs\u003c/a\u003e. You might have expected that to cause a problem with respect to DNSSEC and dynamic records. However, as it turns out, wildcard records don’t present the same problem to DNSSEC that ALIAS records present.\u003c/p\u003e\n\u003cp\u003eLet’s take a closer look at what happened in our two-step zone updating process. First, the master server contains a wildcard ALIAS record that points customer sites to an ELB CNAME:\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e$ORIGIN search.usa.gov.\u003cbr /\u003e *.sites.infr IN ALIAS usasearch-elb.ec2.aws.com\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eAfter outbound ALIAS expansion during a zone AXFR, this becomes\u003c/p\u003e\n\u003cp\u003e\u003ctt\u003e$ORIGIN search.usa.gov.\u003cbr /\u003e *.sites.infr IN A 52.86.186.226\u003cbr /\u003e *.sites.infr IN A 52.0.72.176\u003cbr /\u003e *.sites.infr IN A 52.23.22.193\u003c/tt\u003e\u003c/p\u003e\n\u003cp\u003eThese records, although wildcards, are no longer dynamic. And DNSSEC supports signing \u003ca href=\"https://tools.ietf.org/html/rfc7129#section-5.3\"\u003ewildcard records\u003c/a\u003e, making these results as valid as any other A record that might appear after ALIAS expansion.\u003c/p\u003e\n\u003ch2 id=\"conclusion\"\u003eConclusion\u003c/h2\u003e\n\u003cp\u003eSo, thanks to some tricky multi-tier design, our solution now works as follows:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eThe master DNS server knows nothing about DNSSEC. Its job is just to publish up-to-date zone contents:\n\u003cul\u003e\n\u003cli\u003eThe zone apex search.usa.gov is an ALIAS to an ELB, and gets evaluated into an A record\u003c/li\u003e\n\u003cli\u003eWildcard records are just ALIASes to ELBs, and also get evaluated into wildcard records that are A records\u003c/li\u003e\n\u003cli\u003eAll other records are transferred untouched\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eEvery 60 seconds, the authoritative nameservers for search.usa.gov poll for a current “alias-expanded” version of the search.usa.gov zone file\u003c/li\u003e\n\u003cli\u003eWhenever any change appears in “alias-expanded” search.usa.gov zone, the entire zone file is re-signed and re-published to the Internet-facing authoritative nameservers\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eFor DigitalGov Search agency customers with DNSSEC-signed zones of their own, this setup allows them to select their own customer-specific CNAME, delegate it to \u003ccode\u003esearch.usa.gov\u003c/code\u003e, and operate with confidence that the entire chain of DNS resolution is signed with DNSSEC and safe from cache poisoning or man-in-the-middle attacks.\u003c/p\u003e\n\u003ch3 id=\"series\"\u003e\u003cem\u003eRead more of this 5 part series:\u003c/em\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/08/18/the-right-tools-for-the-job-re-hosting-digitalgov-search-to-a-dynamic-infrastructure-environment/\"\u003eThe Right Tools for the Job: Re-Hosting DigitalGov Search to a Dynamic Infrastructure Environment\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/02/quality-speed-and-lower-costs-yes-you-can-have-it-all/\"\u003eQuality, Speed, and Lower Costs: Yes, You Can Have It All\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/06/a-domain-by-any-other-name-cnames-wildcard-records-and-another-level-of-indirection/\"\u003eA Domain by Any Other Name: CNAMES, Wildcard Records and Another Level of Indirection\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/preview/gsa/digitalgov.gov/bc-archive-content-3/2016/09/07/lets-encrypt-those-cnames-shall-we/\"\u003eLet’s Encrypt those CNAMES, Shall We?\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n"}
  ]
}