While the RRSIG record allows you to verify the authenticity of returned records, there is still a gap remaining. What if the requested name does not exist? Since the DNS server replies with an empty answer section, there is nothing to sign. That’s where NSEC (or its brother NSEC3) comes into play. An NSEC record (Next SECure) states the range of names that do not exist. By signing this NSEC record by a corresponding RRSIG record, one can prove that a domainname does in fact not exist.

For every existing name, there is a corresponding NSEC record. This NSEC record states what types are available for the current name, as well as the next valid name is in the sorted zone file. This brings us to the downside of NSEC: Since the NSEC records essentially chain through the complete zone, it’s possible to do “zone walking”. While every single entry isn’t a secret, handing out the complete zone is different. To quote Paul Albitz and Cricket Liu from their DNS and BIND:

It’s the difference between letting random folks call your company’s switchboard and ask for John Q. Cubicle’s phone number [versus] sending them a copy of your corporate phone directory.

This is where NSEC3 comes into play. The principle is exactly the same as for NSEC, but in the hashed domain. Normal NSEC looks like this:
NSEC3 looks like this. I choose an artificial hash function, normal hash values are much longer.

Because the client knows how the hashes are calculated, it can still verify the assertion.

The proof of non-existance

Let’s assume a request is made for “charlie” for the above zone. The server answers with an NXDOMAIN status, and additionally provides an NSEC record:

beta.example.net. NSEC delta.example.net. A RRSIG NSEC

This asserts that there is no valid name between “beta” and “delta”. In addition it states that the only data available for “beta” is an A, RRSIG and NSEC record. Since “charlie” sorts between “beta” and “delta”, this is a proof that “charlie” does not exist.

The NSEC3 case is similar. First the server calculates the hash for “charlie”, in this example “a10c”. It then provides the following record:

810c.example.net. NSEC 1 0 5 5A17 c73a A RRSIG

This asserts that there is no valid hash between “810c” and “c73a”. It also states the algorithm used to calculate the hash (1) as well as the used salt (5A17) and iterations (5). This allows the client to calculate for itself the hash for “charlie” and verifying that it sorts between the two given hashes in the NSEC3 record.

However, this is not enough: DNS supports a wildcard function! So in addition to proving that the required name does not exist, the servers inserts an additional NSEC(3) record, proving that there is no wildcard that matches the request.