At some point while running DNSSEC, you will wonder if the base64 blobs in RRSIG and DNSKEY records are actually correct. We specify a few procedures that we follow to have Python calculate signatures if that happens to us.

In what follows below, we are assuming that all your RSA public keys are in DNS; or otherwise, that you will be able to dig it up from your HSM. In fact, it is not so much the public key exponent as the modulus that matters — as the public key exponent is usually 65537 or hexadecimal 0x00010001.

Setting up Python

To get Python ready for action, start it up and give the following commands:

Determining the key tag of a DNSKEY record

Sometimes we need a key tag but only have the fullblown DNSKEY record. This is quickly solved if it is stored in a file, like /tmp/k1 with a simple commandline tool that comes with the LDNS library:

ldns-key2ds -n -f -2 /tmp/k1

The output will be printed on stdout and include the key tag. Note the use of -f to force it on ZSKs and not just request the information on KSKs.

Fetch RSA public keys from DNSKEY records

Using your favourite commandline tools, dig up the DNSKEY record that you need to work on. For instance:

ip4afrika.nl. 1229 IN DNSKEY 256 3 8 AwEAAafyW...WVZCTZ3x

This is a ZSK (as shown by the 256 flag) using the RSA/SHA256 algorithm (the 8 identifier). The long rest of the line is the base64 makeup of the key itself. There will usually be more than one key, and DNS will round-robin the records. Usually, this means that it is possible to start counting after the last 257-flagged KSK (when checking a ZSK that signed an RRSIG).

The first line assigns the binary content of the base64-encoded key to k1. The second line should print True if the public key exponent in use is indeed 65537, as is common for RSA. The third line should print the modulus length in bits, and the fourth line strips off the public key and returns the modulus as an integer in k1i.

We now have a key in k1i. It is usually practical to do the same for the other keys being checked. So you should repeat the procedure for k2, k3 and so on.

Fetch signature values from RRSIG records

Using your favourite commandline tool, dig up the RRSIG record that you are mistrusting. It may look like

The first line extracts the binary value from the base64 string. The second line should print the modulus length as the signature length (although it could be smaller if the signature happens to start with zeroes). The third line stores the signature in the s1i variable.

It is always good to also have an s2 with a known-good signature. Even if your signing process is failing, chances are that something older still exists, and could be used as a positive confirmation of the calculation procedures below.

Checking an RSA signature

Usually, problems in systems like these are related to public key handling. So what we are doing is to apply the RSA algorithm to a signature, to see if the outcome looks like it was properly padded. If not, then something about the key or signature is wrong, and trying other keys may be advisable. Also look for correct signatures, just to check your own procedures of getting here.

"%x" % pow (s1i, 65537, k1i)

This should print something that obviously starts with padding, and whose lower bytes constitute the hash, for instance:

'1fffffff...fffffff003031300...0512'

Making several of these calculations is easy, and it can be helpful to see if signature s1 was made with key k1 or another key.

When a zone encounters a problem, it is helpful to dig around in DNS, possibly probing the authoritatives directly to bypass validating resolvers. This will yield information, but not actually check signatures.

The Drill utility is a replacement of Dig, and can actually chase a signature all the way to the root keys of DNSSEC. Assuming you have not got a local version of it and that security of this check is not the most important issue to you now, let’s see how to use Drill:

The first command retrieves the root key(s) without checking anything security-related. Although, if your resolvers are secure, you will not be bypassing that of course. The second command chases the signature chain of trust all the way to those root key(s) by querying the given name server directly, and the final line prints 0 if it went fine.

The lines printed out are prefixed with [T] for trusted information, [B] for bogus and [S] for self-signatures that are OK, but not trusted. Only if all lines are marked with [T] are you free of trouble.

If, like us, you use an HSM to store your DNSSEC key material you may know that it is important to monitor memory usage in your HSM; with a typical DNSSEC key management scheme you may have as many as 5 keys active per signed domain. This can be a burden on your HSM, especially if it has a limited amount of storage like ours do. To automate this monitoring, we have created a little tool called “p11memmon”. It can be integrated as a Nagios check and monitors the amount of available memory through the PKCS #11 interface of your HSM. The tool can give a warning when memory saturation reaches a certain limit (75% is the default setting) and reports a critical status if the memory saturation exceeds a maximum limit (90% by default).

We’ve made this tool available in open source, you can download it here. The source includes a specification file that you can use to build an RPM package.

It is one thing to sign a zone and publish an RRSIG next to each DNS record; it is quite another to integrate the signing process with the parent’s. This is necessary however, to complete the chain of trust that reaches from the root zone . down to your individual RRSIG records.

Although not all parents follow the same procedures, the general idea usually is that you upload a DS or DNSKEY over EPP. To facilitate various rollover procedures, it is often possible to manage multiple such DS/DNSKEY records over EPP. As always with DNSSEC, the big question is when a certain change may be made without risking domain downtime due to cached older results.

Our Approach

Most DNSSEC approaches take the shape of a step-by-step procedure, intended to ensure that all the caches can follow the changes, and thus that all parties on the Internet always see the zone in a valid state.

Another approach is to represent these various states of the zone inside the system, and have the data evolve through that model. This is the approach taken here.

We see the KSK set as a set of DNSKEY records that starts at one point, and then rippples through a number of stages, each time subject to conditions or timers, until it has spread to the end and reaches a stable state. Whenever two subsequent stages have a different KSK set, then apparently some work needs to be done.

We recognise the following levels of progress, and conditions to progress to them from the foregoing stage:

Level

Description

Entry condition for this level

0/signer

Last-sampled signer output

True

1/author

Zone authoritative KSK set

All authoritatives agree on the KSK set

2/mature

Outcome of 1, author has matured

Wait TTL (plus 1h safety margin) after 1/author change

3/parent

Parent received KSK set

After succeeded KSK set upload to parent

4/public

Parent authoritatives publish the DS set

All parent autoritatives publish the DS set

5/dshold

Outcome of 4, public has matured

Wait TTL after 4/public

6/dsseen

OpenDNSSEC has been notified

After successful ds-seen (or ds-unseen)

Implementation

We have implemented this approach in a cron-job that can run as often as we like; any waiting is implemented independently. The state of a parent relationship is stored in a data directory, such as /var/opendnssec/parenting/ as a series of files, one per state. The files hold the DNSKEY set that is taken into account.

We treat a set as a whole. Specifically for timing issues, we look at the timestamp of the last change on the files, so on the KSK set. This means that one theoretic problem could occur: if a zone keeps on changing, it would never be found silent for TTL+1h and so it would never advance into the parent. This is not likely to occur in practice.

When uploading over EPP, it is usually not possible to offer a new DS set or DNSKEY set; instead, separate add/remove commands are usually required. To that end, we download the old set and calculate the changes necessary to get to the desired new state.

Download code

Code of the engine is available as a free download, without any guarantees or maintenance promises; we simply offer it in the hope that it is useful.

You will need to integrate it with your EPP workflow, as that is not included; it is usually very personal to an environment. Our code refers to a “registry_shell” that must implement a few simple EPP commands, and all you need to do is hook those up to your EPP code:

Chances are that eppkeys() and update_keys() are new calls, as they may be your first encounter with DNSSEC in EPP. Please see your parent’s EPP manual regarding its implementation in their registry. If you have multiple parents, you will need to make a slight generalisation of the script, but at least the difficulties of the workflow will have been resolved for you.

Tilburg University (TiU) completed the introduction of DNSSEC in August, making it the first university in The Netherlands to use DNSSEC on a large scale. In February 2011, we began validating incoming DNSSEC information and we are now also publishing that information ourselves.

Architecture

The system at TiU is based on OpenDNSSEC, NSD, and Unbound. NSD and Unbound were already being used, as the authoritative DNS server and the DNS resolver respectively. An effort has been made to minimise changes to the existing infrastructure and interfaces.

Our administrators process the zone file on their own workstation, copy it to the DNSSEC server, and give a signal to OpenDNSSEC. OpenDNSSEC signs this zone, after which the new zone is loaded in NSD. This NSD runs on the DNSSEC machine and acts as a hidden master. The real masters are updated via notify/AXFR. The advantage of this construction is that the front-end DNS servers are as far as possible relieved from having to read and process enormous zone files and can focus entirely on delivering DNS responses as fast as possible.

OpenDNSSEC

Although DNSSEC is itself already a reasonably complex protocol, the real challenge is in management. Steps have to be taken with great regularity to refresh keys and signatures. If you donâ€™t keep track of this for a few days, your domain may disappear off the Internet. And if you have several hundred domains, you can no longer keep track manually.

OpenDNSSEC automates this process and looks after the creation and timely refreshing of keys and signatures.

Architecture at Tilburg University:

HSM

The keys needed for DNSSEC need to be stored securely. Separate hardware is available for this, namely â€śHardware Security Modulesâ€ť (HSMs). HSMs are available in all price classes. OpenDNSSEC communicates directly with the HSM. TiU has decided not to purchase any HSM. Instead, we are using a software implementation of an HSM. This provides enough security for us, and greater speed, flexibility, and convenience.

Monitoring

If you want to use DNSSEC, you canâ€™t just do something once and at a single point. Because a lot of things can go wrong, you need to take measurements at various different points to see whether everything is still going well. For that purpose, we have various Nagios checks to determine, for example, whether a zone has been signed correctly and whether the signatures are at risk of expiring. Something that still works this week may have expired next week; this large-scale expiry is perhaps the main difference to the traditional DNS system.

Day-to-day management

DNS has become significantly more complicated: where two text files and a simple daemon used to be enough, you now need to set up a whole chain. Adding and removing zones has become quite a bit more complicated because there are a number of steps that need to be carried out at a number of different places and in the right order. The NSD configuration can only be altered, for example, after the zone has been signed. This is not a fundamental problem and it can be solved with a bit of script work, but it is something that we regularly need to deal with. Proper monitoring is indispensable.

You need to have constant access to all your registrars because you regularly need to upload a new DS record. We discovered that there were a lot of domains that we couldnâ€™t access (or couldnâ€™t access any longer). There are also domains that have been acquired by third parties without us â€“ the central IT department â€“ being informed. The DNSSEC project has led to more attention being paid to this, and we now have a clearer view of these domains.

Malfunctions and problems

Solving DNSSEC problems takes a lot of time at first. You should set the validity of the signatures so that you have enough time to respond to problems. You also need to take account of weekends and holidays.

Most DNS administrators have hardly any experience of DNSSEC and if there are any problems it can be hard to convince them that something is wrong. In that situation, a diagram with a big red arrow indicating the problem can be a powerful argument! http://www.dnsviz.net can produce diagrams for you (for example http://dnsviz.net/d/www.dnssec-failed.org/dnssec/).

If DNSSEC goes wrong, itâ€™s impossible for most people to determine what the problem is. You donâ€™t get any error message other than that certain domains cannot be found. As an end-user, itâ€™s therefore no easy matter to work around the problem (as you can do with incorrect SSL certificates). Donâ€™t wait too long to familiarise yourself with DNSSEC; at the moment, you can still make mistakes without it having any immediate dramatic consequences.

DNS zones are becoming a lot larger (typically 20 times larger) and are being altered more frequently (several times a week). This significantly increases the load on the DNS servers, which brings up new bugs.

The future

Uploading the DS records is still work that has to be done manually. There are ways to automate this â€“ for example with the EPP protocol â€“ but these differ from one provider to another. From the security perspective, thereâ€™s in fact something to be said for keeping human control part of the process. By no means all TLDs and registrars support DNSSEC. As soon as this becomes possible, we will also provide these domains with DNSSEC. At the moment, we have precisely 200 domains that support DNSSEC and 65 domains that donâ€™t do so for various technical reasons. There are still a lot of TLDs and registrars that donâ€™t support it. As soon as this becomes possible, we will also provide these domains with DNSSEC.

Besides the central DNS resolvers, there are also a number of departments and affiliated organisations that run their own DNS resolvers and that will also be configured as DNSSEC validitors in the coming period. At the moment, we have a single OpenDNSSEC server in production. If it were to go down, we could â€“ under normal circumstances â€“ quickly construct a new one and restore our backups. If there were a bigger disaster, for example a fire, we would probably not have the time. We therefore want to have a reserve machine standing by at our alternative location.

Casper Gielen is ICT Manager for Library and IT Services UNIX at Tilburg University.

During the course of the past year Gijs van den Broek has performed research on DNSSEC and UDP fragmentation for his M.Sc. thesis. His work is now finished and the tools he used to perform his research have been made available so others may reproduce his research or extend it. Read more about where to download and how to build his tools in his document “Replication of DNS Research Results“.

Gijs has written a journal article that we’re hoping will be published before the end of this year. If you want to learn more about his work before that, here are some pointers to presentations about his work:

Over the past week we have published a detailed HOWTO on the signer migration process we have gone through last month on our DNSSEC blog. The full process, including a worksheet (also available separately) to help you during the process, is described in the document that you can download by clicking on the image on this page.

The purpose of this step is to remove the DNSKEY record for the active ZSK from the source signer from the input zone and to resume automated key management. Once this step has taken place, the migration is complete. The situation at the end of this step is shown in the diagram below:

To reach this situation, the following sub-steps need to be taken:

Remove the active ZSK from the source signer from the input zone

If required, update the SOA serial number in your backend system such that it is higher than the SOA serial number that is currently published

Resume automated uploads of the input zone to the destination signer; if you still have zones that are signed and published by the source signer then you can now also resume automated upload of input zones to the source signer

Resume automated key management on the destination signer (in case OpenDNSSEC is used, you can now restart the enforcer component); if you still have zones that are signed and published by the source signer you can also resume automated key management on the source signer

The purpose of this step is to completely switch over to the destination signer. At the end of this step, continuous zone signing has been restarted and will only take place on the destination signer; zone publication will have been resumed and will use the output from the destination signer. The situation at the end of this step is shown in the diagram below (note that the situation on the source signer is greyed-out since the source signer will no longer be in use for the zone that is being migrated by the end of this step).

To reach this situation, the following sub-steps need to be taken:

Make the latest clean (i.e. unmodified by the steps of this process) input zone available as input on the destination signer

Add the DNSKEY record for the active ZSK from the source signer to the input zone

Set the SOA serial number such that it is higher than the SOA serial number of the currently published zone

Remove any signed zone output and intermediate data on the destination signer

Restart automated signing on the destination signer

Do NOT yet restart automated key management or zone upload, this will be done later

Reconfigure your authoritative name servers to source their data from the destination signer

Wait for the newly signed zone from the destination signer to be published on your authoritative name servers

Wait maxTTL(zone) for all the signatures created with the active ZSK from the source signer to disappear from caches and be replaced by signatures created with the active ZSK from the destination signer

Note that the active ZSK from the source signer needs to be published because caches will still contain signatures over resource record sets created with this key. It is therefore still part of the trust chain.

The purpose of this step is to switch the DS for the zone to point to the KSK of the destination signer. The situation at the end of this step is shown in the diagram below:

To reach this situation, the following sub-steps need to be taken:

Contact the parent zone (registry) to submit the new DS and to retract the old DS (some registries may allow you to do this in one interaction, others will have different policies for handling this situation)