DNSSEC KSK Emergency Rollover

In my last blogpost I showed how to perform a DNSSEC KSK rollover. I did it quite slowly and carefully. This time I am looking into an emergency rollover of the KSK. That is: What to do if your KSK is compromised and you must replace it IMMEDIATELY.

I am listing the procedures and commands I used to replace the KSK of my delegated subdomain
dyn.weberdns.de with BIND. And as you might already suggest it, I am showing DNSViz graphs after every step since it greatly reveals the current DNSKEYs etc.

This blogpost is part of a series about DNSSEC. Refer to this list for all articles.

RFC 6781, “DNSSEC Operational Practices, Version 2”, describes all steps in case of a KSK compromise in section 4.2.1. There are two methods how to replace a compromised KSK, by either breaking the chain of trust or by keeping it intact. I decided to train the “Emergency Key Rollover Keeping the Chain of Trust Intact” version. This is basically the same as the normal “Double-Signature KSK rollover” described in section 4.1.2. That is:

introduce a new KSK (so that both KSKs sign the ZSK),

upload the new DS to the parent to replace (!) the DS record immediately,

wait until the TTL of the old DS record expires,

and remove the old KSK from your zone.

That is, there are two time critical things to keep in mind:

Since you must transfer the new DS record to your parent zone, you must wait until they publish it. If you can use an API, you’re lucky. If you must send it manually via mail and must wait for a human response, you’re not lucky…

In my scenario I trained the emergency KSK rollover on one of my own delegated zones,
dyn.weberdns.de. That is: I am controlling the actual zone as well as the parent zone. Good prerequisite for testing purposes. ;) Furthermore my TTL for the DS record was set to 3600 s = 1 h. Hence I was able to rollover (and delete) the compromised DNSKEY within one hour!

It was Friday evening, 2018-03-02. All times are noted as UTC.

19:33: Shit, my KSK for dyn.weberdns.de is compromised!

I was informed that the DNSKEY type 257 (KSK) for my very own zone was compromised. I immediately decided to do an emergency KSK rollover!

At this point DNSViz shows the single DS, a single KSK (grey, 37188) and a single ZSK (white, 58340) for the zone:

19:38: Generating a new KSK

I generated a new KSK for the zone (and corrected the ownership to be readable by BIND, as always):

DNSViz shows this single DS record pointing to the new KSK (45918) while the old and compromised KSK (37188) is still present at the zone:

At this point all new/initial DNS queries will get the new chain of trust, while cached entries all around the world will still use the old (cached) DS record. This is why you MUST NOT delete the compromised KSK until this cached DS record times out!

Now I set the “delete” date of the compromised key to be in one hour, which is the TTL of the DS resource record.

You must set the delete date of the *compromised* key! Be carefull in selecting the appropriate key ID to NOT alter the delete date of the newly created KSK!

To inform BIND that this date has changed you must load the keys again:

1

sudo rndc loadkeys dyn.weberdns.de

20:50: BIND has deleted the compromised KSK

After this one hour of waiting, BIND successfully deleted the old (compromised) KSK out of the DNSKEY set. Hence, only one KSK and one ZSK, along with their two RRSIGs are present:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

weberjoh@nb15-lx:~$dig dyn.weberdns.dednskey+multi+dnssec

;<<>>DiG9.10.3-P4-Ubuntu<<>>dyn.weberdns.dednskey+multi+dnssec

;;global options:+cmd

;;Got answer:

;;->>HEADER<<-opcode:QUERY,status:NOERROR,id:20637

;;flags:qr rd ra ad;QUERY:1,ANSWER:4,AUTHORITY:0,ADDITIONAL:1

;;OPT PSEUDOSECTION:

;EDNS:version:0,flags:do;udp:4096

;;QUESTION SECTION:

;dyn.weberdns.de.INDNSKEY

;;ANSWER SECTION:

dyn.weberdns.de.28INDNSKEY256310(

AwEAAbiC3omZWzDtVx/nlwIEEgY1wrAQktqggJXNq50U

+Xapj2+RQE4s7dNLU7AtJ+jvr/nplax61coz/2Na9YEc

RE1LGpcZdrFH3iDZez+UcOY1GPll5xPGgpSPtSaL1j8M

fUQgAw85ZmFQ6rUptYOpa3/w40n3MrgiOFyu8IoXhP7O

QKICmHFAVcB6nurPZK6m52Vny/5R7CtHBmxGyU/Xcm6D

oimWZPj56+QeQbthiPdSLNYNB8QoLWTPEuzjLdFaz7Jv

Bp1n0tdqPS27OErcWV4eja6as1OyTODreo0b0pkM6Ulp

4xkY/7A/VbVFv9l/bHTIVFW/jXisk6Yr2/LwVy58/lGj

oJ9SfpJnnelxYo0HMbp8zsQ2xGNzvH783NXsc0LeIATl

WeBJrkW3hL4rJFc/5rT71RwHveukofFjXjk008Sgj1TD

6Gyx6I4XVi8AcbhTNajfh4K+elZMnJRnewQqk4g5KYBi

ucyX0gR+JNSejqD4+mWuIyzKTh+cMrUsIBkXYQRzwlRf

S8A0Zs9eVAQXVlKVgPEorj7fp1hnn6Ao8A2P7MeO5pdg

ilFpSEIW3IbwtKqjPWY7wvVOo9Ms2TCdgePC34xxKGub

gNQTBGU9MSlg16K2kvCavSYAgzQRUfzKtknPowr0ODz/

sBNRLH2swqvxU1rkkjZlS3RbW6+n

);ZSK;alg=RSASHA512;key id=58340

dyn.weberdns.de.28INDNSKEY257310(

AwEAAcD4vC7bqZKvApZIOXkuH++bXUBVf5XJj1oQvYn7

i3XjLmoCtOs7JyxElo4PeDnGIa4coAxBTHj4B+iVSa4n

TJ5MK2lvEQVtwr8Pr+ZKOlYKmRs+SeRI1ZGN0OyBpQv/

TxPqZBN8V9RegsJ/adM9Rn5B96Be/hrMFzmkzd0iK2L9

I2TTAPl3YDfZ5vCQzdAMEZG9G56dfZK0GRgebCu3LHQv

D7Wx+/P4CfUknOydZqmu0geww4VRUyveGUY+PLamJGie

G6QopoTtuu5V+DJbjSA+mDPAfIW993iAw2TqMuD4tBha

Q6HSmAA7FHydrq4fcqtuC+QxZaqAhQIEp93L6JM9xcl0

dY5m+OvqjCVBk/X7xRvpW1Xni3pEWs0wMcYfoeVegXXX

w42123kEypd/78h0H7Y9AD5WVNnVnG2M/J0M1eVmsiyB

7xGvyLeFMBGPiuj0YN2bESnsR9A0tGUqgwHKtWcZwcLJ

YqO/r6Ahp0wYqwcqJ2uwvolEjzGUrBOSY7fe4blauqnT

sECWeHDpuOd3Hs+IupSYTIsyFnuI6oFNaSjiD5a2z7KN

25TjpNzYRbcbjnN1J4NabLvziGFwF+ZfSSI1TavipCg2

1MczJ8FJVZPeHIdIFMyt0d5JSL4a/iyHiaRtLw0N8GjC

LDQvS57hJ0J/uQFFo0lPCIUy/iFH

);KSK;alg=RSASHA512;key id=45918

dyn.weberdns.de.28INRRSIG DNSKEY10360(

201804012049472018030219494745918dyn.weberdns.de.

XRjJhJV4/XpZpF6Il+x6hl3mextsfIbHcmCcubNtzELZ

1p+PUx5quS+TmzbUE0MbvHrgJw2EBkAOi5Y+h1ZmDkl5

hu0WaANaxpm4v8NPR7P4A7fVxMYwxgbZ4DO8BVFMb/4s

R0sRYlzr3kAzEEbvIuzetM2kWMt9aNvpW0ZJ2NoubgPY

eSnmy1EoZJC5ZGTvGcoiU5nJfl+jC+qWIo4TnTZ2gyiO

my9qEFiXEDogKWHmbiloQR3u3OLiGUY5WBj8J6FDuBxu

YYSXiEEgusLi2xwxK9tCZoeCSyBOV2mB3KvVwNrLDlgl

yepC1c0iXtgdTjtxBJyUhNS7jnfl75aJ7cZezhGgaC3h

+4CuiSvYsbdIVjDtDGHst0UrDYVvXOe3XWdbMrLY6ueV

+cpxyT5cmXRusFskZjv4GgxwcaTTktDjaZX2umxWrNt+

6CVcoL+wWTY6c3HaO1XZXnx7LQJyTu++G8/ICtVH1CAC

AAxHJPsqCxQoBEdB5odehOPqw1nhaT5Syaz8q4O8NOIi

e9ov0Xn6xtfiyXgjXfc++5SeynRm8MItEqccBzJbN3wp

3u0rDrBanSRxU5rF0T3nWW0bom7+p8KGUZuoremuimXS

uM5kc72OXblvRkRqwWKjduKEmAY9fG6v7IH6sNJc7ugO

Lz39HEeCmODSUsaQpiS5O/Y=)

dyn.weberdns.de.28INRRSIG DNSKEY10360(

201804012049472018030219494758340dyn.weberdns.de.

E4V4ymcuMUb+G5zVTozlgO0l1bRKc1Yw01+IQ671IjB/

j/3m8b8W9yCMr7jWY5NKOSRNP6mzMR30ILseiULVd88W

+hayVNR6ya5jV7BOeVJOrVsiB8O0YrcYTFqBcCuwtZ59

xiiDXRYjWxm1ynxQF3RUgHuDmIeLnzZUh+Q1gVb4TRnZ

ihZCbJMrdKtx8w8dLbtWNrYS5HeWFK6UvWRDiNNmJwI7

fKBOoF3mkaxlyf47o0l+oE082gt2v7x6z8LOsgCVyfd7

EuyCtZNGyjVriEowTqy10ZWe425Xzmu5/dIso/A9jpXd

zplaWQlLmWTUZpdBiRphR3wpmPcdrpeMPZ90L6sPXuQv

yzUzi7whrxpGc4BKVGmRl7Q+xqdxIdqKJ5/O0GBPzhiZ

YKmOQ4vPnyFOysdJtP80pKVU8UQPvfqadnPiX+vI/wKU

2ktPpYkE8vE3AoZZiU4UatMg667bJjMhutuRSfmCEz1N

AmzfKunLOtoMGnhm1fUz0UtcnijIrKUTUP2V1MvmOJVO

uoWXMI683NiNVTNtoYfNa7kjVhI3lJH7o/XwhTm/Kyem

AuZudFRXboQhZqwOJfYgj+2BuENUXs+Fc3qqhtJx01ZL

Jh2cJ353ZWjkIHKk9P9FBE5oWfQl6039hmxB3yxcyM+u

65OxvWXXjdEDbmL2fN2Roi0=)

;;Query time:2msec

;;SERVER:2003:de:2016:120::a08:53#53(2003:de:2016:120::a08:53)

;;WHEN:Fri Mar0221:56:05CET2018

;;MSG SIZE rcvd:2226

weberjoh@nb15-lx:~$

And of course DNSViz greatly shows that the single DS record points to the single KSK:

Conclusion

Since I was able to immediately replace the old DS record at my parents zone with the new DS record, I could delete the compromised KSK just a few moments after the TTL of the old DS timed out, which was about 1 hour in my example. The chain of trust was completely intact during this one hour period.

It is a best practice to train a KSK rollover *before* you must do it in a case of emergency. Write down the commands you need and schedule a KSK rollover on a regular basis to be prepared.