tl;dr A faulty key rollover at DENIC brought down DNS resolution for the whole de TLD. I looked at publicly available data to figure out what happened.

Introduction

As I failed to login on my VPS server yesterday, I blamed the provider, IONOS. A couple of days ago they also had problems with their infrastructure, so I thought it’s the same issue. I refreshed and this time their domain name, login.ionos.de, could not be resolved. So I changed my DNS resolvers to quad9, and was happy to see everything works. Now I blamed my ISP, Deutsche Telekom, for having bogus resolvers. At the end, it turned out that that DENIC e.V., the registry for the de DNS namespace, has misconfigured DNSSEC causing basically every DNS resolution for names in this namespace to fail.

I know a thing or two about DNSSEC, so I thought I’ll take a look at publicly available data to figure out what went wrong last night. I’m using data from dnsviz.net and must note that this data is neither complete nor comprehensive.

If you are familiar with DNSSEC, you can skip the next section and jump right to the incident description or even to what I think was the root cause of the problem.

Background: DNSSEC

The Domain Name System (DNS) is a distributed directory that maps domain names to DNS records, e.g. IP addresses (A and AAAA records). DNS protocol predates the Web and has been around for decades. In its original desing, DNS does not cater for any kind of security (see RFC3833 for a threat model). The DNS Security Extensions (DNSSEC) were introduced to bring integrity, data origin authentication, and authenticated proof of non-existence to the DNS. It basically signs DNS records and creates a signature chain to a trust anchor, i.e., the root zone.

To create digital signatures, cryptographic keys, are used. DNSSEC enabled zones store the public part of the key as DNSKEY records and key the private part on the server. Signatures are created regularely and keys are rotated as part of security hygiene. Both procedures are usually automated and decades of DNSSEC experience makes signing and key rollovers rather trivial.

Incident

On 2026-05-05 around 10PM, DENIC, the registry for the de namespaces, announces issues regarding “reachability of DNSSEC-signed domains .de domains”. This is was incorrect. In fact, by that time you could not resolve a single domain name in the de namespace if you were using a DNSSEC-validating recursive resolver. That is, if your resolver (by default from your ISP) validates DNSSEC, then it would return a bogus result for de domain names.

The de top-level domain (TLD) is DNSSEC secured. To resolve a domain name in this namespace, two scenarios can happen. If the name resides in a sub-namespace (a DNS zone), which is DNSSEC enabled, then there must be valid signature from the root zone to this zone. Otherwise, the signature chain ends at the de zone with an explicit authenticated proof that the signature ends here. In technical terms, de nameservers return a signed NSEC3 record denoting that there exists no secure path (a DS record) for the child zone. In this case, the resolver knows that the child zone is not signed, and continues the resolution without further DNSSEC validation.

Yesterday, both scenarios failed for DENIC. In my case, ionos.de is not secure (see DNSViz). So de returned a signed NSEC3 to the resolver. The signature on this record, however, was invalid. This lead to the resolver considering the chain as broken and returning a SERVFAIL. Now if it was DNSSEC secure, like bund.de. The resolution would still fail because this time we would have a bad signature for a DS record (see DNSViz). A lose/lose situation if you were using a secure resolver.

Cause

On May 5th, DENIC performs a key rollover, that is, they use a new key to sign DNS records (DS, NSEC3, SOA, etc.) for their zone. It seems to me that they are using a pre-pub key rollver. It means that the new key is introduced beforehand, but it is not used to sign any records (see DNSViz). At an apropreate point in time, the old key stops signing, and the new key takes over. Note the actual rollover is more complex, but let’s leave it at that for the sake of simplicity.

It seems that the new key fails to generate valid signatures (see DNSViz). But this problem seems to be overseen by DENIC. It takes at least two hours before DENIC announces the issues. I cannot precisely reconstruct what happenes in the meanwhile and until the issue is resolved. But following DNSViz snapshops (start here and click on the “Next analysis” link), I can see some strange stuff (multiple SOA records, all with bad signatures) and attempts to rollback the key (see DNSViz) with a double-signature period. Finally the rollback succeeds and the new key and all its bogus signatures are removed from the nameserver.

Final words

Seeing how DNSSEC failures can bring down name resolutions for a while TLD zone is actually good news: it shows that resolvers are validating DNSSEC. But it also shows us the miserable user experience of DNSSEC: in the early hours no one was sure what is happening. I blamed IONOS and Telekom and was wrong both times. Moving forward, we must work on this aspect so end users are not left in the dark.