DNSSEC with the Nitrokey HSM

Please read the "Fair warnings"-section first. It will save you potential headaches.

Fair warnings a.k.a. "lessons learned"

Lessons learned from trial and error with the HSM in combination with PowerDNS:

You must run "PowerDNS Authoritative Server - version 4.1.X" from the official repository. Ubuntu 18.04's version does not have PCKS#11 support compiled-in.
If you create new keys in the HSM, restart PowerDNS. It will not properly pick them up otherwise.
The HSM has no concurrency, you must set distributor-threads=1 in your /etc/powerdns/pdns.d/pdns.local.conf to avoid issues.
Consequently, because it has no concurrency, only use it for your Key Signing Key (KSK). Do not use the HSM to sign entire zones, leave that to an additional software ZSK (which is safe because it's signed with the HSM's KSK). More on that later.
Do not use RSA keys. They are way too slow to be good for anything, stick with ECDSA.

You have been warned. If you take this into consideration, it will all be a much smoother experience.

Installing the software

On Ubuntu 18.04, you don't need to do much. Just install opensc. The module used is the default opensc-pkcs11, which means you don't have to set up anything special:

apt install opensc

Generating initial key(s)

By default, PowerDNS signs with algorithm 13, which is "ECDSA Curve P-256 with SHA-256". The HSM supports this too, as EC:prime256v1.
Alternatively, most TLD's use algorithm 8, which is "RSA with SHA-256". This is also supported by the HSM, as RSA:2048.

However, do not use RSA unless you absolutely need to. Elliptic Curve has many advantages over RSA. The keys are (in this case) 8 times smaller, which means:

You can store more keys in the HSM.
The HSM can sign much faster with EC than with RSA (up to 360 signatures per minute).
The resulting signatures fit inside a TCP-packet, with RSA they do not. That means that resolvers and clients, especially with high latency, will have faster responses from you.

Moving on.. use the following to create the keys. 648219 is the HSM's default User PIN, and -a is set to give the key a recognizable name in the HSM (for you!). In this case, I gave it the name powerdns. But for specific domains I label them with the name of the domain, like -a example.com instead:

pkcs11-tool -l -p 648219 -k --key-type EC:prime256v1 -a powerdns

This might take a while, after which you will see a Private+Public keypair shown.

Using the key

You will need the exact label of your token (the name you gave your Nitrokey HSM during initialisation):

lxc :: ~ » pkcs11-tool -T
Available slots:
Slot 0 (0x0): Nitrokey Nitrokey HSM (010000000000000000000000) 00 00
  token label        : SmartCard-HSM (UserPIN)

Now all you need to do is assign them to zones (ECDSA, algorithm 13):

for zone in example.com example.net example.org example.nl;
do 
    pdnsutil create-zone $zone
    pdnsutil hsm assign $zone ecdsap256sha256 ksk opensc-pkcs11 "SmartCard-HSM (UserPIN)" 648219 powerdns/ec
done

Restart PowerDNS to use the new keys

After all this you will want to restart PowerDNS, because for some reason it won't be able to load the PKCS#11 keys otherwise:

systemctl restart pdns

I think that also might have to do with the amount of backend threads like the other bug I ran in to, but I haven't double-checked that yet.

Increasing performance with a ZSK

PowerDNS defaults to (when using secure-zone) creating just one single KSK (Key Signing Key). Which is then shown as a CSK, Combined Signing Key). This is fine if you're just using software keys. A ZSK (Zone Signing Key) doesn't really gain you anything performance-wise (in fact, technically reduces it).

With the HSM, the opposite holds true. The HSM can only generate 360 signatures per minute (average of 6 per second). This might seem like a lot, but it's really not. If you want to increase the serial of a few zones in one go, it'll be busy for a long time.

So, generate a (preferably unique) Zone Signing Key per domain. You can regenerate these whenever you want, given that you only give the KSK's information to your provider or registry, anyway:

pdnsutil add-zone-key example.com zsk ecdsap256sha256 active

This makes it so that your HSM's key (KSK) authorizes this software key (ZSK) to sign the records in your zone. So instead of having to sign a zone with possibly 1000 DNS-records, your HSM only has to sign 2 DNSKEY-records. This means you can let your HSM handle up to 180 zones per minute, instead of maybe not even one, without compromising in security (though the ZSK can leak, while the HSM's KSK can not. Keep an eye out and replace the ZSK once in a while perhaps).

Example output / double-checking

With pdnsutil you can review what's going on:

lxc :: ~ » pdnsutil create-zone example.com
Creating empty zone 'example.com'
lxc :: ~ » pdnsutil hsm assign example.com ecdsap256sha256 ksk opensc-pkcs11 "SmartCard-HSM (UserPIN)" 648219 powerdns/ec
Module opensc-pkcs11 slot SmartCard-HSM (UserPIN) assigned to example.com with key id 775
lxc :: ~ » pdnsutil add-zone-key example.com zsk ecdsap256sha256 active
Added a ZSK with algorithm = 13, active=1
776
lxc :: ~ » pdnsutil list-keys
Zone                          Type    Size    Algorithm    ID   Location    Keytag
----------------------------------------------------------------------------------
example.com                   ZSK     256     ECDSAP256SHA256 776  cryptokeys  31876
example.com                   KSK     256     ECDSAP256SHA256 775  opensc-pkcs11,SmartCard-HSM (UserPIN),powerdns/ec 10293
lxc :: ~ » pdnsutil show-zone example.com
This is a Native zone
Metadata items: None
Zone has NSEC semantics
keys:
ID = 776 (ZSK), flags = 256, tag = 31876, algo = 13, bits = 256   Active ( ECDSAP256SHA256 )
ID = 775 (KSK), flags = 257, tag = 10293, algo = 13, bits = 256   Active ( ECDSAP256SHA256 )
KSK DNSKEY = example.com. IN DNSKEY 257 3 13 Nb5yucqSPw2c8mRdFpRqDekrYuMCGXt+t1jEZjC2FFmukqsle4vHCAsPtxZS332L414vkPd9pUDXH1nL9uRfTQ== ; ( ECDSAP256SHA256 )
DS = example.com. IN DS 10293 13 1 c4f8114fe3afbe5067cf5c9601ac89641beb6f81 ; ( SHA1 digest )
DS = example.com. IN DS 10293 13 2 7bbd57ee4a2acdd64e52825058c261a5b076b4f3c0ac9ce1dd208701625c3e63 ; ( SHA256 digest )
DS = example.com. IN DS 10293 13 4 4733b159196710235b82020e239ea168778b34f2fd57e1327b9c927efd0812677b15b8f21a76e58953b5e546a1ded039 ; ( SHA-384 digest )
lxc :: ~ » dig +dnssec +short dnskey example.com @::1 | sort
256 3 13 HHLZSTlvK5KiKmPu4vHhFaW9C2R9Mtx/PhWdEcuNCmyqqINspuvJJs27 6pBsrZwIwjs/1LRlHsOLVFB39sSQsA==
257 3 13 Nb5yucqSPw2c8mRdFpRqDekrYuMCGXt+t1jEZjC2FFmukqsle4vHCAsP txZS332L414vkPd9pUDXH1nL9uRfTQ==
DNSKEY 13 2 3600 20180906000000 20180816000000 10293 example.com. NkqQfqKBR/yfCAu/dAV9tZuwBfm6xZNXegLYjRtpTTT7u/5yoqcObSA4 MTwiHrbDNjqGTpVHtHKuHDrwzqHXbg==
lxc :: ~ »

As you can see, it's all working. You can see the tags 10293 (the KSK) in the output of pdnsutil show-zone example.com, and later in the signed result-set. Additionally, you can see the ZSK and KSK records, too (respectively 256 and 257).