Nitrokey HSM: timeout on initialization after heavy use

[szszszsz]

Summary

Initialization command for the Nitrokey HSM can time out after it was heavily populated.

Possible Cause

The smart card is engaged in a cleanup operations (like garbage collection), and cannot reply immediately. The timeout extension commands are not sent to the host due to waiting for the smart card reply in a busy loop, which results in a timeout error shown by the OpenSC to the user.

Workaround

While the timeout error is shown to the user, this does not necessarily means the operation failed. The smart card continues execution of the initialization command and should finish after around 30 seconds of the processing (usually 10 seconds later after the timeout error is reported).
The workaround is to ignore the error and wait until the LED on the Nitrokey HSM will stop flashing, which will signalize the reception of the smart card's reply, or add a hard delay in case of the batch execution, and restart the pcscd (OpenSC smart card service) after doing so.

Possible solution

Send timeout extension packets on the bulk endpoint (during the busy wait loop in USART_ByteReceive()) while the smart card is busy and is not doing it by itself. Perhaps by reusing CCID_CheckUsbCommunication().

nitrokey-pro-firmware/src/ccid/smartcard/smartcard.c

Lines 1303 to 1322 in 47dd7f1

	static ErrorStatus USART_ByteReceive (u8 * Data, u32 TimeOut)
	{
	u32 Counter = 0;
	while ((USART_GetFlagStatus (USART1, USART_FLAG_RXNE) == RESET) && (Counter != TimeOut))
	{
	Counter++;
	}
	if (Counter != TimeOut)
	{
	*Data = (u8) USART_ReceiveData (USART1);
	return SUCCESS;
	}
	else
	{
	return ERROR;
	}
	}

See below for CCID protocol details:

• https://usb.org/sites/default/files/DWG_Smart-Card_CCID_Rev110.pdf

Used library:

• https://www.st.com/resource/en/application_note/cd00166510-smartcard-interface-with-stm32f10x-and-stm32l1xx-microcontrollers-stmicroelectronics.pdf

Details

Nitrokey HSM firmware: hsm-2
HSM smart card firmware: v3.4
Frequency: tester1: 1/6, tester2: 1/3

Scenario

1. Fill HSM by generating RSA4096 keys until the device is full, e.g. with:
   pkcs11-tool -l --pin 648219 --keypairgen --key-type rsa:4096 --id $i
2. Reset the device with: sc-hsm-tool --initialize --so-pin 3537363231383830 --pin 648219

Execution takes about 45-60 minutes.
Populating with binary data (like certificates) might be faster, but the occurrence frequency seems to drop. With generating ECC keys the issue was not reproduced.

Populating script

#!/bin/bash
set -xeo

for i in {1..1000}
do
   env OPENSC_DEBUG=2 pkcs11-tool -l --pin 648219 --keypairgen --key-type rsa:4096 --id $i
done