Store keyring in Raft

[schmichael]

Background

With the introduction of Variables in Nomad 1.4, Nomad has had an internal keyring consisting of:

1. A Data Encryption Key (DEK) for encrypting Variables in Raft and on disk
2. A Key Encryption Key (KEK) for encrypting the DEK
3. A Workload Identity signing key (see identity: default to RS256 for new workload ids #18882 for details)

The KEK is stored on disk distinctly from Raft logs and snapshots so that secret material may not be extracted from leaked snapshots.

As of #23580 Nomad will support using an external KMS for the KEK. However the wrapped (encrypted) DEK and WI keys will still be in the on disk keyring outside of snapshots.

The on disk keyring, even when using a KMS, poses an unfortunate challenge for Nomad operators: they must backup the keyring like snapshots, but distinctly. If a single script backs up both snapshots and keyring to the same location, the only remaining benefit to keeping the two distinct is in the case of sharing snapshots with a third party (eg HashiCorp) for debugging.

Nomad engineering has encountered multiple outages caused by users unaware that they need to backup and restore the keyring distinct but alongside the snapshot.

Proposal

The downsides of storing the keyring separate from snapshots outweighs the potential security benefits. The security benefits are likely rarely realized due to the difficulty of splitting backups across multiple processes and locations. This user experience should be fixed by doing the following:

1. By default, without a KMS configured, Nomad should store the unencrypted KEK and other wrapped keys in Raft and snapshots. Server agents should log a warning instructing users their snapshots are functionally unencrypted and using a KMS is desirable (or linking to such docs).
2. When using a KMS, wrapped key material should be stored in Raft and snapshots.
3. The existing behavior of storing the KEK on disk distinctly from Raft and snapshots should be available for users who do not want to rely on an external KMS but also do not want to risk exposing their keys in Raft or snapshots.

In order to support storing key material separate from snapshots, the following features should be added:

1. The Snapshot API and operator snapshot save CLI should gain a parameter that allows generating snapshots without the KEK included (for users without a KMS). Users with a KMS configured will never receive their KEK in a snapshot.
2. A new operator snapshot redact command should remove the KEK from a snapshot if one is present. This command could optionally store the extracted key to a file for backing up distinctly from the snapshot.
3. operator snapshot inspect should warn users if a snapshot contains a KEK.

Attempted Solutions

The Nomad team has attempted to document proper keyring handling, but has encountered multiple instances of operators being unaware of how to properly handle keyrings during backups and restores.

[schmichael]

Proposal
...
3. The existing behavior of storing the KEK on disk distinctly from Raft and snapshots should be available for users who do not want to rely on an external KMS but also do not want to risk exposing their keys in Raft or snapshots.

I don't think this is required. The benefits are just too hard to realize. Security sensitive users must use a KMS. Just offering this option adds a non-trivial amount of cognitive overhead to operators and security teams to find the right solution for their use case, and that effort is better put toward using a KMS.

[tgross]

#23977 has merged. I'll have a separate PR for docs and CLI tools