keyring in raft #23977

tgross · 2024-09-17T15:34:56Z

In Nomad 1.4, we implemented a root keyring to support encrypting Variables and signing Workload Identities. The keyring was originally stored with the AEAD-wrapped DEKs and the KEK together in a JSON keystore file on disk. We recently added support for using an external KMS for the KEK to improve the security model for the keyring. But we've encountered multiple instances of the keystore files not getting backed up separately from the Raft snapshot, resulting in failure to restore clusters from backup.

Move Nomad's root keyring into Raft (encrypted with a KMS/Vault where available) in order to eliminate operational problems with the separate on-disk keystore.

Fixes: #23665
Ref: https://hashicorp.atlassian.net/browse/NET-10523
Ref: https://go.hashi.co/rfc/nmd-203

Notes for reviewers:

I wanted to split this out into a couple of PRs but because we're replacing the existing state store table rather than having it side-by-side, there wasn't a reasonable subset that actually compiled doing it that way.
Documentation updates will be in a separate PR
Tooling updates for redacting key material will be in a separate PR

tgross · 2024-09-17T15:38:25Z

nomad/structs/structs.go

@@ -127,10 +127,13 @@ const (
 	ACLBindingRulesDeleteRequestType             MessageType = 58
 	NodePoolUpsertRequestType                    MessageType = 59
 	NodePoolDeleteRequestType                    MessageType = 60
+	JobVersionTagRequestType                     MessageType = 61


Phil has this message ID reserved in another PR, so I'm trying to avoid creating a conflict for him here 😁

tgross · 2024-09-17T15:39:44Z

helper/backoff.go

+
+// WithBackoffFunc is a helper that runs a function with geometric backoff + a
+// small jitter to a maximum backoff. It returns once the context closes, with
+// the error wrapping over the error from the function.


Note for reviewers: this logic was lifted from the csi_hook, which is reasonably battle-tested at this point. Once we merge this I'll go back and update the csi_hook to use this function.

tgross · 2024-09-17T16:11:10Z

~~Looks like I've got a failing plan applier test... will look into that ASAP~~ Should be fixed; we weren't waiting for the keyring initialization.

~~Landlock test failure is unrelated and fixed here: #23979~~ That's merged. Rebased on main.

nomad/leader.go

schmichael

There's a few errors we need to handle, but otherwise all of my comments are ignorable hand wringing to try to make the keyring as understandable to users and developers as possible.

Very excited to see this happening!

nomad/encrypter.go

nomad/structs/keyring.go

nomad/leader.go

nomad/plan_apply_test.go

schmichael · 2024-09-18T17:59:12Z

nomad/server.go

+	// Block until keys are decrypted
+	s.encrypter.IsReady(s.shutdownCtx)


Hm, so in the KMS case we block starting the RPC server until we've contacted at least 1 KMS? Will there be a log line to indicate this is what's happening? No RPCs means no observability like remote log streaming or goroutine dumps, so I want to make sure a KMS misconfiguration (like a wrong IP or port that a defensive firewall silently drops packets to) is observable from Nomad agent logs.

Hm, so in the KMS case we block starting the RPC server until we've contacted at least 1 KMS?

The RPC server starts before we get here. So we can receive RPCs (which will fail if they need to VerifyClaim), but we won't return from NewServer and won't register the agent with Consul as a result. We could conceivably move this up higher, but I didn't think it made sense to have it land before we start the legacy keyring replication.

Will there be a log line to indicate this is what's happening?

Yeah it'll actually be extremely noisy in the logs with the line keyring is not ready - waiting for keys: <list of IDs> each time it polls. I didn't want to make it the sort of thing that someone can miss a single log message and then not realize it's going to blow up later.

In Nomad 1.4, we implemented a root keyring to support encrypting Variables and signing Workload Identities. The keyring was originally stored with the AEAD-wrapped DEKs and the KEK together in a JSON keystore file on disk. We recently added support for using an external KMS for the KEK to improve the security model for the keyring. But we've encountered multiple instances of the keystore files not getting backed up separately from the Raft snapshot, resulting in failure to restore clusters from backup. Move Nomad's root keyring into Raft (encrypted with a KMS/Vault where available) in order to eliminate operational problems with the separate on-disk keystore. Fixes: #23665 Ref: https://hashicorp.atlassian.net/browse/NET-10523

In #23977 we merged a change to how the keyring was stored. Because keyring initialization takes slightly longer now, this uncovered existing timing bugs in some of our tests where tests that require the keyring (ex. plan applier tests) were waiting for the leader but not the keyring initialization. Fix some of the examples we've seen cause test flakes.

In #23977 we moved the keyring into Raft, which can expose key material in Raft snapshots when using the less-secure AEAD keyring instead of KMS. This changeset adds tools for redacting this material from snapshots: * The `operator snapshot state` command gains the ability to display key metadata (only), which respects the `-filter` option. * The `operator snapshot save` command gains a `-redact` option that removes key material from the snapshot after it's downloaded. * A new `operator snapshot redact` command allows removing key material from an existing snapshot.

In #23977 we moved the keyring into Raft. This changeset documents the operational changes and adds notes to the upgrade guide.

In #23977 we moved the keyring into Raft, which can expose key material in Raft snapshots when using the less-secure AEAD keyring instead of KMS. This changeset adds tools for redacting this material from snapshots: * The `operator snapshot state` command gains the ability to display key metadata (only), which respects the `-filter` option. * The `operator snapshot save` command gains a `-redact` option that removes key material from the snapshot after it's downloaded. * A new `operator snapshot redact` command allows removing key material from an existing snapshot.

In #23977 we moved the keyring into Raft. This changeset documents the operational changes and adds notes to the upgrade guide.

vercel bot deployed to Preview – nomad-ui September 17, 2024 15:35 View deployment

tgross added this to the 1.9.0 milestone Sep 17, 2024

tgross added theme/keyring type/enhancement labels Sep 17, 2024

tgross commented Sep 17, 2024

View reviewed changes

tgross force-pushed the keyring-in-raft branch 2 times, most recently from 428667f to 0cac402 Compare September 17, 2024 15:46

tgross marked this pull request as ready for review September 17, 2024 15:46

tgross requested review from schmichael, pkazmierczak and shoenig September 17, 2024 15:47

vercel bot deployed to Preview – nomad-ui September 17, 2024 15:47 View deployment

tgross force-pushed the keyring-in-raft branch from 0cac402 to 94c16a3 Compare September 17, 2024 15:58

vercel bot deployed to Preview – nomad-ui September 17, 2024 15:59 View deployment

tgross force-pushed the keyring-in-raft branch from 94c16a3 to bfab122 Compare September 17, 2024 16:19

vercel bot deployed to Preview – nomad-ui September 17, 2024 16:21 View deployment

tgross force-pushed the keyring-in-raft branch from bfab122 to 8e1f37d Compare September 17, 2024 20:57

vercel bot deployed to Preview – nomad-ui September 17, 2024 20:58 View deployment

This comment was marked as resolved.

Sign in to view

tgross commented Sep 18, 2024

View reviewed changes

nomad/leader.go Outdated Show resolved Hide resolved

vercel bot deployed to Preview – nomad-ui September 18, 2024 13:57 View deployment

tgross force-pushed the keyring-in-raft branch from 5c73c22 to 4959f20 Compare September 18, 2024 15:06

vercel bot deployed to Preview – nomad-ui September 18, 2024 15:07 View deployment

tgross force-pushed the keyring-in-raft branch from 4959f20 to 602ffbc Compare September 18, 2024 15:09

vercel bot deployed to Preview – nomad-ui September 18, 2024 15:11 View deployment

schmichael approved these changes Sep 18, 2024

View reviewed changes

vercel bot deployed to Preview – nomad-ui September 18, 2024 19:43 View deployment

vercel bot deployed to Preview – nomad-ui September 18, 2024 21:10 View deployment

tgross added 6 commits September 19, 2024 11:07

refactor: rename structs

b02a480

tidying up docstrings and order of function definitions for readability

665a7df

add test for plan apply without ready keyring

a1b8ce1

allow restore to work without encrypter for CLI

6be644d

rename wasMigrated to stateChanged for clarity

adb696c

bump min version to 1.9.0-dev

1e781c2

tgross force-pushed the keyring-in-raft branch from 18fbc57 to 1e781c2 Compare September 19, 2024 15:07

vercel bot deployed to Preview – nomad-ui September 19, 2024 15:08 View deployment

missed waiting for keyring in plan apply test

67579ae

vercel bot deployed to Preview – nomad-ui September 19, 2024 15:33 View deployment

more tests not waiting for keyring

d603d96

vercel bot deployed to Preview – nomad-ui September 19, 2024 15:52 View deployment

tgross merged commit 44f4970 into main Sep 19, 2024
27 checks passed

tgross deleted the keyring-in-raft branch September 19, 2024 17:56

tgross mentioned this pull request Sep 19, 2024

Store keyring in Raft #23665

Closed

tgross mentioned this pull request Sep 20, 2024

test: wait for keyring for plan apply tests #24021

Merged

tgross mentioned this pull request Sep 20, 2024

command line tools for redacting keyring from snapshots #24023

Merged

tgross added a commit that referenced this pull request Sep 20, 2024

docs: update key management docs for keyring-in-Raft

2abdf68

In #23977 we moved the keyring into Raft. This changeset documents the operational changes and adds notes to the upgrade guide.

tgross mentioned this pull request Sep 20, 2024

docs: update key management docs for keyring-in-Raft #24026

Merged

tgross added a commit that referenced this pull request Sep 25, 2024

docs: update key management docs for keyring-in-Raft (#24026)

a3a2028

In #23977 we moved the keyring into Raft. This changeset documents the operational changes and adds notes to the upgrade guide.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

keyring in raft #23977

keyring in raft #23977

tgross commented Sep 17, 2024 •

edited

Loading

tgross Sep 17, 2024

tgross Sep 17, 2024

tgross commented Sep 17, 2024 •

edited

Loading

This comment was marked as resolved.

schmichael left a comment

schmichael Sep 18, 2024

tgross Sep 18, 2024 •

edited

Loading

		// Block until keys are decrypted
		s.encrypter.IsReady(s.shutdownCtx)

keyring in raft #23977

keyring in raft #23977

Conversation

tgross commented Sep 17, 2024 • edited Loading

tgross Sep 17, 2024

Choose a reason for hiding this comment

tgross Sep 17, 2024

Choose a reason for hiding this comment

tgross commented Sep 17, 2024 • edited Loading

This comment was marked as resolved.

schmichael left a comment

Choose a reason for hiding this comment

schmichael Sep 18, 2024

Choose a reason for hiding this comment

tgross Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

tgross commented Sep 17, 2024 •

edited

Loading

tgross commented Sep 17, 2024 •

edited

Loading

tgross Sep 18, 2024 •

edited

Loading