Skip to content

Commit

Permalink
Crypto: deprecate external key storage (#3149)
Browse files Browse the repository at this point in the history
  • Loading branch information
@reinkrul
reinkrul committed May 31, 2024
1 parent 7a5aa1a commit 4dc62e6
Show file tree
Hide file tree
Showing 9 changed files with 25 additions and 192 deletions.
6 changes: 2 additions & 4 deletions README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -180,9 +180,7 @@ The following options can be configured on the server:
verbosity info Log level (trace, debug, info, warn, error)
httpclient.timeout 30s Request time-out for HTTP clients, such as '10s'. Refer to Golang's 'time.Duration' syntax for a more elaborate description of the syntax.
**Crypto**
crypto.storage Storage to use, 'external' for an external backend (experimental), 'fs' for file system (for development purposes), 'vaultkv' for Vault KV store (recommended, will be replaced by external backend in future).
crypto.external.address Address of the external storage service.
crypto.external.timeout 100ms Time-out when invoking the external storage backend, in Golang time.Duration string format (e.g. 1s).
crypto.storage Storage to use, 'fs' for file system (for development purposes), 'vaultkv' for HashiCorp Vault KV store,'external' for an external backend (deprecated).
crypto.vault.address The Vault address. If set it overwrites the VAULT_ADDR env var.
crypto.vault.pathprefix kv The Vault path prefix.
crypto.vault.timeout 5s Timeout of client calls to Vault, in Golang time.Duration string format (e.g. 1s).
Expand All @@ -199,7 +197,7 @@ The following options can be configured on the server:
http.internal.auth.type Whether to enable authentication for /internal endpoints, specify 'token_v2' for bearer token mode or 'token' for legacy bearer token mode.
http.public.address \:8080 Address and port the server will be listening to for public-facing endpoints.
**JSONLD**
jsonld.contexts.localmapping [https://w3id.org/vc/status-list/2021/v1=assets/contexts/w3c-statuslist2021.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson,https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist.
jsonld.contexts.localmapping [https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3id.org/vc/status-list/2021/v1=assets/contexts/w3c-statuslist2021.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist.
jsonld.contexts.remoteallowlist [https://schema.org,https://www.w3.org/2018/credentials/v1,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json,https://w3id.org/vc/status-list/2021/v1] In strict mode, fetching external JSON-LD contexts is not allowed except for context-URLs listed here.
**PKI**
pki.maxupdatefailhours 4 Maximum number of hours that a denylist update can fail
Expand Down
45 changes: 6 additions & 39 deletions crypto/cmd/cmd.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,19 @@ func FlagSet() *pflag.FlagSet {
flags := pflag.NewFlagSet("crypto", pflag.ContinueOnError)

defs := cryptoEngine.DefaultCryptoConfig()
flags.String("crypto.storage", defs.Storage, fmt.Sprintf("Storage to use, '%s' for an external backend (experimental), "+
"'%s' for file system (for development purposes), "+
"'%s' for Vault KV store (recommended, will be replaced by external backend in future).", external.StorageType, fs.StorageType, vault.StorageType))
flags.String("crypto.storage", defs.Storage, fmt.Sprintf("Storage to use, '%s' for file system (for development purposes), "+
"'%s' for HashiCorp Vault KV store,"+
"'%s' for an external backend (deprecated).", fs.StorageType, vault.StorageType, external.StorageType))
flags.String("crypto.vault.token", defs.Vault.Token, "The Vault token. If set it overwrites the VAULT_TOKEN env var.")
flags.String("crypto.vault.address", defs.Vault.Address, "The Vault address. If set it overwrites the VAULT_ADDR env var.")
flags.Duration("crypto.vault.timeout", defs.Vault.Timeout, "Timeout of client calls to Vault, in Golang time.Duration string format (e.g. 1s).")
flags.String("crypto.vault.pathprefix", defs.Vault.PathPrefix, "The Vault path prefix.")
flags.String("crypto.external.address", defs.External.Address, "Address of the external storage service.")
flags.Duration("crypto.external.timeout", defs.External.Timeout, "Time-out when invoking the external storage backend, in Golang time.Duration string format (e.g. 1s).")

_ = flags.MarkDeprecated("crypto.external.address", "Use another key storage backend instead of the external storage backend.")
_ = flags.MarkDeprecated("crypto.external.timeout", "Use another key storage backend instead of the external storage backend.")

return flags
}

Expand All @@ -57,45 +60,9 @@ func ServerCmd() *cobra.Command {
Short: "crypto commands",
}
cmd.AddCommand(fs2VaultCommand())
cmd.AddCommand(fs2ExternalStore())
return cmd
}

func fs2ExternalStore() *cobra.Command {
return &cobra.Command{
Use: "fs2external [directory]",
Short: "Imports private keys from filesystem based storage (located at the given directory) into the storage server.",
Long: "Imports private keys from filesystem based storage into the secret store server. The given directory must contain the private key files. " +
"The Nuts node must be configured to use storage-api as crypto storage. Can only be run on the local Nuts node, from the directory where nuts.yaml resides.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
cmd.Println("Exporting keys from FileSystem storage to the external storage service...")

instance, err := LoadCryptoModule(cmd)
if err != nil {
return err
}
config := instance.Config().(*cryptoEngine.Config)
targetStorage, err := external.NewAPIClient(config.External)
if err != nil {
return fmt.Errorf("unable to set up external crypto API client: %w", err)
}

directory := args[0]
keys, err := fsToOtherStorage(cmd.Context(), directory, targetStorage)
cmd.Println(fmt.Sprintf("Imported %d keys:", len(keys)))
for _, key := range keys {
cmd.Println(" ", key)
}
if err != nil {
cmd.Println("Failed to import all fs keys into external store:", err)
return err
}
return nil
},
}
}

func fs2VaultCommand() *cobra.Command {
return &cobra.Command{
Use: "fs2vault [directory]",
Expand Down
59 changes: 0 additions & 59 deletions crypto/cmd/cmd_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,65 +121,6 @@ func Test_fs2VaultCommand(t *testing.T) {
assert.Contains(t, output, "pk3")
}

func Test_fs2ExternalStore(t *testing.T) {
// tests imports 1 new key, skips a known one, and the server returns an error for the third one
t.Run("ok", func(t *testing.T) {
// Set up webserver that stubs Vault
importRequests := make(map[string]string, 0)

s := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
data, err := io.ReadAll(request.Body)
if err != nil {
t.Fatal(err)
}
importRequests[request.RequestURI] = string(data)
switch request.RequestURI {
case "/secrets/pk1":
writer.WriteHeader(http.StatusOK)
case "/secrets/pk2":
writer.WriteHeader(http.StatusConflict)
case "/secrets/pk3":
writer.WriteHeader(http.StatusBadRequest)
}

}))
defer s.Close()

// Configure target
t.Setenv("NUTS_CRYPTO_STORAGE", "external")
t.Setenv("NUTS_CRYPTO_EXTERNAL_ADDRESS", s.URL)

testDirectory := testIo.TestDirectory(t)
setupFSStoreData(t, testDirectory)

outBuf := new(bytes.Buffer)
cryptoCmd := ServerCmd()
for _, cmd := range cryptoCmd.Commands() {
cmd.Flags().AddFlagSet(core.FlagSet())
cmd.Flags().AddFlagSet(FlagSet())
}
cryptoCmd.SetOut(outBuf)
cryptoCmd.SetArgs([]string{"fs2external", testDirectory})

err := cryptoCmd.Execute()
require.EqualError(t, err, "unable to store private key in Vault (kid=pk3): unable to save private key: server returned HTTP 400")

// Assert 2 keys were imported into Vault on the expected paths
assert.Len(t, importRequests, 3)
assert.Contains(t, importRequests, "/secrets/pk1")
assert.Contains(t, importRequests, "/secrets/pk2")
assert.Contains(t, importRequests, "/secrets/pk3")

// Assert imported keys are logged
output := outBuf.String()
assert.Contains(t, output, "pk1")
// key 2 is skipped because it already exists
assert.NotContains(t, output, "pk2")
// key 3 caused an error
assert.Contains(t, output, "Failed to import all fs keys into external store: unable to store private key in Vault (kid=pk3)")
})
}

// setupFSStoreData creates a directory with 2 keys in it
// Can be used to test the fs2* commands
func setupFSStoreData(t *testing.T, testDirectory string) {
Expand Down
1 change: 1 addition & 0 deletions crypto/crypto.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ func (client *Crypto) setupFSBackend(config core.ServerConfig) error {

func (client *Crypto) setupStorageAPIBackend() error {
log.Logger().Debug("Setting up StorageAPI backend for storage of private key material.")
log.Logger().Warn("External key storage backend is deprecated and will be removed in the future.")
apiBackend, err := external.NewAPIClient(client.config.External)
if err != nil {
return fmt.Errorf("unable to set up external crypto API client: %w", err)
Expand Down
14 changes: 1 addition & 13 deletions docs/pages/deployment/cli-reference.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@ The following options apply to the server commands below:
--auth.irma.schememanager string IRMA schemeManager to use for attributes. Can be either 'pbdf' or 'irma-demo'. (default "pbdf")
--configfile string Nuts config file (default "./config/nuts.yaml")
--cpuprofile string When set, a CPU profile is written to the given path. Ignored when strictmode is set.
--crypto.external.address string Address of the external storage service.
--crypto.external.timeout duration Time-out when invoking the external storage backend, in Golang time.Duration string format (e.g. 1s). (default 100ms)
--crypto.storage string Storage to use, 'external' for an external backend (experimental), 'fs' for file system (for development purposes), 'vaultkv' for Vault KV store (recommended, will be replaced by external backend in future).
--crypto.storage string Storage to use, 'fs' for file system (for development purposes), 'vaultkv' for HashiCorp Vault KV store,'external' for an external backend (deprecated).
--crypto.vault.address string The Vault address. If set it overwrites the VAULT_ADDR env var.
--crypto.vault.pathprefix string The Vault path prefix. (default "kv")
--crypto.vault.timeout duration Timeout of client calls to Vault, in Golang time.Duration string format (e.g. 1s). (default 5s)
Expand Down Expand Up @@ -100,16 +98,6 @@ Prints the current config
nuts config [flags]


nuts crypto fs2external
^^^^^^^^^^^^^^^^^^^^^^^

Imports private keys from filesystem based storage into the secret store server. The given directory must contain the private key files. The Nuts node must be configured to use storage-api as crypto storage. Can only be run on the local Nuts node, from the directory where nuts.yaml resides.

::

nuts crypto fs2external [directory] [flags]


nuts crypto fs2vault
^^^^^^^^^^^^^^^^^^^^

Expand Down
Loading

0 comments on commit 4dc62e6

Please sign in to comment.