diff --git a/README.rst b/README.rst index 528698e0ec..e653f38bea 100644 --- a/README.rst +++ b/README.rst @@ -180,9 +180,7 @@ The following options can be configured on the server: verbosity info Log level (trace, debug, info, warn, error) httpclient.timeout 30s Request time-out for HTTP clients, such as '10s'. Refer to Golang's 'time.Duration' syntax for a more elaborate description of the syntax. **Crypto** - crypto.storage Storage to use, 'external' for an external backend (experimental), 'fs' for file system (for development purposes), 'vaultkv' for Vault KV store (recommended, will be replaced by external backend in future). - crypto.external.address Address of the external storage service. - crypto.external.timeout 100ms Time-out when invoking the external storage backend, in Golang time.Duration string format (e.g. 1s). + crypto.storage Storage to use, 'fs' for file system (for development purposes), 'vaultkv' for HashiCorp Vault KV store,'external' for an external backend (deprecated). crypto.vault.address The Vault address. If set it overwrites the VAULT_ADDR env var. crypto.vault.pathprefix kv The Vault path prefix. crypto.vault.timeout 5s Timeout of client calls to Vault, in Golang time.Duration string format (e.g. 1s). @@ -199,7 +197,7 @@ The following options can be configured on the server: http.internal.auth.type Whether to enable authentication for /internal endpoints, specify 'token_v2' for bearer token mode or 'token' for legacy bearer token mode. http.public.address \:8080 Address and port the server will be listening to for public-facing endpoints. **JSONLD** - jsonld.contexts.localmapping [https://w3id.org/vc/status-list/2021/v1=assets/contexts/w3c-statuslist2021.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson,https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist. + jsonld.contexts.localmapping [https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3id.org/vc/status-list/2021/v1=assets/contexts/w3c-statuslist2021.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist. jsonld.contexts.remoteallowlist [https://schema.org,https://www.w3.org/2018/credentials/v1,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json,https://w3id.org/vc/status-list/2021/v1] In strict mode, fetching external JSON-LD contexts is not allowed except for context-URLs listed here. **PKI** pki.maxupdatefailhours 4 Maximum number of hours that a denylist update can fail diff --git a/crypto/cmd/cmd.go b/crypto/cmd/cmd.go index 01825f66f9..e56e2c9c07 100644 --- a/crypto/cmd/cmd.go +++ b/crypto/cmd/cmd.go @@ -37,9 +37,9 @@ func FlagSet() *pflag.FlagSet { flags := pflag.NewFlagSet("crypto", pflag.ContinueOnError) defs := cryptoEngine.DefaultCryptoConfig() - flags.String("crypto.storage", defs.Storage, fmt.Sprintf("Storage to use, '%s' for an external backend (experimental), "+ - "'%s' for file system (for development purposes), "+ - "'%s' for Vault KV store (recommended, will be replaced by external backend in future).", external.StorageType, fs.StorageType, vault.StorageType)) + flags.String("crypto.storage", defs.Storage, fmt.Sprintf("Storage to use, '%s' for file system (for development purposes), "+ + "'%s' for HashiCorp Vault KV store,"+ + "'%s' for an external backend (deprecated).", fs.StorageType, vault.StorageType, external.StorageType)) flags.String("crypto.vault.token", defs.Vault.Token, "The Vault token. If set it overwrites the VAULT_TOKEN env var.") flags.String("crypto.vault.address", defs.Vault.Address, "The Vault address. If set it overwrites the VAULT_ADDR env var.") flags.Duration("crypto.vault.timeout", defs.Vault.Timeout, "Timeout of client calls to Vault, in Golang time.Duration string format (e.g. 1s).") @@ -47,6 +47,9 @@ func FlagSet() *pflag.FlagSet { flags.String("crypto.external.address", defs.External.Address, "Address of the external storage service.") flags.Duration("crypto.external.timeout", defs.External.Timeout, "Time-out when invoking the external storage backend, in Golang time.Duration string format (e.g. 1s).") + _ = flags.MarkDeprecated("crypto.external.address", "Use another key storage backend instead of the external storage backend.") + _ = flags.MarkDeprecated("crypto.external.timeout", "Use another key storage backend instead of the external storage backend.") + return flags } @@ -57,45 +60,9 @@ func ServerCmd() *cobra.Command { Short: "crypto commands", } cmd.AddCommand(fs2VaultCommand()) - cmd.AddCommand(fs2ExternalStore()) return cmd } -func fs2ExternalStore() *cobra.Command { - return &cobra.Command{ - Use: "fs2external [directory]", - Short: "Imports private keys from filesystem based storage (located at the given directory) into the storage server.", - Long: "Imports private keys from filesystem based storage into the secret store server. The given directory must contain the private key files. " + - "The Nuts node must be configured to use storage-api as crypto storage. Can only be run on the local Nuts node, from the directory where nuts.yaml resides.", - Args: cobra.ExactArgs(1), - RunE: func(cmd *cobra.Command, args []string) error { - cmd.Println("Exporting keys from FileSystem storage to the external storage service...") - - instance, err := LoadCryptoModule(cmd) - if err != nil { - return err - } - config := instance.Config().(*cryptoEngine.Config) - targetStorage, err := external.NewAPIClient(config.External) - if err != nil { - return fmt.Errorf("unable to set up external crypto API client: %w", err) - } - - directory := args[0] - keys, err := fsToOtherStorage(cmd.Context(), directory, targetStorage) - cmd.Println(fmt.Sprintf("Imported %d keys:", len(keys))) - for _, key := range keys { - cmd.Println(" ", key) - } - if err != nil { - cmd.Println("Failed to import all fs keys into external store:", err) - return err - } - return nil - }, - } -} - func fs2VaultCommand() *cobra.Command { return &cobra.Command{ Use: "fs2vault [directory]", diff --git a/crypto/cmd/cmd_test.go b/crypto/cmd/cmd_test.go index de9ccf8f5c..aeeec8f1e0 100644 --- a/crypto/cmd/cmd_test.go +++ b/crypto/cmd/cmd_test.go @@ -121,65 +121,6 @@ func Test_fs2VaultCommand(t *testing.T) { assert.Contains(t, output, "pk3") } -func Test_fs2ExternalStore(t *testing.T) { - // tests imports 1 new key, skips a known one, and the server returns an error for the third one - t.Run("ok", func(t *testing.T) { - // Set up webserver that stubs Vault - importRequests := make(map[string]string, 0) - - s := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { - data, err := io.ReadAll(request.Body) - if err != nil { - t.Fatal(err) - } - importRequests[request.RequestURI] = string(data) - switch request.RequestURI { - case "/secrets/pk1": - writer.WriteHeader(http.StatusOK) - case "/secrets/pk2": - writer.WriteHeader(http.StatusConflict) - case "/secrets/pk3": - writer.WriteHeader(http.StatusBadRequest) - } - - })) - defer s.Close() - - // Configure target - t.Setenv("NUTS_CRYPTO_STORAGE", "external") - t.Setenv("NUTS_CRYPTO_EXTERNAL_ADDRESS", s.URL) - - testDirectory := testIo.TestDirectory(t) - setupFSStoreData(t, testDirectory) - - outBuf := new(bytes.Buffer) - cryptoCmd := ServerCmd() - for _, cmd := range cryptoCmd.Commands() { - cmd.Flags().AddFlagSet(core.FlagSet()) - cmd.Flags().AddFlagSet(FlagSet()) - } - cryptoCmd.SetOut(outBuf) - cryptoCmd.SetArgs([]string{"fs2external", testDirectory}) - - err := cryptoCmd.Execute() - require.EqualError(t, err, "unable to store private key in Vault (kid=pk3): unable to save private key: server returned HTTP 400") - - // Assert 2 keys were imported into Vault on the expected paths - assert.Len(t, importRequests, 3) - assert.Contains(t, importRequests, "/secrets/pk1") - assert.Contains(t, importRequests, "/secrets/pk2") - assert.Contains(t, importRequests, "/secrets/pk3") - - // Assert imported keys are logged - output := outBuf.String() - assert.Contains(t, output, "pk1") - // key 2 is skipped because it already exists - assert.NotContains(t, output, "pk2") - // key 3 caused an error - assert.Contains(t, output, "Failed to import all fs keys into external store: unable to store private key in Vault (kid=pk3)") - }) -} - // setupFSStoreData creates a directory with 2 keys in it // Can be used to test the fs2* commands func setupFSStoreData(t *testing.T, testDirectory string) { diff --git a/crypto/crypto.go b/crypto/crypto.go index fedd80976a..45a220a5c8 100644 --- a/crypto/crypto.go +++ b/crypto/crypto.go @@ -100,6 +100,7 @@ func (client *Crypto) setupFSBackend(config core.ServerConfig) error { func (client *Crypto) setupStorageAPIBackend() error { log.Logger().Debug("Setting up StorageAPI backend for storage of private key material.") + log.Logger().Warn("External key storage backend is deprecated and will be removed in the future.") apiBackend, err := external.NewAPIClient(client.config.External) if err != nil { return fmt.Errorf("unable to set up external crypto API client: %w", err) diff --git a/docs/pages/deployment/cli-reference.rst b/docs/pages/deployment/cli-reference.rst index 078424ad4c..410499a381 100755 --- a/docs/pages/deployment/cli-reference.rst +++ b/docs/pages/deployment/cli-reference.rst @@ -20,9 +20,7 @@ The following options apply to the server commands below: --auth.irma.schememanager string IRMA schemeManager to use for attributes. Can be either 'pbdf' or 'irma-demo'. (default "pbdf") --configfile string Nuts config file (default "./config/nuts.yaml") --cpuprofile string When set, a CPU profile is written to the given path. Ignored when strictmode is set. - --crypto.external.address string Address of the external storage service. - --crypto.external.timeout duration Time-out when invoking the external storage backend, in Golang time.Duration string format (e.g. 1s). (default 100ms) - --crypto.storage string Storage to use, 'external' for an external backend (experimental), 'fs' for file system (for development purposes), 'vaultkv' for Vault KV store (recommended, will be replaced by external backend in future). + --crypto.storage string Storage to use, 'fs' for file system (for development purposes), 'vaultkv' for HashiCorp Vault KV store,'external' for an external backend (deprecated). --crypto.vault.address string The Vault address. If set it overwrites the VAULT_ADDR env var. --crypto.vault.pathprefix string The Vault path prefix. (default "kv") --crypto.vault.timeout duration Timeout of client calls to Vault, in Golang time.Duration string format (e.g. 1s). (default 5s) @@ -100,16 +98,6 @@ Prints the current config nuts config [flags] -nuts crypto fs2external -^^^^^^^^^^^^^^^^^^^^^^^ - -Imports private keys from filesystem based storage into the secret store server. The given directory must contain the private key files. The Nuts node must be configured to use storage-api as crypto storage. Can only be run on the local Nuts node, from the directory where nuts.yaml resides. - -:: - - nuts crypto fs2external [directory] [flags] - - nuts crypto fs2vault ^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/pages/deployment/server_options.rst b/docs/pages/deployment/server_options.rst index 2f49f7d824..523c7ad4ae 100755 --- a/docs/pages/deployment/server_options.rst +++ b/docs/pages/deployment/server_options.rst @@ -15,9 +15,7 @@ verbosity info Log level (trace, debug, info, warn, error) httpclient.timeout 30s Request time-out for HTTP clients, such as '10s'. Refer to Golang's 'time.Duration' syntax for a more elaborate description of the syntax. **Crypto** - crypto.storage Storage to use, 'external' for an external backend (experimental), 'fs' for file system (for development purposes), 'vaultkv' for Vault KV store (recommended, will be replaced by external backend in future). - crypto.external.address Address of the external storage service. - crypto.external.timeout 100ms Time-out when invoking the external storage backend, in Golang time.Duration string format (e.g. 1s). + crypto.storage Storage to use, 'fs' for file system (for development purposes), 'vaultkv' for HashiCorp Vault KV store,'external' for an external backend (deprecated). crypto.vault.address The Vault address. If set it overwrites the VAULT_ADDR env var. crypto.vault.pathprefix kv The Vault path prefix. crypto.vault.timeout 5s Timeout of client calls to Vault, in Golang time.Duration string format (e.g. 1s). @@ -34,7 +32,7 @@ http.internal.auth.type Whether to enable authentication for /internal endpoints, specify 'token_v2' for bearer token mode or 'token' for legacy bearer token mode. http.public.address \:8080 Address and port the server will be listening to for public-facing endpoints. **JSONLD** - jsonld.contexts.localmapping [https://w3id.org/vc/status-list/2021/v1=assets/contexts/w3c-statuslist2021.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson,https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist. + jsonld.contexts.localmapping [https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3id.org/vc/status-list/2021/v1=assets/contexts/w3c-statuslist2021.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist. jsonld.contexts.remoteallowlist [https://schema.org,https://www.w3.org/2018/credentials/v1,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json,https://w3id.org/vc/status-list/2021/v1] In strict mode, fetching external JSON-LD contexts is not allowed except for context-URLs listed here. **PKI** pki.maxupdatefailhours 4 Maximum number of hours that a denylist update can fail diff --git a/docs/pages/deployment/storage.rst b/docs/pages/deployment/storage.rst index 56c3c2610c..a97fdafce1 100644 --- a/docs/pages/deployment/storage.rst +++ b/docs/pages/deployment/storage.rst @@ -81,48 +81,13 @@ In any case, make sure the key-value secret engine exists before trying to migra External Store API ================== -.. note:: - - The external store API is still experimental and may change in the future. .. warning:: + The external store API is deprecated and will be removed in the next major release. Anyone with access to the external store can read/write your private keys, so make sure it's properly secured and only the Nuts node can access it. The Nuts node can be configured to use an external store for private keys. This allows you to use your own key management system. The external store must implement the Nuts Secret store API specification. This OpenAPI specification is available from the `Secret Store API repository `__ on GitHub. - -Configuration -^^^^^^^^^^^^^ - -In order to use an external store, you need to set the ``crypto.storage`` option to ``external``. You also need to configure the ``crypto.external.address`` option to the address of the external store. The following example shows the typical configuration for a Nuts Vault proxy. - -.. code-block:: yaml - - crypto: - storage: external - external: - address: https://localhost:8210 - -Migrating to external storage -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you want to migrate your private keys from the filesystem to an external store, you can use the Nuts command line interface with the ``fs2external`` crypto command. It takes the directory containing the private keys as argument (the example assumes the container is called *nuts-node* and *NUTS_DATADIR=/opt/nuts/data*): - -.. code-block:: shell - - docker exec nuts-node nuts crypto fs2external /opt/nuts/data/crypto - -If you use the `vaultkv` store and want to start using the vault proxy, read the documentation of the Nuts Vault proxy. - - -Available external storage implementations -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The following list contains all the known implementations of the Nuts external store API: - -- `Nuts Vault proxy `__. This is a proxy that integrates with Hashicorp Vault. It uses the Vault KV store to store the keys. The proxy is developed by the Nuts foundation and is available under an open source license. - -If you want to build your own store, take a look at the documentation at :ref:`external-secret-store`. diff --git a/docs/pages/development/5-external-secret-store.rst b/docs/pages/development/5-external-secret-store.rst deleted file mode 100644 index 91e0ed8ae3..0000000000 --- a/docs/pages/development/5-external-secret-store.rst +++ /dev/null @@ -1,35 +0,0 @@ -.. _external-secret-store: - -Integrate with an external secret store -####################################### - -.. note:: - - The following functionality is experimental and subject to change. We encourage developers to test it out on a test network and provide feedback. - -Motivation -********** - -The Nuts-node represents entities in the network. These entities have cryptographic keys that are used to sign and encrypt data. The Nuts-node stores these keys in a secret store. The safety of the Nuts network depends on the security of these keys since the possession of the keys proofs ownership over the identity. For development purposes, the Nuts node contains a simple file based secret store. This is not recommended for production environments, and a safer alternative is advised. Since there are many different secret stores, the Nuts-node has been designed to be able to integrate with any secret store. This is done by using a proxy that acts as a bridge between the node and secret store. The Nuts-node communicates with this proxy using a standardised API. This way, the Nuts-node can be configured to use any secret store. - -Implementing a proxy or building a custom secret store -****************************************************** - -A proxy or secret store should implement the Nuts Secret store API specification. This OpenAPI specification is available from the `Secret Store API repository `__. - -This repository also provides integration tests which can be run against an implementation. - -Consider developing your implementation under an open source license and publish it on a collaborative version control website such as `GitHub `__ or `Gitlab `__ so that other parties can use it. - -Reference implementation -************************ - -The Nuts community provides a supported reference implementation for the HashiCorp Vault secret store. This implementation is available on `GitHub `__. - - -Limitations -*********** - -The API has a few limitations: - -- It does not yet support authentication. This means that the proxy should be secured in such a way that only the Nuts node can access the API. diff --git a/docs/pages/release_notes.rst b/docs/pages/release_notes.rst index 98c72c403a..21dfb38738 100644 --- a/docs/pages/release_notes.rst +++ b/docs/pages/release_notes.rst @@ -69,10 +69,20 @@ To simplify HTTP configuration and proxying and make the default more secure, HT - port ``8081`` for all internal-facing endpoints (``/internal``, ``/status``, ``/metrics``, ``/health``) - port ``8080`` for all public-facing endpoints (all others) -The new HTTP configuration reflects this, - Note that ``8081`` by default maps to ``127.0.0.1`` only, so you might need to configure it to allow it to be accessible from other machines. +Deprecated features +=================== + +The following features have been deprecated: + +- The external key store API has been deprecated and will be removed in the next major release. + It was introduced to allow flexible support for other key storage backends, while reducing the number of dependencies and clients to maintain in the Nuts node. + But, in practice the secret store API is unmaintained itself and lacks features (e.g. authentication/authorization). + Starting v6, the preferred way to support other key storage backends is to directly implement it in the Nuts node itself. + This also reduces the complexity of a Nuts node deployment (one service less to configure and deploy). + Users are recommended to switch to the built-in client of their key storage backend. + ************************ Hazelnut update (v5.4.6) ************************