-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Secrets Encryption Test Plan
K3s currently has only a single command for managing secrets encryption - the --secrets-encryption flag. If this is turned on, a secrets encryption configuration document is generated using a random aescbckey key; this document is stored in the bootstrap data and shared among server nodes via the kube-apiserver --encryption-provider-config CLI flag. There are no provisions for rotating the key, disabling encryption after enabling it, etc. This feature expansion will allow for rotating the encryption key via a multi-step process in a single and multi node cluster. Ref: Design Doc
secrets-encryption keys rotation and enable/disable of encryption. The is all controlled via a new sub command k3s secrets-encrypt. Ref: Ticket
$ k3s secrets-encrypt --help NAME: k3s secrets-encrypt - Control secrets encryption and keys rotation USAGE: k3s secrets-encrypt command [command options] [arguments...] COMMANDS: status Print current status of secrets encryption enable Enable secrets encryption disable Disable secrets encryption prepare Prepare for encryption keys rotation rotate Rotate secrets encryption keys reencrypt Reencrypt all data with new encryption key OPTIONS: --help, -h show help
Note: Stopping/killing and Restarting the service after each command, except status, is required. If not performed then the cluster goes in to bad state
$ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=<commithash> INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644 --secrets-encryption" sh -
OR
Install k3s using parameter `INSTALL_K3S_SKIP_START=true`. Start the service with `--secrets-encryption` flag
$ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=<commithash> INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644" INSTALL_K3S_SKIP_START=true sh - $ sudo k3s server secrets-encrypt
$ kubectl create secret generic secret1 -n default --from-literal=mykey=mydata secret/secret1 created $ kubectl describe secret secret1 -n default Name: secret1 Namespace: default Labels: <none> Annotations: <none> Type: Opaque Data ==== mykey: 6 bytes
| # | Test Case | Steps | Docs Needed? | Expected Result | Pass/Fail Version | Notes /GH issues |
|---|---|---|---|---|---|---|
| 1 |
secrets-encrypt status Print current status of secrets encryption |
# Check status $ sudo k3s secrets-encrypt status |
Yes | Should show the status of secrets-encryption Encryption Status: Enabled Current Rotation Stage: start Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
N/A |
| 2 |
secrets-encrypt enable (Enabled by default) Enable secrets encryption Use flag --force --skip to properly enable |
# Check status $ sudo k3s secrets-encrypt status # Enable secrets encryption $ sudo k3s secrets-encrypt enable # Check status $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s |
Yes | Console output shows secrets-encryption enabled Status should show the correct state of secrets-encryption Encryption Status: Enabled Current Rotation Stage: start Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
N/A |
| 3 |
secrets-encrypt disable Disable secrets encryption Use flag --force --skip to properly disable |
# Check status, ensure it is enabled $ sudo k3s secrets-encrypt status # Disable secrets encryption $ sudo k3s secrets-encrypt disable # Check status $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s # Deploy another secret and ensure it is not encrypted $ kubectl create secret generic secret2 -n default --from-literal=mykey=mydata |
Yes | Console output shows secrets-encryption disabled Status should show the correct state of secrets-encryption Encryption Status: Disabled Current Rotation Stage: start Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey Use ETCDCTL to show secrets being encrypted/not encrypted. Change the below command to point to secret1, which should still show k8s:enc:aescbc:v1: and not have the data of the secret, meaning it is encrypted. secret2 will not show that, and will show the secret data, meaning it is not encrypted sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt get /registry/secrets/default/secret2 | hexdump -C |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|
| 4 |
secrets-encrypt prepare Prepare for encryption keys rotation |
# Check status, ensure its enabled $ sudo k3s secrets-encrypt status # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Check status $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s |
Yes | Console output shows prepare completed successfully Status should show the correct state of secrets-encryption along with new key Encryption Status: Enabled
Current Rotation Stage: prepare
Server Encryption Hashes: All hashes match
Active Key Type Name
------ -------- ----
* AES-CBC aescbckey
AES-CBC aescbckey-2021-12-08T21:34:03Z
|
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|
| 5 |
secrets-encrypt rotate Rotate secrets encryption keys |
# Check status, ensure its enabled $ sudo k3s secrets-encrypt status # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Check status, ensure it is in prepare stage $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s # Rotate secrets encryption $ sudo k3s secrets-encrypt rotate # Check status $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s |
Yes | Console output shows rotate completed successfully Status should show the correct state of secrets-encryption with new key as active Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: All hashes match
Active Key Type Name
------ -------- ----
* AES-CBC aescbckey-2021-12-08T21:34:03Z
AES-CBC aescbckey
|
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|
| 6 |
secrets-encrypt reencrypt Reencrypt all data with new encryption key Use flag -f or --force |
# Check status, ensure its enabled $ sudo k3s secrets-encrypt status # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Check status, ensure it is in prepare stage $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s # Rotate secrets encryption $ sudo k3s secrets-encrypt rotate # Check status, ensure it is in rotate stage $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s # Re-encrypt secrets encryption $ sudo k3s secrets-encrypt reencrypt # Check status, ensure it is in reencrypt_active stage $ sudo k3s secrets-encrypt status # Check status until it is in reencrypt_finished stage $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s |
Yes | Console output shows reencryption started Status should show the correct state of secrets-encryption with new key as active Encryption Status: Enabled
Current Rotation Stage: reencrypt_active
Server Encryption Hashes: All hashes match
Active Key Type Name
------ -------- ----
* AES-CBC aescbckey-2021-12-08T21:34:03Z
AES-CBC aescbckey
Status shows correct state of secrets-encryption with new key as active and old key as removed Encryption Status: Enabled Current Rotation Stage: reencrypt_finished Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey-2021-12-08T21:34:03Z |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
| # | Test Case | Steps | Expected Result | Pass/Fail Version | Notes/GH issue |
|---|---|---|---|---|---|
| 1 | Perform secrets encryption rotation on multi node cluster with 3 servers, 1 agent |
# Start up 3 K3s server $ k3s server --secrets-encryption # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Select ONE server (S1 going forward) to perform the rotate on (doing any stage on any server is supported) # Perform secrets-encrypt prepare on S1 $ sudo k3s secrets-encrypt prepare # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Perform secrets-encrypt rotate on S1 $ sudo k3s secrets-encrypt rotate # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Perform secrets-encrypt reencrypt on S1 $ sudo k3s secrets-encrypt reencrypt # Check status on all the servers $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
||
| 2 | Ensure all server nodes perform reconciliation on restart. |
|
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
Detailed steps to be added later |
| # | Test Case | Steps | Expected Result | Pass/Fail Version | Notes/GH issue |
|---|---|---|---|---|---|
| 1 | Perform secrets encryption rotation on multi node cluster with 3 servers, 1 agent |
# Start up 3 K3s server joined to external DB (Ex: mySQL) $ k3s server --secrets-encryption --datastore-endpoint "mysql://root:mysql@tcp(<server_ip>:3306)/k3s" # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Select ONE server (S1 going forward) to perform the rotate on (doing any stage on any server is supported) # Perform secrets-encrypt prepare on S1 $ sudo k3s secrets-encrypt prepare # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Perform secrets-encrypt rotate on S1 $ sudo k3s secrets-encrypt rotate # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status # Perform secrets-encrypt reencrypt on S1 $ sudo k3s secrets-encrypt reencrypt # Check status on all the servers $ sudo k3s secrets-encrypt status # Stop/Kill and Restart k3s service on S1 $ sudo systemctl stop k3s && sudo systemctl restart k3s # Once S1 is back up, restart the other two servers $ sudo systemctl stop k3s && sudo systemctl restart k3s # Check status on all the servers, ensure all shows the same result $ sudo k3s secrets-encrypt status |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
| # | Test Scenarios | Steps | Expected Result | Pass/Fail Version | Notes/GH issues |
|---|---|---|---|---|---|
| 1 | Perform commands without restarting k3s | # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Rotate secrets encryption $ sudo k3s secrets-encrypt rotate # Re-encrypt secrets encryption $ sudo k3s secrets-encrypt reencrypt # Stop/Kill and Restart k3s service $ sudo systemctl stop k3s && sudo systemctl restart k3s |
The system does not restart and goes in to infinite loop with the error below Dec 08 21:39:09 ip-172-31-41-171 k3s[5579]: E1208 21:39:09.263197 5579 cacher.go:420] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/default-token-f6bkz": no matching key was found for the provided AES transformer; reinitializing... Dec 08 21:39:09 ip-172-31-41-171 k3s[5579]: E1208 21:39:09.941337 5579 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/default-token-f6bkz": no matching key was found for the provided AES transformer |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
This action breaks the cluster so don't try this! This happens because all the secrets gets locked and the key gets thrown away |
| 2 | Perform same command repeatedly / multiple times |
On running `sudo k3s secrets-encrypt enable` |
Console shows `secrets-encryption enabled` without any errors |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|
| On running `sudo k3s secrets-encrypt disable` | Console shows `secrets-encryption enabled` without any errors |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|||
| On running `sudo k3s secrets-encrypt prepare` | Console shows `FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-90363: see server logs for more info` after showing `prepare completed successfully` on the first command run |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|||
| On running `sudo k3s secrets-encrypt rotate` | Console shows `FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-45155: see server logs for more info` after showing `rotate completed successfully` on the first command run |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|||
| On running `sudo k3s secrets-encrypt reencrypt` | Console shows `reencryption started` with no errors but the system gets stuck with rotation stage as `reencrypt_request` and spits out errors in the log Dec 10 18:28:15 ip-172-31-32-180 k3s[2146]: time="2021-12-10T18:28:15Z" level=error msg="error syncing 'ip-172-31-32-180': handler reencrypt-controller: invalid hash: 8d0f2a2ad73ce974c4cc39874ebcb3b31d142774aed2c4771e4b4205c94a028f found on node ip-172-31-32-180, requeuing" |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|||
| 3 | Perform each stage on different servers in HA cluster |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|||
| 4 | Perform commands out of sync |
# Rotate secrets encryption $ sudo k3s secrets-encrypt rotate # Prepare secrets encryption $ sudo k3s secrets-encrypt prepare # Disable secrets encryption $ sudo k3s secrets-encrypt disable # Enable secrets encryption $ sudo k3s secrets-encrypt enable |
Console shows FATA[0000] https://127.0.0.1:6443/v1-k3s/encrypt/config: 400 Bad Request error secrets-encrypt-5012: see server logs for more info |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
|
| 5 | Perform re-encrypt with 1000+ secrets stored in the DB |
||||
| 6 | Perform commands on agent node instead of server |
Install k3s with 1 server and 1 agent node sudo k3s secrets-encrypt status |
Console shows FATA[0000] open /var/lib/rancher/k3s/server/token: no such file or directory |
PASS: v1.21.8+k3s1 PASS: v1.22.5+k3s1 |
</div>