Cannot get External-Secrets to work with AWS EKS and Secrets Manager

[udemy-darnone1]

I set this up and and created a values.yaml to override the default values in the chart. I created a k8s secret called aws-credentials with keys id and key for and IAM user that has admin rights. I deployed the helm chart, created an AWS secret, and External secret following the documentation. The secret never gets created. If I do a describe on the external secret object I see and error:

Status: ERROR, Missing credentials in config

My values file is as follows. Looking for solution.

env:
  AWS_REGION: us-east-2
  AWS_DEFAULT_REGION: us-east-2

# Create environment variables from existing k8s secrets
envVarsFromSecret:
  AWS_ACCESS_KEY_ID:
    secretKeyRef: aws-credentials
    key: id
  AWS_SECRET_ACCESS_KEY:
    secretKeyRef: aws-credentials
    key: key

serviceAccount:
  annotations: 
    "eks.amazonaws.com/role-arn": 'arn:aws:iam::123456789101:role/secrets-role'
  name: webserver-service-account

securityContext:
  fsGroup: 65534

[Flydiverny]

So first of what type of authentication are you trying to use for AWS?
In the values it looks like you are trying to use both ENV injected credentials and IAM Roles for service account credentials.

(I took the liberty of deleting your comment, and editing the issue. As the comment contained 2 copies of your values, one with the AWS account id which you may or may not be concerned with posting)

[udemy-darnone1]

Thanks for modifying my post. So then it is either service accounts or AWS credentials but not both? So I could really uses someone's input I started with just the service account but then I added the AWS credentials because the external secret status reports ERROR, Missing credentials in config. Logging into the deployment the I can see the credentials are properly set. If I change the serviceAccount.create to false and copy out the annotation and name the generation the Kubernetes yam complains with

Error: failed to parse values.yaml: error converting YAML to JSON: yaml: line 17: did not find expected key

If I uncomment the annotation and name then the serviceaccount yaml is not generated but the deployment still references the name of the service account. serviceAccountName: {{ template "kubernetes-external-secrets.serviceAccountName" . }}
Commenting out the service account causes the deployment pod to go status error.

So I am at a loss.

[Flydiverny]

I'm not sure of the exact precedence order but I believe set environment variables will override any other credentials.

There must be a service account, either created, or referenced by name to an already existing service account.
If you do not have any special requirements, I recommend you leave the default serviceAccount.create: true, no need to override the name.

If you are trying to use IAM roles for service account then set the annotation, if you are trying to use env variables for credentials then leave the annotation out.

As listed here https://github.com/godaddy/kubernetes-external-secrets#aws-based-backends there is several different ways to provide authentication details for AWS backends. You can only use one at a time.

And yes most of the docs here could use some love to better present the information it contains :)

[udemy-darnone1]

Markus,

Thanks for your reply. Set the annotation where - in the deployment or the service account? And is a role really required in the external secret. So I let the generation define a service account with an annotation specifying a role that has access to retrieve the secret but external secret complained of no credentials. I am attaching the deployment yamls for your reference. in another comment. If you could point out where I have gone wrong that would be really, really great.

[udemy-darnone1]

This is what I get:

kubectl get externalsecrets.kubernetes-client.io hello-service
NAME LAST SYNC STATUS AGE

hello-service 0s ERROR, Missing credentials in config 11s

[taragurung]

I am stuck in similar issues, Tried with IAM role get same issue, when using environment variable get some certification mis-match issues.

[udemy-darnone1]

Here are the yaml files I am deploying. Tar extracts into templates. Trust relationship and policy are also included along with values.yaml. The only thing not included is the aws-creds secret. I am still getting reds error from the external secret. I definitely appreciate your assistance.

templates.zip

[Flydiverny]

Assuming you have a role arn:aws:iam::123456789101:role/webserver-secrets-role which has the trust relationship set.
You should be able to

Set the annotation on the service account refering to this role

serviceAccount:
  annotations: 
    "eks.amazonaws.com/role-arn": 'arn:aws:iam::123456789101:role/webserver-secrets-role'
  name: webserver-service-account

Remove the secret envs, eg this section:

envVarsFromSecret:
  AWS_ACCESS_KEY_ID:
    secretKeyRef: aws-credentials
    key: id
  AWS_SECRET_ACCESS_KEY:
    secretKeyRef: aws-credentials
    key: key

in your external secret resource remove the

  roleArn: arn:aws:iam::123456789101:role/webserver-secrets-role

roleArn on an external secret is used to specify additional credentials for that specific secret, it will be assumed by the first role.

You should be able to confirm that the credentials are picked up by exec into the pod, AWS provides some examples here
https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html

Hopefully that gets you further! I know there is some trouble getting up and running with IRSA, but it should work given all configuration is correct 😄

[abork-dolby]

Hey @Flydiverny I'm also in the process of setting up external secrets for our EKS cluster and I'm also trying to use IRSA for authentication. Here is what's confusing to me.

This IAM role that you are assigning to your service account webserver-secrets-role is this the same service account that is used by external secrets controller itself?

What I'm wondering is the right way to go to have external secrets controller with it's own service account which has IAM role assigned to it OR I should create service account/IAM role for each SecretStore. Like this for example:

# secretstore-1
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789101:role/secretstore-1-role
  name: secretstore-1-serviceaccount
---
apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  name: secretstore-1
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        jwt:
          serviceAccountRef:
            name: secretstore-1-serviceaccount
---
# secretstore-2
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789101:role/secretstore-2-role
  name: secretstore-2-serviceaccount
---
apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  name: secretstore-2
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        jwt:
          serviceAccountRef:
            name: secretstore-2-serviceaccount

[abork-dolby]

Ah, this is so annoying... Google is constantly directing me to kubernetes-external-secrets, I'm actually using new one external-secrets... Anyway, if you know....

[Flydiverny]

I am not a 100% certain here, but by this config those roles are used only when getting secrets from those individual stores. It gives you more granularity on how you manage access. Like you could have a secret store in one namespace with a role that only has access to /team/bananas/* and in another namespace be limited to another path.

So I think you can do whichever you prefer for your security needs. One IRSA setup for the external secrets controller would be easier, but anyone that can create an external secret would have indirect access to all secrets.

Hope that helps!

[Flydiverny]

https://external-secrets.io/v0.4.4/provider-aws-secrets-manager/ provides some examples for various setups :)

[abork-dolby]

Yeah, you are right... After few more tries I was able to get it to work service account per SecretStore 🙌

[darnone]

Markus,

I followed your instructions above and I think I am getting closer but I am still not quite there yet. Now I am able to create an external secret but the secret is not getting created with the role I am using. looking at the logs on the deployment pod, it seems that my role is not working correctly and the role being used to access secrets is the NodeInstanceRole instead of the role in the service account. I attached my policy to the NodeInstanceRole and the external secret generated the secret as advertised. So I need to figure out what is going on with my role. Code is attached along with a log snippet. I will keep you posted - that you for your assistance.
templates.zip

[Flydiverny]

NodeInstanceRole is the last fallback in the credential chain. Its always a mystery to me why the IRSA credentials aren't picked up!
One issue is the file permissions for the token file that is mounted but that should be solved with

securityContext:
  fsGroup: 65534

Best to verify the configuration works without KES first. So the trust relationship is known to be working.

[rfinner]

Please explain this fsGroup security context and where we need it. thanks!

[darnone]

I think I figured this out. I keep destroying and recreating EKS clusters to keep costs down, but I was forgetting to recreate the IODC identity provider. Duh! I do my best work thinking in the shower. So this is now working in a us-east-2 1.18 EKS test cluster created from a CentOS 7 machine with kubectl 1.18.8 , aws 2.0.57, helm version 3.34, and eksctl 0.30.0. I am going redeploy this cluster with a KMS key acticated. Then I need to move this to our work cluster. it is EKS 1.17 cluster from a WSL Ubuntu running on Windows10, kubectl 1.17.7, aws 2.0.38, helm 3.3.0, and eksctl 0.28.1 (although eksctl does not come into play). I will keep you posted.

[darnone]

Markus,
I got the secret to work again in my test cluster with KMS enabled and I was able to get this to work inmy work cluster. I wanted to really thank you for all of your help, assistance and patience.

David

[vkbera]

@darnone Will it be possible to provide all the deployment file that worked for you. I am also facing similar issue .

[darnone]

This is what I am doing - first create a secret in Secrets Manager.

aws secretsmanager create-secret --region ${AWS_DEFAULT_REGION} \
      --name ${KES_TEST_SECRET}/credentials \
      --secret-string '{"username":"admin","password":"1234"}'

If you are testing this often you can remove the secret immediately with:

aws secretsmanager delete-secret \
      --secret-id ${KES_TEST_SECRET}/credentials \
      --force-delete-without-recovery

Enable the OIDC provider: if not already enable in your cluster

eksctl utils associate-iam-oidc-provider --cluster ${CLUSTER_NAME} --approve

I am using eksctl to create the OIDC IRSA because it generates a lot of stuff but the IRSA looks like the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/oidc.eks.${AWS_DEFAULT_REGION}.amazonaws.com/id/${OIDC_ID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.${AWS_DEFAULT_REGION}.amazonaws.com/id/${OIC_ID}:sub": "system:serviceaccount:${KES_NAMESPACE}:${KES_SERVICE_ACCOUNT_NAME}",
          "oidc.eks${AW_DEFAULT_REGION}.amazonaws.com/id/${OIDC_ID}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

The IRSA above has no policy as the roles for the secret will assume this role and carry along its own policies. Since the Helm chart provides the service account eksctl is only creating the role. If you choose to use eksctl, you cannot create a IRSA without a policy (a missing feature) so you will need "no-op" policy that does nothing to plug that gap. Refer to the EKS workshop for more info.

We are overriding very few variables in the Helm chart. I pass in a values file:

env:
  AWS_REGION: ${AWS_DEFAULT_REGION}
  AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}

  # Caution: setting this frequency may incur additional charges on AWS
  POLLER_INTERVAL_MILLISECONDS: ${KES_POLLING_INTERVAL} # Caution, setting this frequency may incur additional charges on some platforms

  # requires permitted and permitted-key-name in all namespaces
  ENFORCE_NAMESPACE_ANNOTATIONS: true # requires permitted and permitted-key-name in all namespaces

serviceAccount:
  annotations:
    "eks.amazonaws.com/role-arn": 'arn:aws:iam::${AWS_ACCOUNT_ID}:role/${KES_ROLE_NAME}'
  name: ${KES_SERVICE_ACCOUNT_NAME}

securityContext:
  fsGroup: 65534

This file is in an extra directory I create in the chart called files. So deploy the helm chart as follows:

helm install ${KES_HELM_RELEASE} ../charts/kubernetes-external-secrets/ \
      --skip-crds -f ../charts/kubernetes-external-secrets/files/daves-values.yaml \
      -n ${KES_NAMESPACE} --create-namespace

Check the logs and the deloyment to make sure KES is connecting with AWS.

The secret will need a role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/${KES_ROLE_NAME}"
        },
        "Action": "sts:AssumeRole"
        }
    ]
}

and a policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "KESPermissions",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-east-1:${AWS_ACCOUNT_ID}:secret:${KES_TEST_SECRET}"
            ]
        }
    ]
}

The secret deployment consists of a namespace that is annotated:

apiVersion: v1
kind: Namespace
metadata:
  name: ${KES_TEST_SECRET}-namespace
  # annotation permitted is configurable regular expresssion to match
  # the role or roles that are permitted to be used by pods in this
  # namespace
  
  annotations:
    # annotation iam.amazonaws.com/permitted is takes a configurable
    # regular expresssion that limites the range of roles that can
    # be assumed by resources in this namespace.
    iam.amazonaws.com/permitted: "arn:aws:iam::${AWS_ACCOUNT_ID}:role/.*${TEST_SECRET_ROLE}"
    # annotation externalsecrets.kubernetes-client.io/permitted-key-name
    # takes a configurable regular expresssion to limit the secrets that
    # can be acessed by resources in this namespace.
    # namespace, ie defender/credentials.
    externalsecrets.kubernetes-client.io/permitted-key-name: "${KES_TEST_SECRET}/.*"
spec: {}
status: {}

and the external secret itself:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: ${KES_TEST_SECRET}
  namespace: "${KES_TEST_SECRET}-namespace"
spec:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${TEST_SECRET_ROLE}
  # optional: specify region
  region: ${AWS_DEFAULT_REGION}
  data:
    - key: ${KES_TEST_SECRET}/credentials
      name: password
      property: password
    - key: ${KES_TEST_SECRET}/credentials
      name: username
      property: username

There is a strange variable in the above that is from my automation. the TEST_SECRET-ROLE can be retrieved with:

export TEST_SECRET_ROLE=$(aws iam get-role --role-name ${KES_TEST_SECRET}-role | \
      jq .Role | grep RoleName | cut -d "\"" -f 4)

I believe it the same as ${KES_ROLE_NAME}

deploy the namespace and then deploy the external secret, The secret should appear in the ${KES_TEST_SECRET}-namespace. I hope this helps.

[vkbera]

@darnone Thanks for your detailed input. Can you please provide the ekstl command for creating IRSA without any policy. I tried but didn't able to find.

[darnone]

Check out the eksclt examples, the issue I created, and the IAM service account section of the doc. For the policy, you can define an inline policy to the eksctl config that essentially does nothing. Something like the following. Instance i-1234567890abcdef0' doesn't and most likely would never exist.

attachPolicy: # inline policy can be defined along with `attachPolicyARNs`
      Version: "2012-10-17"
      Statement:
      - Sid: DoNothing
        Effect: Deny
        Action:
        - "ec2:RunInstances"
        Resource: 'arn:aws:ec2:*:*:instance/i-1234567890abcdef0'

[udemy-darnone1]

Markus,

I do have two remaining questions. 1.) Can this be setup for all namespaces in the cluster or does External Secrets need to deploy to each namespace that requires a secret. 2.) Does this support rotating passwords?

David

[Flydiverny]

1. KES will watch in all namespaces, currently there's no support for limiting it to a single namespace.

2. By default KES will poll your backend every 10 seconds, you probably want to tweak that for your needs as it can get expensive depending on your amount of secrets, number of keys to fetch, and backend type.
   So your secret will be updated if the value changes in the secret store. Having the pods restart or so when the secret updates is not part of KES concerns tho. You could probably utilize something like https://github.com/stakater/Reloader

[udemy-darnone1]

Markus,

How do I change the polling? Is there a parameter in values.yml to do that? So if I understand correctly, external secrets will watch across all namespaces even if it is in a namespace of its own? If that is the case, how does this work in a cluster with multiple apps each having an external secret. Would I add multiple roles to the service account and then restrict which roles a namespace is permitted to assume?

[udemy-darnone1]

Markus,

I see where I can change polling - I should have take closer look at the deployment. I have been playing around with deploying ket into it's own namespace and creating secrets others. It is pretty cool. Now I just need to work out the question above with the service account/namespace/password restriction.

[udemy-darnone1]

Markus,

Another set of questions if you are willing to answer - these are really design questions. I want to limit access from different namespaces to specific secrets. First, since this implementation works across all namespaces, there is only one service account. I presume there are two different ways to approach this. First, multiple roles pointing to the same serviceaccount for the KES deployment each with different secret policies. Then the service account would have a list of roles as annotations. The second, one role with a set of policies, one for each secret. The former would restrict namespaceaccess to a role and the latter would allow namespaces to restrict what password they have access to. I have not tried either of these - is one better than the other? Also, is KES centralized across nodegroups? Again, I appreciate your help.

[Flydiverny]

You can do a setup with multiple IAM roles where the role used by the service account (SA) is allowed to assume the other roles.

You can specify a roleArn in an external secret resource to have it be assumed by the SA role when that secret is fetched.
Further you can set annotations on the namespaces iam.amazonaws.com/permitted with a regex to limit which roleArn pattern or value is allowed for the respective namespace. (details here)

Likewise you can use externalsecrets.kubernetes-client.io/permitted-key-name to limit which keys are allowed to be fetched in a namespace. (more in readme here)

So given a secret key naming pattern of something like /dev/<namespace>/db-credentials you could in your namespace staging set externalsecrets.kubernetes-client.io/permitted-key-name: /dev/staging/.* to only allow keys with that a path matching that to be fetched for those external secrets.

The annotation names are configurable.

Additionally if you use those annotations you might want to set the environment variable ENFORCE_NAMESPACE_ANNOTATIONS=true so secrets wont be allowed to be fetched in namespaces where those annotations are missing. Looks like that key is missing in our docs 😢

[udemy-darnone1]

Markus,
Thanks for you reply. I am new to AWS so I am difficulty trying to visualize your first two sentences. To make sure I understand, the suggestion is to create a role unique to each external secret that the service account role is allowed to assume. What kind or role would that external secret have - AWS Service, Another AWS account, Web Identity, etc? Would the policy for that external secret role be the same as what is provided in README for hello-service/password and attached to the external secret role or the service account?

[udemy-darnone1]

I guess what you are talking about is this optional config in the external-secret:

optional: specify role to assume when retrieving the data
roleArn: arn:aws:iam::123456789012:role/test-role

but what would the role look like and that policy would be attached? I appreciate your time.

[Flydiverny]

I guess what you are talking about is this optional config in the external-secret:

Yes! :)

The role would have trust relationship to your service account role (the one with the webidentity) and the policy would provide access to a subset of your secret manager resources.
So if you use the web UI to create the role you would select 'Another AWS account', I suppose you could select AWS service as well since you will need to edit the trust relationship regardless.

Example trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789101:role/webserver-secrets-role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Policy could look something like this, to limit it to only get resources whose names starting with /development/

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:Get*",
            ],
            "Resource": "arn:aws:secretsmanager:us-east-2:123456789101:secret:/development/*"
        }
    ]
}

[Flydiverny]

Correct, if you will be using roleArn everywhere, then service account role doesn't need any policy document

[udemy-darnone1]

Yeah! I was writing something when you just replied and I lost it. I was asking what the final service account policy should look like since in the README.md secrets were referenced directly. But now we are getting policies from the external secrets roles. Thus the SA just becomes a role with no policy and everything we need comes from the policy attached to the role in the external secret. Also the ENFORCE_NAMESPACE_ANNOTATIONS=true parameter - does that require both annotations or one or the other? Markus, I really appreciate your time an answers. I will keep you posted. - David

[Flydiverny]

Looks like it requires both annotations to be present.

[udemy-darnone1]

Markus,

I got the implementation to work. Now onto the iam.amazonaws.com/permitted, externalsecrets.kubernetes-client.io/permitted-key-name:, and ENFORCE_NAMESPACE_ANNOTATIONS=true feature. I thank you for you assistance - I will keep you posted. - David

[udemy-darnone1]

Markus,
I just wanted to know that I hot the annotations to work fully - iam.amazonaws.com/permitted, externalsecrets.kubernetes-client.io/permitted-key-name:, and ENFORCE_NAMESPACE_ANNOTATIONS=true. pretty slick. I want to thank you for all of you assistance. Hopefully, the tread provides some ideas o how to make your readme rock solid including fully examples beginning to end. I will be ding this for my own purposes and would be willing to have if you are interested. Have a great weekend - David

[Flydiverny]

PRs welcome! We should really try to set up a docs site to be able to provide better structure for the docs but lack of time is apparently a thing :)