PV is not deleted and getting AWS: access denied on VolumeDelete

[IonitaCatalin]

/kind bug

What happened?

When trying to delete a PV that was provisioned using the EFS driver, version mentioned below, currently getting the following error:

rpc error: code = **Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: 
Access denied**

Thus making the PV getting stuck in a perpetual state of Termination, as for controller logs:

efs-csi-controller-86c5454cd-74btz csi-provisioner E1113 14:53:42.363877       1 controller.go:1025] error syncing volume "efs-pv-**": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied

The policy attached to the role used by the efs-controller is the one managed by AWS: AmazonEFSCSIDriverPolicy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowDescribe",
            "Effect": "Allow",
            "Action": [
                "elasticfilesystem:DescribeAccessPoints",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:DescribeMountTargets",
                "ec2:DescribeAvailabilityZones"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowCreateAccessPoint",
            "Effect": "Allow",
            "Action": [
                "elasticfilesystem:CreateAccessPoint"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/efs.csi.aws.com/cluster": "false"
                },
                "ForAllValues:StringEquals": {
                    "aws:TagKeys": "efs.csi.aws.com/cluster"
                }
            }
        },
        {
            "Sid": "AllowTagNewAccessPoints",
            "Effect": "Allow",
            "Action": [
                "elasticfilesystem:TagResource"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "elasticfilesystem:CreateAction": "CreateAccessPoint"
                },
                "Null": {
                    "aws:RequestTag/efs.csi.aws.com/cluster": "false"
                },
                "ForAllValues:StringEquals": {
                    "aws:TagKeys": "efs.csi.aws.com/cluster"
                }
            }
        },
        {
            "Sid": "AllowDeleteAccessPoint",
            "Effect": "Allow",
            "Action": "elasticfilesystem:DeleteAccessPoint",
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/efs.csi.aws.com/cluster": "false"
                }
            }
        }
    ]
}

I assumed something changed since this started malfunctioning recently.

What you expected to happen?

The PV should be deleted successfully. From my perspective there can't be issues with the trust relationship and such as volumes are mounted successfully only the deletion received Access Denied.

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version): v.130.4-eks-a737599
• Driver version: v1.36.0-eksbuild.1
• karpenter cluster

Please also attach debug logs to help us better diagnose

• Instructions to gather debug logs can be found here

efs-csi-controller-84b4775c98-4ns99 csi-provisioner I1113 15:11:50.336718       1 controller.go:1258] "shouldDelete" PV="efs-pv-**"
efs-csi-controller-84b4775c98-4ns99 csi-provisioner I1113 15:11:50.336739       1 controller.go:1288] "shouldDelete is true" PV="efs-pv-**"
efs-csi-controller-84b4775c98-4ns99 csi-provisioner I1113 15:11:50.336754       1 controller.go:1132] "shouldDelete" PV="efs-pv-**"
efs-csi-controller-84b4775c98-4ns99 csi-provisioner I1113 15:11:50.336809       1 controller.go:1548] "Started" PV="efs-pv-**"
efs-csi-controller-84b4775c98-4ns99 csi-provisioner E1113 15:11:50.378097       1 controller.go:1558] "Volume deletion failed" err="rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied" PV="efs-pv-**"
efs-csi-controller-84b4775c98-4ns99 csi-provisioner I1113 15:11:50.378189       1 controller.go:1007] "Retrying syncing volume" key="efs-pv-**" failures=6
efs-csi-controller-84b4775c98-4ns99 csi-provisioner E1113 15:11:50.378221       1 controller.go:1025] error syncing volume "efs-pv-**": rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
efs-csi-controller-84b4775c98-4ns99 csi-provisioner I1113 15:11:50.378267       1 event.go:389] "Event occurred" object="efs-pv-**" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied"