Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to rotate Aurora Postgres secret if using the ReadOnly endpoint as Host #146

Open
LukaHedtSV opened this issue Aug 26, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@LukaHedtSV
Copy link

Hi there,

I'm attempting to set up an Aurora v2 Cluster with the following users with Auto-Rotate password turned on for all of them:

  • postgres (admin)
  • lambda_writer (connects to the Aurora Writer endpoint)
  • lambda_reader (connects to the Aurora Reader endpoint)

lambda_writer's setup is the default, and it seems to work properly.

I've set up the Database Secret so that lambda_reader's host value is the ReadOnly endpoint for Aurora, as I don't want any users connecting with this secret to tax a potentially stressed Writer instance.

Unfortunately, Aurora v2 Reader instances aren't consider by the RDS client to be "Read Replicas", so when the Multi-User Rotation lambda goes to set the secret, and sees that the hostnames for the master and child secrets differ, the "isReadReplica" (see below) function fails:

def is_rds_replica_database(replica_dict, master_dict):
"""Validates that the database of a secret is a replica of the database of the master secret
This helper function validates that the database of a secret is a replica of the database of the master secret.
Args:
replica_dict (dictionary): The secret dictionary containing the replica database
primary_dict (dictionary): The secret dictionary containing the primary database
Returns:
isReplica : whether or not the database is a replica
Raises:
ValueError: If the new username length would exceed the maximum allowed
"""
# Setup the client
rds_client = boto3.client('rds')
# Get instance identifiers from endpoints
replica_instance_id = replica_dict['host'].split(".")[0]
master_instance_id = master_dict['host'].split(".")[0]
try:
describe_response = rds_client.describe_db_instances(DBInstanceIdentifier=replica_instance_id)
except Exception as err:
logger.warning("Encountered error while verifying rds replica status: %s" % err)
return False
instances = describe_response['DBInstances']
# Host from current secret cannot be found
if not instances:
logger.info("Cannot verify replica status - no RDS instance found with identifier: %s" % replica_instance_id)
return False
# DB Instance identifiers are unique - can only be one result
current_instance = instances[0]
return master_instance_id == current_instance.get('ReadReplicaSourceDBInstanceIdentifier')

I've checked as to why with the following setup:

  • An Aurora v2 DbCluster with:
    • 1 Writer instance
    • 1 Reader instance

And then I got the following in the CLI:

aws rds describe-db-instances --region ap-southeast-2 | grep Replica
            "ReadReplicaDBInstanceIdentifiers": [],
            "ReadReplicaDBInstanceIdentifiers": [],

I think the potential solution is to check that the host value is an Endpoint for the given cluster potentially, since Secret.attach adds the dbClusterIdentifier to the Secret Value for Aurora DB Connections, but I'm not sure tbh.

Thanks!

@LukaHedtSV
Copy link
Author

Additional note: Our org is set up to need almost everything to be done as part of CDK code, and Ideally I don't want to be writing my own custom CR to do this task that looks like CDK should handle natively.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants
@simonmarty @LukaHedtSV and others