CFDB stack

[conradbzura]

Summary

Deploy CFDB on ECS Fargate with DocumentDB, replacing the unused X.509 certificate auth with SCRAM (username/password) over TLS.

• Replace X.509 authentication with SCRAM/TLS in the Python API and Rust materializer — credentials are now parsed from DATABASE_URL, and DocumentDB's retryWrites=false requirement is handled via MONGODB_RETRY_WRITES
• Add GET /health endpoint for ECS target group health checks
• Simplify Dockerfile.mongodb to dev-only (remove TLS/X.509 branch); add curl and AWS DocumentDB CA bundle to Dockerfile.api
• Remove all X.509 artifacts: cert generation script, mongo TLS config, user creation script, -prod Makefile targets
• Add three CloudFormation templates (cloudformation/): network (VPC, subnets, security groups), database (DocumentDB + Secrets Manager), backend (ECS, ALB, ACM, Route 53)
• Add ECS deployment step to CI/CD — images tagged with both :latest and short git SHA, then aws ecs update-service --force-new-deployment

Local dev workflow (make mongodb && make api) is unchanged.

Post-merge deployment steps

1. Deploy CloudFormation stacks (in order)

# Network
aws cloudformation deploy \
  --stack-name cfdb-network \
  --template-file cloudformation/network.yml \
  --capabilities CAPABILITY_IAM \
  --region us-east-2

# Database
aws cloudformation deploy \
  --stack-name cfdb-database \
  --template-file cloudformation/database.yml \
  --capabilities CAPABILITY_IAM \
  --region us-east-2 \
  --parameter-overrides \
    NetworkStackName=cfdb-network

# Backend (sync API key is auto-generated in Secrets Manager)
aws cloudformation deploy \
  --stack-name cfdb-backend \
  --template-file cloudformation/backend.yml \
  --capabilities CAPABILITY_IAM \
  --region us-east-2 \
  --parameter-overrides \
    NetworkStackName=cfdb-network \
    DatabaseStackName=cfdb-database \
    ImageURI=605134458779.dkr.ecr.us-east-2.amazonaws.com/cfdb:latest

2. Set GitHub repo secrets

The CI/CD workflow needs two new secrets:

# ECS_CLUSTER is the cluster name (deterministic from stack name)
# → cfdb-backend-cluster

# ECS_SERVICE comes from the backend stack output:
aws cloudformation describe-stacks \
  --stack-name cfdb-backend \
  --region us-east-2 \
  --query 'Stacks[0].Outputs[?OutputKey==`ECSServiceName`].OutputValue' \
  --output text

Set in GitHub → Settings → Secrets and variables → Actions:

Secret	Value
ECS_CLUSTER	cfdb-backend-cluster
ECS_SERVICE	(output from command above)

3. Retrieve the sync API key

The sync API key is auto-generated at stack creation. Retrieve it for use in X-API-Key headers:

aws secretsmanager get-secret-value \
  --secret-id cfdb-backend-sync-api-key \
  --region us-east-2 \
  --query SecretString \
  --output text

4. Update IAM deploy role permissions

The existing AWS_IAM_ROLE used by GitHub Actions needs these additional permissions:

• ecs:UpdateService
• ecs:DescribeServices

5. Initial data load

DocumentDB starts empty. Restore data from a host with VPC access (bastion or ECS Exec):

# Get the cluster endpoint
aws docdb describe-db-clusters \
  --db-cluster-identifier cfdb-database-cluster \
  --region us-east-2 \
  --query 'DBClusters[0].Endpoint' \
  --output text

# Get the master password
aws secretsmanager get-secret-value \
  --secret-id cfdb-database-master-credentials \
  --region us-east-2 \
  --query 'SecretString' \
  --output text | python3 -c "import sys,json; print(json.load(sys.stdin)['password'])"

# Restore (from a host inside the VPC)
mongorestore --gzip \
  --uri="mongodb://cfdbadmin:<password>@<endpoint>:27017/cfdb?tls=true&tlsCAFile=/etc/cfdb/certs/global-bundle.pem" \
  /path/to/database/dump

6. Verify

# Health check
curl https://cfdb.vis-api.link/health

# API smoke test
curl -X POST https://cfdb.vis-api.link/metadata \
  -H 'Content-Type: application/json' \
  -d '{"query": "{ files(first: 1) { edges { node { filename } } } }"}'

Test plan

• Local dev: make mongodb && make api starts successfully, curl http://localhost:8000/health returns 200
• Docker build: docker build -f Dockerfile.api . succeeds (CA bundle downloaded)
• CloudFormation: stacks deploy in order without errors
• ECS: ALB target group shows healthy targets
• HTTPS: curl https://cfdb.vis-api.link/health returns 200

[conradbzura]

Virtually a direct copy of https://github.com/hms-dbmi/cvh/blob/john-conroy/yaml-envs/cloudformation/network.yml