Feat/aws ci cd

.github/workflows/build-test-publish.yaml

@@ -49,6 +49,7 @@ jobs:
           git tag "v$VERSION"
 
   lint-format-and-static-code-checks:
+    name: Lint, Format, and Static Code Quality Checks
     runs-on: ubuntu-latest
     steps:
       - name: Checkout Repository
@@ -68,7 +69,8 @@ jobs:
       - name: Run oasdiff, Lint, Format, and other static code quality checks
         run: SKIP=no-commit-to-branch uvx --from pre-commit pre-commit run --all-files
 
-  build-wheel-and-sdist:
+  build-wheel:
+    name: Build Wheel Dist & Upload as Artifact
     runs-on: ubuntu-latest
     steps:
       - name: Checkout Repository
@@ -87,8 +89,9 @@ jobs:
           path: ./dist/*
 
   execute-tests:
+    name: Execute Tests against Wheel
     needs:
-      - build-wheel-and-sdist
+      - build-wheel
     runs-on: ubuntu-latest
     steps:
       - name: Checkout Repository
@@ -111,12 +114,54 @@ jobs:
           uv pip install --group test
           ./run run-tests
 
-  publish:
+  deploy:
+    name: Deploy to AWS
     needs:
-      - execute-tests
-      - build-wheel-and-sdist
-      - lint-format-and-static-code-checks
-      - check-version
+      [
+        execute-tests,
+        build-wheel,
+        lint-format-and-static-code-checks,
+        check-version,
+      ]
+    permissions:
+      id-token: write
+      contents: read
+    runs-on: ubuntu-latest
+    # IDEAL WAY to do this would be deploy to dev/staging on PR merge, then either
+    # Deploy to prod on merge to main or have a manual approval step.
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main' || github.event_name == 'workflow_dispatch' && github.event.inputs.deploy == 'true'
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: Setup Node
+        uses: actions/setup-node@v6
+
+      - name: Setup Docker Buildx
+        uses: docker/setup-buildx-action@v3
+        # ^^^needed for building and deploying Lambda Layer
+
+      - name: Configure AWS Credentials
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          role-to-assume: ${{ secrets.AWS_OIDC_ROLE_TO_ASSUME }}
+          aws-region: us-west-2
+
+      - name: Install AWS CDK CLI
+        run: npm install -g aws-cdk
+
+      - name: CDK Synthesize
+        run: bash +x ./run cdk-synth
+
+      - name: CDK Deploy
+        run: bash +x ./run cdk-deploy
+
+  publish:
+    name: Publish Version Tag
+    needs: [deploy]
     runs-on: ubuntu-latest
     # if - this is a merge to main or push directly to the main branch
     if: github.event_name == 'push' && github.ref == 'refs/heads/main'

.gitignore

@@ -12,6 +12,8 @@ cdk.context.json
 backup/
 notebooks/code/
 
+**/*cache*/
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

README.md

@@ -43,13 +43,17 @@ This project is a more polished version of the [cloud-engineering-project](https
 - [x] Setup the Dockerfile with the recommended way of using [uv in Docker](https://docs.astral.sh/uv/guides/integration/docker/).
   - [x] CDK rebuilds the Lambda Layer Docker image on every deployment. Is it possible to cache it locally and only rebuild when there are changes to files like `pyproject.toml` or `uv.lock`?
   - [ ] Try Docker multi-stage builds and configure [watch](https://docs.astral.sh/uv/guides/integration/docker/#configuring-watch-with-docker-compose) with docker compose.
-- [ ] Implement API versioning strategy (like v1 in the path).
-- [ ] Setup CI/CD pipeline to deploy the API to AWS using GitHub Actions.
-- [ ] Deployment Stratgies like Blue-Green, Canary deployments, etc.
-- [ ] Implement authentication (API keys or AWS Cognito) and secure Swagger UI page and possiblly the API endpoints as well.
-  - [ ] Add rate limiting to the API using API Gateway
-  - [ ] Implement multi-environment deployment pipeline (dev/prod) with approval gates
+- [x] Setup CI/CD pipeline to deploy the API to AWS using GitHub Actions.
+- [ ] API Gateway Improvements:
+  - [ ] Implement API versioning strategy (like v1 in the path).
+  - [ ] API Gateway with multiple stages (dev/prod). [`aws-samples/aws-cdk-examples/api-stages-lambda`](https://github.com/aws-samples/aws-cdk-examples/tree/main/python/api-stages-lambda)
+  - [ ] API Throttling and Usage Plans to manage API consumption and prevent abuse.
+  - [ ] Deployment Stratgies like Blue-Green, Canary deployments, etc.
+  - [ ] Implement authentication (API keys with Usage Plans/AWS Cognito/custom JWT/IAM Auth) and secure Swagger UI page and possiblly the API endpoints as well.
+    - [ ] Add rate limiting to the API using API Gateway
+    - [ ] Implement multi-environment deployment pipeline (dev/prod) with approval gates
 - [ ] Observability & Monitoring improvements:
+  - [ ] Cloudwatch Dashboard using CDK to visualize custom metrics, logs and traces in one place. [*ref*](https://github.com/cdk-patterns/serverless/tree/main/the-cloudwatch-dashboard/python)
   - [ ] Use OpenTelemetry for tracing instead of AWS X-Ray, [ref](https://aws.amazon.com/blogs/mt/aws-x-ray-sdks-daemon-migration-to-opentelemetry/).
   - [ ] Setup Grafana dashboards with CloudWatch data sources for enhanced monitoring
   - [ ] Replace Cloudwatch with Grafana Stack -- logs, metrics and traces

docs/aws/API-Gateway.md

@@ -0,0 +1,80 @@
+Optimizing tool selection...Looking at your infra.py, I can see you have a basic Lambda REST API setup. Here are the key production patterns and concepts you should learn:
+
+## API Gateway Deployment Patterns to Study
+
+**1. API Types & When to Use Them**
+- REST API vs HTTP API (cost/feature tradeoffs)
+- WebSocket APIs for real-time communication
+- Regional vs Edge-Optimized vs Private endpoints
+
+**2. Authentication & Authorization**
+- IAM authorization
+- Lambda authorizers (custom auth logic)
+- Cognito User Pools integration
+- API Keys and Usage Plans
+- OAuth 2.0 / JWT validation
+
+**3. Traffic Management**
+- Throttling and rate limiting (per-client quotas)
+- Usage plans for different customer tiers
+- API Keys management
+- Request/Response validation
+
+**4. Deployment Strategies**
+- Stage variables for environment configuration
+- Canary deployments (gradual traffic shifting)
+- Multiple stages (dev/staging/prod)
+- Stage-specific configuration
+
+**5. Observability & Monitoring**
+- CloudWatch Logs (access logs vs execution logs)
+- X-Ray tracing integration (you have this!)
+- Custom CloudWatch metrics
+- Alarms for error rates and latency
+
+**6. Cost Optimization**
+- Caching strategies at API Gateway level
+- Regional vs Edge endpoints
+- HTTP API vs REST API cost comparison
+
+**7. Security Best Practices**
+- WAF (Web Application Firewall) integration
+- Resource policies to restrict access
+- TLS/SSL certificate management
+- CORS configuration
+- Request validation to block malformed requests
+
+**8. Domain & DNS**
+- Custom domain names
+- Certificate management (ACM)
+- Base path mappings
+- API versioning strategies
+
+## Recommended Resources
+
+**AWS Documentation:**
+- [API Gateway Best Practices](https://docs.aws.amazon.com/apigateway/latest/developerguide/best-practices.html)
+- [Choosing Between HTTP and REST APIs](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-vs-rest.html)
+- [API Gateway Security Best Practices](https://docs.aws.amazon.com/apigateway/latest/developerguide/security-best-practices.html)
+
+**Architecture Patterns:**
+- [AWS Serverless Application Lens](https://docs.aws.amazon.com/wellarchitected/latest/serverless-applications-lens/) - comprehensive guide
+- [API Gateway Deployment Best Practices whitepaper](https://aws.amazon.com/blogs/compute/best-practices-for-organizing-larger-serverless-applications/)
+
+**CDK Specific:**
+- [AWS CDK API Gateway L2 Constructs](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_apigateway-readme.html)
+- CDK Patterns for API Gateway: https://github.com/cdk-patterns/serverless
+
+**Hands-On Learning:**
+- AWS Workshops: https://catalog.workshops.aws/
+- Serverless Land: https://serverlessland.com/patterns?framework=CDK
+
+**Key Topics for Your Current Setup:**
+1. Add request/response validation models
+2. Implement proper error responses with Gateway Responses
+3. Add CloudWatch access logging
+4. Consider usage plans if this is a public API
+5. Implement custom domain with Route53
+6. Add Lambda authorizer if you need custom auth
+
+Start with the AWS Well-Architected Serverless Lens - it covers all five pillars (operational excellence, security, reliability, performance, cost) specifically for serverless applications.

docs/aws/CDK-CI-CD-Github-Actions.md

@@ -0,0 +1,52 @@
+Here's a learning path for CI/CD with AWS CDK:
+
+## Core Concepts to Learn
+
+**1. AWS CDK Fundamentals**
+- CDK Pipelines construct
+- Stacks vs Stages vs Applications
+- CDK synthesis and bootstrapping
+- Cross-account/cross-region deployments
+
+**2. CI/CD Patterns**
+- Pipeline stages: Source → Build → Test → Deploy
+- Blue/Green and Canary deployments
+- Self-mutating pipelines (pipelines that update themselves)
+- Manual approval gates
+
+**3. AWS Services Integration**
+- CodePipeline, CodeBuild, CodeCommit
+- GitHub Actions with OIDC for AWS
+- AWS Secrets Manager for credentials
+- CloudWatch for monitoring pipeline metrics
+
+**4. Testing Strategies**
+- CDK assertions and snapshot testing
+- Integration tests in pipeline stages
+- Security scanning (cdk-nag)
+- Infrastructure validation pre-deployment
+
+## Recommended Resources
+
+**Official AWS Documentation:**
+- [CDK Pipelines Documentation](https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html)
+- [CDK Workshop - CI/CD Module](https://cdkworkshop.com/)
+- [AWS CDK Examples - Pipelines](https://github.com/aws-samples/aws-cdk-examples)
+
+**Video Courses:**
+- AWS re:Invent sessions on "CDK Pipelines" (YouTube)
+- A Cloud Guru / Pluralsight CDK courses
+
+**Hands-On:**
+- [CDK Patterns - CI/CD Patterns](https://cdkpatterns.com/)
+- AWS Well-Architected Labs for CI/CD
+
+**GitHub Actions specific:**
+- [AWS Actions for GitHub](https://github.com/aws-actions)
+- [Configure AWS Credentials Action](https://github.com/aws-actions/configure-aws-credentials)
+
+**Best Practices:**
+- AWS Well-Architected Framework - Operational Excellence Pillar
+- CDK Best Practices guide
+
+Start with the official CDK Pipelines documentation, then work through a hands-on tutorial to deploy your files-api project through a pipeline.