Skip to content

Update staging compose for demo/preview deployments#1181

Open
mihow wants to merge 5 commits intomainfrom
feat/update-staging-compose
Open

Update staging compose for demo/preview deployments#1181
mihow wants to merge 5 commits intomainfrom
feat/update-staging-compose

Conversation

@mihow
Copy link
Collaborator

@mihow mihow commented Mar 18, 2026

Summary

  • Update docker-compose.staging.yml to serve as the standard config for staging, demo, and branch preview environments: local Redis, RabbitMQ, NATS containers with an external database via DATABASE_IP
  • Add compose/staging/docker-compose.db.yml as an optional convenience for running a local PostgreSQL container
  • Upgrade gunicorn from 20.1.0 to 23.0.0 (fixes ModuleNotFoundError: No module named 'pkg_resources' on fresh Docker builds)

Staging compose changes

  • Remove local Postgres service (DB is always external, set DATABASE_IP in .envs/.production/.compose)
  • Add RabbitMQ 3.13 container for Celery task broker
  • Add NATS to depends_on (was commented out)
  • Add restart: always to all services
  • Switch from .envs/.local/.postgres to .envs/.production/.postgres
  • Remove hardcoded container_name on NATS (allows multiple instances on same host / worktrees)
  • Remove awscli service (backups handled by TeamCity)
  • RabbitMQ credentials configured via .envs/.production/.django, not hardcoded in compose

Usage

# Optional: start a local database
docker compose -f compose/staging/docker-compose.db.yml up -d

# Start the staging stack
docker compose -f docker-compose.staging.yml --env-file .envs/.production/.compose up -d

Environments this supports

Environment Database Redis/RabbitMQ/NATS
Demo (Arbutus) External ami-db-primary Local containers
Staging (Beluga) External fieldguide-db Local containers (or keep current external)
ood (Arbutus) Local via docker-compose.db.yml Local containers
Branch previews Any external or local DB Local containers

Relates to RolnickLab/ami-devops#1, RolnickLab/ami-admin#66
Closes #1180

Test plan

  • Validated staging compose with docker compose config
  • Started full stack locally: Django, Celery worker/beat, Flower, Redis, RabbitMQ, NATS, external Postgres
  • Ran migrations successfully
  • Created demo project with fixtures (50 captures, 20 occurrences, 6 taxa)
  • Verified API responds on :5001, NATS healthy on :8222, RabbitMQ management on :15672
  • CI tests pass
  • Verify gunicorn 23 works with existing production start script and uvicorn workers

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Staging infrastructure: Redis, RabbitMQ, and NATS can run locally to improve development and testing.
    • Optional local PostgreSQL support added for staging environments.
    • Staging services aligned with production-like port behavior and made safer for multiple instances.
    • Production dependency: gunicorn upgraded to a newer stable version.

mihow and others added 2 commits March 18, 2026 14:04
Update docker-compose.staging.yml to serve as the standard config for
staging, demo, and branch preview environments:

- Remove local Postgres (DB is always external via DATABASE_IP)
- Add RabbitMQ container for Celery task broker
- Add NATS container (was present but commented out in depends_on)
- Add restart:always to all services
- Switch from .envs/.local/.postgres to .envs/.production/.postgres
- Remove hardcoded container_name on NATS (allows multiple instances)
- Remove awscli service (backups handled by TeamCity)
- RabbitMQ credentials configured via .envs/.production/.django, not
  hardcoded in compose

Add compose/staging/docker-compose.db.yml as an optional convenience
for running a local PostgreSQL container when no external DB is
available (e.g., ood environment, local testing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
gunicorn 20.x requires pkg_resources from setuptools, which was
removed in setuptools 82+. Fresh Docker image builds fail with
ModuleNotFoundError on startup.

gunicorn 23 drops the pkg_resources dependency entirely.

Closes #1180

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings March 18, 2026 21:52
@netlify
Copy link

netlify bot commented Mar 18, 2026

Deploy Preview for antenna-ssec canceled.

Name Link
🔨 Latest commit 7f75f98
🔍 Latest deploy log https://app.netlify.com/projects/antenna-ssec/deploys/69bb237d92f2940007ea5efa

@netlify
Copy link

netlify bot commented Mar 18, 2026

Deploy Preview for antenna-preview canceled.

Name Link
🔨 Latest commit 7f75f98
🔍 Latest deploy log https://app.netlify.com/projects/antenna-preview/deploys/69bb237d26cebf0008ce26d5

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 18, 2026

Warning

Rate limit exceeded

@mihow has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 5 minutes and 53 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: db4f7296-9f60-454e-98d6-3cb9249a6437

📥 Commits

Reviewing files that changed from the base of the PR and between d5e8f6f and 7f75f98.

📒 Files selected for processing (1)
  • compose/staging/README.md
📝 Walkthrough

Walkthrough

Staging compose now provides an optional local Postgres file and moves Redis, RabbitMQ, and NATS to local containers while Django uses an external DATABASE_IP for Postgres. Gunicorn bumped from 20.1.0 to 23.0.0 in requirements.

Changes

Cohort / File(s) Summary
Staging DB compose
compose/staging/docker-compose.db.yml
Adds an optional local PostgreSQL Compose file with named volumes, env_file, ports, and restart policy for staging.
Staging Compose services
docker-compose.staging.yml
Removes local postgres; adds local redis, rabbitmq, and nats; Django now uses DATABASE_IP via extra_hosts and updated env_files; depends_on includes rabbitmq and nats; celery/flower services gain restart: always and flower gains a data volume; port variables parameterized.
Dependencies
requirements/base.txt
Bumps gunicorn from 20.1.0 to 23.0.0.

Sequence Diagram(s)

sequenceDiagram
    participant Dev as Developer / CI
    participant Compose as docker-compose (staging)
    participant Django as Django container
    participant Postgres as Postgres (external or compose.db)
    participant Redis as Redis
    participant Rabbit as RabbitMQ
    participant NATS as NATS

    Dev->>Compose: up -d (docker-compose.staging.yml)
    Compose->>Redis: start
    Compose->>Rabbit: start
    Compose->>NATS: start
    Compose->>Django: start (depends_on checks)
    Django->>Postgres: connect via DATABASE_IP / env_file
    Django->>Redis: cache/session usage
    Django->>Rabbit: publish/consume tasks
    Django->>NATS: pub/sub telemetry
    Rabbit->>Celery: deliver tasks (celery workers)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers

  • annavik

Poem

🐰 I hopped through compose with a carrot and grin,
Redis, RabbitMQ, and NATS tucked in,
Postgres optional—external by name,
Gunicorn leapt versions, faster the game,
Staging now hums — a rabbit's small win. 🥕

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Update staging compose for demo/preview deployments' clearly summarizes the main change of updating the staging Docker Compose configuration for multiple deployment scenarios.
Description check ✅ Passed The description includes all required sections from the template: Summary, List of Changes, Related Issues, Detailed Description with deployment considerations, Test plan, and verification status.
Linked Issues check ✅ Passed The PR successfully implements the primary objective from issue #1180: upgrading gunicorn from 20.1.0 to 23.0.0 to fix the pkg_resources ModuleNotFoundError. The staging compose changes support the broader deployment infrastructure goals.
Out of Scope Changes check ✅ Passed All changes are scope-aligned: the gunicorn upgrade directly addresses issue #1180, while the Docker Compose updates support the staging environment infrastructure improvements documented in the PR objectives.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/update-staging-compose
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
docker-compose.staging.yml (1)

68-70: Bind RabbitMQ management UI to localhost by default.

Line 69 exposes 15672 broadly; in staging/demo this is safer as localhost-bound unless remote admin access is explicitly needed.

Proposed hardening
     ports:
-      - "15672:15672"
+      - "127.0.0.1:15672:15672"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docker-compose.staging.yml` around lines 68 - 70, The ports mapping currently
exposes the RabbitMQ management UI publicly via the line with "15672:15672";
change the ports entry so the management port is bound to localhost only (e.g.,
use a host IP prefix like 127.0.0.1:15672:15672) under the same ports block so
the service still restarts as configured (restart: always) but the management UI
is only accessible from the host machine.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@compose/staging/docker-compose.db.yml`:
- Around line 5-17: The comments are ambiguous about DB connectivity modes
(Docker network vs host bridge). Update the header comments in
compose/staging/docker-compose.db.yml to clearly state which mode the stack
expects: if the app uses POSTGRES_HOST=db (container-to-container via Docker
network) remove or reword the DATABASE_IP/host-bridge instructions;
alternatively, if the intended workflow requires the host bridge and
DATABASE_IP, change the POSTGRES_HOST guidance accordingly and explain when to
start the DB with docker compose -f compose/staging/docker-compose.db.yml up -d
vs when to set DATABASE_IP for the app compose. Ensure you reference
POSTGRES_HOST and DATABASE_IP and mention the two compose files
(compose/staging/docker-compose.db.yml and docker-compose.staging.yml) so
readers know which mode each file supports.
- Around line 35-36: The compose file currently publishes PostgreSQL on all
interfaces via the ports mapping `ports: - "5432:5432"`; change this to bind to
localhost by replacing it with `ports: - "127.0.0.1:5432:5432"` (or remove the
ports mapping entirely and rely on an internal network) so Postgres only listens
on loopback for staging/demo unless external access is explicitly required;
update any documentation or scripts that expect an externally accessible port
accordingly.

In `@docker-compose.staging.yml`:
- Around line 14-15: The rabbitmq service is missing an env_file so it doesn't
pick up RABBITMQ_DEFAULT_USER/RABBITMQ_DEFAULT_PASS from
.envs/.production/.django; update the rabbitmq service definition to add an
env_file pointing to .envs/.production/.django (so the RABBITMQ_DEFAULT_USER and
RABBITMQ_DEFAULT_PASS values are loaded) and remove or override any conflicting
environment: entries if present; ensure the service name "rabbitmq" and the
variables RABBITMQ_DEFAULT_USER / RABBITMQ_DEFAULT_PASS are used consistently
with Django's CELERY_BROKER_URL.

---

Nitpick comments:
In `@docker-compose.staging.yml`:
- Around line 68-70: The ports mapping currently exposes the RabbitMQ management
UI publicly via the line with "15672:15672"; change the ports entry so the
management port is bound to localhost only (e.g., use a host IP prefix like
127.0.0.1:15672:15672) under the same ports block so the service still restarts
as configured (restart: always) but the management UI is only accessible from
the host machine.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3cbcc483-c4ef-463f-8d20-ba97a05053c7

📥 Commits

Reviewing files that changed from the base of the PR and between 59d2c21 and b3d9771.

📒 Files selected for processing (3)
  • compose/staging/docker-compose.db.yml
  • docker-compose.staging.yml
  • requirements/base.txt

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the staging Docker Compose setup to be a shared baseline for staging/demo/branch-preview deployments by running Redis/RabbitMQ/NATS locally while connecting to an external Postgres via DATABASE_IP, and upgrades Gunicorn to avoid fresh-build failures on slim Python images.

Changes:

  • Upgrade gunicorn to 23.0.0.
  • Revise docker-compose.staging.yml to remove the local Postgres service and add local RabbitMQ + NATS (with restarts and updated env-file usage).
  • Add an optional compose/staging/docker-compose.db.yml for running a local Postgres container.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File Description
requirements/base.txt Bumps Gunicorn to 23.0.0.
docker-compose.staging.yml Reworks staging compose to use external DB + local Redis/RabbitMQ/NATS.
compose/staging/docker-compose.db.yml Adds an optional local Postgres compose for staging-like setups.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mihow and others added 3 commits March 18, 2026 15:09
- Add env_file to rabbitmq service so it picks up
  RABBITMQ_DEFAULT_USER/RABBITMQ_DEFAULT_PASS from .django env
- Use ${DATABASE_IP:?} required-variable syntax for fail-fast on
  missing config
- Bind local Postgres to 127.0.0.1 instead of 0.0.0.0
- Clarify DB compose comments: document host-bridge connectivity
  via DATABASE_IP, remove ambiguous "Docker network" wording

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Internal services (Redis, RabbitMQ, NATS) don't need host port
exposure — only the app containers talk to them via the Docker
network. Removing host ports means multiple instances (branch
previews, worktrees) never conflict on these ports.

Django and Flower ports are now configurable via DJANGO_PORT and
FLOWER_PORT env vars (default 5001 and 5550).

Also use host-gateway (works on all platforms) instead of
platform-specific Docker bridge IPs in DB compose docs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Setup instructions for single and multi-instance staging deployments,
covering environment configuration, database options, migrations,
sample data, and port management for running multiple instances on
the same host.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade gunicorn from 20.1.0 to 23.x

2 participants