From 980efc4ff54f616d32a4b00689f8a00739773a62 Mon Sep 17 00:00:00 2001 From: tianzhou Date: Thu, 30 Oct 2025 16:15:47 +0800 Subject: [PATCH 1/3] docs: document external database support for plan command MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update documentation to reflect the new external database option for plan generation: - .env.example: Add PGSCHEMA_PLAN_* environment variables with usage examples and explanations - CLAUDE.md: Document the external database feature in architecture, environment variables, and key patterns sections The external database feature allows users to provide their own PostgreSQL database instead of using embedded postgres, which is useful for environments where embedded postgres has limitations (e.g., ARM architectures, containerized environments). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- .env.example | 23 ++++++++++++++++++++++- CLAUDE.md | 20 ++++++++++++++++++-- 2 files changed, 40 insertions(+), 3 deletions(-) diff --git a/.env.example b/.env.example index d2cedd98..6c4adec6 100644 --- a/.env.example +++ b/.env.example @@ -2,6 +2,7 @@ # These environment variables are automatically used by pgschema commands # when the corresponding CLI flags are not provided +# Target Database Connection (for plan and apply commands) # Database server host (default: localhost) PGHOST=localhost @@ -19,4 +20,24 @@ PGPASSWORD=your_password_here # Application name for database connection (default: pgschema) # This appears in pg_stat_activity and can help identify connections -PGAPPNAME=pgschema \ No newline at end of file +PGAPPNAME=pgschema + +# Plan Database Connection (optional - for using external database instead of embedded postgres) +# If PGSCHEMA_PLAN_HOST is provided, the plan command will use an external database +# to validate the desired state instead of spinning up an embedded PostgreSQL instance. +# This is useful for environments where embedded postgres has limitations. + +# Plan database host (if not provided, uses embedded postgres) +#PGSCHEMA_PLAN_HOST=localhost + +# Plan database port (default: 5432) +#PGSCHEMA_PLAN_PORT=5432 + +# Plan database name (required if PGSCHEMA_PLAN_HOST is set) +#PGSCHEMA_PLAN_DB=pgschema_plan + +# Plan database user (required if PGSCHEMA_PLAN_HOST is set) +#PGSCHEMA_PLAN_USER=postgres + +# Plan database password +#PGSCHEMA_PLAN_PASSWORD=your_plan_db_password \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 2a1a5f5a..28756ec3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -98,6 +98,10 @@ PGPASSWORD=testpwd1 - `dump/` - Schema dump formatting and output - `fingerprint/` - Schema fingerprinting for change detection - `include/` - Include file processing for modular schemas +- `postgres/` - Database provider implementations (embedded and external) + - `desired_state.go` - DesiredStateProvider interface + - `embedded.go` - Embedded PostgreSQL implementation + - `external.go` - External database implementation - `color/` - Terminal output colorization - `logger/` - Structured logging - `version/` - Version information @@ -106,7 +110,7 @@ PGPASSWORD=testpwd1 **Schema Representation**: Uses an Intermediate Representation (IR) to normalize schema objects from database introspection. Both desired state (from user SQL files) and current state (from target database) are extracted by inspecting PostgreSQL databases. -**Embedded Postgres for Desired State**: The `plan` command spins up a temporary embedded PostgreSQL instance, applies the user's SQL files to it, then inspects that database to get the desired state IR. This ensures both desired and current states come from the same source (database inspection), eliminating parser/inspector format differences. +**Embedded Postgres for Desired State**: The `plan` command spins up a temporary embedded PostgreSQL instance (by default) or connects to an external database (if `--plan-host` is provided), applies the user's SQL files to it, then inspects that database to get the desired state IR. This ensures both desired and current states come from the same source (database inspection), eliminating parser/inspector format differences. External database support is useful for environments where embedded postgres has limitations (e.g., ARM architectures, containerized environments). **Migration Planning**: The `diff` package compares IR representations to generate a sequence of migration steps with proper dependency ordering (topological sort). @@ -114,6 +118,12 @@ PGPASSWORD=testpwd1 **Inspector-Only Approach**: Both desired state (from user SQL files) and current state (from target database) are obtained through database inspection. The plan command spins up an embedded PostgreSQL instance, applies user SQL files, then inspects it to get the desired state IR. This eliminates the need for SQL parsing and ensures consistency. +**External Database for Plan Generation**: As an alternative to embedded postgres, users can provide an external PostgreSQL database using `--plan-host` flags or `PGSCHEMA_PLAN_*` environment variables. The external database approach: +- Creates temporary schemas with timestamp suffixes (e.g., `pgschema_plan_20251030_154501_123456789`) +- Validates major version compatibility with target database (exact match required) +- Cleans up temporary schemas after use (best effort) +- Useful for environments where embedded postgres has limitations (ARM architectures, containerized environments) + ## Common Development Workflows ### Adding New Schema Object Support @@ -158,7 +168,13 @@ The tool supports comprehensive PostgreSQL schema objects (see `ir/ir.go` for co ## Environment Variables -- **Database connection**: `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, `PGDATABASE` +- **Target database connection**: `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, `PGDATABASE` +- **Plan database connection** (optional - for external database instead of embedded postgres): + - `PGSCHEMA_PLAN_HOST` - If set, uses external database for plan generation + - `PGSCHEMA_PLAN_PORT` - Plan database port (default: 5432) + - `PGSCHEMA_PLAN_DB` - Plan database name (required if PGSCHEMA_PLAN_HOST is set) + - `PGSCHEMA_PLAN_USER` - Plan database user (required if PGSCHEMA_PLAN_HOST is set) + - `PGSCHEMA_PLAN_PASSWORD` - Plan database password - **Environment files**: `.env` - automatically loaded by main.go - **Test filtering**: `PGSCHEMA_TEST_FILTER` - run specific test cases (e.g., `"create_table/"` or `"create_table/add_column"`) - **Postgres version**: `PGSCHEMA_POSTGRES_VERSION` - test against specific versions (14, 15, 16, 17) From 8e0ca779d8021cb6d22e7f99c6244ee2ff0a56e1 Mon Sep 17 00:00:00 2001 From: tianzhou Date: Thu, 30 Oct 2025 16:23:30 +0800 Subject: [PATCH 2/3] docs: add comprehensive external plan database documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Create detailed documentation for the external plan database feature covering: - Overview and when to use external database vs embedded - Configuration options (CLI flags and environment variables) - Real-world examples: - Docker Compose setup - GitHub Actions CI/CD pipeline - Kubernetes deployment - Version compatibility requirements - Database permissions needed - Best practices (dedicated database, cleanup, monitoring, pooling) - Troubleshooting common issues - Comparison table: embedded vs external database The documentation provides production-ready examples for containerized environments, CI/CD pipelines, and Kubernetes deployments where the external database feature is most valuable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- docs/cli/plan-db.mdx | 501 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 501 insertions(+) create mode 100644 docs/cli/plan-db.mdx diff --git a/docs/cli/plan-db.mdx b/docs/cli/plan-db.mdx new file mode 100644 index 00000000..3f3b2970 --- /dev/null +++ b/docs/cli/plan-db.mdx @@ -0,0 +1,501 @@ +--- +title: "External Plan Database" +--- + +The `plan` command can use an external PostgreSQL database instead of the default embedded PostgreSQL instance for validating desired state schemas. This is useful in environments where embedded PostgreSQL has limitations. + +## Overview + +By default, the `plan` command spins up a temporary embedded PostgreSQL instance to apply and validate your desired state SQL. However, you can optionally provide your own PostgreSQL database using the `--plan-*` flags or `PGSCHEMA_PLAN_*` environment variables. + +### When to Use External Database + +Use an external database for plan generation when: + +- Running on **ARM architectures** where embedded PostgreSQL support is limited +- Working in **containerized environments** (Docker, Kubernetes) where embedded postgres may have issues +- You have **resource constraints** and want to reuse an existing database +- You need more control over the PostgreSQL version or configuration +- Running in **CI/CD pipelines** where spinning up embedded postgres is slow + +### How It Works + +When using an external database: + +1. **Temporary Schema Creation**: pgschema creates a temporary schema with a unique timestamp (e.g., `pgschema_plan_20251030_154501_123456789`) +2. **SQL Application**: Your desired state SQL is applied to the temporary schema +3. **Schema Inspection**: The temporary schema is inspected to extract the desired state +4. **Comparison**: The desired state is compared with your target database's current state +5. **Cleanup**: The temporary schema is dropped (best effort) after plan generation + +## Basic Usage + +```bash +# Use external database for plan generation +pgschema plan \ + --file schema.sql \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres + +# With all options specified +pgschema plan \ + --file schema.sql \ + --host localhost --port 5432 --db myapp --user postgres \ + --plan-host localhost --plan-port 5432 --plan-db pgschema_plan --plan-user postgres --plan-password secret +``` + +## Configuration Options + +### Using Command-Line Flags + + + Plan database server host. If provided, uses external database instead of embedded PostgreSQL. + + Environment variable: `PGSCHEMA_PLAN_HOST` + + + + Plan database server port. + + Environment variable: `PGSCHEMA_PLAN_PORT` + + + + Plan database name. Required when `--plan-host` is provided. + + Environment variable: `PGSCHEMA_PLAN_DB` + + + + Plan database user name. Required when `--plan-host` is provided. + + Environment variable: `PGSCHEMA_PLAN_USER` + + + + Plan database password. Can also be provided via `PGSCHEMA_PLAN_PASSWORD` environment variable. + + Environment variable: `PGSCHEMA_PLAN_PASSWORD` + + +### Using Environment Variables + + +```bash .env File (Recommended) +# Target database connection +PGHOST=localhost +PGPORT=5432 +PGDATABASE=myapp +PGUSER=postgres +PGPASSWORD=mypassword + +# Plan database connection (optional) +PGSCHEMA_PLAN_HOST=localhost +PGSCHEMA_PLAN_PORT=5432 +PGSCHEMA_PLAN_DB=pgschema_plan +PGSCHEMA_PLAN_USER=postgres +PGSCHEMA_PLAN_PASSWORD=planpassword + +# Run plan with external database +pgschema plan --file schema.sql +``` + +```bash Environment Variables +# Set environment variables +export PGHOST=localhost +export PGDATABASE=myapp +export PGUSER=postgres +export PGPASSWORD=mypassword + +export PGSCHEMA_PLAN_HOST=localhost +export PGSCHEMA_PLAN_DB=pgschema_plan +export PGSCHEMA_PLAN_USER=postgres +export PGSCHEMA_PLAN_PASSWORD=planpassword + +# Run plan +pgschema plan --file schema.sql +``` + +```bash Command Line Only +# All options as flags (no environment variables) +pgschema plan \ + --file schema.sql \ + --host localhost \ + --db myapp \ + --user postgres \ + --password mypassword \ + --plan-host localhost \ + --plan-db pgschema_plan \ + --plan-user postgres \ + --plan-password planpassword +``` + + +## Examples + +### Docker Compose Setup + +```yaml docker-compose.yml +version: '3.8' + +services: + target-db: + image: postgres:16 + environment: + POSTGRES_DB: myapp + POSTGRES_USER: postgres + POSTGRES_PASSWORD: secret + ports: + - "5432:5432" + + plan-db: + image: postgres:16 + environment: + POSTGRES_DB: pgschema_plan + POSTGRES_USER: postgres + POSTGRES_PASSWORD: secret + ports: + - "5433:5432" + + pgschema: + image: pgschema/pgschema:latest + environment: + # Target database + PGHOST: target-db + PGPORT: 5432 + PGDATABASE: myapp + PGUSER: postgres + PGPASSWORD: secret + + # Plan database + PGSCHEMA_PLAN_HOST: plan-db + PGSCHEMA_PLAN_PORT: 5432 + PGSCHEMA_PLAN_DB: pgschema_plan + PGSCHEMA_PLAN_USER: postgres + PGSCHEMA_PLAN_PASSWORD: secret + volumes: + - ./schema.sql:/schema.sql + command: plan --file /schema.sql +``` + +### CI/CD Pipeline (GitHub Actions) + +```yaml .github/workflows/plan.yml +name: Schema Plan + +on: + pull_request: + paths: + - 'schema.sql' + +jobs: + plan: + runs-on: ubuntu-latest + + services: + postgres: + image: postgres:16 + env: + POSTGRES_DB: myapp + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + ports: + - 5432:5432 + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + + plan-db: + image: postgres:16 + env: + POSTGRES_DB: pgschema_plan + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + ports: + - 5433:5432 + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + + steps: + - uses: actions/checkout@v4 + + - name: Download pgschema + run: | + curl -L https://github.com/pgschema/pgschema/releases/latest/download/pgschema-linux-amd64 -o pgschema + chmod +x pgschema + + - name: Generate Plan + env: + PGHOST: localhost + PGPORT: 5432 + PGDATABASE: myapp + PGUSER: postgres + PGPASSWORD: postgres + PGSCHEMA_PLAN_HOST: localhost + PGSCHEMA_PLAN_PORT: 5433 + PGSCHEMA_PLAN_DB: pgschema_plan + PGSCHEMA_PLAN_USER: postgres + PGSCHEMA_PLAN_PASSWORD: postgres + run: | + ./pgschema plan --file schema.sql --output-human plan.txt + cat plan.txt + + - name: Comment PR with Plan + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const plan = fs.readFileSync('plan.txt', 'utf8'); + github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: '## Schema Migration Plan\n\n```\n' + plan + '\n```' + }); +``` + +### Kubernetes Deployment + +```yaml kubernetes-job.yml +apiVersion: batch/v1 +kind: Job +metadata: + name: pgschema-plan +spec: + template: + spec: + containers: + - name: pgschema + image: pgschema/pgschema:latest + command: ["pgschema", "plan", "--file", "/schema/schema.sql"] + env: + # Target database connection + - name: PGHOST + value: "postgres-service" + - name: PGPORT + value: "5432" + - name: PGDATABASE + value: "myapp" + - name: PGUSER + valueFrom: + secretKeyRef: + name: postgres-credentials + key: username + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: postgres-credentials + key: password + + # Plan database connection + - name: PGSCHEMA_PLAN_HOST + value: "postgres-plan-service" + - name: PGSCHEMA_PLAN_PORT + value: "5432" + - name: PGSCHEMA_PLAN_DB + value: "pgschema_plan" + - name: PGSCHEMA_PLAN_USER + valueFrom: + secretKeyRef: + name: postgres-plan-credentials + key: username + - name: PGSCHEMA_PLAN_PASSWORD + valueFrom: + secretKeyRef: + name: postgres-plan-credentials + key: password + + volumeMounts: + - name: schema + mountPath: /schema + + volumes: + - name: schema + configMap: + name: schema-files + + restartPolicy: Never +``` + +## Version Compatibility + +**Important**: The plan database must have the **same major version** as the target database. + +```bash +# ✅ Both databases are PostgreSQL 16.x - This works +Target DB: PostgreSQL 16.9 +Plan DB: PostgreSQL 16.4 + +# ❌ Different major versions - This fails +Target DB: PostgreSQL 16.9 +Plan DB: PostgreSQL 15.13 + +# Error: version mismatch: plan database is PostgreSQL 15, +# but target database is PostgreSQL 16 (exact major version match required) +``` + +pgschema automatically detects the target database version and validates that the plan database matches before proceeding. + +## Database Permissions + +The plan database user needs the following permissions: + +```sql +-- Minimum required permissions +GRANT CREATE ON DATABASE pgschema_plan TO your_plan_user; +GRANT USAGE ON SCHEMA public TO your_plan_user; + +-- Or simply use a superuser for simplicity +ALTER USER your_plan_user WITH SUPERUSER; +``` + +The user must be able to: +- Create and drop schemas +- Create tables, indexes, functions, and other schema objects +- Set search_path + +## Best Practices + + + + Create a dedicated database for plan operations to avoid conflicts with other workloads. + + ```sql + CREATE DATABASE pgschema_plan; + ``` + + + + While pgschema attempts to clean up temporary schemas, you may want to periodically clean up any leftover schemas: + + ```sql + -- Find old temporary schemas (older than 1 hour) + SELECT schemaname + FROM pg_catalog.pg_namespace n + JOIN pg_catalog.pg_stat_user_tables s ON n.nspname = s.schemaname + WHERE schemaname LIKE 'pgschema_plan_%' + AND pg_stat_file('base/'||n.oid)::record->>'modification' < now() - interval '1 hour'; + + -- Drop them + DO $$ + DECLARE + schema_name text; + BEGIN + FOR schema_name IN + SELECT nspname + FROM pg_namespace + WHERE nspname LIKE 'pgschema_plan_%' + LOOP + EXECUTE 'DROP SCHEMA IF EXISTS ' || quote_ident(schema_name) || ' CASCADE'; + END LOOP; + END $$; + ``` + + + + Plan operations can be resource-intensive for large schemas. Monitor your plan database: + + ```sql + -- Check active sessions + SELECT * FROM pg_stat_activity + WHERE datname = 'pgschema_plan'; + + -- Check schema sizes + SELECT schemaname, + pg_size_pretty(sum(pg_total_relation_size(schemaname||'.'||tablename))::bigint) + FROM pg_tables + WHERE schemaname LIKE 'pgschema_plan_%' + GROUP BY schemaname + ORDER BY sum(pg_total_relation_size(schemaname||'.'||tablename)) DESC; + ``` + + + + For high-frequency plan operations, consider using connection pooling (like PgBouncer) to reduce connection overhead. + + + + Always use environment variables or secret management for passwords, never hardcode them in scripts: + + ```bash + # ✅ Good - Using environment variables + export PGSCHEMA_PLAN_PASSWORD=$(vault read -field=password secret/pgschema) + + # ❌ Bad - Hardcoded password + pgschema plan --plan-password hardcodedpassword + ``` + + + +## Troubleshooting + +### Version Mismatch Error + +``` +Error: version mismatch: plan database is PostgreSQL 15, but target database is PostgreSQL 16 +``` + +**Solution**: Ensure both databases are running the same major version of PostgreSQL. + +### Connection Refused + +``` +Error: failed to connect to external database: connection refused +``` + +**Solution**: +- Verify the plan database is running and accessible +- Check firewall rules and network connectivity +- Verify host and port are correct + +### Permission Denied + +``` +Error: failed to create temporary schema: permission denied for database +``` + +**Solution**: Grant CREATE permission to the plan database user: + +```sql +GRANT CREATE ON DATABASE pgschema_plan TO your_plan_user; +``` + +### Temporary Schema Not Cleaned Up + +If temporary schemas are not being cleaned up automatically: + +```sql +-- List all temporary schemas +SELECT nspname +FROM pg_namespace +WHERE nspname LIKE 'pgschema_plan_%'; + +-- Manually drop old schemas +DROP SCHEMA IF EXISTS pgschema_plan_20251030_154501_123456789 CASCADE; +``` + +## Comparison: Embedded vs External Database + +| Feature | Embedded PostgreSQL | External Database | +|---------|---------------------|-------------------| +| **Setup** | Automatic | Requires configuration | +| **Startup Time** | 2-5 seconds | Instant (already running) | +| **Platform Support** | Limited (x86_64 primarily) | Universal | +| **Resource Usage** | Creates new process | Reuses existing database | +| **Network Required** | No | Yes | +| **Container Friendly** | Limited | Yes | +| **ARM Support** | Limited | Yes | +| **Version Control** | Automatic (matches target) | Manual (must match target) | +| **Cleanup** | Automatic | Best effort | +| **Use Case** | Local development | CI/CD, production, containers | + +## See Also + +- [Plan Command](/cli/plan) - Main plan command documentation +- [Apply Command](/cli/apply) - Applying migration plans +- [Environment Variables](/cli/dotenv) - Managing environment configuration From 5970f007e6f3320af2a7dd981288fac4d0270550 Mon Sep 17 00:00:00 2001 From: tianzhou Date: Thu, 30 Oct 2025 16:43:29 +0800 Subject: [PATCH 3/3] docs: update plan-db.mdx with use cases and cross-reference from plan.mdx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add "Common Use Cases" section with practical examples for: - Using PostgreSQL extensions (issue #121) - Handling cross-schema foreign keys (issue #122) - Update plan.mdx to reference external plan database feature: - Add explanation to Overview section - Add "Plan Database Options" section with link - Add example showing external database usage - Streamline plan-db.mdx by focusing on main use cases - Update docs.json navigation to include plan-db page 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- docs/cli/plan-db.mdx | 437 +++++++++---------------------------------- docs/cli/plan.mdx | 7 + docs/docs.json | 2 +- 3 files changed, 92 insertions(+), 354 deletions(-) diff --git a/docs/cli/plan-db.mdx b/docs/cli/plan-db.mdx index 3f3b2970..3ba03044 100644 --- a/docs/cli/plan-db.mdx +++ b/docs/cli/plan-db.mdx @@ -12,11 +12,8 @@ By default, the `plan` command spins up a temporary embedded PostgreSQL instance Use an external database for plan generation when: -- Running on **ARM architectures** where embedded PostgreSQL support is limited -- Working in **containerized environments** (Docker, Kubernetes) where embedded postgres may have issues -- You have **resource constraints** and want to reuse an existing database -- You need more control over the PostgreSQL version or configuration -- Running in **CI/CD pipelines** where spinning up embedded postgres is slow +- Your schema uses **PostgreSQL extensions** (like `hstore`, `postgis`, `uuid-ossp`, etc.) - The embedded database doesn't have extensions pre-installed, causing plan generation to fail with "type does not exist" errors ([#121](https://github.com/pgschema/pgschema/issues/121)) +- Your schema has **cross-schema foreign key references** - The embedded approach only loads one schema at a time, breaking foreign key constraints that reference tables in other schemas ([#122](https://github.com/pgschema/pgschema/issues/122)) ### How It Works @@ -44,6 +41,88 @@ pgschema plan \ --plan-host localhost --plan-port 5432 --plan-db pgschema_plan --plan-user postgres --plan-password secret ``` +## Common Use Cases + +### Using PostgreSQL Extensions + +If your schema uses extensions like `hstore`, `postgis`, or `uuid-ossp`, you need to install them in the plan database first: + +```sql +-- In your plan database, install required extensions +CREATE EXTENSION IF NOT EXISTS hstore; +CREATE EXTENSION IF NOT EXISTS postgis; +CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; +``` + +Then run plan with the external database: + +```bash +# Install extensions in plan database +psql -h localhost -U postgres -d pgschema_plan -c "CREATE EXTENSION IF NOT EXISTS hstore;" + +# Now run plan - it will work because extensions are available +pgschema plan \ + --file schema.sql \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres +``` + +Your `schema.sql` can now use extension types: + +```sql +CREATE TABLE products ( + id SERIAL PRIMARY KEY, + attributes HSTORE, -- Works because hstore extension is installed + location GEOGRAPHY(POINT, 4326) -- Works because postgis is installed +); +``` + +### Handling Cross-Schema Foreign Keys + +If your schema has foreign keys that reference tables in other schemas, you need to create those schemas in the plan database: + +```sql +-- In your plan database, create referenced schemas and tables +CREATE SCHEMA IF NOT EXISTS auth; +CREATE TABLE IF NOT EXISTS auth.users ( + id SERIAL PRIMARY KEY, + email TEXT NOT NULL +); + +CREATE SCHEMA IF NOT EXISTS billing; +CREATE TABLE IF NOT EXISTS billing.customers ( + id SERIAL PRIMARY KEY, + user_id INTEGER REFERENCES auth.users(id) +); +``` + +Then run plan: + +```bash +# Set up referenced schemas in plan database +psql -h localhost -U postgres -d pgschema_plan << 'EOF' +CREATE SCHEMA IF NOT EXISTS auth; +CREATE TABLE IF NOT EXISTS auth.users (id SERIAL PRIMARY KEY, email TEXT NOT NULL); +EOF + +# Now run plan for your main schema that references auth.users +pgschema plan \ + --file schema.sql \ + --schema public \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres +``` + +Your `schema.sql` can now reference tables in other schemas: + +```sql +CREATE TABLE orders ( + id SERIAL PRIMARY KEY, + user_id INTEGER REFERENCES auth.users(id), -- Cross-schema FK works + total DECIMAL(10,2) +); +``` + ## Configuration Options ### Using Command-Line Flags @@ -131,216 +210,6 @@ pgschema plan \ ``` -## Examples - -### Docker Compose Setup - -```yaml docker-compose.yml -version: '3.8' - -services: - target-db: - image: postgres:16 - environment: - POSTGRES_DB: myapp - POSTGRES_USER: postgres - POSTGRES_PASSWORD: secret - ports: - - "5432:5432" - - plan-db: - image: postgres:16 - environment: - POSTGRES_DB: pgschema_plan - POSTGRES_USER: postgres - POSTGRES_PASSWORD: secret - ports: - - "5433:5432" - - pgschema: - image: pgschema/pgschema:latest - environment: - # Target database - PGHOST: target-db - PGPORT: 5432 - PGDATABASE: myapp - PGUSER: postgres - PGPASSWORD: secret - - # Plan database - PGSCHEMA_PLAN_HOST: plan-db - PGSCHEMA_PLAN_PORT: 5432 - PGSCHEMA_PLAN_DB: pgschema_plan - PGSCHEMA_PLAN_USER: postgres - PGSCHEMA_PLAN_PASSWORD: secret - volumes: - - ./schema.sql:/schema.sql - command: plan --file /schema.sql -``` - -### CI/CD Pipeline (GitHub Actions) - -```yaml .github/workflows/plan.yml -name: Schema Plan - -on: - pull_request: - paths: - - 'schema.sql' - -jobs: - plan: - runs-on: ubuntu-latest - - services: - postgres: - image: postgres:16 - env: - POSTGRES_DB: myapp - POSTGRES_USER: postgres - POSTGRES_PASSWORD: postgres - ports: - - 5432:5432 - options: >- - --health-cmd pg_isready - --health-interval 10s - --health-timeout 5s - --health-retries 5 - - plan-db: - image: postgres:16 - env: - POSTGRES_DB: pgschema_plan - POSTGRES_USER: postgres - POSTGRES_PASSWORD: postgres - ports: - - 5433:5432 - options: >- - --health-cmd pg_isready - --health-interval 10s - --health-timeout 5s - --health-retries 5 - - steps: - - uses: actions/checkout@v4 - - - name: Download pgschema - run: | - curl -L https://github.com/pgschema/pgschema/releases/latest/download/pgschema-linux-amd64 -o pgschema - chmod +x pgschema - - - name: Generate Plan - env: - PGHOST: localhost - PGPORT: 5432 - PGDATABASE: myapp - PGUSER: postgres - PGPASSWORD: postgres - PGSCHEMA_PLAN_HOST: localhost - PGSCHEMA_PLAN_PORT: 5433 - PGSCHEMA_PLAN_DB: pgschema_plan - PGSCHEMA_PLAN_USER: postgres - PGSCHEMA_PLAN_PASSWORD: postgres - run: | - ./pgschema plan --file schema.sql --output-human plan.txt - cat plan.txt - - - name: Comment PR with Plan - uses: actions/github-script@v7 - with: - script: | - const fs = require('fs'); - const plan = fs.readFileSync('plan.txt', 'utf8'); - github.rest.issues.createComment({ - issue_number: context.issue.number, - owner: context.repo.owner, - repo: context.repo.repo, - body: '## Schema Migration Plan\n\n```\n' + plan + '\n```' - }); -``` - -### Kubernetes Deployment - -```yaml kubernetes-job.yml -apiVersion: batch/v1 -kind: Job -metadata: - name: pgschema-plan -spec: - template: - spec: - containers: - - name: pgschema - image: pgschema/pgschema:latest - command: ["pgschema", "plan", "--file", "/schema/schema.sql"] - env: - # Target database connection - - name: PGHOST - value: "postgres-service" - - name: PGPORT - value: "5432" - - name: PGDATABASE - value: "myapp" - - name: PGUSER - valueFrom: - secretKeyRef: - name: postgres-credentials - key: username - - name: PGPASSWORD - valueFrom: - secretKeyRef: - name: postgres-credentials - key: password - - # Plan database connection - - name: PGSCHEMA_PLAN_HOST - value: "postgres-plan-service" - - name: PGSCHEMA_PLAN_PORT - value: "5432" - - name: PGSCHEMA_PLAN_DB - value: "pgschema_plan" - - name: PGSCHEMA_PLAN_USER - valueFrom: - secretKeyRef: - name: postgres-plan-credentials - key: username - - name: PGSCHEMA_PLAN_PASSWORD - valueFrom: - secretKeyRef: - name: postgres-plan-credentials - key: password - - volumeMounts: - - name: schema - mountPath: /schema - - volumes: - - name: schema - configMap: - name: schema-files - - restartPolicy: Never -``` - -## Version Compatibility - -**Important**: The plan database must have the **same major version** as the target database. - -```bash -# ✅ Both databases are PostgreSQL 16.x - This works -Target DB: PostgreSQL 16.9 -Plan DB: PostgreSQL 16.4 - -# ❌ Different major versions - This fails -Target DB: PostgreSQL 16.9 -Plan DB: PostgreSQL 15.13 - -# Error: version mismatch: plan database is PostgreSQL 15, -# but target database is PostgreSQL 16 (exact major version match required) -``` - -pgschema automatically detects the target database version and validates that the plan database matches before proceeding. - ## Database Permissions The plan database user needs the following permissions: @@ -349,9 +218,6 @@ The plan database user needs the following permissions: -- Minimum required permissions GRANT CREATE ON DATABASE pgschema_plan TO your_plan_user; GRANT USAGE ON SCHEMA public TO your_plan_user; - --- Or simply use a superuser for simplicity -ALTER USER your_plan_user WITH SUPERUSER; ``` The user must be able to: @@ -359,141 +225,6 @@ The user must be able to: - Create tables, indexes, functions, and other schema objects - Set search_path -## Best Practices - - - - Create a dedicated database for plan operations to avoid conflicts with other workloads. - - ```sql - CREATE DATABASE pgschema_plan; - ``` - - - - While pgschema attempts to clean up temporary schemas, you may want to periodically clean up any leftover schemas: - - ```sql - -- Find old temporary schemas (older than 1 hour) - SELECT schemaname - FROM pg_catalog.pg_namespace n - JOIN pg_catalog.pg_stat_user_tables s ON n.nspname = s.schemaname - WHERE schemaname LIKE 'pgschema_plan_%' - AND pg_stat_file('base/'||n.oid)::record->>'modification' < now() - interval '1 hour'; - - -- Drop them - DO $$ - DECLARE - schema_name text; - BEGIN - FOR schema_name IN - SELECT nspname - FROM pg_namespace - WHERE nspname LIKE 'pgschema_plan_%' - LOOP - EXECUTE 'DROP SCHEMA IF EXISTS ' || quote_ident(schema_name) || ' CASCADE'; - END LOOP; - END $$; - ``` - - - - Plan operations can be resource-intensive for large schemas. Monitor your plan database: - - ```sql - -- Check active sessions - SELECT * FROM pg_stat_activity - WHERE datname = 'pgschema_plan'; - - -- Check schema sizes - SELECT schemaname, - pg_size_pretty(sum(pg_total_relation_size(schemaname||'.'||tablename))::bigint) - FROM pg_tables - WHERE schemaname LIKE 'pgschema_plan_%' - GROUP BY schemaname - ORDER BY sum(pg_total_relation_size(schemaname||'.'||tablename)) DESC; - ``` - - - - For high-frequency plan operations, consider using connection pooling (like PgBouncer) to reduce connection overhead. - - - - Always use environment variables or secret management for passwords, never hardcode them in scripts: - - ```bash - # ✅ Good - Using environment variables - export PGSCHEMA_PLAN_PASSWORD=$(vault read -field=password secret/pgschema) - - # ❌ Bad - Hardcoded password - pgschema plan --plan-password hardcodedpassword - ``` - - - -## Troubleshooting - -### Version Mismatch Error - -``` -Error: version mismatch: plan database is PostgreSQL 15, but target database is PostgreSQL 16 -``` - -**Solution**: Ensure both databases are running the same major version of PostgreSQL. - -### Connection Refused - -``` -Error: failed to connect to external database: connection refused -``` - -**Solution**: -- Verify the plan database is running and accessible -- Check firewall rules and network connectivity -- Verify host and port are correct - -### Permission Denied - -``` -Error: failed to create temporary schema: permission denied for database -``` - -**Solution**: Grant CREATE permission to the plan database user: - -```sql -GRANT CREATE ON DATABASE pgschema_plan TO your_plan_user; -``` - -### Temporary Schema Not Cleaned Up - -If temporary schemas are not being cleaned up automatically: - -```sql --- List all temporary schemas -SELECT nspname -FROM pg_namespace -WHERE nspname LIKE 'pgschema_plan_%'; - --- Manually drop old schemas -DROP SCHEMA IF EXISTS pgschema_plan_20251030_154501_123456789 CASCADE; -``` - -## Comparison: Embedded vs External Database - -| Feature | Embedded PostgreSQL | External Database | -|---------|---------------------|-------------------| -| **Setup** | Automatic | Requires configuration | -| **Startup Time** | 2-5 seconds | Instant (already running) | -| **Platform Support** | Limited (x86_64 primarily) | Universal | -| **Resource Usage** | Creates new process | Reuses existing database | -| **Network Required** | No | Yes | -| **Container Friendly** | Limited | Yes | -| **ARM Support** | Limited | Yes | -| **Version Control** | Automatic (matches target) | Manual (must match target) | -| **Cleanup** | Automatic | Best effort | -| **Use Case** | Local development | CI/CD, production, containers | - ## See Also - [Plan Command](/cli/plan) - Main plan command documentation diff --git a/docs/cli/plan.mdx b/docs/cli/plan.mdx index 2afd9c7b..5db20f15 100644 --- a/docs/cli/plan.mdx +++ b/docs/cli/plan.mdx @@ -8,11 +8,14 @@ The `plan` command generates a migration plan to apply a desired schema state to The plan command follows infrastructure-as-code principles similar to Terraform: 1. Read the desired state from a SQL file (with include directive support) +1. Apply the desired state SQL to a temporary PostgreSQL instance (embedded by default, or external via `--plan-*` flags) 1. Connect to the target database and analyze current state of the specified schema 1. Compare the two states 1. Generate a detailed migration plan with proper dependency ordering 1. Display the plan without making any changes +By default, pgschema uses an embedded PostgreSQL instance to validate your desired state SQL. For schemas using PostgreSQL extensions or cross-schema references, you can use an external database instead. See [External Plan Database](/cli/plan-db) for details. + ## Basic Usage ```bash @@ -117,6 +120,10 @@ pgschema plan --host localhost --db myapp --user postgres --password mypassword Schema name to target for comparison +## Plan Database Options + +By default, the plan command uses an embedded PostgreSQL instance to validate your desired state SQL. For schemas that require PostgreSQL extensions or have cross-schema references, you can provide an external database. See [External Plan Database](/cli/plan-db) for complete documentation. + ## Plan Options diff --git a/docs/docs.json b/docs/docs.json index 340d2b5b..76ffb56e 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -45,7 +45,7 @@ }, { "group": "Configuration", - "pages": ["cli/ignore", "cli/dotenv"] + "pages": ["cli/plan-db", "cli/ignore", "cli/dotenv"] } ] },