diff --git a/.env.example b/.env.example index d2cedd98..6c4adec6 100644 --- a/.env.example +++ b/.env.example @@ -2,6 +2,7 @@ # These environment variables are automatically used by pgschema commands # when the corresponding CLI flags are not provided +# Target Database Connection (for plan and apply commands) # Database server host (default: localhost) PGHOST=localhost @@ -19,4 +20,24 @@ PGPASSWORD=your_password_here # Application name for database connection (default: pgschema) # This appears in pg_stat_activity and can help identify connections -PGAPPNAME=pgschema \ No newline at end of file +PGAPPNAME=pgschema + +# Plan Database Connection (optional - for using external database instead of embedded postgres) +# If PGSCHEMA_PLAN_HOST is provided, the plan command will use an external database +# to validate the desired state instead of spinning up an embedded PostgreSQL instance. +# This is useful for environments where embedded postgres has limitations. + +# Plan database host (if not provided, uses embedded postgres) +#PGSCHEMA_PLAN_HOST=localhost + +# Plan database port (default: 5432) +#PGSCHEMA_PLAN_PORT=5432 + +# Plan database name (required if PGSCHEMA_PLAN_HOST is set) +#PGSCHEMA_PLAN_DB=pgschema_plan + +# Plan database user (required if PGSCHEMA_PLAN_HOST is set) +#PGSCHEMA_PLAN_USER=postgres + +# Plan database password +#PGSCHEMA_PLAN_PASSWORD=your_plan_db_password \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 2a1a5f5a..28756ec3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -98,6 +98,10 @@ PGPASSWORD=testpwd1 - `dump/` - Schema dump formatting and output - `fingerprint/` - Schema fingerprinting for change detection - `include/` - Include file processing for modular schemas +- `postgres/` - Database provider implementations (embedded and external) + - `desired_state.go` - DesiredStateProvider interface + - `embedded.go` - Embedded PostgreSQL implementation + - `external.go` - External database implementation - `color/` - Terminal output colorization - `logger/` - Structured logging - `version/` - Version information @@ -106,7 +110,7 @@ PGPASSWORD=testpwd1 **Schema Representation**: Uses an Intermediate Representation (IR) to normalize schema objects from database introspection. Both desired state (from user SQL files) and current state (from target database) are extracted by inspecting PostgreSQL databases. -**Embedded Postgres for Desired State**: The `plan` command spins up a temporary embedded PostgreSQL instance, applies the user's SQL files to it, then inspects that database to get the desired state IR. This ensures both desired and current states come from the same source (database inspection), eliminating parser/inspector format differences. +**Embedded Postgres for Desired State**: The `plan` command spins up a temporary embedded PostgreSQL instance (by default) or connects to an external database (if `--plan-host` is provided), applies the user's SQL files to it, then inspects that database to get the desired state IR. This ensures both desired and current states come from the same source (database inspection), eliminating parser/inspector format differences. External database support is useful for environments where embedded postgres has limitations (e.g., ARM architectures, containerized environments). **Migration Planning**: The `diff` package compares IR representations to generate a sequence of migration steps with proper dependency ordering (topological sort). @@ -114,6 +118,12 @@ PGPASSWORD=testpwd1 **Inspector-Only Approach**: Both desired state (from user SQL files) and current state (from target database) are obtained through database inspection. The plan command spins up an embedded PostgreSQL instance, applies user SQL files, then inspects it to get the desired state IR. This eliminates the need for SQL parsing and ensures consistency. +**External Database for Plan Generation**: As an alternative to embedded postgres, users can provide an external PostgreSQL database using `--plan-host` flags or `PGSCHEMA_PLAN_*` environment variables. The external database approach: +- Creates temporary schemas with timestamp suffixes (e.g., `pgschema_plan_20251030_154501_123456789`) +- Validates major version compatibility with target database (exact match required) +- Cleans up temporary schemas after use (best effort) +- Useful for environments where embedded postgres has limitations (ARM architectures, containerized environments) + ## Common Development Workflows ### Adding New Schema Object Support @@ -158,7 +168,13 @@ The tool supports comprehensive PostgreSQL schema objects (see `ir/ir.go` for co ## Environment Variables -- **Database connection**: `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, `PGDATABASE` +- **Target database connection**: `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, `PGDATABASE` +- **Plan database connection** (optional - for external database instead of embedded postgres): + - `PGSCHEMA_PLAN_HOST` - If set, uses external database for plan generation + - `PGSCHEMA_PLAN_PORT` - Plan database port (default: 5432) + - `PGSCHEMA_PLAN_DB` - Plan database name (required if PGSCHEMA_PLAN_HOST is set) + - `PGSCHEMA_PLAN_USER` - Plan database user (required if PGSCHEMA_PLAN_HOST is set) + - `PGSCHEMA_PLAN_PASSWORD` - Plan database password - **Environment files**: `.env` - automatically loaded by main.go - **Test filtering**: `PGSCHEMA_TEST_FILTER` - run specific test cases (e.g., `"create_table/"` or `"create_table/add_column"`) - **Postgres version**: `PGSCHEMA_POSTGRES_VERSION` - test against specific versions (14, 15, 16, 17) diff --git a/docs/cli/plan-db.mdx b/docs/cli/plan-db.mdx new file mode 100644 index 00000000..3ba03044 --- /dev/null +++ b/docs/cli/plan-db.mdx @@ -0,0 +1,232 @@ +--- +title: "External Plan Database" +--- + +The `plan` command can use an external PostgreSQL database instead of the default embedded PostgreSQL instance for validating desired state schemas. This is useful in environments where embedded PostgreSQL has limitations. + +## Overview + +By default, the `plan` command spins up a temporary embedded PostgreSQL instance to apply and validate your desired state SQL. However, you can optionally provide your own PostgreSQL database using the `--plan-*` flags or `PGSCHEMA_PLAN_*` environment variables. + +### When to Use External Database + +Use an external database for plan generation when: + +- Your schema uses **PostgreSQL extensions** (like `hstore`, `postgis`, `uuid-ossp`, etc.) - The embedded database doesn't have extensions pre-installed, causing plan generation to fail with "type does not exist" errors ([#121](https://github.com/pgschema/pgschema/issues/121)) +- Your schema has **cross-schema foreign key references** - The embedded approach only loads one schema at a time, breaking foreign key constraints that reference tables in other schemas ([#122](https://github.com/pgschema/pgschema/issues/122)) + +### How It Works + +When using an external database: + +1. **Temporary Schema Creation**: pgschema creates a temporary schema with a unique timestamp (e.g., `pgschema_plan_20251030_154501_123456789`) +2. **SQL Application**: Your desired state SQL is applied to the temporary schema +3. **Schema Inspection**: The temporary schema is inspected to extract the desired state +4. **Comparison**: The desired state is compared with your target database's current state +5. **Cleanup**: The temporary schema is dropped (best effort) after plan generation + +## Basic Usage + +```bash +# Use external database for plan generation +pgschema plan \ + --file schema.sql \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres + +# With all options specified +pgschema plan \ + --file schema.sql \ + --host localhost --port 5432 --db myapp --user postgres \ + --plan-host localhost --plan-port 5432 --plan-db pgschema_plan --plan-user postgres --plan-password secret +``` + +## Common Use Cases + +### Using PostgreSQL Extensions + +If your schema uses extensions like `hstore`, `postgis`, or `uuid-ossp`, you need to install them in the plan database first: + +```sql +-- In your plan database, install required extensions +CREATE EXTENSION IF NOT EXISTS hstore; +CREATE EXTENSION IF NOT EXISTS postgis; +CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; +``` + +Then run plan with the external database: + +```bash +# Install extensions in plan database +psql -h localhost -U postgres -d pgschema_plan -c "CREATE EXTENSION IF NOT EXISTS hstore;" + +# Now run plan - it will work because extensions are available +pgschema plan \ + --file schema.sql \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres +``` + +Your `schema.sql` can now use extension types: + +```sql +CREATE TABLE products ( + id SERIAL PRIMARY KEY, + attributes HSTORE, -- Works because hstore extension is installed + location GEOGRAPHY(POINT, 4326) -- Works because postgis is installed +); +``` + +### Handling Cross-Schema Foreign Keys + +If your schema has foreign keys that reference tables in other schemas, you need to create those schemas in the plan database: + +```sql +-- In your plan database, create referenced schemas and tables +CREATE SCHEMA IF NOT EXISTS auth; +CREATE TABLE IF NOT EXISTS auth.users ( + id SERIAL PRIMARY KEY, + email TEXT NOT NULL +); + +CREATE SCHEMA IF NOT EXISTS billing; +CREATE TABLE IF NOT EXISTS billing.customers ( + id SERIAL PRIMARY KEY, + user_id INTEGER REFERENCES auth.users(id) +); +``` + +Then run plan: + +```bash +# Set up referenced schemas in plan database +psql -h localhost -U postgres -d pgschema_plan << 'EOF' +CREATE SCHEMA IF NOT EXISTS auth; +CREATE TABLE IF NOT EXISTS auth.users (id SERIAL PRIMARY KEY, email TEXT NOT NULL); +EOF + +# Now run plan for your main schema that references auth.users +pgschema plan \ + --file schema.sql \ + --schema public \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres +``` + +Your `schema.sql` can now reference tables in other schemas: + +```sql +CREATE TABLE orders ( + id SERIAL PRIMARY KEY, + user_id INTEGER REFERENCES auth.users(id), -- Cross-schema FK works + total DECIMAL(10,2) +); +``` + +## Configuration Options + +### Using Command-Line Flags + + + Plan database server host. If provided, uses external database instead of embedded PostgreSQL. + + Environment variable: `PGSCHEMA_PLAN_HOST` + + + + Plan database server port. + + Environment variable: `PGSCHEMA_PLAN_PORT` + + + + Plan database name. Required when `--plan-host` is provided. + + Environment variable: `PGSCHEMA_PLAN_DB` + + + + Plan database user name. Required when `--plan-host` is provided. + + Environment variable: `PGSCHEMA_PLAN_USER` + + + + Plan database password. Can also be provided via `PGSCHEMA_PLAN_PASSWORD` environment variable. + + Environment variable: `PGSCHEMA_PLAN_PASSWORD` + + +### Using Environment Variables + + +```bash .env File (Recommended) +# Target database connection +PGHOST=localhost +PGPORT=5432 +PGDATABASE=myapp +PGUSER=postgres +PGPASSWORD=mypassword + +# Plan database connection (optional) +PGSCHEMA_PLAN_HOST=localhost +PGSCHEMA_PLAN_PORT=5432 +PGSCHEMA_PLAN_DB=pgschema_plan +PGSCHEMA_PLAN_USER=postgres +PGSCHEMA_PLAN_PASSWORD=planpassword + +# Run plan with external database +pgschema plan --file schema.sql +``` + +```bash Environment Variables +# Set environment variables +export PGHOST=localhost +export PGDATABASE=myapp +export PGUSER=postgres +export PGPASSWORD=mypassword + +export PGSCHEMA_PLAN_HOST=localhost +export PGSCHEMA_PLAN_DB=pgschema_plan +export PGSCHEMA_PLAN_USER=postgres +export PGSCHEMA_PLAN_PASSWORD=planpassword + +# Run plan +pgschema plan --file schema.sql +``` + +```bash Command Line Only +# All options as flags (no environment variables) +pgschema plan \ + --file schema.sql \ + --host localhost \ + --db myapp \ + --user postgres \ + --password mypassword \ + --plan-host localhost \ + --plan-db pgschema_plan \ + --plan-user postgres \ + --plan-password planpassword +``` + + +## Database Permissions + +The plan database user needs the following permissions: + +```sql +-- Minimum required permissions +GRANT CREATE ON DATABASE pgschema_plan TO your_plan_user; +GRANT USAGE ON SCHEMA public TO your_plan_user; +``` + +The user must be able to: +- Create and drop schemas +- Create tables, indexes, functions, and other schema objects +- Set search_path + +## See Also + +- [Plan Command](/cli/plan) - Main plan command documentation +- [Apply Command](/cli/apply) - Applying migration plans +- [Environment Variables](/cli/dotenv) - Managing environment configuration diff --git a/docs/cli/plan.mdx b/docs/cli/plan.mdx index 2afd9c7b..5db20f15 100644 --- a/docs/cli/plan.mdx +++ b/docs/cli/plan.mdx @@ -8,11 +8,14 @@ The `plan` command generates a migration plan to apply a desired schema state to The plan command follows infrastructure-as-code principles similar to Terraform: 1. Read the desired state from a SQL file (with include directive support) +1. Apply the desired state SQL to a temporary PostgreSQL instance (embedded by default, or external via `--plan-*` flags) 1. Connect to the target database and analyze current state of the specified schema 1. Compare the two states 1. Generate a detailed migration plan with proper dependency ordering 1. Display the plan without making any changes +By default, pgschema uses an embedded PostgreSQL instance to validate your desired state SQL. For schemas using PostgreSQL extensions or cross-schema references, you can use an external database instead. See [External Plan Database](/cli/plan-db) for details. + ## Basic Usage ```bash @@ -117,6 +120,10 @@ pgschema plan --host localhost --db myapp --user postgres --password mypassword Schema name to target for comparison +## Plan Database Options + +By default, the plan command uses an embedded PostgreSQL instance to validate your desired state SQL. For schemas that require PostgreSQL extensions or have cross-schema references, you can provide an external database. See [External Plan Database](/cli/plan-db) for complete documentation. + ## Plan Options diff --git a/docs/docs.json b/docs/docs.json index 340d2b5b..76ffb56e 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -45,7 +45,7 @@ }, { "group": "Configuration", - "pages": ["cli/ignore", "cli/dotenv"] + "pages": ["cli/plan-db", "cli/ignore", "cli/dotenv"] } ] },