diff --git a/CLAUDE.md b/CLAUDE.md index 28756ec3..af03fcc3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,12 +12,14 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co - **Apply**: Execute the migration with safety features like concurrent change detection, transaction-adaptive execution, and lock timeout control The tool is written in Go 1.24+ (toolchain go1.24.7) and uses: + - Cobra for CLI commands - embedded-postgres v1.29.0 for plan command (temporary instances) and testing (no Docker required) - pgx/v5 v5.7.5 for database connections - Supports PostgreSQL versions 14-17 Key differentiators: + - Comprehensive Postgres support for virtually all schema-level objects - State-based Terraform-like workflow (no migration history table) - Schema-level focus for single-schema apps to multi-tenant architectures @@ -66,6 +68,7 @@ go test -v ./... For interactive database validation, see the **Validate with Database** skill (`.claude/skills/validate_db/SKILL.md`). Connection details are in `.env`: + ``` PGHOST=localhost PGDATABASE=employee @@ -78,6 +81,7 @@ PGPASSWORD=testpwd1 ### Core Components **CLI Commands** (`cmd/`): + - `dump/` - Schema extraction from live database - `plan/` - Migration planning by comparing schemas - `apply/` - Migration execution with safety checks @@ -85,6 +89,7 @@ PGPASSWORD=testpwd1 - `root.go` - Main CLI setup with Cobra **Core Packages**: + - `ir/` - Intermediate Representation (IR) package - Schema objects (tables, indexes, functions, procedures, triggers, policies, etc.) - Database inspector using pgx (queries pg_catalog for schema extraction) @@ -93,6 +98,7 @@ PGPASSWORD=testpwd1 - Note: Parser removed in favor of embedded-postgres approach **Internal Packages** (`internal/`): + - `diff/` - Schema comparison and migration DDL generation - `plan/` - Migration plan structures and execution - `dump/` - Schema dump formatting and output @@ -119,7 +125,8 @@ PGPASSWORD=testpwd1 **Inspector-Only Approach**: Both desired state (from user SQL files) and current state (from target database) are obtained through database inspection. The plan command spins up an embedded PostgreSQL instance, applies user SQL files, then inspects it to get the desired state IR. This eliminates the need for SQL parsing and ensures consistency. **External Database for Plan Generation**: As an alternative to embedded postgres, users can provide an external PostgreSQL database using `--plan-host` flags or `PGSCHEMA_PLAN_*` environment variables. The external database approach: -- Creates temporary schemas with timestamp suffixes (e.g., `pgschema_plan_20251030_154501_123456789`) + +- Creates temporary schemas with timestamp suffixes (e.g., `pgschema_tmp_20251030_154501_123456789`) - Validates major version compatibility with target database (exact match required) - Cleans up temporary schemas after use (best effort) - Useful for environments where embedded postgres has limitations (ARM architectures, containerized environments) @@ -182,30 +189,36 @@ The tool supports comprehensive PostgreSQL schema objects (see `ir/ir.go` for co ## Important Implementation Notes **Trigger Features**: + - Full support for WHEN conditions using `pg_get_expr(t.tgqual, t.tgrelid, false)` from `pg_catalog.pg_trigger` - Constraint triggers with deferrable options - REFERENCING OLD TABLE / NEW TABLE for statement-level triggers **Online Migration Support**: + - CREATE INDEX CONCURRENTLY for non-blocking index creation - ALTER TABLE ... ADD CONSTRAINT ... NOT VALID for online constraint addition - Proper transaction handling - some operations must run outside transactions **pgschema Directives**: + - Special SQL comments control behavior: `--pgschema-lock-timeout`, `--pgschema-no-transaction` - Handled in `cmd/apply/directive.go` **Reference Implementations**: + - PostgreSQL's pg_dump serves as reference for system catalog queries (see **pg_dump Reference** skill) - PostgreSQL's gram.y defines canonical SQL syntax (see **PostgreSQL Syntax Reference** skill) ## Key Files Reference **Entry Point & CLI**: + - `main.go` - Entry point, loads .env and calls cmd.Execute() - `cmd/root.go` - Root CLI with global flags **IR Package** (`./ir`): + - `ir/ir.go` - Core IR data structures for all schema objects - `ir/inspector.go` - Database introspection using pgx (queries pg_catalog) - `ir/normalize.go` - Schema normalization (version-specific differences, type mappings) @@ -213,10 +226,12 @@ The tool supports comprehensive PostgreSQL schema objects (see `ir/ir.go` for co - Note: `ir/parser.go` removed - now using embedded-postgres for desired state **Diff Package** (`internal/diff/`): + - `diff.go` - Main diff logic, topological sorting - `table.go`, `index.go`, `trigger.go`, `view.go`, `function.go`, `procedure.go`, `sequence.go`, `type.go`, `policy.go`, `aggregate.go` - Object-specific diff operations **Testing**: + - `cmd/migrate_integration_test.go` - Main integration test suite (TestPlanAndApply) - `testdata/diff/` - 100+ test cases covering all schema object types - See **Run Tests** skill for complete testing workflows @@ -224,6 +239,7 @@ The tool supports comprehensive PostgreSQL schema objects (see `ir/ir.go` for co ## Test Data Structure Tests are organized in `testdata/diff/` by object type: + - `comment/` (8 tests), `create_domain/` (3), `create_function/` (4), `create_index/` (1) - `create_materialized_view/` (3), `create_policy/` (8), `create_procedure/` (3), `create_sequence/` (3) - `create_table/` (40 tests), `create_trigger/` (7), `create_type/` (3), `create_view/` (6) diff --git a/cmd/apply/apply.go b/cmd/apply/apply.go index b23eaaf4..1658d40f 100644 --- a/cmd/apply/apply.go +++ b/cmd/apply/apply.go @@ -31,6 +31,13 @@ var ( applyNoColor bool applyLockTimeout string applyApplicationName string + + // Plan database connection flags (optional - for using external database instead of embedded postgres) + applyPlanDBHost string + applyPlanDBPort int + applyPlanDBDatabase string + applyPlanDBUser string + applyPlanDBPassword string ) var ApplyCmd = &cobra.Command{ @@ -63,6 +70,13 @@ func init() { ApplyCmd.Flags().StringVar(&applyLockTimeout, "lock-timeout", "", "Maximum time to wait for database locks (e.g., 30s, 5m, 1h)") ApplyCmd.Flags().StringVar(&applyApplicationName, "application-name", "pgschema", "Application name for database connection (visible in pg_stat_activity) (env: PGAPPNAME)") + // Plan database connection flags (optional - for using external database instead of embedded postgres when using --file) + ApplyCmd.Flags().StringVar(&applyPlanDBHost, "plan-host", "", "Plan database host (env: PGSCHEMA_PLAN_HOST). If provided, uses external database instead of embedded postgres for validating desired state schema") + ApplyCmd.Flags().IntVar(&applyPlanDBPort, "plan-port", 5432, "Plan database port (env: PGSCHEMA_PLAN_PORT)") + ApplyCmd.Flags().StringVar(&applyPlanDBDatabase, "plan-db", "", "Plan database name (env: PGSCHEMA_PLAN_DB)") + ApplyCmd.Flags().StringVar(&applyPlanDBUser, "plan-user", "", "Plan database user (env: PGSCHEMA_PLAN_USER)") + ApplyCmd.Flags().StringVar(&applyPlanDBPassword, "plan-password", "", "Plan database password (env: PGSCHEMA_PLAN_PASSWORD)") + // Mark file and plan as mutually exclusive ApplyCmd.MarkFlagsMutuallyExclusive("file", "plan") } @@ -286,6 +300,22 @@ func RunApply(cmd *cobra.Command, args []string) error { // Using --file flag, will need desired state provider config.File = applyFile + // Apply environment variables to plan database flags (only needed for File Mode) + util.ApplyPlanDBEnvVars(cmd, &applyPlanDBHost, &applyPlanDBDatabase, &applyPlanDBUser, &applyPlanDBPassword, &applyPlanDBPort) + + // Validate plan database flags if plan-host is provided + if err := util.ValidatePlanDBFlags(applyPlanDBHost, applyPlanDBDatabase, applyPlanDBUser); err != nil { + return err + } + + // Derive final plan database password + finalPlanPassword := applyPlanDBPassword + if finalPlanPassword == "" { + if envPassword := os.Getenv("PGSCHEMA_PLAN_PASSWORD"); envPassword != "" { + finalPlanPassword = envPassword + } + } + // Create desired state provider (embedded postgres or external database) planConfig := &planCmd.PlanConfig{ Host: applyHost, @@ -296,6 +326,12 @@ func RunApply(cmd *cobra.Command, args []string) error { Schema: applySchema, File: applyFile, ApplicationName: applyApplicationName, + // Plan database configuration + PlanDBHost: applyPlanDBHost, + PlanDBPort: applyPlanDBPort, + PlanDBDatabase: applyPlanDBDatabase, + PlanDBUser: applyPlanDBUser, + PlanDBPassword: finalPlanPassword, } provider, err = planCmd.CreateDesiredStateProvider(planConfig) if err != nil { diff --git a/cmd/apply/apply_integration_test.go b/cmd/apply/apply_integration_test.go index aebb2804..d6b6ef3b 100644 --- a/cmd/apply/apply_integration_test.go +++ b/cmd/apply/apply_integration_test.go @@ -9,9 +9,12 @@ import ( "testing" planCmd "github.com/pgschema/pgschema/cmd/plan" + "github.com/pgschema/pgschema/cmd/util" "github.com/pgschema/pgschema/internal/plan" "github.com/pgschema/pgschema/internal/postgres" "github.com/pgschema/pgschema/testutil" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" ) var ( @@ -1085,3 +1088,256 @@ func TestApplyCommand_WaitDirective(t *testing.T) { t.Fatal("Index idx_users_email_status should be valid after wait directive completion") } } + +// TestApplyCommand_WithExternalPlanDatabase tests that apply command works with external plan database +func TestApplyCommand_WithExternalPlanDatabase(t *testing.T) { + if testing.Short() { + t.Skip("Skipping integration test in short mode") + } + + // Setup: Create two embedded postgres instances + // One is the target database, one is the external plan database + targetDB := testutil.SetupPostgres(t) + defer targetDB.Stop() + + externalPlanDB := testutil.SetupPostgres(t) + defer externalPlanDB.Stop() + + // Get connection details + targetConn, targetHost, targetPort, targetDatabase, targetUser, targetPassword := testutil.ConnectToPostgres(t, targetDB) + defer targetConn.Close() + + planHost, planPort, planDatabase, planUser, planPassword := externalPlanDB.GetConnectionDetails() + + // Create test schema file + schemaSQL := ` +CREATE TABLE departments ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_departments_name ON departments(name); + +CREATE TABLE employees ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL, + department_id INTEGER REFERENCES departments(id), + hired_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +` + + tmpDir := t.TempDir() + schemaFile := filepath.Join(tmpDir, "schema.sql") + err := os.WriteFile(schemaFile, []byte(schemaSQL), 0644) + if err != nil { + t.Fatalf("Failed to write schema file: %v", err) + } + + // Create plan config with external database + planConfig := &planCmd.PlanConfig{ + Host: targetHost, + Port: targetPort, + DB: targetDatabase, + User: targetUser, + Password: targetPassword, + Schema: "public", + File: schemaFile, + ApplicationName: "pgschema-test", + // External database configuration + PlanDBHost: planHost, + PlanDBPort: planPort, + PlanDBDatabase: planDatabase, + PlanDBUser: planUser, + PlanDBPassword: planPassword, + } + + // Create external database provider + provider, err := planCmd.CreateDesiredStateProvider(planConfig) + if err != nil { + t.Fatalf("Failed to create external database provider: %v", err) + } + defer provider.Stop() + + // Verify it's using external database (not embedded) + _, ok := provider.(*postgres.ExternalDatabase) + if !ok { + t.Fatal("Provider should be ExternalDatabase when plan-host is provided") + } + + // Create apply config + applyConfig := &ApplyConfig{ + Host: targetHost, + Port: targetPort, + DB: targetDatabase, + User: targetUser, + Password: targetPassword, + Schema: "public", + File: schemaFile, + AutoApprove: true, // Auto-approve for testing + NoColor: true, + ApplicationName: "pgschema-test", + } + + // Apply migration using external database provider + err = ApplyMigration(applyConfig, provider) + if err != nil { + t.Fatalf("Failed to apply migration: %v", err) + } + + // Verify changes were applied to target database + // Check that departments table exists + var tableName string + err = targetConn.QueryRow("SELECT tablename FROM pg_tables WHERE schemaname = 'public' AND tablename = 'departments'").Scan(&tableName) + if err != nil { + t.Fatalf("Failed to query departments table: %v", err) + } + if tableName != "departments" { + t.Errorf("Expected table 'departments', got '%s'", tableName) + } + + // Check that index exists + var indexName string + err = targetConn.QueryRow("SELECT indexname FROM pg_indexes WHERE schemaname = 'public' AND indexname = 'idx_departments_name'").Scan(&indexName) + if err != nil { + t.Fatalf("Failed to query index: %v", err) + } + if indexName != "idx_departments_name" { + t.Errorf("Expected index 'idx_departments_name', got '%s'", indexName) + } + + // Check that employees table exists with foreign key + var employeeTableName string + err = targetConn.QueryRow("SELECT tablename FROM pg_tables WHERE schemaname = 'public' AND tablename = 'employees'").Scan(&employeeTableName) + if err != nil { + t.Fatalf("Failed to query employees table: %v", err) + } + if employeeTableName != "employees" { + t.Errorf("Expected table 'employees', got '%s'", employeeTableName) + } + + // Verify foreign key constraint exists + var constraintName string + err = targetConn.QueryRow(` + SELECT conname + FROM pg_constraint + WHERE conrelid = 'public.employees'::regclass + AND contype = 'f' + `).Scan(&constraintName) + if err != nil { + t.Fatalf("Failed to query foreign key constraint: %v", err) + } + if constraintName == "" { + t.Error("Expected foreign key constraint on employees table") + } +} + +// TestApplyCommand_PlanMode_IgnoresPlanDBFlags tests that Plan Mode doesn't validate plan database flags +func TestApplyCommand_PlanMode_IgnoresPlanDBFlags(t *testing.T) { + // Skip in short mode + if testing.Short() { + t.Skip("skipping integration test") + } + + // Setup target database + targetDB := testutil.SetupPostgres(t) + defer targetDB.Stop() + + targetHost, targetPort, targetDatabase, targetUser, targetPassword := targetDB.GetConnectionDetails() + + // Create a simple schema file + schemaSQL := ` +CREATE TABLE users ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL +); +` + + tmpDir := t.TempDir() + schemaFile := filepath.Join(tmpDir, "schema.sql") + err := os.WriteFile(schemaFile, []byte(schemaSQL), 0644) + require.NoError(t, err) + + // First, generate a plan using file mode (without external plan database) + planConfig := &planCmd.PlanConfig{ + Host: targetHost, + Port: targetPort, + DB: targetDatabase, + User: targetUser, + Password: targetPassword, + Schema: "public", + File: schemaFile, + ApplicationName: "pgschema-test", + } + + provider, err := planCmd.CreateDesiredStateProvider(planConfig) + require.NoError(t, err, "should create provider") + defer provider.Stop() + + generatedPlan, err := planCmd.GeneratePlan(planConfig, provider) + require.NoError(t, err, "should generate plan") + + // Save plan to JSON file + planJSON, err := generatedPlan.ToJSON() + require.NoError(t, err, "should serialize plan to JSON") + + planFile := filepath.Join(tmpDir, "plan.json") + err = os.WriteFile(planFile, []byte(planJSON), 0644) + require.NoError(t, err, "should write plan file") + + // Now apply using Plan Mode with INVALID plan database flags set + // This should NOT fail because Plan Mode shouldn't validate plan database flags + config := &ApplyConfig{ + Host: targetHost, + Port: targetPort, + DB: targetDatabase, + User: targetUser, + Password: targetPassword, + Schema: "public", + AutoApprove: true, + NoColor: true, + ApplicationName: "pgschema-test", + } + + // Load the plan + planData, err := os.ReadFile(planFile) + require.NoError(t, err, "should read plan file") + + migrationPlan, err := plan.FromJSON(planData) + require.NoError(t, err, "should load plan from JSON") + + config.Plan = migrationPlan + + // Set environment variables with INVALID plan database configuration + // This would normally fail validation, but should be ignored in Plan Mode + os.Setenv("PGSCHEMA_PLAN_HOST", "invalid-host-that-does-not-exist") + os.Setenv("PGSCHEMA_PLAN_DB", "") // Missing required field - would fail validation + defer func() { + os.Unsetenv("PGSCHEMA_PLAN_HOST") + os.Unsetenv("PGSCHEMA_PLAN_DB") + }() + + // Apply should succeed because Plan Mode doesn't validate plan database flags + err = ApplyMigration(config, nil) + require.NoError(t, err, "apply with plan mode should ignore invalid plan database flags") + + // Verify the table was created + connConfig := &util.ConnectionConfig{ + Host: targetHost, + Port: targetPort, + Database: targetDatabase, + User: targetUser, + Password: targetPassword, + SSLMode: "prefer", + ApplicationName: "pgschema-test", + } + + targetConn, err := util.Connect(connConfig) + require.NoError(t, err, "should connect to target database") + defer targetConn.Close() + + var tableName string + err = targetConn.QueryRow("SELECT tablename FROM pg_tables WHERE schemaname = 'public' AND tablename = 'users'").Scan(&tableName) + require.NoError(t, err, "should find users table") + assert.Equal(t, "users", tableName, "table should be created") +} diff --git a/cmd/apply/apply_test.go b/cmd/apply/apply_test.go index 42d5daf7..ea76d2c7 100644 --- a/cmd/apply/apply_test.go +++ b/cmd/apply/apply_test.go @@ -496,3 +496,48 @@ func TestApplyCommandFileError(t *testing.T) { t.Error("Expected error when file doesn't exist, but got none") } } + +func TestApplyCommand_PlanDatabaseFlags(t *testing.T) { + flags := ApplyCmd.Flags() + + // Test plan database flags + planHostFlag := flags.Lookup("plan-host") + if planHostFlag == nil { + t.Error("Expected --plan-host flag to be defined") + } + if planHostFlag.DefValue != "" { + t.Errorf("Expected default plan-host to be empty, got '%s'", planHostFlag.DefValue) + } + + planPortFlag := flags.Lookup("plan-port") + if planPortFlag == nil { + t.Error("Expected --plan-port flag to be defined") + } + if planPortFlag.DefValue != "5432" { + t.Errorf("Expected default plan-port to be '5432', got '%s'", planPortFlag.DefValue) + } + + planDBFlag := flags.Lookup("plan-db") + if planDBFlag == nil { + t.Error("Expected --plan-db flag to be defined") + } + if planDBFlag.DefValue != "" { + t.Errorf("Expected default plan-db to be empty, got '%s'", planDBFlag.DefValue) + } + + planUserFlag := flags.Lookup("plan-user") + if planUserFlag == nil { + t.Error("Expected --plan-user flag to be defined") + } + if planUserFlag.DefValue != "" { + t.Errorf("Expected default plan-user to be empty, got '%s'", planUserFlag.DefValue) + } + + planPasswordFlag := flags.Lookup("plan-password") + if planPasswordFlag == nil { + t.Error("Expected --plan-password flag to be defined") + } + if planPasswordFlag.DefValue != "" { + t.Errorf("Expected default plan-password to be empty, got '%s'", planPasswordFlag.DefValue) + } +} diff --git a/cmd/plan/external_db_integration_test.go b/cmd/plan/external_db_integration_test.go index 195899cd..e2454ffd 100644 --- a/cmd/plan/external_db_integration_test.go +++ b/cmd/plan/external_db_integration_test.go @@ -77,7 +77,7 @@ CREATE INDEX idx_users_email ON users(email); // Verify temporary schema name is returned tempSchema := provider.GetSchemaName() assert.NotEmpty(t, tempSchema, "temporary schema name should not be empty") - assert.Contains(t, tempSchema, "pgschema_plan_", "temporary schema should have timestamp prefix") + assert.Contains(t, tempSchema, "pgschema_tmp_", "temporary schema should have timestamp prefix") // Generate plan migrationPlan, err := GeneratePlan(config, provider) diff --git a/cmd/plan/plan.go b/cmd/plan/plan.go index 93abfd0b..ab6173bb 100644 --- a/cmd/plan/plan.go +++ b/cmd/plan/plan.go @@ -12,6 +12,7 @@ import ( "github.com/pgschema/pgschema/internal/include" "github.com/pgschema/pgschema/internal/plan" "github.com/pgschema/pgschema/internal/postgres" + "github.com/pgschema/pgschema/ir" "github.com/spf13/cobra" ) @@ -260,9 +261,9 @@ func GeneratePlan(config *PlanConfig, provider postgres.DesiredStateProvider) (* // Inspect the provider database to get desired state IR providerHost, providerPort, providerDB, providerUsername, providerPassword := provider.GetConnectionDetails() - // Determine which schema to inspect - // For external database: use the temporary schema name - // For embedded postgres: use the config.Schema (GetSchemaName returns empty string) + // Get the temporary schema name where desired state SQL was applied. + // Both embedded and external database providers use temporary schemas with unique timestamps + // (e.g., pgschema_tmp_20251030_154501_123456789) to ensure isolation and prevent conflicts. schemaToInspect := provider.GetSchemaName() if schemaToInspect == "" { schemaToInspect = config.Schema @@ -273,6 +274,15 @@ func GeneratePlan(config *PlanConfig, provider postgres.DesiredStateProvider) (* return nil, fmt.Errorf("failed to get desired state: %w", err) } + // Normalize schema names in the IR from temporary schema to target schema. + // At this point, the IR contains schema names like "pgschema_tmp_20251030_154501_123456789" + // because that's where objects were created. We need to replace these with the target + // schema name (e.g., "public") so that generated DDL references the correct schema. + // Without this normalization, DDL would reference non-existent temporary schemas and fail. + if schemaToInspect != config.Schema { + normalizeSchemaNames(desiredStateIR, schemaToInspect, config.Schema) + } + // Generate diff (current -> desired) using IR directly diffs := diff.GenerateMigration(currentStateIR, desiredStateIR, config.Schema) @@ -366,6 +376,118 @@ func processOutput(migrationPlan *plan.Plan, output outputSpec, cmd *cobra.Comma return nil } +// normalizeSchemaNames replaces all occurrences of fromSchema with toSchema in the IR. +// +// Context: +// During plan generation, desired state SQL is applied to a temporary schema with a unique +// timestamped name (e.g., pgschema_tmp_20251030_154501_123456789). This temporary schema +// ensures isolation and prevents conflicts when running concurrent plan operations or when +// using an external database for plan validation. +// +// When the database is inspected after applying the SQL, the IR will contain schema names +// matching the temporary schema. However, the generated DDL needs to reference the target +// schema (e.g., "public") where the migration will actually be applied. +// +// This function performs a comprehensive schema name replacement across all IR objects: +// - Tables, views, functions, procedures, types, sequences, aggregates +// - Constraints (including foreign key referenced schemas) +// - Indexes, triggers, policies +// - Dependencies and cross-references +// +// Without this normalization, generated DDL would reference non-existent temporary schemas +// and fail when applied to the target database. +func normalizeSchemaNames(irData *ir.IR, fromSchema, toSchema string) { + // Normalize schema names in Schemas map + if schema, exists := irData.Schemas[fromSchema]; exists { + delete(irData.Schemas, fromSchema) + schema.Name = toSchema + irData.Schemas[toSchema] = schema + + // Normalize schema names in all objects within this schema + // Tables + for _, table := range schema.Tables { + table.Schema = toSchema + + // Normalize constraint schemas + for _, constraint := range table.Constraints { + // Normalize the constraint's own schema field + if constraint.Schema == fromSchema { + constraint.Schema = toSchema + } + // Normalize referenced schema in foreign key constraints + if constraint.ReferencedSchema == fromSchema { + constraint.ReferencedSchema = toSchema + } + } + + // Normalize schema references in table dependencies + for i := range table.Dependencies { + if table.Dependencies[i].Schema == fromSchema { + table.Dependencies[i].Schema = toSchema + } + } + + // Normalize schema names in indexes + for _, index := range table.Indexes { + if index.Schema == fromSchema { + index.Schema = toSchema + } + } + + // Normalize schema names in triggers + for _, trigger := range table.Triggers { + if trigger.Schema == fromSchema { + trigger.Schema = toSchema + } + } + + // Normalize schema names in RLS policies + for _, policy := range table.Policies { + if policy.Schema == fromSchema { + policy.Schema = toSchema + } + } + } + + // Views + for _, view := range schema.Views { + view.Schema = toSchema + + // Normalize schema names in materialized view indexes + for _, index := range view.Indexes { + if index.Schema == fromSchema { + index.Schema = toSchema + } + } + } + + // Functions + for _, fn := range schema.Functions { + fn.Schema = toSchema + } + + // Procedures + for _, proc := range schema.Procedures { + proc.Schema = toSchema + } + + // Types + for _, typ := range schema.Types { + typ.Schema = toSchema + } + + // Sequences + for _, seq := range schema.Sequences { + seq.Schema = toSchema + } + + // Aggregates + for _, agg := range schema.Aggregates { + agg.Schema = toSchema + } + } +} + // ResetFlags resets all global flag variables to their default values for testing func ResetFlags() { planHost = "localhost" diff --git a/docs/cli/apply.mdx b/docs/cli/apply.mdx index dd4c2d9a..21ff17d1 100644 --- a/docs/cli/apply.mdx +++ b/docs/cli/apply.mdx @@ -10,11 +10,14 @@ The apply command supports two execution modes: ### File Mode (Generate and Apply) 1. Read the desired state from a SQL file (with include directive support) +1. Apply the desired state SQL to a temporary PostgreSQL instance (embedded by default, or external via `--plan-*` flags) 1. Compare it with the current database state of the target schema 1. Generate a migration plan with proper dependency ordering 1. Display the plan for review 1. Apply the changes (with optional confirmation and safety checks) +By default, File Mode uses an embedded PostgreSQL instance to validate the desired state. For schemas using PostgreSQL extensions or cross-schema references, you can use an external database instead via the `--plan-*` flags. See [External Plan Database](/cli/plan-db) for details. + ### Plan Mode (Execute Pre-generated Plan) 1. Load a pre-generated plan from JSON file 1. Validate plan version compatibility and schema fingerprints @@ -115,6 +118,12 @@ pgschema apply --host localhost --db myapp --user postgres --password mypassword Schema name to apply changes to +## Plan Database Options + +When using File Mode (`--file`), the apply command generates a plan internally using a temporary PostgreSQL instance. By default, this uses embedded PostgreSQL. For schemas that require PostgreSQL extensions or have cross-schema references, you can provide an external database. See [External Plan Database](/cli/plan-db) for complete documentation. + +**Note**: These options only apply when using `--file` mode. When using `--plan` mode, the plan has already been generated. + ## Apply Options diff --git a/docs/cli/plan-db.mdx b/docs/cli/plan-db.mdx index 3ba03044..817e5c38 100644 --- a/docs/cli/plan-db.mdx +++ b/docs/cli/plan-db.mdx @@ -2,11 +2,13 @@ title: "External Plan Database" --- -The `plan` command can use an external PostgreSQL database instead of the default embedded PostgreSQL instance for validating desired state schemas. This is useful in environments where embedded PostgreSQL has limitations. +The `plan` and `apply` commands can use an external PostgreSQL database instead of the default embedded PostgreSQL instance for validating desired state schemas. This is useful in environments where embedded PostgreSQL has limitations. ## Overview -By default, the `plan` command spins up a temporary embedded PostgreSQL instance to apply and validate your desired state SQL. However, you can optionally provide your own PostgreSQL database using the `--plan-*` flags or `PGSCHEMA_PLAN_*` environment variables. +By default, the `plan` command (and `apply` command in File Mode) spins up a temporary embedded PostgreSQL instance to apply and validate your desired state SQL. However, you can optionally provide your own PostgreSQL database using the `--plan-*` flags or `PGSCHEMA_PLAN_*` environment variables. + +**Note**: For the `apply` command, these options only apply when using File Mode (`--file`). When using Plan Mode (`--plan`), the plan has already been generated, so plan database options are not applicable. ### When to Use External Database @@ -19,7 +21,7 @@ Use an external database for plan generation when: When using an external database: -1. **Temporary Schema Creation**: pgschema creates a temporary schema with a unique timestamp (e.g., `pgschema_plan_20251030_154501_123456789`) +1. **Temporary Schema Creation**: pgschema creates a temporary schema with a unique timestamp (e.g., `pgschema_tmp_20251030_154501_123456789`) 2. **SQL Application**: Your desired state SQL is applied to the temporary schema 3. **Schema Inspection**: The temporary schema is inspected to extract the desired state 4. **Comparison**: The desired state is compared with your target database's current state @@ -27,6 +29,8 @@ When using an external database: ## Basic Usage +### With Plan Command + ```bash # Use external database for plan generation pgschema plan \ @@ -41,6 +45,20 @@ pgschema plan \ --plan-host localhost --plan-port 5432 --plan-db pgschema_plan --plan-user postgres --plan-password secret ``` +### With Apply Command (File Mode) + +```bash +# Use external database when applying from file +pgschema apply \ + --file schema.sql \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres \ + --auto-approve + +# The apply command generates a plan internally using the external database, +# then applies the changes to the target database +``` + ## Common Use Cases ### Using PostgreSQL Extensions @@ -54,7 +72,7 @@ CREATE EXTENSION IF NOT EXISTS postgis; CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; ``` -Then run plan with the external database: +Then run plan or apply with the external database: ```bash # Install extensions in plan database @@ -65,6 +83,13 @@ pgschema plan \ --file schema.sql \ --host localhost --db myapp --user postgres \ --plan-host localhost --plan-db pgschema_plan --plan-user postgres + +# Or use apply command directly (File Mode) +pgschema apply \ + --file schema.sql \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres \ + --auto-approve ``` Your `schema.sql` can now use extension types: @@ -96,7 +121,7 @@ CREATE TABLE IF NOT EXISTS billing.customers ( ); ``` -Then run plan: +Then run plan or apply: ```bash # Set up referenced schemas in plan database @@ -111,6 +136,14 @@ pgschema plan \ --schema public \ --host localhost --db myapp --user postgres \ --plan-host localhost --plan-db pgschema_plan --plan-user postgres + +# Or use apply command to plan and apply in one step (File Mode) +pgschema apply \ + --file schema.sql \ + --schema public \ + --host localhost --db myapp --user postgres \ + --plan-host localhost --plan-db pgschema_plan --plan-user postgres \ + --auto-approve ``` Your `schema.sql` can now reference tables in other schemas: @@ -177,6 +210,9 @@ PGSCHEMA_PLAN_PASSWORD=planpassword # Run plan with external database pgschema plan --file schema.sql + +# Or apply (File Mode) with external database +pgschema apply --file schema.sql --auto-approve ``` ```bash Environment Variables @@ -193,10 +229,13 @@ export PGSCHEMA_PLAN_PASSWORD=planpassword # Run plan pgschema plan --file schema.sql + +# Or apply (File Mode) +pgschema apply --file schema.sql --auto-approve ``` ```bash Command Line Only -# All options as flags (no environment variables) +# Plan command - all options as flags (no environment variables) pgschema plan \ --file schema.sql \ --host localhost \ @@ -207,6 +246,19 @@ pgschema plan \ --plan-db pgschema_plan \ --plan-user postgres \ --plan-password planpassword + +# Apply command (File Mode) - all options as flags +pgschema apply \ + --file schema.sql \ + --host localhost \ + --db myapp \ + --user postgres \ + --password mypassword \ + --plan-host localhost \ + --plan-db pgschema_plan \ + --plan-user postgres \ + --plan-password planpassword \ + --auto-approve ``` diff --git a/internal/postgres/desired_state.go b/internal/postgres/desired_state.go index 5e5dc968..414915b6 100644 --- a/internal/postgres/desired_state.go +++ b/internal/postgres/desired_state.go @@ -2,7 +2,14 @@ // This file defines the interface for desired state providers (embedded or external databases). package postgres -import "context" +import ( + "context" + "crypto/rand" + "encoding/hex" + "fmt" + "regexp" + "time" +) // DesiredStateProvider is an interface that abstracts the desired state database provider. // It can be implemented by either embedded PostgreSQL or an external database connection. @@ -12,8 +19,8 @@ type DesiredStateProvider interface { GetConnectionDetails() (string, int, string, string, string) // GetSchemaName returns the actual schema name to inspect. - // For embedded postgres: returns the user-provided schema name - // For external database: returns the temporary schema name with timestamp + // For embedded postgres: returns the temporary schema name (pgschema_tmp_*) + // For external database: returns the temporary schema name (pgschema_tmp_*) GetSchemaName() string // ApplySchema applies the desired state SQL to a schema. @@ -26,3 +33,85 @@ type DesiredStateProvider interface { // For external database: drops temporary schema (best effort) and closes connection Stop() error } + +// GenerateTempSchemaName creates a unique temporary schema name for plan operations. +// The format is: pgschema_tmp_YYYYMMDD_HHMMSS_RRRRRRRR +// where RRRRRRRR is a random 8-character hex string for uniqueness. +// The "_tmp_" marker makes it distinctive and prevents accidental matching with user schemas. +// +// Example: pgschema_tmp_20251030_154501_a3f9d2e1 +// +// Panics if random number generation fails (indicates serious system issue). +func GenerateTempSchemaName() string { + timestamp := time.Now().Format("20060102_150405") + + // Add random suffix for uniqueness (4 bytes = 8 hex characters) + randomBytes := make([]byte, 4) + if _, err := rand.Read(randomBytes); err != nil { + // If crypto/rand fails, something is seriously wrong with the system + panic(fmt.Sprintf("failed to generate random schema name: %v", err)) + } + randomSuffix := hex.EncodeToString(randomBytes) + + return fmt.Sprintf("pgschema_tmp_%s_%s", timestamp, randomSuffix) +} + +// stripSchemaQualifications removes schema qualifications from SQL statements for the specified target schema. +// +// Purpose: +// When applying user-provided SQL to temporary schemas during the plan command, we need to ensure +// that objects are created in the temporary schema (e.g., pgschema_tmp_20251030_154501_123456789) +// rather than in explicitly qualified schemas. PostgreSQL's search_path only affects unqualified +// object names - explicit schema qualifications always override search_path. +// +// Input SQL Sources: +// - pgschema dump command produces schema-agnostic output (no schema qualifications for target schema) +// - Users may manually edit SQL files and add schema qualifications (e.g., public.table) +// - Users may provide SQL from other sources that contains schema qualifications +// +// Behavior: +// This function strips schema qualifications ONLY for the target schema (specified by schemaName), +// while preserving qualifications for other schemas. This allows: +// 1. Target schema objects to be created in temporary schemas via search_path +// 2. Cross-schema references to be preserved correctly +// +// Examples: +// When target schema is "public": +// - public.employees -> employees (stripped - will use search_path) +// - "public".employees -> employees (stripped - handles quoted identifiers) +// - public."employees" -> "employees" (stripped - preserves quoted object names) +// - other_schema.users -> other_schema.users (preserved - cross-schema reference) +// +// It handles both quoted and unquoted schema names: +// - public.table -> table +// - "public".table -> table +// - public."table" -> "table" +// - "public"."table" -> "table" +// +// Only qualifications matching the specified schemaName are stripped. +// All other schema qualifications are preserved as intentional cross-schema references. +func stripSchemaQualifications(sql string, schemaName string) string { + if schemaName == "" { + return sql + } + + // Escape the schema name for use in regex + escapedSchema := regexp.QuoteMeta(schemaName) + + // Pattern matches: optional quote + schemaName + optional quote + dot + captured object name + // This handles all four combinations of quoted/unquoted schema and object names + + // Pattern for unquoted identifier after dot + pattern1 := fmt.Sprintf(`"?%s"?\.([a-zA-Z_][a-zA-Z0-9_$]*)`, escapedSchema) + re1 := regexp.MustCompile(pattern1) + + // Pattern for quoted identifier after dot + pattern2 := fmt.Sprintf(`"?%s"?\.(\"[^"]+\")`, escapedSchema) + re2 := regexp.MustCompile(pattern2) + + result := sql + result = re1.ReplaceAllString(result, "$1") + result = re2.ReplaceAllString(result, "$1") + + return result +} diff --git a/internal/postgres/embedded.go b/internal/postgres/embedded.go index 78f46f55..d5ceee6e 100644 --- a/internal/postgres/embedded.go +++ b/internal/postgres/embedded.go @@ -11,7 +11,6 @@ import ( "net" "os" "path/filepath" - "time" embeddedpostgres "github.com/fergusstrange/embedded-postgres" _ "github.com/jackc/pgx/v5/stdlib" @@ -32,6 +31,7 @@ type EmbeddedPostgres struct { username string password string runtimePath string + tempSchema string // temporary schema name with timestamp for uniqueness } // EmbeddedPostgresConfig holds configuration for starting embedded PostgreSQL @@ -68,9 +68,9 @@ func DetectPostgresVersionFromDB(host string, port int, database, user, password // StartEmbeddedPostgres starts a temporary embedded PostgreSQL instance func StartEmbeddedPostgres(config *EmbeddedPostgresConfig) (*EmbeddedPostgres, error) { - // Create unique runtime path with timestamp (using nanoseconds for uniqueness) - timestamp := time.Now().Format("20060102_150405.000000000") - runtimePath := filepath.Join(os.TempDir(), fmt.Sprintf("pgschema-plan-%s", timestamp)) + // Create unique runtime path and schema name + tempSchema := GenerateTempSchemaName() + runtimePath := filepath.Join(os.TempDir(), tempSchema) // Find an available port port, err := findAvailablePort() @@ -134,11 +134,20 @@ func StartEmbeddedPostgres(config *EmbeddedPostgresConfig) (*EmbeddedPostgres, e username: config.Username, password: config.Password, runtimePath: runtimePath, + tempSchema: tempSchema, }, nil } // Stop stops and cleans up the embedded PostgreSQL instance func (ep *EmbeddedPostgres) Stop() error { + // Drop the temporary schema (best effort - don't fail if this errors) + if ep.db != nil && ep.tempSchema != "" { + ctx := context.Background() + dropSchemaSQL := fmt.Sprintf("DROP SCHEMA IF EXISTS \"%s\" CASCADE", ep.tempSchema) + // Ignore errors - this is best effort cleanup + _, _ = ep.db.ExecContext(ctx, dropSchemaSQL) + } + // Close database connection if ep.db != nil { ep.db.Close() @@ -170,41 +179,44 @@ func (ep *EmbeddedPostgres) GetConnectionDetails() (host string, port int, datab return ep.host, ep.port, ep.database, ep.username, ep.password } -// GetSchemaName returns the schema name to inspect. -// For embedded postgres, this is managed externally, so we return empty string -// and rely on the caller to track the schema name. +// GetSchemaName returns the temporary schema name used for desired state validation. +// This returns the timestamped schema name that was created by ApplySchema. func (ep *EmbeddedPostgres) GetSchemaName() string { - // Embedded postgres doesn't track schema name - it's provided by the caller in ApplySchema - // The caller (GeneratePlan) needs to use config.Schema for inspection - return "" + return ep.tempSchema } // ApplySchema resets a schema (drops and recreates it) and applies SQL to it. // This ensures a clean state before applying the desired schema definition. +// Note: The schema parameter is ignored - we always use the temporary schema name. func (ep *EmbeddedPostgres) ApplySchema(ctx context.Context, schema string, sql string) error { - // Drop the schema if it exists (CASCADE to drop all objects) - dropSchemaSQL := fmt.Sprintf("DROP SCHEMA IF EXISTS \"%s\" CASCADE", schema) + // Drop the temporary schema if it exists (CASCADE to drop all objects) + dropSchemaSQL := fmt.Sprintf("DROP SCHEMA IF EXISTS \"%s\" CASCADE", ep.tempSchema) if _, err := ep.db.ExecContext(ctx, dropSchemaSQL); err != nil { - return fmt.Errorf("failed to drop schema %s: %w", schema, err) + return fmt.Errorf("failed to drop temporary schema %s: %w", ep.tempSchema, err) } - // Create the schema - createSchemaSQL := fmt.Sprintf("CREATE SCHEMA \"%s\"", schema) + // Create the temporary schema + createSchemaSQL := fmt.Sprintf("CREATE SCHEMA \"%s\"", ep.tempSchema) if _, err := ep.db.ExecContext(ctx, createSchemaSQL); err != nil { - return fmt.Errorf("failed to create schema %s: %w", schema, err) + return fmt.Errorf("failed to create temporary schema %s: %w", ep.tempSchema, err) } - // Set search_path to the target schema - setSearchPathSQL := fmt.Sprintf("SET search_path TO \"%s\"", schema) + // Set search_path to the temporary schema + setSearchPathSQL := fmt.Sprintf("SET search_path TO \"%s\"", ep.tempSchema) if _, err := ep.db.ExecContext(ctx, setSearchPathSQL); err != nil { return fmt.Errorf("failed to set search_path: %w", err) } + // Strip schema qualifications from SQL before applying to temporary schema + // This ensures that objects are created in the temporary schema via search_path + // rather than being explicitly qualified with the original schema name + schemaAgnosticSQL := stripSchemaQualifications(sql, schema) + // Execute the SQL directly // Note: Desired state SQL should never contain operations like CREATE INDEX CONCURRENTLY // that cannot run in transactions. Those are migration details, not state declarations. - if _, err := ep.db.ExecContext(ctx, sql); err != nil { - return fmt.Errorf("failed to apply schema SQL: %w", err) + if _, err := ep.db.ExecContext(ctx, schemaAgnosticSQL); err != nil { + return fmt.Errorf("failed to apply schema SQL to temporary schema %s: %w", ep.tempSchema, err) } return nil diff --git a/internal/postgres/external.go b/internal/postgres/external.go index 91b97c09..589d1fa0 100644 --- a/internal/postgres/external.go +++ b/internal/postgres/external.go @@ -5,7 +5,6 @@ import ( "context" "database/sql" "fmt" - "time" _ "github.com/jackc/pgx/v5/stdlib" ) @@ -13,23 +12,23 @@ import ( // ExternalDatabase manages an external PostgreSQL database for desired state validation. // It creates temporary schemas with timestamp suffixes to avoid conflicts. type ExternalDatabase struct { - db *sql.DB - host string - port int - database string - username string - password string - tempSchema string // Temporary schema name with timestamp suffix - targetMajorVersion int // Expected major version (from target database) + db *sql.DB + host string + port int + database string + username string + password string + tempSchema string // Temporary schema name with timestamp suffix + targetMajorVersion int // Expected major version (from target database) } // ExternalDatabaseConfig holds configuration for connecting to an external database type ExternalDatabaseConfig struct { - Host string - Port int - Database string - Username string - Password string + Host string + Port int + Database string + Username string + Password string TargetMajorVersion int // Expected major version to match } @@ -69,20 +68,17 @@ func NewExternalDatabase(config *ExternalDatabaseConfig) (*ExternalDatabase, err ) } - // Generate temporary schema name with timestamp including nanoseconds for uniqueness - // Format: pgschema_plan_YYYYMMDD_HHMMSS.NNNNNNNNN - // Note: The period is required in Go's time format for fractional seconds - timestamp := time.Now().Format("20060102_150405.000000000") - tempSchema := fmt.Sprintf("pgschema_plan_%s", timestamp) + // Generate temporary schema name with unique timestamp + tempSchema := GenerateTempSchemaName() return &ExternalDatabase{ - db: db, - host: config.Host, - port: config.Port, - database: config.Database, - username: config.Username, - password: config.Password, - tempSchema: tempSchema, + db: db, + host: config.Host, + port: config.Port, + database: config.Database, + username: config.Username, + password: config.Password, + tempSchema: tempSchema, targetMajorVersion: config.TargetMajorVersion, }, nil } @@ -115,10 +111,15 @@ func (ed *ExternalDatabase) ApplySchema(ctx context.Context, schema string, sql return fmt.Errorf("failed to set search_path: %w", err) } + // Strip schema qualifications from SQL before applying to temporary schema + // This ensures that objects are created in the temporary schema via search_path + // rather than being explicitly qualified with the original schema name + schemaAgnosticSQL := stripSchemaQualifications(sql, schema) + // Execute the SQL directly // Note: Desired state SQL should never contain operations like CREATE INDEX CONCURRENTLY // that cannot run in transactions. Those are migration details, not state declarations. - if _, err := ed.db.ExecContext(ctx, sql); err != nil { + if _, err := ed.db.ExecContext(ctx, schemaAgnosticSQL); err != nil { return fmt.Errorf("failed to apply schema SQL to temporary schema %s: %w", ed.tempSchema, err) } diff --git a/testdata/diff/dependency/table_to_function/plan.json b/testdata/diff/dependency/table_to_function/plan.json index e7f65556..14432a86 100644 --- a/testdata/diff/dependency/table_to_function/plan.json +++ b/testdata/diff/dependency/table_to_function/plan.json @@ -15,7 +15,7 @@ "path": "public.documents" }, { - "sql": "CREATE OR REPLACE FUNCTION get_document_count()\nRETURNS integer\nLANGUAGE plpgsql\nSECURITY INVOKER\nVOLATILE\nAS $$\nBEGIN\n RETURN (SELECT COUNT(*) FROM public.documents);\nEND;\n$$;", + "sql": "CREATE OR REPLACE FUNCTION get_document_count()\nRETURNS integer\nLANGUAGE plpgsql\nSECURITY INVOKER\nVOLATILE\nAS $$\nBEGIN\n RETURN (SELECT COUNT(*) FROM documents);\nEND;\n$$;", "type": "function", "operation": "create", "path": "public.get_document_count" diff --git a/testdata/diff/dependency/table_to_function/plan.sql b/testdata/diff/dependency/table_to_function/plan.sql index bf16996a..69753cb4 100644 --- a/testdata/diff/dependency/table_to_function/plan.sql +++ b/testdata/diff/dependency/table_to_function/plan.sql @@ -13,6 +13,6 @@ SECURITY INVOKER VOLATILE AS $$ BEGIN - RETURN (SELECT COUNT(*) FROM public.documents); + RETURN (SELECT COUNT(*) FROM documents); END; $$; diff --git a/testdata/diff/dependency/table_to_function/plan.txt b/testdata/diff/dependency/table_to_function/plan.txt index c3558420..40591717 100644 --- a/testdata/diff/dependency/table_to_function/plan.txt +++ b/testdata/diff/dependency/table_to_function/plan.txt @@ -28,6 +28,6 @@ SECURITY INVOKER VOLATILE AS $$ BEGIN - RETURN (SELECT COUNT(*) FROM public.documents); + RETURN (SELECT COUNT(*) FROM documents); END; $$;