Skip to content

Commit 9cbcd29

Browse files
faizanazim11faizanazim11
authored
Merge pull request #2 from TechPrismatica/copilot/fix-f110b891-961e-4db7-9061-1b980694cf77
Implement precreate functionality in session management
2 parents 9142480 + 1ace79b commit 9cbcd29

File tree

5 files changed

+432
-4
lines changed

5 files changed

+432
-4
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
# sql-db-utils - GitHub Copilot Instructions
2+
3+
**SQL database utilities package for Python developers that provides declarative model generation and session management.**
4+
5+
Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.
6+
7+
## Working Effectively
8+
9+
### Bootstrap, Build, and Test Repository:
10+
- `pip install -e .` -- installs the package in development mode. NEVER CANCEL: Takes 10-60 seconds, may timeout due to network issues. Set timeout to 120+ seconds (extra margin for slow mirrors or network issues).
11+
- `pip install coverage pre-commit pytest pytest-cov ruff` -- installs development dependencies. NEVER CANCEL: Takes 30-120 seconds. Set timeout to 180+ seconds (extra margin for slow mirrors or network issues).
12+
- `python -m pytest tests/ -v` -- runs unit tests (when tests are available)
13+
- `ruff check .` -- runs linting (takes ~0.01 seconds)
14+
- `ruff format --check .` -- checks code formatting (takes ~0.01 seconds)
15+
16+
### Environment Configuration:
17+
- Package requires Python >= 3.13
18+
- Core dependencies: SQLAlchemy >= 2.0.38, sqlalchemy-utils, psycopg, python-dateutil, whenever
19+
- Optional dependencies available: polars, pandas, async, binary, codegen
20+
- Database support: PostgreSQL (primary), with pooling and connection management
21+
- Configuration via environment variables and PostgreSQL connection strings
22+
23+
### Run Integration Tests with Real Database:
24+
- Start PostgreSQL: `docker run -d --name test-postgres -p 5432:5432 -e POSTGRES_PASSWORD=test postgres:15-alpine` (NEVER CANCEL: Takes 30-60 seconds for first download)
25+
- Wait for startup: `sleep 10`
26+
- Run integration tests: `POSTGRES_URI=postgresql://postgres:test@localhost:5432/test python -c "from sql_db_utils import SQLSessionManager; print('SQL integration test passed')"` (basic session management test)
27+
- Clean up: `docker stop test-postgres && docker rm test-postgres`
28+
29+
## Validation Scenarios
30+
31+
### Always Test After Making Changes:
32+
1. **Import Test**: `python -c "from sql_db_utils import SQLSessionManager; from sql_db_utils.asyncio import SQLSessionManager as AsyncSQLSessionManager; print('Import successful')"`
33+
2. **Basic Session Management Test** (requires PostgreSQL running):
34+
```bash
35+
POSTGRES_URI=postgresql://postgres:test@localhost:5432/test python -c "
36+
from sql_db_utils import SQLSessionManager
37+
manager = SQLSessionManager()
38+
print('Session Manager created successfully')
39+
"
40+
```
41+
3. **Precreate/Postcreate Functionality Test**:
42+
```bash
43+
python -c "
44+
from sql_db_utils import SQLSessionManager
45+
manager = SQLSessionManager()
46+
47+
@manager.register_precreate('test_db')
48+
def test_precreate(tenant_id):
49+
return 'SELECT 1;'
50+
51+
@manager.register_postcreate('test_db')
52+
def test_postcreate(tenant_id):
53+
return 'SELECT 2;'
54+
55+
print('Precreate/Postcreate registration successful')
56+
"
57+
```
58+
4. **Run Full Test Suite**: `python -m pytest tests/ -v --cov=sql_db_utils --cov-report=term-missing` (when tests are available)
59+
5. **Linting**: `ruff check . && ruff format --check .`
60+
61+
### Manual Testing Requirements:
62+
- ALWAYS test session management functionality after code changes
63+
- Test both synchronous and asynchronous implementations
64+
- Verify precreate and postcreate hooks work correctly
65+
- Test with different PostgreSQL configurations and connection parameters
66+
- Test database creation, connection pooling, and engine management
67+
68+
## Common Tasks
69+
70+
### Repository Structure:
71+
```
72+
sql-db-utils/
73+
├── .github/workflows/ # CI/CD pipelines
74+
├── sql_db_utils/ # Main package source
75+
│ ├── __init__.py # Main exports
76+
│ ├── config.py # Configuration settings
77+
│ ├── constants.py # Package constants
78+
│ ├── datetime_utils.py # Date/time utilities
79+
│ ├── session_management.py # Core session management (sync)
80+
│ ├── sql_creations.py # SQL table creation utilities
81+
│ ├── sql_extras.py # Additional SQL utilities
82+
│ ├── sql_retry_handler.py # Query retry mechanisms
83+
│ ├── sql_utils.py # General SQL utilities
84+
│ ├── declarative_utils.py # Declarative model utilities
85+
│ ├── declaratives.py # Base declarative classes
86+
│ ├── codegen.py # Code generation utilities
87+
│ ├── aggrid/ # AG Grid integration utilities
88+
│ │ ├── date_filters.py
89+
│ │ ├── number_filters.py
90+
│ │ └── text_filters.py
91+
│ └── asyncio/ # Asynchronous implementations
92+
│ ├── __init__.py
93+
│ ├── session_management.py # Async session management
94+
│ ├── sql_creations.py # Async SQL creation utilities
95+
│ ├── sql_creation_helper.py # Async creation helpers
96+
│ ├── sql_retry_handler.py # Async retry mechanisms
97+
│ ├── sql_utils.py # Async SQL utilities
98+
│ ├── declarative_utils.py # Async declarative utilities
99+
│ ├── declaratives.py # Async base classes
100+
│ ├── codegen.py # Async code generation
101+
│ └── inspector_utils.py # Database inspection utilities
102+
├── tests/ # Test files (when available)
103+
├── pyproject.toml # Project configuration
104+
└── README.md # Documentation
105+
```
106+
107+
### Key Files to Check After Changes:
108+
- Always verify `sql_db_utils/__init__.py` after changing main exports
109+
- Check `sql_db_utils/config.py` after modifying configuration handling
110+
- Verify sync/async parity between `sql_db_utils/session_management.py` and `sql_db_utils/asyncio/session_management.py`
111+
- Update `sql_db_utils/sql_creations.py` and `sql_db_utils/asyncio/sql_creations.py` for SQL creation changes
112+
- Test declarative utilities in both sync and async versions
113+
- Verify codegen functionality if making changes to code generation features
114+
- Update tests when adding new functionality
115+
- Run integration tests with real PostgreSQL database
116+
117+
### Development Dependencies:
118+
- **Testing**: pytest, pytest-cov, coverage
119+
- **Linting**: ruff (replaces black, flake8, isort)
120+
- **Git hooks**: pre-commit
121+
- **Type checking**: Built into package development
122+
- **Core Dependencies**: SQLAlchemy, sqlalchemy-utils, psycopg, python-dateutil, whenever
123+
124+
### Build and Package:
125+
- `python -m build` -- builds distribution packages. NEVER CANCEL: May fail due to network timeouts depending on the configured build backend and network environment. Package requires Python >= 3.13.
126+
- Package metadata in `pyproject.toml`
127+
- Uses hatchling as build backend
128+
- **Note**: Package requires specific Python version (>=3.13) which may not be available in all environments
129+
130+
### Session Management Features:
131+
- **Precreate Hooks**: Execute before database/table creation for setup tasks
132+
- **Postcreate Hooks**: Execute after database/table creation for initialization
133+
- **Auto Hooks**: Return SQL statements to be executed automatically
134+
- **Manual Hooks**: Receive session objects for custom operations
135+
- **Multi-database Support**: Register hooks for single or multiple databases
136+
- **Tenant Support**: All hooks receive tenant_id parameter for multi-tenant applications
137+
138+
## Database Features and Testing
139+
140+
### Supported Database Features:
141+
- **PostgreSQL**: Primary database with full feature support
142+
- **Connection Pooling**: Configurable via PostgresConfig.PG_ENABLE_POOLING
143+
- **Connection Retry**: Built-in retry mechanisms with PostgresConfig.PG_MAX_RETRY
144+
- **Database Creation**: Automatic database creation if not exists
145+
- **Schema Support**: Multi-schema support with declarative utilities
146+
- **Transaction Management**: Automatic transaction handling in hooks
147+
148+
### Session Management Testing:
149+
- **Engine Creation**: Test `_get_engine()` method with various configurations
150+
- **Hook Execution**: Verify precreate/postcreate hooks execute in correct order
151+
- **Connection Pooling**: Test with/without pooling enabled
152+
- **Multi-tenant**: Test with different tenant_id values
153+
- **Error Handling**: Test connection failures and retry mechanisms
154+
155+
### Setting up Test Database with Docker:
156+
- PostgreSQL: `docker run -d --name test-postgres -p 5432:5432 -e POSTGRES_PASSWORD=test postgres:15-alpine`
157+
- Create test database: `docker exec -it test-postgres psql -U postgres -c "CREATE DATABASE test;"`
158+
159+
## CI/CD Pipeline (.github/workflows)
160+
161+
### Linter Pipeline:
162+
- Runs on pull requests and pushes
163+
- Uses ruff for linting and formatting
164+
- ALWAYS run `ruff check .` and `ruff format --check .` before committing
165+
- Pre-commit hooks should handle formatting automatically
166+
167+
### Package Publishing:
168+
- Triggers on git tags
169+
- Builds with hatchling backend
170+
- Publishes to PyPI
171+
- Requires Python >= 3.13 environment
172+
173+
## Critical Notes
174+
175+
### Session Management Execution Order:
176+
1. **Engine Creation**: Database connection and engine setup
177+
2. **Precreate Hooks**: Execute custom setup before table creation
178+
3. **Table Creation**: `create_default_psql_dependencies()` creates tables/metadata
179+
4. **Postcreate Hooks**: Execute initialization after table creation
180+
181+
### Hook Implementation Patterns:
182+
- **Auto Hooks**: Return SQL strings or lists of SQL strings
183+
- **Manual Hooks**: Receive (session, tenant_id) parameters for custom logic
184+
- **Registration**: Use `@manager.register_precreate()` or `@manager.register_precreate_manual()`
185+
- **Multi-Database**: Pass list of database names to register for multiple databases
186+
187+
### Async/Sync Parity:
188+
- Both implementations must have identical API and functionality
189+
- Async version uses `async`/`await` patterns appropriately
190+
- Session types differ: `Session` vs `AsyncSession`
191+
- Engine types differ: `Engine` vs `AsyncEngine`
192+
193+
### Configuration and Environment:
194+
- PostgresConfig class manages all database configuration
195+
- Environment variables control connection parameters
196+
- ModuleConfig handles application-level settings
197+
- Connection pooling and retry settings are configurable
198+
199+
## Troubleshooting
200+
201+
### Common Issues:
202+
1. **Import Error**: Check Python version (>=3.13 required)
203+
2. **Connection Failures**: Verify PostgreSQL is running and accessible
204+
3. **Linting Failures**: Run `ruff format .` to auto-fix formatting issues
205+
4. **Missing Dependencies**: Run `pip install -e .` to reinstall package
206+
5. **Hook Execution**: Verify hooks are registered before engine creation
207+
208+
### Session Management Issues:
209+
- **Engine Not Created**: Check database URI and connection parameters
210+
- **Hooks Not Executing**: Ensure registration occurs before `get_session()` calls
211+
- **Transaction Errors**: Verify database permissions and connection state
212+
- **Pooling Issues**: Check PostgresConfig.PG_ENABLE_POOLING setting
213+
214+
### Development Environment:
215+
- Python 3.13+ required (check with `python --version`)
216+
- PostgreSQL server required for integration testing
217+
- Docker recommended for consistent database testing
218+
- Pre-commit hooks help maintain code quality
219+
220+
### Async/Sync Coordination:
221+
- Changes to sync version should be mirrored in async version
222+
- Test both implementations when making session management changes
223+
- Verify async patterns use proper `await` keywords
224+
- Check that both versions handle errors consistently

‎README.md‎

Lines changed: 109 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ A powerful SQL database utilities package for Python developers that provides de
55
## Features
66

77
- Automatic declarative model generation for SQL databases
8-
- Schema-aware session management
8+
- Schema-aware session management with precreate and postcreate hooks
99
- Support for both synchronous and asynchronous operations
1010
- Project-based database isolation
1111
- Security-enabled database connections
@@ -32,6 +32,114 @@ pip install sql-db-utils
3232
- SQLAlchemy >= 2.0.38
3333
- Additional dependencies based on optional features
3434

35+
## Session Management Hooks
36+
37+
The `SQLSessionManager` provides two types of hooks for customizing database setup:
38+
39+
### Precreate Hooks
40+
41+
Precreate hooks execute **before** database tables and meta information are created. These are useful for:
42+
- Setting up database extensions
43+
- Creating custom schemas
44+
- Preparing database environment
45+
- Initial database configuration
46+
47+
#### Automatic Precreate
48+
Returns SQL statements to be executed automatically:
49+
50+
```python
51+
from sql_db_utils import SQLSessionManager
52+
53+
session_manager = SQLSessionManager()
54+
55+
@session_manager.register_precreate("my_database")
56+
def setup_database_extensions(tenant_id):
57+
return "CREATE EXTENSION IF NOT EXISTS uuid-ossp;"
58+
59+
# Or for multiple databases
60+
@session_manager.register_precreate(["db1", "db2"])
61+
def setup_multiple_databases(tenant_id):
62+
return [
63+
"CREATE EXTENSION IF NOT EXISTS uuid-ossp;",
64+
"CREATE EXTENSION IF NOT EXISTS pgcrypto;"
65+
]
66+
```
67+
68+
#### Manual Precreate
69+
Receives a session object for custom operations:
70+
71+
```python
72+
@session_manager.register_precreate_manual("my_database")
73+
def custom_precreate_setup(session, tenant_id):
74+
# Custom logic with session
75+
session.execute("CREATE SCHEMA IF NOT EXISTS custom_schema;")
76+
# Additional setup logic here
77+
```
78+
79+
### Postcreate Hooks
80+
81+
Postcreate hooks execute **after** database tables and meta information are created. These are useful for:
82+
- Seeding initial data
83+
- Creating triggers and procedures
84+
- Setting up initial user permissions
85+
- Post-creation optimizations
86+
87+
#### Automatic Postcreate
88+
Returns SQL statements to be executed automatically:
89+
90+
```python
91+
@session_manager.register_postcreate("my_database")
92+
def seed_initial_data(tenant_id):
93+
return "INSERT INTO users (name) VALUES ('admin') ON CONFLICT DO NOTHING;"
94+
95+
# Or for multiple databases
96+
@session_manager.register_postcreate(["db1", "db2"])
97+
def setup_multiple_databases(tenant_id):
98+
return [
99+
"INSERT INTO settings (key, value) VALUES ('version', '1.0') ON CONFLICT DO NOTHING;",
100+
"INSERT INTO roles (name) VALUES ('admin') ON CONFLICT DO NOTHING;"
101+
]
102+
```
103+
104+
#### Manual Postcreate
105+
Receives a session object for custom operations:
106+
107+
```python
108+
@session_manager.register_postcreate_manual("my_database")
109+
def custom_postcreate_setup(session, tenant_id):
110+
# Custom logic with session
111+
session.execute("INSERT INTO initial_data (tenant_id) VALUES (:tenant_id);", {"tenant_id": tenant_id})
112+
# Additional setup logic here
113+
```
114+
115+
### Execution Order
116+
117+
1. **Precreate hooks** (before table creation)
118+
- `register_precreate` functions are executed
119+
- `register_precreate_manual` functions are executed
120+
2. **Database table creation** (`create_default_psql_dependencies`)
121+
3. **Postcreate hooks** (after table creation)
122+
- `register_postcreate` functions are executed
123+
- `register_postcreate_manual` functions are executed
124+
125+
### Async Support
126+
127+
All hooks work identically with the async session manager:
128+
129+
```python
130+
from sql_db_utils.asyncio import SQLSessionManager
131+
132+
async_session_manager = SQLSessionManager()
133+
134+
@async_session_manager.register_precreate("my_database")
135+
def setup_extensions(tenant_id):
136+
return "CREATE EXTENSION IF NOT EXISTS uuid-ossp;"
137+
138+
@async_session_manager.register_postcreate_manual("my_database")
139+
async def seed_data(session, tenant_id):
140+
await session.execute("INSERT INTO users (tenant_id) VALUES (:tenant_id);", {"tenant_id": tenant_id})
141+
```
142+
35143
## Authors
36144

37145
- Faizan (faizanazim11@gmail.com)

‎sql_db_utils/__version__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = "1.1.1"
1+
__version__ = "1.2.0"

0 commit comments

Comments
 (0)