Connector Generator AI Service

Smart Integration Microservice for scraping, digester and CodeGen built with FastAPI.

Project structure

Directory structure:

• src - production source code
• src/modules - domain oriented modules with API routes and implementation
• src/common - common code and utils
• test/unit - unit tests
• test/integration - integration tests

Important files:

• server.py - uvicorn server entry point
• src/app.py - FastAPI entry point
• src/config.py - project configuration
• pyproject.toml - dependencies, tools, tasks
• Dockerfile - docker file

API Documentation

App exposes API documentation on the URLs below:

• OpenAPI UI: http://localhost:8090/docs
• ReDoc: http://localhost:8090/redoc

Configuration

App is configured using environment variables, see the default configuration in src/config.py and samples in .env-example.

For local development you can take advantage of dotenv plugins that read .env file from the project root.

# copy and use example dev configuration
cp .env-example .env

# copy and use configuration for unit/integration tests
cp .env.test-example .env.test

Running with Docker

Requirements

• Docker
• Docker Compose

Run using docker compose

# build image
docker compose build

# start container
docker compose up

Local Development

Prerequisites

• Python 3.12+
• UV dependency manager
• PostgreSQL 15

NOTE: tasks are run with a poethepoet tool and configured in pyproject.toml

Database Setup

The application requires PostgreSQL database. The database is automatically configured when using Docker Compose.

1. Using Docker Compose (Recommended)

The docker-compose.yaml includes a PostgreSQL service that is automatically configured with the environment variables from your .env file:

# Start all services (app + database)
docker compose up

# Start only the database
docker compose up db

The database service uses the following environment variables from .env:

• DATABASE__HOST - Database host (default: localhost)
• DATABASE__PORT - Database port (default: 5432)
• DATABASE__USER - Database user
• DATABASE__PASSWORD - Database password
• DATABASE__NAME - Database name

2. Manual PostgreSQL Setup (Alternative)

If you prefer to run PostgreSQL outside Docker Compose:

docker run --name postgres \
  -e POSTGRES_USER=user \
  -e POSTGRES_PASSWORD=password \
  -e POSTGRES_DB=db \
  -p 5432:5432 \
  -d postgres:15-alpine

3. Configure Database Connection

Ensure your .env file has the correct database configuration:

DATABASE__HOST=localhost
DATABASE__PORT=5432
DATABASE__USER=user
DATABASE__PASSWORD=password
DATABASE__NAME=db

# Full connection string for Alembic and SQLAlchemy
DATABASE_URL=postgresql+asyncpg://${DATABASE__USER}:${DATABASE__PASSWORD}@${DATABASE__HOST}:${DATABASE__PORT}/${DATABASE__NAME}

4. Run Database Migrations

The application uses Alembic for database migrations.

# Run all pending migrations
uv run alembic upgrade head

# Check current migration version
uv run alembic current

# View migration history
uv run alembic history

5. Common Alembic Commands

# Create a new migration (auto-generate from models)
uv run alembic revision --autogenerate -m "description of changes"

# Upgrade to a specific revision
uv run alembic upgrade <revision_id>

# Downgrade one revision
uv run alembic downgrade -1

# Downgrade to base (drop all tables)
uv run alembic downgrade base

# Show current revision
uv run alembic current

# Show SQL without executing
uv run alembic upgrade head --sql

Run the app

uv run poe start
# access the service at http://localhost:8090
# e.g. `curl http://0.0.0.0:8090/health`

Installing dependencies

dev dependency group is used to distinguish from production ones.

# install production dependency
uv add mydep

# install dev dependency
uv add --dev mydep

Quality assurance

All code should adhere to the quality checks below. It is highly recommended to instal pre-commit hook and also integrate these tools below in the IDE.

Linting and formatting

Quality checks using ruff and mypy.

uv run poe typecheck
uv run poe lint
uv run poe stylecheck

# optionally run all quality checks (including unit tests)
uv run poe qa

# attempt to fix formatting and lint errors
uv run poe fix

Tests

Running tests using pytest.

# run all tests
uv run poe test

# run in watch mode
uv run poe test-watch

# run unit tests only
uv run poe test test/unit

# run integration tests only
uv run poe test test/integration

Pre-commit hooks

This project uses pre-commit to ensure consistent code style and other quality checks.

# install the hooks from .pre-commit-config.yaml
# this needs to be done just once when setting up project
uv run pre-commit install

Once installed, the hooks will automatically run every time you commit changes. If any issues are found or files are modified, the commit will be aborted until fixed.

Langfuse configuration

For development and testing purposes is every api request and llm call traced with Langfuse. By default is langfuse tracing disabled, you can enable it by configuration:

# configure correct langfuse project keys
LANGFUSE__SECRET_KEY=project-secret-key
LANGFUSE__PUBLIC_KEY=project-public-key
# when using for development define your own environment
LANGFUSE__ENVIRONMENT=dev-myname
# enable langfuse
LANGFUSE__TRACING_ENABLED=true

Technical notes

• API endpoints and parameters have to follow camel case convention
• LLM model can be configured via any OpenAI-compatible chat API