Data Catalog Core

A service image to generate and serve dbt documentation (dbt docs) for Fast.BI user console. Automates docs generation on a schedule and serves an interactive docs site.

Overview

This image clones a dbt project, installs dependencies, generates dbt docs on a cron schedule, and serves them over HTTP. It supports multiple warehouses (BigQuery, Snowflake, Redshift, Fabric) and integrates with GCP secrets and Kubernetes-mounted secrets for credentials.

Architecture

Core Components

• Docs Generator: Runs dbt deps and dbt docs generate on a configurable schedule.
• Docs Server: Serves the interactive dbt docs website on port 8001.
• Secrets Manager: Reads credentials from GCP Secret Manager or mounted secrets.
• Logging & Cron: Structured logs and cron-based scheduling with health checks.

Docker Image

Base Image

• Base: Python 3.11.11-slim-bullseye

Build

# Build the image
./build.sh

# Or manually
docker build -t data-catalog-core .

Environment Variables

• GITLINK_SECRET - Git URL (with token) to clone the dbt project
• DBT_REPO_NAME - Repo subdirectory containing dbt project
• DBT_PROJECT_NAME - Logical project name
• CRON_TIME - Cron expression for docs regeneration (default: 0 6 * * *)
• GCP_CE_ACC - Use GCP secret manager for env (true/false)
• DBT_SECRETS - Secret name in GCP Secret Manager (base64 env bundle)
• GCP_SECRET_KEY - Backward compatibility mode for SA JSON flow (true/false)
• SA_EMAIL / SA_SECRET / PROFILE_SECRET_NAME - GCP auth and profiles.yml sources
• DATA_WAREHOUSE_PLATFORM - bigquery | snowflake | redshift | fabric

Mounted secrets supported at /fastbi/secrets/* for warehouse credentials.

Main Functionality

1. Clone dbt repo and install dependencies
2. Configure credentials via GCP or mounted secrets
3. Generate docs via cron and on startup
4. Serve docs on port 8001, with logs persisted under /data/logs

Health Checks

# Check container health
docker inspect --format='{{.State.Health.Status}}' data-catalog-core

# View health check logs
docker inspect --format='{{range .State.Health.Log}}{{.Output}}{{end}}' data-catalog-core

Troubleshooting

• Failed to clone repo: Verify GITLINK_SECRET and network access
• Missing profiles: Ensure PROFILE_SECRET_NAME or mounted profiles are available
• Warehouse auth issues: Confirm mounted secrets or GCP auth settings
• Docs not regenerating: Check CRON_TIME and cron logs in /data/logs/

Getting Help

• Documentation: https://wiki.fast.bi
• Issues: https://github.com/fast-bi/data-catalog-core/issues
• Email: support@fast.bi

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.