Skip to content

Commit

Permalink
VPC deployment reference.
Browse files Browse the repository at this point in the history
  • Loading branch information
Sebastian Sassi committed Jul 26, 2024
1 parent bbc882b commit 4a27e40
Show file tree
Hide file tree
Showing 4 changed files with 39 additions and 0 deletions.
2 changes: 2 additions & 0 deletions docs/_sidebar.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@
- [Airflow Config Defaults](/reference/airflow/airflow-config-defaults.md)
- [DAG Generators](/reference/airflow/dag-generators.md)
- [Datacoves Operators](/reference/airflow/datacoves-operator.md)
- [Datacoves](/reference/datacoves/)
- [VPC Deployment](/reference/datacoves/vpc-deployment.md)
- [VS Code](/reference/vscode/)
- [Datacoves Environment Variables](/reference/vscode/datacoves-env-vars.md)
- [Metrics & Logs](/reference/metrics-and-logs/)
Expand Down
1 change: 1 addition & 0 deletions docs/reference/datacoves/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Datacoves reference
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
36 changes: 36 additions & 0 deletions docs/reference/datacoves/vpc-deployment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# VPC Deployment

Datacoves is designed to work on Public or Private Virtual Clouds.
The following diagram shows the main services required by Datacoves when deployed on a VPC.

## Datacoves Architecture

![Datacoves Architecture](./assets/datacoves-architecture.png)

## Dependencies

| Service | Purpose | Requirements | Quantity |
|--------------|-----------------|--------------------------|------------|
| Database | Datacoves core services | PostgreSQL > v.14, Minimum 2vcpu, 16Gb memory __(*)__ | 1 server |
| Database | Datacoves stack services | PostgreSQL > v.14, Minimum 4vcpu, 32Gb memory __(*)__ | 1 server |
| Blob storage | DBT artifacts | N/A | 1 bucket |
| Blob storage | Grafana logs | Lifetime policy with 30 days retention | 1 bucket |
| Blob storage | Airflow DAGs | N/A | 1 bucket per Airflow instance |
| Blob storage | Airbyte logs | N/A | 1 bucket per Airbyte instance |
| File storage | Airflow logs | Sub-millisecond first byte read latency | 1 file system per Airflow instance |
| OIDC provider | Datacoves SSO | Oauth 2.0 OIDC compliant provider | 1 provider |
| Git server | DBT development version control | > v2.33 | 1 server |
| CI/CD server | DBT development | N/A | 1 server |

__(*)__ min. requirements may vary depending on the number of environments

### Optional dependencies

| Service | Purpose | Requirements | Quantity |
|--------------|-----------------|--------------------------|------------|
| Docker Registry | Docker images registry | Any docker API compliant image registry | 1 service account |
| SMTP account | Airflow notifications | N/A | 1 service account |
| Slack account | Airflow notifications | N/A | 1 account |
| MS Teams account | Airflow notifications | N/A | 1 account |


0 comments on commit 4a27e40

Please sign in to comment.