Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add drawdb module #124

Merged
merged 4 commits into from
Sep 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# dbterd

Generate the ERD-as-a-code ([DBML](https://dbdiagram.io/d), [Mermaid](https://mermaid-js.github.io/mermaid-live-editor/), [PlantUML](https://plantuml.com/ie-diagram), [GraphViz](https://graphviz.org/), [D2](https://d2lang.com/)) from dbt artifact files (`dbt Core`) or from dbt metadata (`dbt Cloud`)
Generate the ERD-as-a-code ([DBML](https://dbdiagram.io/d), [Mermaid](https://mermaid-js.github.io/mermaid-live-editor/), [PlantUML](https://plantuml.com/ie-diagram), [GraphViz](https://graphviz.org/), [D2](https://d2lang.com/), [DrawDB](https://drawdb.vercel.app/)) from dbt artifact files (`dbt Core`) or from dbt metadata (`dbt Cloud`)

Entity Relationships are configurably detected by ([docs](https://dbterd.datnguyen.de/latest/nav/guide/cli-references.html#dbterd-run-algo-a)):

Expand Down
176 changes: 176 additions & 0 deletions dbterd/adapters/targets/drawdb.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
import json
from typing import List, Tuple

from dbterd.adapters import adapter
from dbterd.adapters.meta import Table
from dbterd.types import Catalog, Manifest


def run(manifest: Manifest, catalog: Catalog, **kwargs) -> Tuple[str, str]:
"""Parse dbt artifacts and export DDB file

Args:
manifest (dict): Manifest json
catalog (dict): Catalog json

Returns:
Tuple(str, str): File name and the DDB (json) content
"""
output_file_name = kwargs.get("output_file_name") or "output.ddb"
return (output_file_name, parse(manifest, catalog, **kwargs))


def parse(manifest: Manifest, catalog: Catalog, **kwargs) -> str:
"""Get the DDB content from dbt artifacts

Args:
manifest (dict): Manifest json
catalog (dict): Catalog json

Returns:
str: DDB (json) content
"""

algo_module = adapter.load_algo(name=kwargs["algo"])
tables, relationships = algo_module.parse(
manifest=manifest, catalog=catalog, **kwargs
)

# Build DDB content
graphic_tables = get_graphic_tables(tables=tables)
drawdb = dict(
author="dbterd",
title=kwargs.get("output_file_name") or "Generated by dbterd",
date=str(manifest.metadata.generated_at),
tables=[
dict(
id=idx,
name=x.name,
x=graphic_tables.get(x.name, {}).get("x"),
y=graphic_tables.get(x.name, {}).get("y"),
comment=x.description,
indices=[],
color="#175e7a",
fields=[
dict(
id=idc,
name=c.name,
type=c.data_type,
default="",
check="",
primary=False, # TODO
unique=False, # TODO
notNull=False, # TODO
increment=False,
comment=c.description,
)
for idc, c in enumerate(x.columns)
],
)
for idx, x in enumerate(tables)
],
relationships=[
dict(
id=idx,
name=f"fk__{x.table_map[1]}_{x.table_map[0]}__{x.column_map[1]}",
cardinality=get_rel_symbol(x.type),
startTableId=graphic_tables.get(x.table_map[1], {}).get("id"),
endTableId=graphic_tables.get(x.table_map[0], {}).get("id"),
startFieldId=(
graphic_tables.get(x.table_map[1], {})
.get("fields")
.get(x.column_map[1], {})
.get("id")
),
endFieldId=(
graphic_tables.get(x.table_map[0], {})
.get("fields")
.get(x.column_map[0], {})
.get("id")
),
updateConstraint="No action",
deleteConstraint="No action",
)
for idx, x in enumerate(relationships)
],
notes=[],
subjectAreas=[],
database="generic",
types=[],
)

return json.dumps(drawdb)


def get_y(
tables: List[Table], idx: int, graphic_tables: dict, column_size: int = 4
) -> float:
"""Get y value of a table

`y = S x (T's no of columns) + (T's y value if any)`

- T: the prev table in the same graph column
- S: the height value of a graphic column, default = 50

Args:
tables (List[Table]): Parsed tables
idx (int): Current table index
graphic_tables (dict): Mutable caculated graphic tables dict
column_size (int): Graphic column size, default = 4

Returns:
float: y value
"""
if idx < column_size:
return 0

col_len = len(tables[idx - column_size].columns) + 1 # plus title row
y = (50 * col_len) * int(0 if idx < column_size else 1)

if idx - column_size >= 0:
prev_table_name = tables[idx - column_size].name
y += graphic_tables[prev_table_name].get("y", 0)

return y


def get_graphic_tables(tables: List[Table]) -> dict:
"""Return the indexed and pre-layouted tables

Args:
tables (List[Table]): List of parsed tables

Returns:
dict: Indexed and Layouted tables
"""
graphic_tables = dict()
for idx, x in enumerate(tables):
idx_fields = dict()
graphic_tables[x.name] = dict(
id=idx,
x=500 * (idx % 4),
y=get_y(tables, idx, graphic_tables),
fields=idx_fields,
)
for idc, c in enumerate(x.columns):
idx_fields[c.name] = dict(id=idc)

return graphic_tables


def get_rel_symbol(relationship_type: str) -> str:
"""Get DDB relationship symbol

Args:
relationship_type (str): relationship type

Returns:
str: Relation symbol supported in DDB
"""
if relationship_type in ["01", "11"]:
return "One to one"
if relationship_type in ["0n", "1n"]:
return "One to many"
if relationship_type in ["nn"]:
return "Many to many"
return "Many to one" # n1
Binary file added docs/assets/images/import-ddb.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# dbterd

CLI to generate Diagram-as-a-code file ([DBML](https://dbdiagram.io/d), [Mermaid](https://mermaid-js.github.io/mermaid-live-editor/), [PlantUML](https://plantuml.com/ie-diagram), [GraphViz](https://graphviz.org/), [D2](https://d2lang.com/)) from dbt artifact files.
CLI to generate Diagram-as-a-code file ([DBML](https://dbdiagram.io/d), [Mermaid](https://mermaid-js.github.io/mermaid-live-editor/), [PlantUML](https://plantuml.com/ie-diagram), [GraphViz](https://graphviz.org/), [D2](https://d2lang.com/), [DrawDB](https://drawdb.vercel.app/)) from dbt artifact files.

Entity Relationships are configurably detected by ([docs](https://dbterd.datnguyen.de/latest/nav/guide/cli-references.html#dbterd-run-algo-a)):

Expand Down
69 changes: 69 additions & 0 deletions docs/nav/guide/targets/generate-drawdb.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Generate DrawDB

## 1. Produce dbt artifact files

Let's use [Jaffle-Shop](https://github.com/dbt-labs/jaffle-shop) as the example.

Clone it, then perform the `dbt docs generate` in order to generate the `/target` folder containing:

- `manifest.json`
- `catalog.json`

Or we can use the generated files found in the [samples](https://github.com/datnguye/dbterd/tree/main/samples/jaffle-shop)

## 2. Generate DrawDB (.ddb) file

In the same dbt project directory, let's run `dbterd` command to generate the `.ddb` file which is the supported import file format, actually it is `json` file

```shell
dbterd run -t drawdb -enf table
```

There we go, here is the sample output content:

```json
{
"author": "dbterd",
"title": "erd",
"date": "2024-07-28T01:54:24.620460Z",
"tables": [
...
{
"id": 3,
"name": "order_items",
...
}
...
{
"id": 4,
"name": "orders",
...
}
],
"relationships": [
{
"id": 0,
"name": "fk__order_items_orders__order_id",
"cardinality": "Many to one",
"startTableId": 3,
"endTableId": 4,
"startFieldId": 1,
"endFieldId": 0,
...
},
...
```

> Check full sample at [samples/jaffle-shop/erd.ddb](https://github.com/datnguye/dbterd/blob/main/samples/jaffle-shop/erd.ddb)

## 3. Import to Draw DB Editor

Go to the [Draw DB Editor](https://drawdb.vercel.app/editor) playaround:

- Files > Import diagram
- Choose the generated file e.g. `erd.ddb`
- Click `Import`

Voila 🎉, here the result:

![import-ddb.png](./../../../assets/images/import-ddb.png)
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ nav:
- PlantUML: nav/guide/targets/generate-plantuml.md
- D2: nav/guide/targets/generate-d2.md
- GraphViz: nav/guide/targets/generate-graphviz.md
- DrawDB: nav/guide/targets/generate-drawdb.md
- Metadata:
- Ignore Tests: nav/metadata/ignore_in_erd.md
- Relationship Types: nav/metadata/relationship_type.md
Expand Down
Loading
Loading