For knowledge document of this organization, you can follow this 📖 Data Developer & Engineer document.
Note
I delegate the tools docs to Data Dev & Eng Lab
flowchart LR
1([📌 ddeutil]) --> 2([🚅 ddeutil-io]) --> 3([🏃 ddeutil-workflow]) ---> 4
subgraph observe
4([📡 ddeutil-observe])
6([🔭 ddeutil-observe<br>streamlit])
end
3 --> 5([🏗️ ddeutil-extensions])
3 --> 6
0([✂️ fmtutil]) -.-> 2
0 -.-> 5
This organization has the propose to make lightweight data orchestration framework for small - middle data platform project (🏃Around 10K workflows).
Firstly, I will implement base projects, 📌 Core (utility functions) and 🚅 IO (Input/Output transport utility objects) for the first dependency packages because it has a lot of base code to make main package and I do not want to develop this code on the main package, for example, it do not good if I want fix bug on the merge key function that no relate with the workflow package
🎯 The main package of this organize orchestration framework has 2 layers and I split it with 2 projects for optional installation requirement (you can only use just one of these layers without raise error).
- 🏃 Workflow - Lightweight workflow orchestration in Python with less dependencies.
- 📡 Observe (FastAPI) - Lightweight observation application with FastAPI for the workflow package.
- 🔭 Observe (Streamlit) - Lightweight observation application with Streamlit for the workflow package.
Note
I have some 3rd-party projects, 🏗️ Extensions, for keeping
an additional practices to use any 3rd API connect data source, like polars, duckdb, etc.
It is dynamic data processing & transformation functions and objects from external vendor packages.
It can plug-in to the Workflow package on the hook stage.
- ⚡ DeFlow - Lightweight Declarative Data Workflow Framework.
This organize has some mini-projects that develop for specific usecase:
- data-orchestra - Full-Stack Data Orchestration from Yaml template with Flask & HTMX
- load-routing - Routing Application Service deploy to On-Premise server with FastAPI
Warning
The above projects have a lot of bugs and need times to fix and refactor the code. So, you should not use these projects.