Skip to content
datacorner edited this page Dec 2, 2023 · 24 revisions

The pipelite Project

The purpose of this solution is to build and execute Data Pipelines. Nothing new under the sun of course, however it aims to make this task very easy and by just using some configuration to build those data pipelines. in other words by using pipelite you can create quite powerful data intergation stuff without typing any lines of code !

pipelite is a Data Pipeline solution by design !

The way this solution is built is also totally extensible and enables all developers to extend its capabilities by addin new Data connectors and/or data pipelite.transformers.

πŸš€ Currently this solution provides data access and load from these data sources :
πŸ“„ External file (csv)
πŸ“‘ External Excel Spreadsheet (xls, xlsx, xlsm, xlsb, odf, ods and odt) (read only) πŸ“ƒ External XES File (read only)
πŸ“€ ODBC Data Sources (checked with SQL Server, SQLite) by using an configurable SQL query (Read Only)
🏒 SAP Read Table via SAP RFC (Read Only)
🎒 ABBYY Timeline PI(write only in Repository)

πŸš€ And provides those transformers
πŸ”€ Pass Through (Ex. just to change the Data Sources names IN-OUT)
πŸ“Ά Dataset Profiling
πŸ”‚ Concat 2 Data sources
πŸ†– SubString
πŸ†’ Column Transformation
πŸ”ƒ Join data sources
πŸ”€ Rename Column Name

This is the beggining and pipelite is designed to be extensible ... So if you have in mind some new good stuff in mind you'd like to add, just join the community ;-)

🏠 Home
πŸ”‘ Main concepts
πŸ’» Installation
πŸ”¨ Configuration
πŸš€ Running

Supported Data Sources
πŸ“„ CSV File
πŸ“‘ XES File
πŸ“ƒ Excel File
πŸ“€ ODBC
🏒 SAP
🎒 ABBYY Timeline

Supported Transformations
πŸ”€ Pass Through
πŸ“Ά Dataset Profiling
πŸ”‚ Concat 2 Data sources
πŸ†– SubString
πŸ†’ Column Transformation
πŸ”ƒ Join data sources
πŸ”ƒ Lookup
πŸ”€ Rename Column Name

Extending pipelite
βœ… how to
βœ… Adding new Data sources
βœ… Adding new Transformers
βœ… Adding new Pipelines

Clone this wiki locally