This project aims to build Apache Beam® I/O Connectors using the Python SDK. Visit the project website for more details.
We keep looking into adding new connectors. If you have a new idea, please add a comment to this discussion.
I/O connectors for selective AWS services. They are developed by interfacing with the boto3 package.
| Connector name | Source Supported | Sink Supported | Batch Supported | Streaming Supported |
|---|---|---|---|---|
| firehose_pyio | ❌ | ✔️ | ✔️ | ✔️ |
| sqs_pyio | 📅 | ✔️ | ✔️ | ✔️ |
| sns_pyio | ❌ | 📅 | 📅 | 📅 |
| dynamodb_pyio | 📅 | ✔️ | ✔️ | ✔️ |
| eventbridge_pyio | ❌ | 📅 | 📅 | 📅 |
It connects to over 20 databases, data warehouses and query engines by integrating with the Daft package. See the SQL integration page for details.
| Connector name | Source Supported | Sink Supported | Batch Supported | Streaming Supported |
|---|---|---|---|---|
| sql_pyio | 📅 | 📅 | 📅 | ❌ |
These packages connect to popular open table formats including Apache Iceberg, Apache Hudi and Delta Lake. They are implemented by integrating with the Daft package. See the integrations page for details.
| Connector name | Source Supported | Sink Supported | Batch Supported | Streaming Supported |
|---|---|---|---|---|
| iceberg_pyio | 📅 | 📅 | 📅 | ❌ |
| hudi_pyio | 📅 | 📅 | 📅 | ❌ |
| deltalake_pyio | 📅 | 📅 | 📅 | ❌ |
All Apache Beam Python I/O Connectors projects are governed by the project Code of Conduct. Please review that and keep it in mind in all interactions.
