[Feature][tools] X2SeaTunnel

[Adamyuanyuan]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

X2SeaTunnel Design Document

Background Overview

X2SeaTunnel is a universal configuration conversion tool designed to convert configuration files from various data integration tools (such as DataX, Sqoop, etc.) into SeaTunnel's HOCON or JSON configuration files, helping users migrate smoothly to the SeaTunnel platform.

Design Philosophy

Core Principles

• Simple and Lightweight: Keep the tool efficient and focused on configuration file format conversion
• Unified Framework: Build a common framework that supports configuration conversion from multiple data integration tools
• Extensibility: Adopt a plugin-based design for easy extension to support more tools in the future
• Usability: Provide multiple usage methods, offering SDK, command-line interface, supporting both single script and batch processing to meet different scenario requirements
  

Conversion Process

Source Tool Config(DataX json) → Parsing → Unified Model → Mapping Conversion → Generate SeaTunnel Config

Usage Methods

Simple Command Line Interface

# Basic usage
sh bin/x2seatunnel.sh -t datax -i /path/to/config.json -o /path/to/output.conf

# Specify tool type, input/output and format
sh bin/x2seatunnel.sh -t datax -i input.json -o output.conf -f hocon

# Batch conversion
sh bin/x2seatunnel.sh -t datax -d /input/dir/ -o /output/dir/

YAML Command Line Interface

# Using YAML configuration file
sh bin/x2seatunnel.sh --config conversion.yaml

YAML Configuration File Example

# X2SeaTunnel configuration file
metadata:
  # Configuration file format version
  configVersion: "1.0"
  # Description (optional)
  description: "DataX to SeaTunnel conversion configuration"

# Tool configuration
tool:
  # Source tool type: datax, sqoop, etc.
  sourceType: "datax"
  sourceVersion: "2.1.2"
  # Target SeaTunnel version
  targetVersion: "2.3.11"

# Input configuration
input:
  # Source configuration path (file or directory)
  path: "/path/to/configs"
  # Whether to process subdirectories recursively
  recursive: true
  # File matching pattern
  pattern: "*.json"

# Output configuration
output:
  # Output path
  path: "/path/to/output"
  # Output format: hocon or json
  format: "hocon" 
  # Filename conversion rule
  namePattern: "${filename}_seatunnel.conf"

# Mapping configuration
mapping:
  # Custom mapping rules path (optional)
  rulesPath: "/path/to/custom/rules.json"
  
# Validation configuration
validation:
  # Whether to enable validation
  enabled: true
  # Validation failure behavior: warn, error, ignore
  
# Logging configuration
logging:
  # Log level: debug, info, warn, error
  level: "info"
  # Log output path
  path: "./logs"
  # Log filename pattern
  filePattern: "x2seatunnel-%d{yyyy-MM-dd}.log"
  # Whether to output to console simultaneously
  console: true

SDK Integration Method

// Create specific tool converter
X2SeaTunnelConverter converter = X2SeaTunnelFactory.createConverter("datax");

// Configure conversion options
ConversionOptions options = new ConversionOptions.Builder()
    .outputFormat("hocon")
    .targetVersion("2.3.11")
    .build();
    
// Execute conversion
String seatunnelConfig = converter.convert(sourceConfigContent, options);

Implementation Roadmap

1. Phase One: Basic framework and DataX support, MySQL data source usable
   • Core interface design
   • Support for common DataX connectors (MySQL, Hive)
   • Basic command line tools
   • Batch processing capability
   • Implementation of unit tests and e2e tests
   • Summarize AI-based prompts for implementing different connectors
2. Phase Two: Enhance support for more DataX data sources
   • Extend DataX connector support (PostgreSQL, ES, Kafka, etc.)
   • Version adaptation functionality
3. Phase Three: Expand support for other tools and continuous optimization
   • Sqoop support implementation
   • More advanced features

Summary

The X2SeaTunnel tool adopts a unified framework design, supporting configuration conversion from multiple data integration tools to SeaTunnel. Through its plugin-based architecture, it ensures both lightweight efficiency and good extensibility. This tool helps users migrate smoothly to the SeaTunnel platform by reducing migration costs, thereby improving data integration efficiency.

The tool provides both command line and SDK usage methods to meet requirements in different scenarios. The core design focuses on the accuracy and universality of configuration mapping, ensuring that the generated SeaTunnel configurations can be used directly. The overall architecture supports the future expansion of conversion capabilities for more data integration tools.

---

X2SeaTunnel 设计文档

背景概述

X2SeaTunnel 是一个通用配置转换工具，用于将多种数据集成工具（如 DataX、Sqoop 等）的配置文件转换为 SeaTunnel 的 HOCON 或 JSON 配置文件，帮助用户平滑迁移到 SeaTunnel 平台。

设计思路

核心理念

• 简单轻量：保持工具轻量高效，专注于配置文件格式转换
• 统一框架：构建一个通用框架，支持多种数据集成工具的配置转换
• 可扩展性：采用插件式设计，便于后续扩展支持更多工具
• 易用性：提供多种使用方式，提供SDK，提供命令行方式，支持单脚本和批量，满足不同场景需求

转换流程

源工具配置(DataX json) → 解析 → 统一模型 → 映射转换 → 生成 SeaTunnel 配置

使用方式

简单命令行方式

# 基本用法
sh bin/x2seatunnel.sh -t datax -i /path/to/config.json -o /path/to/output.conf

# 指定工具类型、输入输出和格式
sh bin/x2seatunnel.sh -t datax -i input.json -o output.conf -f hocon

# 批量转换
sh bin/x2seatunnel.sh -t datax -d /input/dir/ -o /output/dir/

Yaml命令行方式

# 使用YAML配置文件
sh bin/x2seatunnel.sh --config conversion.yaml

YAML配置文件示例

# X2SeaTunnel配置文件
metadata:
  # 配置文件格式版本
  configVersion: "1.0"
  # 描述（可选）
  description: "DataX到SeaTunnel转换配置"

# 工具配置
tool:
  # 源工具类型：datax, sqoop等
  sourceType: "datax"
  sourceVersion: "2.1.2"
  # 目标SeaTunnel版本
  targetVersion: "2.3.11"

# 输入配置
input:
  # 源配置路径（文件或目录）
  path: "/path/to/configs"
  # 是否递归处理子目录
  recursive: true
  # 文件匹配模式
  pattern: "*.json"

# 输出配置
output:
  # 输出路径
  path: "/path/to/output"
  # 输出格式：hocon或json
  format: "hocon" 
  # 文件名转换规则
  namePattern: "${filename}_seatunnel.conf"

# 映射配置
mapping:
  # 自定义映射规则路径（可选）
  rulesPath: "/path/to/custom/rules.json"
  
# 验证配置
validation:
  # 是否启用验证
  enabled: true
  # 验证失败行为：warn, error, ignore
  
# 日志配置
logging:
  # 日志级别：debug, info, warn, error
  level: "info"
  # 日志输出路径
  path: "./logs"
  # 日志文件名模式
  filePattern: "x2seatunnel-%d{yyyy-MM-dd}.log"
  # 是否同时输出到控制台
  console: true

SDK方式集成

// 创建特定工具转换器
X2SeaTunnelConverter converter = X2SeaTunnelFactory.createConverter("datax");

// 配置转换选项
ConversionOptions options = new ConversionOptions.Builder()
    .outputFormat("hocon")
    .targetVersion("2.3.11")
    .build();
    
// 执行转换
String seatunnelConfig = converter.convert(sourceConfigContent, options);

实施路线图

1. 第一阶段：基础框架及DataX支持，Mysql数据源可使用
   • 核心接口设计
   • DataX常用连接器支持(MySQL, Hive)
   • 基本命令行工具
   • 批量处理功能
   • 实现单元测试与e2e测试
   • 总结基于AI实现不同连接器的prompt。
2. 第二阶段：完善DataX更多数据源支持
   • 扩展DataX连接器支持(PostgreSQL，ES, Kafka等)
   • 版本适配功能
3. 第三阶段：扩展其他工具支持与持续优化
   • Sqoop支持实现
   • 更多高级功能

总结

X2SeaTunnel工具采用统一框架设计，支持多种数据集成工具配置向SeaTunnel的转换。通过插件式架构，既保证了工具的轻量高效，又提供了良好的扩展性。该工具通过降低迁移成本，帮助用户平滑迁移到SeaTunnel平台，提高数据集成效率。

工具同时提供命令行和SDK两种使用方式，满足不同场景需求。核心设计着重于配置映射的准确性和通用性，确保生成的SeaTunnel配置可直接使用。整体架构支持未来扩展更多数据集成工具的转换能力。

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[davidzollo]

Good feature.
I think it's okay to get one way to work first.

[Adamyuanyuan]

Thanks, I'll add the implementation ideas later. Once the discussion is roughly settled, I'll proceed to implement the first version.

[e-mhui]

Support, how can I participate in your design.

[Adamyuanyuan]

Implementation Approach

Adopting a "configuration-driven, logic-based design" can reduce code volume, lower expansion difficulty, and is suitable for migration conversion scenarios. This is because:
● The target system SeaTunnel's configuration specifications are well-defined
● We need to ensure the completeness and correctness of the migrated configuration
● We need to identify which original configurations cannot be migrated, not pursuing perfection, requiring manual processing
See the specific solution details below.

As shown above, the overall logic includes the following steps:

1. Script Invocation and Tool Triggering
Execute sh bin/x2seatunnel.sh --config conversion.yaml to call the X2Seatunnel jar package tool, which relies on the conversion.yaml configuration (optional) or command-line parameters to start the data conversion tool process.

2. Jar Package Core Initialization
When the jar package runs, it infers the SeaTunnel Connector types to be converted based on the DataX (or Sqoop, etc.) configuration files and related parameters, clarifying the component direction for conversion adaptation, laying the foundation for subsequent field matching and file conversion.

3. Rule Matching and Field Population Phase
Traverse through Connectors, extract and populate corresponding field values from DataX json files using the mapping rule library, while outputting the matching status of fields and Connectors, clearly identifying the adaptation status of various elements during the conversion process.

4. Conversion Output Phase
4.1 Configuration File Conversion
Transform Connector objects into HOCON/JSON files suitable for SeaTunnel, outputting to the specified directory;
4.2 Conversion Report Generation
Generate conversion reports, recording conversion details and matching results; for manual inspection and confirmation, ensuring conversion quality.

5. Rule Iteration Phase
Based on actual conversion scenarios, continuously improve the mapping rule library, cover more data conversion requirements, optimize the adaptation capability of the X2Seatunnel tool, making the process more accurate and efficient across diverse scenarios.
As the rule engine gradually becomes stable through iteration, adding new conversion rules only requires modifying the mapping rule library to quickly add conversion for new types of data sources.
By using summarized prompts, AI large models can quickly generate mapping rules.
The entire process, driven by rules and verified by manual checks, assists in migrating data synchronization tasks to SeaTunnel, supporting tool functionality implementation and iteration.

---

实现思路

采用“配置驱动、取用逻辑的设计”，可以减少代码量，降低扩展难度，适合迁移转换场景。因为：
● 目标系统Seatunnel的配置规范是确定的
● 需要确保迁移后配置的完整性和正确性
● 需要识别哪些原有配置无法迁移，不追求完美，需要人工处理
具体选型依据见后文。

如上图，整体逻辑包含如下几步：

1. 脚本调用与工具触发
执行 sh bin/x2seatunnel.sh --config conversion.yaml ，调用 X2Seatunnel jar包工具，依托 conversion.yaml 配置（可选）或命令行参数，启动数据转换工具流程 。
2. Jar 包核心初始化
Jar 包运行时，根据 DataX(或sqoop等)的配置文件，以及相关参数，推断待转换的 SeaTunnel Connector 类型，明确转换适配的组件方向，为后续字段匹配、文件转换奠定基础。
3. 规则匹配与字段填充阶段
遍历 Connector，借助映射规则库，从 DataX 的 json 文件中提取并填充对应字段值，同时输出字段、Connector 的匹配情况，明确转换过程中各元素的适配状态。
4. 转换输出阶段
4.1 配置文件转换
将 Connector 对象转化为 SeaTunnel 适用的 HOCON/JSON 文件，输出到指定目录；
4.2 输出转换报告
生成转换报告（convert report），记录转换详情与匹配结果；供人工检查和确认，保障转换质量。
5. 规则迭代阶段
基于实际转换场景，可持续完善映射规则库，覆盖更多数据转换需求，优化 X2Seatunnel 工具的适配能力，让流程在多样场景下更精准、高效。
待规则引擎逐步迭代稳定后，后续新增转换规则，只需要修改映射规则库，即可快速添加新类型数据源的转换。
通过总结的prompt，可以利用AI大模型，快速生成映射规则。
整个流程通过规则驱动、人工校验，助力数据同步任务向 SeaTunnel 迁移，支撑工具功能落地与迭代 。

[Adamyuanyuan]

Three Implementation Approaches

There are several implementation methods for X2Seatunnel, primarily these three approaches:

1. Object Mapping Route: Strong typing, conversion through object models, coding-focused
2. Declarative Mapping Logic (Push-based): Traversing source configuration, mapping to target, configuration-focused
3. Retrieval Logic (Pull-based): Traversing target requirements, retrieving from source, template-focused

The following table illustrates the characteristics of different implementation approaches:

Characteristic	Object Mapping Route	Declarative Mapping Logic (Push-based)	Retrieval Logic (Pull-based)
Basic Principle	DataX JSON → DataX Object → SeaTunnel Object → SeaTunnel JSON	DataX JSON → Traverse source config keys → Map to target keys → SeaTunnel JSON	DataX JSON → Traverse target required keys → Retrieve values from source → SeaTunnel JSON
Type Safety	✅ Strong typing, compile-time checking	❌ Weak typing, runtime checking	❌ Weak typing, runtime checking
Extension Difficulty	❌ High (requires object model definition for each tool) Leads to very large code volume	✅ Low (only requires adding mapping configurations)	✅ Low (only requires adding template configurations) But demands strong abstraction capability in the core framework
Complex Conversions	✅ Java code handles complex logic	❌ Difficult to handle complex logic	🟡 Can be processed through converters or through rule specifications
Configuration Completeness	🟡 Depends on development implementation	❌ May omit target configuration items	✅ Naturally ensures target configuration completeness
Error Detection	✅ Checkable at compile-time	❌ Only checkable at runtime	✅ Can check required fields in advance
Mapping Direction	Source → Target (indirect)	Source → Target (direct)	Target → Source (reverse)

Essential Differences Between Implementation Approaches

1. Object Mapping Route: Strong typing, conversion through object models, coding-focused

DataXConfig dataX = JsonUtils.parse(jsonStr, DataXConfig.class);
SeaTunnelConfig st = converter.convert(dataX);
String stJson = JsonUtils.toString(st);

2. Declarative Mapping Logic (Push-based): Traversing source configuration, mapping to target, configuration-focused

// Traverse each field in the source configuration
for (String srcPath : mappingRules.keySet()) {
    String targetPath = mappingRules.get(srcPath);
    Object value = JsonPath.read(sourceJson, srcPath);
    JsonPath.set(targetJson, targetPath, value);
}

3. Retrieval Logic (Pull-based): Traversing target requirements, retrieving from source, template-focused

// Traverse each field needed in the target template
for (TemplateField field : targetTemplate.getFields()) {
    String sourcePath = field.getSourcePath();
    Object value = sourcePath != null ? 
        JsonPath.read(sourceJson, sourcePath) : field.getDefault();
    targetJson.put(field.getName(), value);
}

Essential Difference Between Push-based and Pull-based

These two methods may seem similar (both use mapping engines), but their directions are completely opposite:

• Push-based: Starting from the source, "I give you what I have", may omit target fields
• Pull-based: Starting from the target, "I take what I need from you", ensures target completeness

Best Practice Recommendations

Based on analysis, a hybrid approach is most suitable, combining the advantages of all three approaches:

1. Using pull-based mapping as the core: Ensures completeness of target configuration

# Template-driven mapping configuration
seatunnel_mysql_source:
  required_fields:
    url:
      source_path: "job.content[0].reader.parameter.connection[0].jdbcUrl[0]"

2. Using object handling for complex conversions: Handles conversions requiring programming logic

// Complex SQL generation and conversion logic
String buildSelectQuery(String tableName, List<String> columns, String where) {
  // Complex logic implementation
}

3. Configuration-driven extensions: Adding tool support mainly through configuration files

Conclusion

We recommend adopting a hybrid approach with "pull-based mapping" as the core, supplemented by a small amount of object mapping to handle complex logic. This approach ensures the completeness of the target configuration while maintaining good extensibility and maintainability, and can also handle complex conversion scenarios.

---

三种实现思路探讨

X2Seatunnel的实现方式有很多种，主要有以下三种实现方式：

1. 对象映射路线：强类型，通过对象模型转换，编码为主
2. 声明映射逻辑（推送式）：遍历源配置，映射到目标，配置为主
3. 取用逻辑（拉取式）：遍历目标需求，从源获取，模板为主

下面用一个表格来说明不同实现思路的特点：

特点	对象映射路线	声明映射逻辑（推送式）	取用逻辑（拉取式）
基本原理	DataX JSON → DataX对象 → SeaTunnel对象 → SeaTunnel JSON	DataX JSON → 遍历源配置key → 映射到目标key → SeaTunnel JSON	DataX JSON → 遍历目标需要的key → 从源取值 → SeaTunnel JSON
类型安全	✅ 强类型，编译期检查	❌ 弱类型，运行时检查	❌ 弱类型，运行时检查
扩展难度	❌ 高（需要为每种工具定义对象模型） 会导致代码量特别大	✅ 低（只需添加映射配置）	✅ 低（只需添加模板配置） 但对核心框架的抽象能力有要求
复杂转换	✅ Java代码处理复杂逻辑	❌ 较难处理复杂逻辑	🟡 可通过转换器处理 或者通过一些规则规范
配置完整性	🟡 取决于开发实现	❌ 可能遗漏目标配置项	✅ 天然确保目标配置完整性
错误检测	✅ 编译期可检查	❌ 运行时才能检查	✅ 可提前检查必填字段
映射方向	源→目标（间接）	源→目标（直接）	目标→源（反向）

各实现思路本质区别

1. 对象映射路线：强类型，通过对象模型转换，编码为主

DataXConfig dataX = JsonUtils.parse(jsonStr, DataXConfig.class);
SeaTunnelConfig st = converter.convert(dataX);
String stJson = JsonUtils.toString(st);

2. 声明映射逻辑（推送式）：遍历源配置，映射到目标，配置为主

// 遍历源配置中的每个字段
for (String srcPath : mappingRules.keySet()) {
    String targetPath = mappingRules.get(srcPath);
    Object value = JsonPath.read(sourceJson, srcPath);
    JsonPath.set(targetJson, targetPath, value);
}

3. 取用逻辑（拉取式）：遍历目标需求，从源获取，模板为主

// 遍历目标模板中需要的每个字段
for (TemplateField field : targetTemplate.getFields()) {
    String sourcePath = field.getSourcePath();
    Object value = sourcePath != null ? 
        JsonPath.read(sourceJson, sourcePath) : field.getDefault();
    targetJson.put(field.getName(), value);
}

推送式与拉取式的本质区别

这两种方式看似相似（都用映射引擎），但方向完全相反：

• 推送式：从源出发，"我有什么给你什么"，可能遗漏目标字段
• 拉取式：从目标出发，"我需要什么从你那拿什么"，确保目标完整

最佳实践建议

根据分析，混合方案最为合适，结合三种思路的优点：

1. 以拉取式映射为核心：确保目标配置的完整性

# 模板驱动的映射配置
seatunnel_mysql_source:
  required_fields:
    url:
      source_path: "job.content[0].reader.parameter.connection[0].jdbcUrl[0]"

2. 复杂转换用对象处理：处理需要编程逻辑的转换
   这个到时候具体看，我觉得基于简单的字符串拼接规则应该就ok了。

3. 配置驱动扩展：新增工具支持主要通过配置文件

结论

推荐采用以"拉取式映射"为核心，辅以少量对象映射处理复杂逻辑的混合方案。这种方式既确保了目标配置的完整性，又保持了良好的扩展性和维护性，同时能够应对复杂的转换场景。

[Zzih]

I have developed a project for converting datax to st configuration before. Supports fewer connectors. I can contribute it for reference.

[yujiaji03]

Very good suggestion. I've developed a small tool to convert DataX JSON into SparkSQL, extending it to Datatunnel statements, and I hope to get involved.

[Shlpeng]

Has any development work started on this feature yet?

[Adamyuanyuan]

#9664

[Adamyuanyuan]

Has any development work started on this feature yet?

Yes, I have completed the development. It's ready to use out of the box with complete functionality. And with the help of this tool, We have converted over 1,000 DataX tasks.