Skip to content

Conversation

@Adamyuanyuan
Copy link
Contributor

Purpose of this pull request

X2SeaTunnel is a tool for converting DataX and other configuration files to SeaTunnel configuration files, designed to help users quickly migrate from other data integration platforms to SeaTunnel.
This is the implementation of the first version.

Does this PR introduce any user-facing change?

Added a new tool with the following functions:

  • Standard Configuration Conversion: DataX → SeaTunnel configuration file conversion
  • Custom Template Conversion: Support for user-defined conversion templates
  • Detailed Conversion Reports: Generate Markdown format conversion reports
  • Regular Expression Variable Extraction: Extract variables from configuration using regex, supporting custom scenarios
  • Batch Conversion Mode: Support directory and file wildcard batch conversion, automatic report and summary report generation

Basic Usage

# Standard conversion: Use default template system with built-in common Sources and Sinks
./bin/x2seatunnel.sh -s examples/source/datax-mysql2hdfs.json -t examples/target/mysql2hdfs-result.conf -r examples/report/mysql2hdfs-report.md

# Custom task: Implement customized conversion requirements through custom templates
# Scenario: MySQL → Hive (DataX doesn't have HiveWriter)
# DataX configuration: MySQL → HDFS Custom task: Convert to MySQL → Hive
./bin/x2seatunnel.sh -s examples/source/datax-mysql2hdfs2hive.json -t examples/target/mysql2hive-result.conf -r examples/report/mysql2hive-report.md -T templates/datax/custom/mysql-to-hive.conf

# YAML configuration method (equivalent to above command line parameters)
./bin/x2seatunnel.sh -c examples/yaml/datax-mysql2hdfs2hive.yaml

# Batch conversion mode: Process by directory
./bin/x2seatunnel.sh -d examples/source -o examples/target2 -R examples/report2

# Batch mode supports wildcard filtering
./bin/x2seatunnel.sh -d examples/source -o examples/target3 -R examples/report3 --pattern "*-full.json" --verbose

# View help
./bin/x2seatunnel.sh --help

Conversion Report

After conversion is completed, view the generated Markdown report file, which includes:

  • Basic Information: Conversion time, source/target file paths, connector types, conversion status, etc.
  • Conversion Statistics: Counts and percentages of direct mappings, smart transformations, default values used, and unmapped fields
  • Detailed Field Mapping Relationships: Source values, target values, filters used for each field
  • Default Value Usage: List of all fields using default values
  • Unmapped Fields: Fields present in DataX but not converted
  • Possible Error and Warning Information: Issue prompts during conversion process

For batch conversions, a batch summary report summary.md will be generated in the batch report directory, including:

  • Conversion Overview: Overall statistics, success rate, duration, etc.
  • Successful Conversion List: Complete list of successfully converted files
  • Failed Conversion List: Failed files and error messages (if any)

How was this patch tested?

Yes, including unit tests, local tests, and production tests.

Check list

@Adamyuanyuan
Copy link
Contributor Author

apache/seatunnel#9507

you need to open this annotation and change the dependency of config-shade to project.
<module>seatunnel-config</module>
-->
<module>seatunnel</module>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert this. We should use sub module seatunnel as dependency, not release version.

pom.xml Outdated

<!-- External dependency versions for submodules (x2seatunnel) -->
<!-- SeaTunnel released artifacts used by tools (adjust as needed or override with -Dseatunnel.version=...) -->
<seatunnel.version>2.3.11</seatunnel.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use sub module seatunnel.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, thanks

@Hisoka-X Hisoka-X merged commit 5c1ced1 into apache:main Aug 25, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants