Skip to content

Commit ab695e3

Browse files
sunxiaojiansunxiaojian
committed
Supports SQLServer cdc
1 parent 60d7ca1 commit ab695e3

37 files changed

+4914
-9
lines changed

‎docs/content/cdc-ingestion/overview.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,8 @@ We currently support the following sync ways:
3838
5. Kafka Synchronizing Database: synchronize one Kafka topic containing multiple tables or multiple topics containing one table each into one Paimon database.
3939
6. MongoDB Synchronizing Collection: synchronize one Collection from MongoDB into one Paimon table.
4040
7. MongoDB Synchronizing Database: synchronize the whole MongoDB database into one Paimon database.
41-
41+
8. SQLServer Synchronizing Table: synchronize one or multiple tables from SQLServer into one Paimon table.
42+
9. SQLServer Synchronizing Database: synchronize the whole SQLServer database into one Paimon database.
4243
## What is Schema Evolution
4344

4445
Suppose we have a MySQL table named `tableA`, it has three fields: `field_1`, `field_2`, `field_3`. When we want to load

‎docs/content/cdc-ingestion/sqlserver-cdc.md

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
---
2+
title: "SQLServer CDC"
3+
weight: 2
4+
type: docs
5+
aliases:
6+
- /cdc-ingestion/sqlserver-cdc.html
7+
---
8+
<!--
9+
Licensed to the Apache Software Foundation (ASF) under one
10+
or more contributor license agreements. See the NOTICE file
11+
distributed with this work for additional information
12+
regarding copyright ownership. The ASF licenses this file
13+
to you under the Apache License, Version 2.0 (the
14+
"License"); you may not use this file except in compliance
15+
with the License. You may obtain a copy of the License at
16+
17+
http://www.apache.org/licenses/LICENSE-2.0
18+
19+
Unless required by applicable law or agreed to in writing,
20+
software distributed under the License is distributed on an
21+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22+
KIND, either express or implied. See the License for the
23+
specific language governing permissions and limitations
24+
under the License.
25+
-->
26+
27+
# SQLServer CDC
28+
29+
Paimon supports synchronizing changes from different databases using change data capture (CDC). This feature requires Flink and its [CDC connectors](https://ververica.github.io/flink-cdc-connectors/).
30+
31+
## Prepare CDC Bundled Jar
32+
33+
```
34+
flink-sql-connector-sqlserver-cdc-*.jar
35+
```
36+
37+
## Synchronizing Tables
38+
39+
By using [SqlServerSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/sqlserver/SqlServerSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from SQLServer into one Paimon table.
40+
41+
To use this feature through `flink run`, run the following shell command.
42+
43+
```bash
44+
<FLINK_HOME>/bin/flink run \
45+
/path/to/paimon-flink-action-{{< version >}}.jar \
46+
sqlserver-sync-table
47+
--warehouse <warehouse-path> \
48+
--database <database-name> \
49+
--table <table-name> \
50+
[--partition-keys <partition-keys>] \
51+
[--primary-keys <primary-keys>] \
52+
[--type-mapping <option1,option2...>] \
53+
[--computed-column <'column-name=expr-name(args[, ...])'> [--computed-column ...]] \
54+
[--metadata-column <metadata-column>] \
55+
[--sqlserver-conf <sqlserver-cdc-source-conf> [--sqlserver-conf <sqlserver-cdc-source-conf> ...]] \
56+
[--catalog-conf <paimon-catalog-conf> [--catalog-conf <paimon-catalog-conf> ...]] \
57+
[--table-conf <paimon-table-sink-conf> [--table-conf <paimon-table-sink-conf> ...]]
58+
```
59+
60+
{{< generated/sqlserver_sync_table >}}
61+
62+
Currently, only one database is supported for synchronization. Regular matching of 'database name' is not supported.
63+
64+
If the Paimon table you specify does not exist, this action will automatically create the table. Its schema will be derived from all specified SQLServer tables. If the Paimon table already exists, its schema will be compared against the schema of all specified SQLServer tables.
65+
66+
Example 1: synchronize tables into one Paimon table
67+
68+
```bash
69+
<FLINK_HOME>/bin/flink run \
70+
/path/to/paimon-flink-action-{{< version >}}.jar \
71+
sqlserver-sync-table \
72+
--warehouse hdfs:///path/to/warehouse \
73+
--database test_db \
74+
--table test_table \
75+
--partition-keys pt \
76+
--primary-keys pt,uid \
77+
--computed-column '_year=year(age)' \
78+
--sqlserver-conf hostname=127.0.0.1 \
79+
--sqlserver-conf username=root \
80+
--sqlserver-conf password=123456 \
81+
--sqlserver-conf database-name='source_db' \
82+
--sqlserver-conf schema-name='dbo' \
83+
--sqlserver-conf table-name='dbo.source_table1|dbo.source_table2' \
84+
--catalog-conf metastore=hive \
85+
--catalog-conf uri=thrift://hive-metastore:9083 \
86+
--table-conf bucket=4 \
87+
--table-conf changelog-producer=input \
88+
--table-conf sink.parallelism=4
89+
```
90+
91+
As example shows, the sqlserver-conf's table-name supports regular expressions to monitor multiple tables that satisfy
92+
the regular expressions. The schemas of all the tables will be merged into one Paimon table schema.
93+
94+
Example 2: synchronize shards into one Paimon table
95+
96+
You can also use regular expressions to set the "schema_name" to capture multiple schemas. A typical scenario is to split the table "source_table" into databases "source_dbo1" and "source_dbo2"..., Then all the data of "source_table" can be synchronized to a Paimon table.
97+
98+
```bash
99+
<FLINK_HOME>/bin/flink run \
100+
/path/to/paimon-flink-action-{{< version >}}.jar \
101+
mysql-sync-table \
102+
--warehouse hdfs:///path/to/warehouse \
103+
--database test_db \
104+
--table test_table \
105+
--partition-keys pt \
106+
--primary-keys pt,uid \
107+
--computed-column '_year=year(age)' \
108+
--sqlserver-conf hostname=127.0.0.1 \
109+
--sqlserver-conf username=root \
110+
--sqlserver-conf password=123456 \
111+
--sqlserver-conf database-name='source_db' \
112+
--sqlserver-conf schema-name='source_dbo.+' \
113+
--sqlserver-conf table-name='source_table' \
114+
--catalog-conf metastore=hive \
115+
--catalog-conf uri=thrift://hive-metastore:9083 \
116+
--table-conf bucket=4 \
117+
--table-conf changelog-producer=input \
118+
--table-conf sink.parallelism=4
119+
```
120+
121+
## Synchronizing Databases
122+
123+
By using [SqlServerSyncDatabaseAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/mysql/SqlServerSyncDatabaseAction) in a Flink DataStream job or directly through `flink run`, users can synchronize the whole SQLServer database into one Paimon database.
124+
125+
To use this feature through `flink run`, run the following shell command.
126+
127+
```bash
128+
<FLINK_HOME>/bin/flink run \
129+
/path/to/paimon-flink-action-{{< version >}}.jar \
130+
sqlserver-sync-database
131+
--warehouse <warehouse-path> \
132+
--database <database-name> \
133+
[--ignore-incompatible <true/false>] \
134+
[--merge-shards <true/false>] \
135+
[--table-prefix <paimon-table-prefix>] \
136+
[--table-suffix <paimon-table-suffix>] \
137+
[--including-tables <sqlserver-table-name|name-regular-expr>] \
138+
[--excluding-tables <sqlserver-table-name|name-regular-expr>] \
139+
[--mode <sync-mode>] \
140+
[--metadata-column <metadata-column>] \
141+
[--type-mapping <option1,option2...>] \
142+
[--sqlserver-conf <sqlserver-cdc-source-conf> [--sqlserver-conf <sqlserver-cdc-source-conf> ...]] \
143+
[--catalog-conf <paimon-catalog-conf> [--catalog-conf <paimon-catalog-conf> ...]] \
144+
[--table-conf <paimon-table-sink-conf> [--table-conf <paimon-table-sink-conf> ...]]
145+
```
146+
147+
{{< generated/sqlserver_sync_database >}}
148+
149+
Currently, only one database is supported for synchronization. Regular matching of 'database_name' is not supported
150+
151+
Only tables with primary keys will be synchronized.
152+
153+
For each SQLServer table to be synchronized, if the corresponding Paimon table does not exist, this action will automatically create the table. Its schema will be derived from all specified SQLServer tables. If the Paimon table already exists, its schema will be compared against the schema of all specified SQLServer tables.
154+
155+
Example 1: synchronize entire database
156+
157+
```bash
158+
<FLINK_HOME>/bin/flink run \
159+
/path/to/paimon-flink-action-{{< version >}}.jar \
160+
sqlserver-sync-database \
161+
--warehouse hdfs:///path/to/warehouse \
162+
--database test_db \
163+
--sqlserver-conf hostname=127.0.0.1 \
164+
--sqlserver-conf username=root \
165+
--sqlserver-conf password=123456 \
166+
--sqlserver-conf database-name=source_db \
167+
--sqlserver-conf schema-name=dbo \
168+
--catalog-conf metastore=hive \
169+
--catalog-conf uri=thrift://hive-metastore:9083 \
170+
--table-conf bucket=4 \
171+
--table-conf changelog-producer=input \
172+
--table-conf sink.parallelism=4
173+
```
174+
175+
Example 2: synchronize and merge multiple shards
176+
177+
Let's say you have multiple schema shards `schema1`, `schema2`, ... and each schema has tables `tbl1`, `tbl2`, .... You can
178+
synchronize all the `schema.+.tbl.+` into tables `test_db.tbl1`, `test_db.tbl2` ... by following command:
179+
180+
```bash
181+
<FLINK_HOME>/bin/flink run \
182+
/path/to/paimon-flink-action-{{< version >}}.jar \
183+
sqlserver-sync-database \
184+
--warehouse hdfs:///path/to/warehouse \
185+
--database test_db \
186+
--sqlserver-conf hostname=127.0.0.1 \
187+
--sqlserver-conf username=root \
188+
--sqlserver-conf password=123456 \
189+
--sqlserver-conf database-name='source_db' \
190+
--sqlserver-conf schema-name='db.+' \
191+
--catalog-conf metastore=hive \
192+
--catalog-conf uri=thrift://hive-metastore:9083 \
193+
--table-conf bucket=4 \
194+
--table-conf changelog-producer=input \
195+
--table-conf sink.parallelism=4 \
196+
--including-tables 'tbl.+'
197+
```
198+
199+
By setting schema-name to a regular expression, the synchronization job will capture all tables under matched schemas
200+
and merge tables of the same name into one table.
201+
202+
{{< hint info >}}
203+
You can set `--merge-shards false` to prevent merging shards. The synchronized tables will be named to 'databaseName_tableName'
204+
to avoid potential name conflict.
205+
{{< /hint >}}
206+
207+
## FAQ
208+
209+
1. Chinese characters in records ingested from MySQL are garbled.
210+
* Try to set `env.java.opts: -Dfile.encoding=UTF-8` in `flink-conf.yaml`
211+
(the option is changed to `env.java.opts.all` since Flink-1.17).

‎docs/layouts/shortcodes/generated/sqlserver_sync_database.html

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
{{/*
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
*/}}
19+
{{ $ref := ref . "maintenance/configurations.md" }}
20+
<table class="configuration table table-bordered">
21+
<thead>
22+
<tr>
23+
<th class="text-left" style="width: 15%">Configuration</th>
24+
<th class="text-left" style="width: 85%">Description</th>
25+
</tr>
26+
</thead>
27+
<tbody>
28+
<tr>
29+
<td><h5>--warehouse</h5></td>
30+
<td>The path to Paimon warehouse.</td>
31+
</tr>
32+
<tr>
33+
<td><h5>--database</h5></td>
34+
<td>The database name in Paimon catalog.</td>
35+
</tr>
36+
<tr>
37+
<td><h5>--ignore-incompatible</h5></td>
38+
<td>It is default false, in this case, if SQLServer table name exists in Paimon and their schema is incompatible,an exception will be thrown. You can specify it to true explicitly to ignore the incompatible tables and exception.</td>
39+
</tr>
40+
<tr>
41+
<td><h5>--merge-shards</h5></td>
42+
<td>It is default true, in this case, if some tables in different schemas have the same name, their table schemas will be merged and their records will be synchronized into one Paimon table. Otherwise, each table's records will be synchronized to a corresponding Paimon table, and the Paimon table will be named to 'databaseName_schemaName_tableName' to avoid potential name conflict.</td>
43+
</tr>
44+
<tr>
45+
<td><h5>--table-prefix</h5></td>
46+
<td>The prefix of all Paimon tables to be synchronized. For example, if you want all synchronized tables to have "ods_" as prefix, you can specify "--table-prefix ods_".</td>
47+
</tr>
48+
<tr>
49+
<td><h5>--table-suffix</h5></td>
50+
<td>The suffix of all Paimon tables to be synchronized. The usage is same as "--table-prefix".</td>
51+
</tr>
52+
<tr>
53+
<td><h5>--including-tables</h5></td>
54+
<td>It is used to specify which source tables are to be synchronized. You must use '|' to separate multiple tables.Because '|' is a special character, a comma is required, for example: 'a|b|c'.Regular expression is supported, for example, specifying "--including-tables test|paimon.*" means to synchronize table 'test' and all tables start with 'paimon'.</td>
55+
</tr>
56+
<tr>
57+
<td><h5>--excluding-tables</h5></td>
58+
<td>It is used to specify which source tables are not to be synchronized. The usage is same as "--including-tables". "--excluding-tables" has higher priority than "--including-tables" if you specified both.</td>
59+
</tr>
60+
<tr>
61+
<td><h5>--mode</h5></td>
62+
<td>It is used to specify synchronization mode.<br />Possible values:<ul><li>"divided" (the default mode if you haven't specified one): start a sink for each table, the synchronization of the new table requires restarting the job.</li><li>"combined": start a single combined sink for all tables, the new table will be automatically synchronized.</li></ul></td>
63+
</tr>
64+
<tr>
65+
<td><h5>--metadata-column</h5></td>
66+
<td>--metadata-column is used to specify which metadata columns to include in the output schema of the connector. Metadata columns provide additional information related to the source data, such as the `table_name`, `schema_name`, `database_name`, and `op_ts`. Each configuration should be specified in the format "key=value". See its <a href="https://ververica.github.io/flink-cdc-connectors/master/content/connectors/sqlserver-cdc.html#available-metadata">document</a> for a complete list of available metadata.</td>
67+
</tr>
68+
<tr>
69+
<td><h5>--type-mapping</h5></td>
70+
<td>It is used to specify how to map SQLServer data type to Paimon type.<br />
71+
Supported options:
72+
<ul>
73+
<li>"tinyint1-not-bool": maps SQLServer TINYINT(1) to TINYINT instead of BOOLEAN.</li>
74+
<li>"to-nullable": ignores all NOT NULL constraints (except for primary keys).</li>
75+
<li>"to-string": maps all SQLServer types to STRING.</li>
76+
<li>"char-to-string": maps SQLServer CHAR(length)/VARCHAR(length)/NCHAR(length)/NVARCHAR(length) types to STRING.</li>
77+
</ul>
78+
</td>
79+
</tr>
80+
<tr>
81+
<td><h5>--sqlserver-conf</h5></td>
82+
<td>The configuration for Flink CDC SQLServer sources. Each configuration should be specified in the format "key=value". hostname, username, password, database-name, schema-name and table-name are required configurations, others are optional. See its <a href="https://ververica.github.io/flink-cdc-connectors/master/content/connectors/sqlserver-cdc.html#connector-options">document</a> for a complete list of configurations.</td>
83+
</tr>
84+
<tr>
85+
<td><h5>--catalog-conf</h5></td>
86+
<td>The configuration for Paimon catalog. Each configuration should be specified in the format "key=value". See <a href="{{ $ref }}">here</a> for a complete list of catalog configurations.</td>
87+
</tr>
88+
<tr>
89+
<td><h5>--table-conf</h5></td>
90+
<td>The configuration for Paimon table sink. Each configuration should be specified in the format "key=value". See <a href="{{ $ref }}">here</a> for a complete list of table configurations.</td>
91+
</tr>
92+
</tbody>
93+
</table>

0 commit comments

Comments
 (0)