使用 oracle cdc抽取数据到doris，同步到3500w条数据（17.8G），程序同步停止，如何解决？

[austinuMyaccount]

运行环境 Centos7
source 端
DB oracle 11g
connnector oracle cdc 2.3.0
sink 端
DB doris 1.3.0
connector flink-doris-connector-1.16
程序运行的 flink web ui 如下图所示

说明，并行度只能为1

[PatrickRen]

@austinuMyaccount Could you provide more details about the issue, like code snippet, exception, logs or steps to reproduce? It's hard to give any clue with information you provided.

[austinuMyaccount]

ok, there are 3 kinds of log detials about the issue as follow :
1.taskmanager log
2.jobmanager log
3.doris FE log

1.taskmanager log
2023-01-31 18:13:29,743 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Exported 35680300 records for table 'ORCL.BMCCCNP.NO_DETAIL_B' after 00:34:30.809
2023-01-31 18:13:39,744 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Exported 35857553 records for table 'ORCL.BMCCCNP.NO_DETAIL_B' after 00:34:40.81
2023-01-31 18:13:42,518 INFO org.apache.http.impl.execchain.RetryExec [] - I/O exception (java.net.SocketException) caught when processing request to {}->http://192.168.0.221:8040: 断开的管道

2.jobmanager log
2023-01-31 18:05:41,197 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint 2 of job 382d58ea4105345aacefc223fd0f70ec expired before completing.
2023-01-31 18:05:41,198 WARN org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 2 for job 382d58ea4105345aacefc223fd0f70ec. (1 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2165) [FlinkCDC-1.0.0-jar-with-dependencies.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_60]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_60]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
2023-01-31 18:05:41,201 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 3 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1675159541199 for job 382d58ea4105345aacefc223fd0f70ec.
2023-01-31 18:15:41,201 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint 3 of job 382d58ea4105345aacefc223fd0f70ec expired before completing.
2023-01-31 18:15:41,202 WARN org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 3 for job 382d58ea4105345aacefc223fd0f70ec. (2 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2165) [FlinkCDC-1.0.0-jar-with-dependencies.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_60]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_60]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
2023-01-31 18:15:41,204 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 4 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1675160141203 for job 382d58ea4105345aacefc223fd0f70ec.

3.doris FE log
2023-01-31 17:21:25,191 WARN (mysql-nio-pool-27|7154) [StmtExecutor.execute():587] execute Exception. stmt[235, 56b184c834bd4d53-b3347aa5f5512abb], errCode = 2, detailMessage = Unknown table 'PROFILING'
2023-01-31 17:51:28,331 WARN (connect-scheduler-check-timer-0|105) [ConnectContext.checkTimeout():562] kill wait timeout connection, remote: 192.168.0.116:48915, wait timeout: 28800
2023-01-31 17:51:28,332 WARN (connect-scheduler-check-timer-0|105) [ConnectContext.kill():532] kill query from 192.168.0.116:48915, kill connection: true
2023-01-31 17:58:00,331 WARN (connect-scheduler-check-timer-0|105) [ConnectContext.checkTimeout():562] kill wait timeout connection, remote: 192.168.0.116:48916, wait timeout: 28800
2023-01-31 17:58:00,331 WARN (connect-scheduler-check-timer-0|105) [ConnectContext.kill():532] kill query from 192.168.0.116:48916, kill connection: true

[Hub-Wling]

解决了吗？遇到一样的问题

[HGHNice]

请问下 是调整谁的哪个参数呀

[jueding920]

我也遇到了，请问如何解决？

[HGHNice]

我也遇到了，请问如何解决？
我是修改flink的相关参数，doris的be和fe的有关load参数都加大，看你机器磁盘性能。这样得到解决

[jueding920]

我也遇到了，请问如何解决？
我是修改flink的相关参数，doris的be和fe的有关load参数都加大，看你机器磁盘性能。这样得到解决

能说下具体的参数吗？

[HGHNice]

我也遇到了，请问如何解决？
我是修改flink的相关参数，doris的be和fe的有关load参数都加大，看你机器磁盘性能。这样得到解决

能说下具体的参数吗？

https://doris.apache.org/zh-CN/docs/admin-manual/config/be-config/ 搜索load的相关的 可以扩大streaming_load_max_mb像这种参数等