[Improvement]: Upgrade to iceberg 1.6.0 (#3084)

[czy006]

Why are the changes needed?

#3084.

Brief change log

• Remove Flink 1.15/1.16/1.17 Mixed Format Support
• Mixed Format Flink 1.18 Base Support Iceberg 1.6.X
• Upgrade iceberg to 1.6.X
• Fix some Mixed Format Unit Test Failed

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible
• Add screenshots for manual tests if appropriate
• Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? (yes)
• If yes, how is the feature documented? (not documented)

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

ConradJam seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[czy006]

• https://github.com/apache/iceberg/blob/main/flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java#L161
• https://github.com/apache/iceberg/blob/apache-iceberg-1.4.3/flink/v1.15/flink/src/main/java/org/apache/iceberg/flink/source/ScanContext.java#L149

Diff ScanContext in Mixed Format , Run TestFlinkSource.testUnkeyedSnapshotRead Have Exception

Do We need to set end-snapshot-id must be in streaming reader? cc @zhoujinsong

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 23.17%. Comparing base (9def89f) to head (47781b0).
Report is 8 commits behind head on master.

Files	Patch %	Lines
...java/org/apache/amoro/table/BasicUnkeyedTable.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3071      +/-   ##
============================================
- Coverage     27.64%   23.17%   -4.47%     
+ Complexity     2496     2330     -166     
============================================
  Files           360      368       +8     
  Lines         37133    37914     +781     
  Branches       5372     5456      +84     
============================================
- Hits          10265     8787    -1478     
- Misses        25908    28353    +2445     
+ Partials        960      774     -186     

Flag	Coverage Δ
trino	23.17% <0.00%> (-4.47%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[czy006]

ops,some time ci was blocker have long time.Maybe we should check

[baiyangtx]

It looks like there will be many engine versions that will no longer be supported.

@zhoujinsong PTAL

[zhoujinsong]

Maybe we should follow Iceberg 1.6, support:

• Flink 1.17/1.18/1.19
• Spark 3.3/3.4/3.5

What do you think?

[czy006]

Details

This will be supported in the future PR, Flink 1.20 as the LTS long-term support version, I recommend as the default version option.

Maybe we should follow Iceberg 1.6, support:

• Flink 1.17/1.18/1.19
• Spark 3.3/3.4/3.5

What do you think?

This will be support in the future PR, Flink 1.20 as the LTS long-term support version, I recommend as the default version option.Now Default Spark is 3.3,Flink is 1.18

[czy006]

Optimizing Task Run Normal

[zhongqishang]

I think it is not a good idea to remove so many versions support.
What is the reason for deleting the support? Is it just that the official does not support it?

Maybe we can keep the combination of old flink and old iceberg versions. @zhoujinsong @baiyangtx @czy006

And It is best to consult users on the mailing list or wechat group and make unsupported plans.

[czy006]

I think it is not a good idea to remove so many versions support. What is the reason for deleting the support? Is it just that the official does not support it?

Maybe we can keep the combination of old flink and old iceberg versions. @zhoujinsong @baiyangtx @czy006

mixed format relies on the spark and flink versions, and currently the iceberg community only supports the latest 3 versions. For example, they have to give up spark 3.2 their support.the last support in 1.4.3, iceberg if we upgrade version, we must give up its support it.flink version seems(I will add flink1.19 1.20 in the next pr),Spark now just support 3.3

[czy006]

I have initiated relevant discussions on the email list:
https://lists.apache.org/thread/1crbstg7prjxz8xokd7dtj1o5mgv1034

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.