Skip to content

[add][plugin][icebergwriter] add icebergwriter #1208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 50 commits into from
May 16, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
a1b83cd
set hive jdbc connection loginTimeout ,fixed the hive connection time…
awol2005ex Nov 11, 2024
69f806e
rdbmsreader jdbc_url is string not list
awol2005ex Nov 25, 2024
16dcda5
S3writer: add pathStyleAccessEnabled config for fixed error 400 when …
awol2005ex Nov 25, 2024
fe0b7c8
Merge remote-tracking branch 'github/master' into awol2005x-my
awol2005ex Nov 25, 2024
796703c
[bugfix][plugin][s3writer]S3writer:don't submit upload when no record…
awol2005ex Nov 25, 2024
6d329da
Merge remote-tracking branch 'github/master' into awol2005x-my
awol2005ex Nov 26, 2024
663d9ae
S3 textwriter
awol2005ex Nov 26, 2024
a12a4f1
S3 orcwriter
awol2005ex Nov 26, 2024
357ab19
S3 parquetwriter
awol2005ex Nov 27, 2024
8667e8f
DataBaseType should not be call set method,so that the driver is erro…
awol2005ex Nov 28, 2024
d150110
add other dbs drivers
awol2005ex Nov 28, 2024
218a738
Merge branch 'master' into awol2005x-my
awol2005ex Jan 17, 2025
b697d54
[feature][plugin][paimonwriter] add paimon format file writer
awol2005ex Jan 21, 2025
d5d53f3
fixed timestamp,decimal,null type value bug
awol2005ex Jan 21, 2025
c9c14b9
Merge branch 'master' into awol2005x-my
awol2005ex Jan 21, 2025
e74ef7b
1.change author
awol2005ex Jan 22, 2025
10a7ac8
Merge branch 'master' into awol2005x-my
awol2005ex Jan 23, 2025
992d7a3
[add] add paimonwriter.json to docs
awol2005ex Jan 23, 2025
9fe9180
Merge branch 'master' into awol2005x-my
awol2005ex Feb 5, 2025
c0dedee
fixed: timestamp time zone error
awol2005ex Feb 10, 2025
4089d91
Merge branch 'master' into awol2005x-my
awol2005ex Feb 10, 2025
561c5a5
Merge branch 'master' into awol2005x-my
awol2005ex Feb 12, 2025
1665649
Merge branch 'master' into awol2005x-my
awol2005ex Feb 13, 2025
c017309
merge with master
awol2005ex Feb 13, 2025
9f738d1
Merge remote-tracking branch 'github/master' into awol2005x-my
awol2005ex Feb 14, 2025
ecc045b
[add][plugin][icebergwriter] Add support for writing files in Iceberg…
awol2005ex Feb 19, 2025
8ac19e4
[add][plugin][icebergwriter] fixed date type insert error
awol2005ex Feb 19, 2025
1444931
Merge remote-tracking branch 'github/master' into awol2005x-my
awol2005ex Feb 19, 2025
b8d8aa0
[add][plugin][icebergwriter] package.xml add fileSet
awol2005ex Feb 20, 2025
0accdef
[add][plugin][icebergwriter] add docs
awol2005ex Feb 20, 2025
b21da1b
[add][plugin][icebergwriter] add build scripts to docs
awol2005ex Feb 20, 2025
c1b876b
[add][plugin][icebergwriter] docs fixed
awol2005ex Feb 20, 2025
dd744a7
[add][plugin][icebergwriter] docs
awol2005ex Feb 20, 2025
ab8bd11
[add][plugin][icebergwriter] docs name error
awol2005ex Feb 20, 2025
a9b2fe2
[add][plugin][icebergwriter] add icebergwriter code
awol2005ex Feb 20, 2025
bab8a0c
[add][plugin][icebergwriter] change to streamreader at example
awol2005ex Feb 25, 2025
6583174
[add][plugin][icebergwriter] fixed bug when partition iceberg table
awol2005ex Mar 3, 2025
34f7e6d
[add][plugin][icebergwriter] fixed bug when partition iceberg table
awol2005ex Mar 3, 2025
b1c96a8
Merge remote-tracking branch 'origin/awol2005ex-br-icebergwriter-2025…
awol2005ex Mar 26, 2025
d598f14
Merge remote-tracking branch 'github/master' into awol2005x-my
awol2005ex Mar 26, 2025
323cf31
merge master
awol2005ex Mar 26, 2025
4c8cd9c
merge master
awol2005ex Mar 26, 2025
5ae32be
Merge remote-tracking branch 'github/master' into awol2005x-my
awol2005ex Apr 1, 2025
5a6625d
Merge remote-tracking branch 'github/master' into awol2005x-my
awol2005ex Apr 3, 2025
24ea3ce
Merge remote-tracking branch 'github/master' into awol2005x-my
awol2005ex Apr 17, 2025
0193c92
Merge branch 'master' into awol2005x-my
awol2005ex May 15, 2025
a83e9ec
merge master
awol2005ex May 15, 2025
408fa57
[bugfix][writer][icebergwriter] fix dependency errors
awol2005ex May 15, 2025
104dfae
[bugfix][writer][icebergwriter] remove profile for the jdk is upgrade…
awol2005ex May 16, 2025
15b1b18
[bugfix][writer][icebergwriter] merge master
awol2005ex May 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions docs/assets/jobs/icebergwriter.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
{
"job": {
"setting": {
"speed": {
"channel": 3
},
"errorLimit": {
"record": 0,
"percentage": 0
}
},
"content": [
{
"reader": {
"name": "streamreader",
"parameter": {
"column": [
{
"value": "1",
"type": "long"
},
{
"value": "1989-06-04 00:00:00",
"type": "timestamp"
},
{
"value": "test1",
"type": "string"
}
],
"sliceRecordCount": 1000
}
},
"writer": {
"name": "icebergwriter",
"parameter": {
"tableName": "test.test1",
"writeMode": "truncate",
"catalogType": "hadoop",
"warehouse": "s3a://pvc-91d1e2cd-4d25-45c9-8613-6c4f7bf0a4cc/iceberg",
"hadoopConfig": {
"fs.s3a.endpoint": "http://localhost:9000",
"fs.s3a.access.key": "gy0dX5lALP176g6c9fYf",
"fs.s3a.secret.key": "ReuUrCzzu5wKWAegtswoHIWV389BYl9AB1ZQbiKr",
"fs.s3a.connection.ssl.enabled": "false",
"fs.s3a.path.style.access": "true",
"fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"
}
}
}
}
]
}
}
Loading