Skip to content

Commit

Permalink
feat(io): IO Implementation using Go CDK (#176)
Browse files Browse the repository at this point in the history 
* IO Implementation using Go CDK

Signed-off-by: Loïc Alleyne <loicalleyne@gmail.com>

* fix comment

Signed-off-by: Loïc Alleyne <loicalleyne@gmail.com>

* refactor CreateBlobFileIO

Signed-off-by: Loïc Alleyne <loicalleyne@gmail.com>

* line endings to LF

* remove deprecated endpoint resolver

* unexport createBlobFileIO

* remove unused urlToBucketPath

* reinit go.mod go.sum

* run go mod tidy to update deps

* clean-up and fixing buckets

* somehow lost the license in go.mod

* fix spacing due to upgraded dependency

* missed a close for the parquet reader

* add config option to force s3 virtual addressing

---------

Signed-off-by: Loïc Alleyne <loicalleyne@gmail.com>
Co-authored-by: Daniel Wilson <dwilson@seerai.space>
Co-authored-by: Matt Topol <zotthewizard@gmail.com>
  • Loading branch information
@loicalleyne @dwilson1988 @zeroshade
3 people authored Dec 16, 2024
1 parent 88bbae3 commit a2c1d2e
Show file tree
Hide file tree
Showing 8 changed files with 609 additions and 130 deletions.
39 changes: 19 additions & 20 deletions cmd/iceberg/output_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,13 @@ var testArgs = []struct {
"snapshot-log": [ ],
"metadata-log": [ ],
"refs": { }
}`,
`Table format version | 2
Metadata location |
}`,
`Table format version | 2
Metadata location |
Table UUID | 9c12d441-03fe-4693-9a96-a0705ddf69c1
Last updated | 1602638573590
Sort Order | 0: []
Partition Spec | []
Last updated | 1602638573590
Sort Order | 0: []
Partition Spec | []
Current Schema, id=0
└──1: x: required long
Expand All @@ -79,7 +79,7 @@ Current Snapshot |
Snapshots
Properties
key | value
key | value
----------------------------------
read.split.target.size | 134217728
Expand Down Expand Up @@ -145,18 +145,18 @@ read.split.target.size | 134217728
],
"metadata-log": [{"metadata-file": "s3://bucket/.../v1.json", "timestamp-ms": 1515100}],
"refs": {"test": {"snapshot-id": 3051729675574597004, "type": "tag", "max-ref-age-ms": 10000000}}
}`,
`Table format version | 2
Metadata location |
}`,
`Table format version | 2
Metadata location |
Table UUID | 9c12d441-03fe-4693-9a96-a0705ddf69c1
Last updated | 1602638573590
Sort Order | 3: [
| 2 asc nulls-first
| bucket[4](3) desc nulls-last
| ]
Partition Spec | [
| 1000: x: identity(1)
| ]
Last updated | 1602638573590
Sort Order | 3: [
| 2 asc nulls-first
| bucket[4](3) desc nulls-last
| ]
Partition Spec | [
| 1000: x: identity(1)
| ]
Current Schema, id=1
├──1: x: required long
Expand All @@ -170,14 +170,13 @@ Snapshots
└──Snapshot 3055729675574597004, schema 1: s3://a/b/2.avro
Properties
key | value
key | value
----------------------------------
read.split.target.size | 134217728
`},
}


func TestDescribeTable(t *testing.T) {
var buf bytes.Buffer
pterm.SetDefaultOutput(&buf)
Expand Down
71 changes: 50 additions & 21 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,33 +24,42 @@ toolchain go1.23.2
require (
github.com/apache/arrow-go/v18 v18.0.1-0.20241029153821-f0c5d9939d3f
github.com/aws/aws-sdk-go-v2 v1.32.6
github.com/aws/aws-sdk-go-v2/config v1.28.5
github.com/aws/aws-sdk-go-v2/credentials v1.17.46
github.com/aws/aws-sdk-go-v2/service/glue v1.102.0
github.com/aws/aws-sdk-go-v2/config v1.28.6
github.com/aws/aws-sdk-go-v2/credentials v1.17.47
github.com/aws/aws-sdk-go-v2/service/glue v1.103.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.71.0
github.com/aws/smithy-go v1.22.1
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815
github.com/google/uuid v1.6.0
github.com/hamba/avro/v2 v2.27.0
github.com/pterm/pterm v0.12.79
github.com/pterm/pterm v0.12.80
github.com/stretchr/testify v1.10.0
github.com/substrait-io/substrait-go v1.2.0
github.com/twmb/murmur3 v1.1.8
github.com/wolfeidau/s3iofs v1.5.2
gocloud.dev v0.40.0
golang.org/x/sync v0.10.0
google.golang.org/api v0.211.0
gopkg.in/yaml.v3 v3.0.1
)

require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
cloud.google.com/go v0.116.0 // indirect
cloud.google.com/go/auth v0.12.1 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.6 // indirect
cloud.google.com/go/compute/metadata v0.5.2 // indirect
cloud.google.com/go/iam v1.2.2 // indirect
cloud.google.com/go/storage v1.43.0 // indirect
github.com/JohnCGriffin/overflow v0.0.0-20211019200055-46fa312c352c // indirect
github.com/alecthomas/participle/v2 v2.1.0 // indirect
github.com/andybalholm/brotli v1.1.1 // indirect
github.com/apache/thrift v0.21.0 // indirect
github.com/aws/aws-sdk-go v1.55.5 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.7 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.20 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.21 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.11 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.25 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.25 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
Expand All @@ -59,26 +68,35 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.4.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.6 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.6 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.7 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.6 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.2 // indirect
github.com/containerd/console v1.0.3 // indirect
github.com/creasty/defaults v1.8.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/fatih/color v1.15.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/goccy/go-json v0.10.3 // indirect
github.com/goccy/go-yaml v1.11.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/google/wire v0.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/gax-go/v2 v2.14.0 // indirect
github.com/gookit/color v1.5.4 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/asmfmt v1.3.2 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/klauspost/cpuid/v2 v2.2.8 // indirect
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
Expand All @@ -91,15 +109,26 @@ require (
github.com/substrait-io/substrait v0.57.1 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
go.opentelemetry.io/otel v1.29.0 // indirect
go.opentelemetry.io/otel/metric v1.29.0 // indirect
go.opentelemetry.io/otel/trace v1.29.0 // indirect
golang.org/x/crypto v0.30.0 // indirect
golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/term v0.25.0 // indirect
golang.org/x/text v0.19.0 // indirect
golang.org/x/tools v0.26.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
golang.org/x/mod v0.22.0 // indirect
golang.org/x/net v0.32.0 // indirect
golang.org/x/oauth2 v0.24.0 // indirect
golang.org/x/sys v0.28.0 // indirect
golang.org/x/term v0.27.0 // indirect
golang.org/x/text v0.21.0 // indirect
golang.org/x/time v0.8.0 // indirect
golang.org/x/tools v0.28.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto v0.0.0-20241113202542-65e8d215514f // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241206012308-a4fef0638583 // indirect
google.golang.org/grpc v1.67.1 // indirect
google.golang.org/protobuf v1.35.1 // indirect
google.golang.org/protobuf v1.35.2 // indirect
)
Loading

0 comments on commit a2c1d2e

Please sign in to comment.