rfc: Modularize `iceberg` Implementations #1854

Xuanwo · 2025-11-13T10:39:26Z

Which issue does this PR close?

Part of Modularize iceberg Implementations #1819

What changes are included in this PR?

Add RFC for iceberg-kernel

Are these changes tested?

Signed-off-by: Xuanwo <github@xuanwo.io>

github-actions

license-eye has checked 374 files.

Valid	Invalid	Ignored	Fixed
306	1	67	0

Click to see the invalid file list

docs/rfcs/0001_kernel.md

Use this command to fix any missing license headers

```bash

docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix

</details>

docs/rfcs/0001_kernel.md

Signed-off-by: Xuanwo <github@xuanwo.io>

ZENOTME · 2025-11-13T14:53:39Z

I'm thinking is reasonable to split out spec part as a single minimum module? It just provide the memory representation of iceberg spec and de/serialize way. It can be evolve quickly and as a minimum module reuse by external user or inner module.

Xuanwo · 2025-11-14T10:22:47Z

I'm thinking is reasonable to split out spec part as a single minimum module? It just provide the memory representation of iceberg spec and de/serialize way. It can be evolve quickly and as a minimum module reuse by external user or inner module.

I think that's an interesting idea. It’s fine to just expose a spec crate, but how useful would it be? For reading snapshots, manifest lists, and manifests, users still need a FileIO. Is there a use case that users just want to ser/de a manifest file?

alamb

I think this looks like a great plan personally -- though I am not a direct user of the crate so I don't have a whole lot of gravitas.

I will also try and hype this PR up some more to get some more commetns

docs/rfcs/0001_kernel.md

kevinjqliu · 2025-11-15T16:59:16Z

btw #1857 should resolve the CI issue

kevinjqliu · 2025-11-16T17:29:30Z

merged main to fix ci

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Signed-off-by: Xuanwo <github@xuanwo.io>

docs/rfcs/0001_kernel.md

CTTY · 2025-11-18T00:47:43Z

docs/rfcs/0001_kernel.md

+
+- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.).
+- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol.
+- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates.


Should this be Storage trait rather than FileIO trait?

From the code example below it seems like FileIO is a struct container for Storage and registry

I'm thinking that we have a iceberg-storage crate, which contains all fileio related things, including FileIO/InputFile/OutputFile struct, Storage trait, some builtin storage implemention(enabled independently by feature).

liurenjie1024

Thanks @Xuanwo for this proposal. I think the overall direction is correct, but I'm not sure we need a new kernel crate. What we need to do is move something like fileio, runtime api, local table execution out of iceberg crate, and this way it serves as the kernel crate, WDYT?

liurenjie1024 · 2025-11-18T09:04:15Z

docs/rfcs/0001_kernel.md

+
+- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.).
+- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol.
+- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates.


I'm thinking that we have a iceberg-storage crate, which contains all fileio related things, including FileIO/InputFile/OutputFile struct, Storage trait, some builtin storage implemention(enabled independently by feature).

docs/rfcs/0001_kernel.md

liurenjie1024 · 2025-11-18T09:08:33Z

docs/rfcs/0001_kernel.md

+- **Full read & write coverage**: the kernel must contain every protocol component required for both scan planning and transactional writes (append, rewrite, commit, etc.).
+- **No default runtime dependency**: the kernel defines a `Runtime` trait instead of depending on Tokio or Smol.
+- **No default storage dependency**: the kernel defines `FileIO` traits only; concrete implementations (for example `iceberg-fileio-opendal`) live in dedicated crates.
+- **Stable facade for existing users**: the top-level `iceberg` crate continues to expose the familiar API by re-exporting the kernel plus a default engine feature.


I agree that we should keep public api as stable as possible, but it maybe not practical before we reach 1.0. Currently we reply on review to ensure it, but after reaching 1.0 we should use some autonomous checks to ensure that.

docs/rfcs/0001_kernel.md

liurenjie1024 · 2025-11-18T09:23:20Z

docs/rfcs/0001_kernel.md

+   - Introduce standalone crates (`iceberg-runtime-tokio`, `iceberg-fileio-opendal`, etc.) that implement the new traits.
+
+3. **Phase 3 – Detach Arrow/execution**
+   - Move `arrow` helpers and `ArrowReaderBuilder` into a reference executor crate (e.g. `iceberg-engine-arrow`).


In fact, there are several parts in the to_arrow method:

ArrowReaderBuilder which is used to read one split of planned task. This is in fact compute engine independent, and we should leave it in iceberg crate.

An executor to use runtime to scan table in parallel. I think this part should be move to the new default engine crate.

Signed-off-by: Xuanwo <github@xuanwo.io>

github-actions

license-eye has checked 374 files.

Valid	Invalid	Ignored	Fixed
306	1	67	0

Click to see the invalid file list

docs/rfcs/0001_modularize_iceberg_implementations.md

Use this command to fix any missing license headers

```bash

docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix

</details>

docs/rfcs/0001_modularize_iceberg_implementations.md

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Xuanwo · 2025-11-19T10:13:28Z

I had an offline discussion with @liurenjie1024 and we’ve agreed to refactor our efforts to modularize the iceberg implementations instead of adding a new iceberg-kernel crate.

In short, we’ll expose all our core traits in iceberg and make it modular so users can plug in whatever they need. At the same time, we’ll provide a pre-built stack to make it easier for users to start scanning an iceberg table.

In the past, we thought it was better to split into more crates, but we realized it just added to our maintenance burden keeping those tiny crates up to date and most of the time we need to use them all at once.

liurenjie1024

Thanks @Xuanwo for raising this. I general I think the direction is right, we just need to refine the action items.

docs/rfcs/0001_modularize_iceberg_implementations.md

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo · 2025-11-24T08:16:13Z

cc @liurenjie1024 and @kevinjqliu please review again.

Signed-off-by: Xuanwo <github@xuanwo.io>

liurenjie1024

Thanks @Xuanwo for this rfc, LGTM!

liurenjie1024 · 2025-11-25T09:55:13Z

Let's keep this for a while to have more eyes on it.

docs: Add RFC for iceberg-kernel

5a4b1c6

Signed-off-by: Xuanwo <github@xuanwo.io>

github-actions bot reviewed Nov 13, 2025

View reviewed changes

docs/rfcs/0001_kernel.md Outdated Show resolved Hide resolved

Xuanwo added 2 commits November 13, 2025 18:45

Add license:

891fd75

Signed-off-by: Xuanwo <github@xuanwo.io>

Merge branch 'main' into kernel

5026942

alamb approved these changes Nov 15, 2025

View reviewed changes

Merge branch 'main' into kernel

9130b89

Xuanwo and others added 3 commits November 17, 2025 17:32

Update docs/rfcs/0001_kernel.md

5cb95ad

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Merge branch 'main' into kernel

27cece8

Remove spawn_blocking

f817034

Signed-off-by: Xuanwo <github@xuanwo.io>

mbutrovich reviewed Nov 17, 2025

View reviewed changes

docs/rfcs/0001_kernel.md Outdated Show resolved Hide resolved

CTTY reviewed Nov 18, 2025

View reviewed changes

liurenjie1024 reviewed Nov 18, 2025

View reviewed changes

Refactor

eaf5268

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo changed the title ~~docs: Add RFC for iceberg-kernel~~ rfc: Modularize iceberg Implementations Nov 19, 2025

github-actions bot reviewed Nov 19, 2025

View reviewed changes

docs/rfcs/0001_modularize_iceberg_implementations.md Show resolved Hide resolved

Update docs/rfcs/0001_modularize_iceberg_implementations.md

09317fc

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Merge branch 'main' into kernel

c472132

liurenjie1024 reviewed Nov 20, 2025

View reviewed changes

Xuanwo added 5 commits November 24, 2025 16:09

Remove detailed traits out of this RFC

621ecba

Signed-off-by: Xuanwo <github@xuanwo.io>

polish details

62e1fc0

Signed-off-by: Xuanwo <github@xuanwo.io>

Merge remote-tracking branch 'refs/remotes/origin/kernel' into kernel

7c30117

Add notes for arrow

54d4dde

Signed-off-by: Xuanwo <github@xuanwo.io>

Merge branch 'main' into kernel

9eefe7c

polish

a65640a

Signed-off-by: Xuanwo <github@xuanwo.io>

Merge remote-tracking branch 'refs/remotes/origin/kernel' into kernel

c64cb46

liurenjie1024 approved these changes Nov 25, 2025

View reviewed changes

liurenjie1024 requested a review from Fokko November 26, 2025 01:33

rfc: Modularize iceberg Implementations #1854

Are you sure you want to change the base?

rfc: Modularize iceberg Implementations #1854

Conversation

Xuanwo commented Nov 13, 2025

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ZENOTME commented Nov 13, 2025

Uh oh!

Xuanwo commented Nov 14, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevinjqliu commented Nov 15, 2025

Uh oh!

kevinjqliu commented Nov 16, 2025

Uh oh!

Uh oh!

CTTY Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

liurenjie1024 Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liurenjie1024 Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Xuanwo commented Nov 19, 2025

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Xuanwo commented Nov 24, 2025

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

rfc: Modularize `iceberg` Implementations #1854

rfc: Modularize `iceberg` Implementations #1854