Delta Kernel Draft PR

[vaibhavk1992]

Important Read

• Please ensure the GitHub issue is mentioned at the beginning of the PR

What is the purpose of the pull request

This draft introduces the Delta Kernel to read the tables instead of using spark sessions.

Brief change log

• Implemented DeltaKernelConversionSource to read the table using kernel library
• Added Delta Kernel API and Delta Kernel Defaults as compile-time dependencies
• Added DeltaKernelTableExtractor for metadata extraction
• Implemented DeltaKernelSchemaExtractor for schema conversion
• Added DeltaKernelPartitionExtractor for partition handling
• Implemented DeltaKernelStatsExtractor for statistics extraction
• Added unit tests, integration test to verify all the edge cases

Verify this pull request

This pull request is a trivial rework , added unit tests, integration test to verify all the edge cases

-Unit Tests
-TestDeltaKernelSchemaExtractor

• Tested primitive type conversions
• Validated complex type handling (structs, arrays, maps)
• Verified metadata preservation
• Tested timestamp and decimal handling
• Validated field metadata (comments, nullability)

-TestDeltaKernelPartitionExtractor

• Tested partitioned and unpartitioned tables
• Verified partition value extraction
• Validated date and timestamp partition handling
• Tested nested partition fields

-TestDeltaKernelStatsExtractor

• Tested statistics extraction

• Validated range and value statistics

• Tested null count handling

• Added integration tests for end-to-end.

[vinishjail97]

This is great progress @vaibhavk1992, added some comments.

[vaibhavk1992]

@the-other-tim-brown @vinishjail97 @rahil-c All the comments have been addressed and the build is passing too.
Please review for the final merge.

[the-other-tim-brown]

This is eagerly pulling all the files into memory instead of using the iterator pattern. Can the code be updated to iterate through these values when next is called instead of eagerly materializing them?

[vaibhavk1992]

@the-other-tim-brown
the scanFiles.hasNext() Returns true if the iteration has more elements. (In other words, returns true if next would return an element rather than throwing an exception.). I am not sure what do you mean by materializing here and what we have to do exactly.
I have followed this convention from the suggestion by the doc form kernel community. Please check this.
https://github.com/delta-io/delta/blob/master/kernel/USER_GUIDE.md

[the-other-tim-brown]

The idea is that the DeltaDataFileIterator is an iterator so we don't need to compute all the values in the constructor. We can instead compute the next value when next is called.

[vaibhavk1992]

I have updated the code, please check if that make sense. @the-other-tim-brown

[the-other-tim-brown]

@vaibhavk1992 make sure to fill out the PR template

[rahil-c]

@vaibhavk1992 Thanks for pushing on this, I took a brief scan and the PR looks close to landing. My main question is around testing, did you happen to test conversion end to end using your delta kernel source to hudi or to iceberg?

I believe the IT class ITConversionController registers the sources in order to trigger conversions from different sourceFormats to different targetFormats, see https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java#L191.

I am wondering if we should try to add some similar IT flow as the above class to confirm Delta Kernel works correctly end to end with other target formats or at the very least wanted to know if you did some manual testing to confirm you can convert to an iceberg or hudi table using your delta kernel source? I think we can also add this end to end IT test plan as a fast follow in a separate pr.

[vaibhavk1992]

@rahil-c As discussed ITConversionController would be part of the follow PR. I have resolved the 2 comments from you as well. If things are looking good can you and @the-other-tim-brown please approve the changes.