2024-05-30 kernel meeting notes

[zachschuermann]

summary

close to delta-rs integration!

action items

PRs in flight:

• new attribute proc macro that will make handles easy to define (annotate a struct) coming from @scovich
• [WIP] updates for delta-rs #189 WIP from @roeap
• Have an ffi c example that actually reads data #203 (end-to-end read table print schema etc.) some decisions about how to expose engine functionality across FFI.
• don't fully enforce matching schema #210

attendees

@zachschuermann @roeap @rtyler @hntd187

notes stream

• interesting bit from @hntd187: duckdb doesn't filter at parquet read level. going to simplify feat: better ways to use deletion vectors #215
• ask: @roeap are you blocked on delta-rs integration at all?
  • not blocked!
  • can we do filtering on nested struct values? (delta-rs)
  • possible once we integrate? might need little follow-up on kernel.
  • PR is green (points at main need release then all good)
• note: @nicklan can release 0.0.2 anytime!
• note ryan on leave now.
• tyler: advanced use cases (seen in slack)
  • people want to see metadata
  • for robert: if someone wants to read a table txn log, should this use delta-rs or kernel only?
  • robert: behind developer-features flag (in kernel) we want to support traversing the log. two kinds of users: peek at the data, OR intimate deep dive into table/log/everything.
  • keep developing in delta-rs: ahead of kernel now. then after upstreaming decide. long-term use kernel.
  • tyler: 3-4 slack threads came up from python users. deltalake python package: need to expose this 'advanced usage'
  • deltalake python package pulls in lots of deps. already dependency problems.
  • kernel: parse the log
  • delta-rs: interpret?
  • support two major use cases: (1) simple: hide APIs, (2) advanced: support playing with log, etc.
    • e.g. DeltaTable / DeltaLog
  • see kafka delta ingest.
  • related:
    • feat: implement transaction identifiers - continued delta-io/delta-rs#2539
    • feat: adopt kernel schema types delta-io/delta-rs#2495
  • robert todo: adopt kernel expressions to push out datafusion further