[Feature Request] Make write.object-storage.enabled optional for Databricks Iceberg REST integration #3197
Description
Who is this feature for?
People who want to use AutoMQ Table Topic with Iceberg tables managed by Databricks Unity Catalog via the Iceberg REST catalog
What problem are they facing today?
Currently, AutoMQ sets write.object-storage.enabled=true when creating Iceberg tables. However, this is incompatible with Databricks-managed Iceberg tables because Unity Catalog forbids modifying this parameter, resulting in an error.
To work around this, I had to:
- Pre-create the Iceberg table directly in Databricks (via SQL or REST).
- Point AutoMQ to that existing table.
- Set
table.topic.expire.snapshot.enabled=false.
While AutoMQ works with the Databricks Iceberg REST catalog for append-only table topics under these conditions, having to pre-create the Iceberg table is not ideal.
Why is solving this impactful?
It allows creating Iceberg tables automatically in Databricks, instead of having to manually pre-create them for append-only table topics
Proposed solution
I think we can take inspiration from #3077.
Additional notes
No response