dbt-elementary's metadata table fails to create with dbt-trino with a Delta Lake catalog

[arvinki]

Describe the bug
I ran dbt run --select elementary after installing the latest version of dbt-elementary and hit an error creating the metadata table

To Reproduce
Steps to reproduce the behavior:

1. Have a Delta Lake catalog in a trino cluster
2. Set up the DBT project's database/schema to the Delta Lake catalog
3. Run dbt run --select elementary
4. See error

19:42:36  14 of 28 ERROR creating sql table model arvind_i_beba1fb1_7ddb_4a56_a529_dc985f65e83d_elementary.metadata  [ERROR in 7.73s]

Below is the error in the trino cluster

io.trino.spi.TrinoException: Renaming managed tables is not allowed with current metastore configuration
	at io.trino.plugin.deltalake.DeltaLakeMetadata.renameTable(DeltaLakeMetadata.java:2448)
	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.renameTable(ClassLoaderSafeConnectorMetadata.java:496)
	at io.trino.tracing.TracingConnectorMetadata.renameTable(TracingConnectorMetadata.java:412)
	at io.trino.metadata.MetadataManager.renameTable(MetadataManager.java:794)
	at io.trino.tracing.TracingMetadata.renameTable(TracingMetadata.java:406)
	at io.trino.execution.RenameTableTask.execute(RenameTableTask.java:116)
	at io.trino.execution.RenameTableTask.execute(RenameTableTask.java:45)
	at io.trino.execution.DataDefinitionExecution.start(DataDefinitionExecution.java:145)
	at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:256)
	at io.trino.dispatcher.LocalDispatchQuery.startExecution(LocalDispatchQuery.java:145)
	at io.trino.dispatcher.LocalDispatchQuery.lambda$waitForMinimumWorkers$2(LocalDispatchQuery.java:129)
	at io.airlift.concurrent.MoreFutures.lambda$addSuccessCallback$12(MoreFutures.java:568)
	at io.airlift.concurrent.MoreFutures$3.onSuccess(MoreFutures.java:543)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1133)
	at io.trino.$gen.Trino_433____20240724_173957_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

The SQL that was being run at that time was

/* {"app": "dbt", "dbt_version": "1.7.13", "profile_name": "opus", "target_name": "opus", "node_id": "model.elementary.metadata"} */
alter table "development_dl"."arvind_i_beba1fb1_7ddb_4a56_a529_dc985f65e83d_elementary"."metadata__dbt_tmp" rename to "development_dl"."arvind_i_beba1fb1_7ddb_4a56_a529_dc985f65e83d_elementary"."metadata"

Expected behavior
All the dbt-elementary artifacts should be created without errors

Environment (please complete the following information):

• Elementary CLI (edr) version: [e.g. 0.5.3], can be found by running pip show elementary-data: 0.15.1
• Elementary dbt package version: [e.g. 0.4.1], can be found in packages.yml file

- package: elementary-data/elementary
    version: 0.15.2

• dbt version you're using [e.g. 1.8.1]

dbt --version
Core:
  - installed: 1.7.13
  - latest:    1.8.4  - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - trino: 1.7.0 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

• Data warehouse [e.g. snowflake] - S3, Trino & Delta Lake catalog
• Infrastructure details (e.g. operating system, prod / dev / staging, deployment infra, CI system, etc) - Dev Catalog

Additional context
We are evaluating using dbt-elementary for our cases, in providing freshness/gaps-in-data/running-tests for our data sets, and I am trying out dbt-elementary in my dev set up.

Would you be willing to contribute a fix for this issue?
No

[ofek1weiss]

Hey @arvinki, that is weird indeed, but from the data you provided it seems that this should be an issue with all models with the materialization of table as what appears to happen is the following:

• dbt creates the elementary.metadata model which it does by:
  • creating a table named elementary.metadata__dbt_tmp (to avoid overwriting existing data if the table creation fails).
  • dropping the existing elementary.metadata table if it exists.
  • renaming the temp table to the non temp one (here it fails, "Renaming managed tables is not allowed with current metastore configuration" in your trino output)

This process is the way in which all table models are created in dbt, and is not modified (and cannot be modified) by elementary.

Do you have models with the table materialization in your project? do they work?

[arvinki]

Hey @ofek1weiss, we don't have models with table materialization in our project.

Also, this works if my catalog is a hive catalog. This catalog I am trying to run elementary in, is a delta-lake catalog. According to Trino's documentation, ALTER TABLE RENAME TO is supported only for external table-types.

We only have incremental models.

[ofek1weiss]

@arvinki If that is the case, this issue is not with elementary, but with the dbt-trino adapter, i recommend opening an issue over there
https://github.com/starburstdata/dbt-trino/issues

[rafafr]

Hey @ofek1weiss, we don't have models with table materialization in our project.

Also, this works if my catalog is a hive catalog. This catalog I am trying to run elementary in, is a delta-lake catalog. According to Trino's documentation, ALTER TABLE RENAME TO is supported only for external table-types.

We only have incremental models.

Hi @arvinki , try using Iceberg format. We had problems with Delta format but Iceberg is running well.