Intermittent error when creating tables: NOT_FOUND: Requested entity was not found

[MasterDDT]

We are using Spark 3.3, BigQuery connector 0.30.0, Scala 2.12, Java 17. We are creating tables via DataFrame write() API with these options:

    "viewsEnabled" -> "true",
    "writeMethod" -> "direct",
    "dataset" -> datasetName,
    "materializationDataset" -> datasetName,
    "table" -> s"${datasetName}.${tableName}",

When writing large tables sometimes (unclear if number of rows or number of columns), we see this error:

Caused by: java.util.concurrent.ExecutionException: com.google.cloud.spark.bigquery.repackaged.com.google.api.gax.rpc.NotFoundException: com.google.cloud.spark.bigquery.repackaged.io.grpc.StatusRuntimeException: NOT_FOUND: Requested entity was not found.
	at com.google.cloud.spark.bigquery.repackaged.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
	at com.google.cloud.spark.bigquery.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
	at com.google.cloud.bigquery.connector.common.BigQueryDirectDataWriterHelper.retryCallable(BigQueryDirectDataWriterHelper.java:131)
	at com.google.cloud.bigquery.connector.common.BigQueryDirectDataWriterHelper.retryCreateWriteStream(BigQueryDirectDataWriterHelper.java:98)
	at com.google.cloud.bigquery.connector.common.BigQueryDirectDataWriterHelper.<init>(BigQueryDirectDataWriterHelper.java:77)
	... 19 more

Full callstack here https://gist.github.com/MasterDDT/da7e93cbf14d34a6e88acfe7213d1360. We cannot repro this consistently. Any ideas how to debug it?

[davidrabinowitz]

Can you please upgrade first to the latest connector (spark-3.3-bigquery-0.41.0) to see if it helps?

[MasterDDT]

Can you please upgrade first to the latest connector (spark-3.3-bigquery-0.41.0) to see if it helps?

Haha I was waiting for that answer. Yeah we'll try but do you have an idea what fix we are missing? We can fork and patch that. I searched issues and didnt see this error msg.

[MasterDDT]

We figured this out, it was a race condition with closing tables