0.24.2

Bug Fixes

• PR #580: Fixed shaded artifacts version flattening, the version appears
  correctly in the released POM
• PR #583: netty-tcnative is taken from the Netty BOM
• PR #584: CVE-2020-36518 - Upgraded jackson

0.24.1

Bug Fixes

• PR #576: Fixed error running on Datapoc clusters where conscrypt is disabled
  (the propertydataproc.conscrypt.provider.enable set to false)

0.24.0

New Features

• PR #518: Cache expiration time can be configured now.
• PR #561: Added support for adding trace ID to the BigQuery reads and writes.
  The trace Id will be of the format Spark:ApplicateName:JobID. The
  application name must be set by the user, job ID is defaults to Dataproc job
  ID if exists, otherwise it is set to spark.app.id.
• PR #568: Added support for BigQuery jobs labels

Bug Fixes

• PR #563: Fixed a bug where using writeMethod=DIRECT and SaveMode=Append the
  destination table may have been deleted in case abort() has been called.
• Issue #530: Treating Field.mode==null as Nullable

Dependency Updates

• BigQuery API has been upgraded to version 2.9.4
• BigQuery Storage API has been upgraded to version 2.11.0
• gRPC has been upgraded to version 1.44.1
• Netty has been upgraded to version 4.1.73.Final

0.23.2

New Features

• PR #521: Added Arrow compression options to the spark-bigquery-with-dependencies_2.* connectors
• PR #526: Added the option to use parent project for the metadata/jobs API as well

Dependency Updtyes

• BigQuery API has been upgraded to version 2.3.3
• BigQuery Storage API has been upgraded to version 2.4.2
• gRPC has been upgraded to version 1.42.1
• Netty has been upgraded to version 4.1.70.Final

0.23.1

Bug Fixes

• Issue #501: Fixed using Apache Avro as an intermediate type for writing.

0.23.0

New Features

• New connector: This release adds a new connector (spark-2.4-bigquery) which have a better integration with the new Spark 2.4 internal APIs. This connector is implemented in Java, so it works for both Scala versions. This connector is released in preview version, comments are welcome.
• PR #469: Added support for the BigQuery Storage Write API, allowing faster writes (Spark 2.4 connector only). Activated by setting the option writeMethod to direct, this should give better write performance.
• Issue #481: Added configuration option to use compression from the READ API for Arrow

Dependency Updates

• BigQuery API has been upgraded to version 2.1.8
• BigQuery Storage API has been upgraded to version 2.1.2
• gRPC has been upgraded to version 1.41.0

0.22.2

Bug Fixes

• Issue #446: BigNumeric values are properly written to BigQuery
• Issue #452: Adding the option to clean BigQueryClient.destinationTableCache

Dependency Upgrades

• BigQuery API has been upgraded to version 2.1.8
• BigQuery Storage API has been upgraded to version 2.1.2
• gRPC has been upgraded to version 1.4.0

0.22.1

Bug Fixes

• Issue #444: allowing unpartitioned clustered table

0.22.0

New Features

• PR #404: Added support for BigNumeric
• PR #430: Added HTTP and gRPC proxy support

Bug Fixes

• Issue #273: Resolved the streaming write issue for spark 3.x

0.21.1

New Features

• PR #413: Pushing all filters to BigQuery Storage API

Bug Fixes

• Issue #412: Supporting WITH queries
• Issue #409: Allowing all whitespaces after the select
• PR #419: Fix a bug where background threads > 2 cases would miss pages (DSv2)
• PR #416: Moved zstd-jni library to be provided in order to solve Spark 2.4 compatibility (DSv2)
• PR #417: Added back column projection to DSv2