Skip to content

Commit

Permalink
preparing release 0.24.1
Browse files Browse the repository at this point in the history
  • Loading branch information
@davidrabinowitz
davidrabinowitz committed Mar 29, 2022
1 parent 9aabf21 commit ddb3d94
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 21 deletions.
4 changes: 4 additions & 0 deletions CHANGES.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Release Notes

## 0.24.1 - 2022-03-29
* PR #576: Fixed error running on Datapoc clusters where conscrypt is disabled
(the property`dataproc.conscrypt.provider.enable` set to `false`)

## 0.24.0 - 2022-03-23
* Issue #530: Treating Field.mode==null as Nullable
* PR #518: Cache expiration time can be configured now.
Expand Down
42 changes: 21 additions & 21 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,9 @@ The latest version of the connector is publicly available in the following links

| version | Link |
| --- | --- |
| Scala 2.11 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.0.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.0.jar)) |
| Scala 2.12 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.24.0.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.24.0.jar)) |
| Spark 2.4 | `gs://spark-lib/bigquery/spark-2.4-bigquery-0.24.0-preview.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-2.4-bigquery-0.24.0-preview.jar)) |
| Scala 2.11 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.1.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.1.jar)) |
| Scala 2.12 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.24.1.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.24.1.jar)) |
| Spark 2.4 | `gs://spark-lib/bigquery/spark-2.4-bigquery-0.24.1-preview.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-2.4-bigquery-0.24.1-preview.jar)) |

The only difference between first two connectors is that the former is a Scala 2.11 based connector, targeting Spark 2.3
and 2.4 using Scala 2.11 whereas the latter is a Scala 2.12 based connector, targeting Spark 2.4 and 3.x using Scala 2.12.
Expand All @@ -78,9 +78,9 @@ repository. It can be used using the `--packages` option or the

| version | Connector Artifact |
| --- | --- |
| Scala 2.11 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.24.0` |
| Scala 2.12 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.24.0` |
| Spark 2.4 | `com.google.cloud.spark:spark-2.4-bigquery:0.24.0-preview` |
| Scala 2.11 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.24.1` |
| Scala 2.12 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.24.1` |
| Spark 2.4 | `com.google.cloud.spark:spark-2.4-bigquery:0.24.1-preview` |

If you want to keep up with the latest version of the connector the following links can be used. Notice that for
production environments where the connector version should be pinned, one of the above links should be used.
Expand Down Expand Up @@ -252,7 +252,7 @@ supported at this moment by the direct write method.
**Important:** Please refer to the [data ingestion pricing](https://cloud.google.com/bigquery/pricing#data_ingestion_pricing)
page regarding the BigQuery Storage Write API pricing.

**Important:** Please use version 0.24.0 and above for direct writes, as previous
**Important:** Please use version 0.24.1 and above for direct writes, as previous
versions have a bug that may cause a table deletion in certain cases.

#### Indirect write
Expand Down Expand Up @@ -862,9 +862,9 @@ creating the job or added during runtime. See examples below:
1) Adding python files while launching pyspark
```
# use appropriate version for jar depending on the scala version
pyspark --jars gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.0.jar
--py-files gs://spark-lib/bigquery/spark-bigquery-support-0.24.0.zip
--files gs://spark-lib/bigquery/spark-bigquery-support-0.24.0.zip
pyspark --jars gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.1.jar
--py-files gs://spark-lib/bigquery/spark-bigquery-support-0.24.1.zip
--files gs://spark-lib/bigquery/spark-bigquery-support-0.24.1.zip
```

2) Adding python files in Jupyter Notebook
Expand All @@ -874,14 +874,14 @@ from pyspark import SparkFiles
# use appropriate version for jar depending on the scala version
spark = SparkSession.builder\
.appName('BigNumeric')\
.config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.0.jar')\
.config('spark.submit.pyFiles', 'gs://spark-lib/bigquery/spark-bigquery-support-0.24.0.zip')\
.config('spark.files', 'gs://spark-lib/bigquery/spark-bigquery-support-0.24.0.zip')\
.config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.1.jar')\
.config('spark.submit.pyFiles', 'gs://spark-lib/bigquery/spark-bigquery-support-0.24.1.zip')\
.config('spark.files', 'gs://spark-lib/bigquery/spark-bigquery-support-0.24.1.zip')\
.getOrCreate()
# extract the spark-bigquery-support zip file
import zipfile
with zipfile.ZipFile(SparkFiles.get("spark-bigquery-support-0.24.0.zip")) as zf:
with zipfile.ZipFile(SparkFiles.get("spark-bigquery-support-0.24.1.zip")) as zf:
zf.extractall()
```

Expand All @@ -890,10 +890,10 @@ with zipfile.ZipFile(SparkFiles.get("spark-bigquery-support-0.24.0.zip")) as zf:
# use appropriate version for jar depending on the scala version
spark = SparkSession.builder\
.appName('BigNumeric')\
.config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.0.jar')\
.config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.24.1.jar')\
.getOrCreate()
spark.sparkContext.addPyFile("gs://spark-lib/bigquery/spark-bigquery-support-0.24.0.zip")
spark.sparkContext.addPyFile("gs://spark-lib/bigquery/spark-bigquery-support-0.24.1.zip")
```

Usage Example:
Expand Down Expand Up @@ -1001,7 +1001,7 @@ using the following code:
```python
from pyspark.sql import SparkSession
spark = SparkSession.builder
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.24.0")
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.24.1")
.getOrCreate()
df = spark.read.format("bigquery")
.load("dataset.table")
Expand All @@ -1010,15 +1010,15 @@ df = spark.read.format("bigquery")
**Scala:**
```python
val spark = SparkSession.builder
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.24.0")
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.24.1")
.getOrCreate()
val df = spark.read.format("bigquery")
.load("dataset.table")
```

In case Spark cluster is using Scala 2.12 (it's optional for Spark 2.4.x,
mandatory in 3.0.x), then the relevant package is
com.google.cloud.spark:spark-bigquery-with-dependencies_**2.12**:0.24.0. In
com.google.cloud.spark:spark-bigquery-with-dependencies_**2.12**:0.24.1. In
order to know which Scala version is used, please run the following code:

**Python:**
Expand All @@ -1042,14 +1042,14 @@ To include the connector in your project:
<dependency>
<groupId>com.google.cloud.spark</groupId>
<artifactId>spark-bigquery-with-dependencies_${scala.version}</artifactId>
<version>0.24.0</version>
<version>0.24.1</version>
</dependency>
```

### SBT

```sbt
libraryDependencies += "com.google.cloud.spark" %% "spark-bigquery-with-dependencies" % "0.24.0"
libraryDependencies += "com.google.cloud.spark" %% "spark-bigquery-with-dependencies" % "0.24.1"
```

## FAQ
Expand Down

0 comments on commit ddb3d94

Please sign in to comment.