From 4484dac74c7f7555f4f59f5bbbe4cda5e55187e8 Mon Sep 17 00:00:00 2001 From: Enrico Minack Date: Fri, 5 Mar 2021 15:17:55 +0100 Subject: [PATCH] Releasing 0.6.0 --- CHANGELOG.md | 15 +++++++++++---- README.md | 12 ++++++------ examples/scala/pom.xml | 2 +- pom.xml | 2 +- 4 files changed, 19 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c54d03d..b5005966 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,17 +3,24 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). -## [UNRELEASED] - YYYY-MM-DD +## [0.6.0] - 2021-03-05 ### Added - Adds support to read string predicates with language tags like `` ([issue #63](https://github.com/G-Research/spark-dgraph-connector/issues/63)). This works with any source and mode except the node source in wide mode. Note that reading into GraphFrames is based on the wide mode, so only the untagged language strings can be read there. - Filter pushdown is not supported yet for multi-language predicates ([issue #68](https://github.com/G-Research/spark-dgraph-connector/issues/68)). + Filter pushdown is not supported for multi-language predicates yet ([issue #68](https://github.com/G-Research/spark-dgraph-connector/issues/68)). +- Adds readable exception and suggests next steps when GRPC fails with `RESOURCE_EXHAUSTED` code. +- Missing `maxLeaseId` in cluster state response defaults to `1000L` to avoid an exception. ### Changed -- Upgraded all dependencies to latest versions +- Improves predicate partitioning on projection pushdown as it creates full partitions. +- Fixes bug that did not push predicate value filter correctly down to Dgraph causing incorrect results ([issue #82](https://github.com/G-Research/spark-dgraph-connector/pull/82)) +- Fixes bug in reading `geo` and `password` data types. +- Tests against Dgraph 20.03, 20.07 and 20.11. +- Moved Java Dgraph client to 20.11.0. +- Upgraded all dependencies to latest versions. ## [0.5.0] - 2020-10-21 @@ -28,7 +35,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Changed - Refactored connector API, renamed `spark.read.dgraph*` methods to `spark.read.dgraph.*`. - Moved `triples`, `edges` and `nodes` sources from package `uk.co.gresearch.spark.dgraph.connector` to `uk.co.gresearch.spark.dgraph`. -- Moved Dgraph client to 20.03.1 and Dgraph test cluster to 20.07.0. +- Moved Java Dgraph client to 20.03.1 and Dgraph test cluster to 20.07.0. ## [0.4.0] - 2020-07-24 diff --git a/README.md b/README.md index 79fa9ef8..89d220ae 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ The connector is under continuous development. It has the following known limita The Spark Dgraph Connector is available for Spark 2.4, Spark 3.0 and Spark 3.1, all with Scala 2.12. Use Maven artifact ID `spark-dgraph-connector_2.12`. The Spark version is part of the package version, -e.g. 0.5.0-2.4, 0.5.0-3.0 and 0.5.0-3.1, respectively. +e.g. 0.6.0-2.4, 0.6.0-3.0 and 0.6.0-3.1, respectively. Minor versions are kept in sync between those two packages, such that identical minor versions contain identical feature sets (where supported by the respective Spark version). @@ -69,7 +69,7 @@ such that identical minor versions contain identical feature sets Add this line to your `build.sbt` file to use the latest version for Spark 2.4: ```sbt -libraryDependencies += "uk.co.gresearch.spark" %% "spark-dgraph-connector" % "0.5.0-2.4" +libraryDependencies += "uk.co.gresearch.spark" %% "spark-dgraph-connector" % "0.6.0-2.4" ``` ### Maven @@ -80,7 +80,7 @@ Add this dependency to your `pom.xml` file to use the latest version: uk.co.gresearch.spark spark-dgraph-connector_2.12 - 0.5.0-2.4 + 0.6.0-2.4 ``` @@ -89,7 +89,7 @@ Add this dependency to your `pom.xml` file to use the latest version: Launch the Scala Spark REPL (Spark ≥2.4.0) with the Spark Dgraph Connector dependency (version ≥0.5.0) as follows: ```shell script -spark-shell --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.5.0-2.4 +spark-shell --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.6.0-2.4 ``` ### PySpark Shell and Python script @@ -97,13 +97,13 @@ spark-shell --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.5.0-2 Launch the Python Spark REPL (pyspark 2.4.2 and ≥3.0) with the Spark Dgraph Connector dependency (version ≥0.5.0) as follows: ```shell script -pyspark --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.5.0-2.4 +pyspark --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.6.0-2.4 ``` Run your Python scripts that use PySpark (pyspark 2.4.2 and ≥3.0) and the Spark Dgraph Connector (version ≥0.5.0) via `spark-submit`: ```shell script -spark-submit --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.5.0-2.4 [script.py] +spark-submit --packages uk.co.gresearch.spark:spark-dgraph-connector_2.12:0.6.0-2.4 [script.py] ``` ## Examples diff --git a/examples/scala/pom.xml b/examples/scala/pom.xml index 8ff3a3dd..7859e46f 100644 --- a/examples/scala/pom.xml +++ b/examples/scala/pom.xml @@ -3,7 +3,7 @@ uk.co.gresearch.spark spark-dgraph-connector-examples_2.12 - 0.6.0-2.4-SNAPSHOT + 0.6.0-2.4 Spark Dgraph Connector Example diff --git a/pom.xml b/pom.xml index 466a8c93..1bc1e3ef 100644 --- a/pom.xml +++ b/pom.xml @@ -2,7 +2,7 @@ 4.0.0 uk.co.gresearch.spark spark-dgraph-connector_2.12 - 0.6.0-2.4-SNAPSHOT + 0.6.0-2.4 Spark Dgraph Connector A Spark connector for Dgraph databases 2020