Skip to content

Commit

Permalink
[MINOR] Upgrade to Spark 3.5.2 in the docs and CI
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

### Why are the changes needed?

Using the latest Spark 3.5.2 in the CI and docs

### Does this PR introduce _any_ user-facing change?

No
### How was this patch tested?

Closes #43 from haoxins/chore-3-5-2.

Authored-by: Xin Hao <haoxinst@gmail.com>
Signed-off-by: Martin Grund <martin.grund@databricks.com>
  • Loading branch information
haoxins authored and grundprinzip committed Aug 15, 2024
1 parent 0ed4200 commit 7070ad0
Show file tree
Hide file tree
Showing 4 changed files with 5 additions and 5 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ on:
- master

env:
SPARK_VERSION: '3.5.1'
SPARK_VERSION: '3.5.2'
HADOOP_VERSION: '3'

permissions:
Expand Down Expand Up @@ -84,7 +84,7 @@ jobs:
tar -xzf spark-${{ env.SPARK_VERSION }}-bin-hadoop${{ env.HADOOP_VERSION }}.tgz
# Delete the old file
rm spark-${{ env.SPARK_VERSION }}-bin-hadoop${{ env.HADOOP_VERSION }}.tgz
# Setup the Environment Variables
echo "Apache Spark is ready to use"
echo "SPARK_HOME=${GITHUB_WORKSPACE}/deps/spark-${{ env.SPARK_VERSION }}-bin-hadoop${{ env.HADOOP_VERSION }}" >> "$GITHUB_ENV"
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Step 4: Setup the Spark Driver on localhost.
2. Start the Spark Connect server with the following command (make sure to use a package version that matches your Spark distribution):

```
sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.0
sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.2
```

Step 5: Run the example Go application.
Expand Down
2 changes: 1 addition & 1 deletion internal/tests/integration/spark_runner.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ func StartSparkConnect() (int64, error) {

cmd := exec.Command("./sbin/start-connect-server.sh", "--wait", "--conf",
"spark.log.structuredLogging.enabled=false", "--packages",
"org.apache.spark:spark-connect_2.12:3.5.1")
"org.apache.spark:spark-connect_2.12:3.5.2")
cmd.Dir = sparkHome

stdout, _ := cmd.StdoutPipe()
Expand Down
2 changes: 1 addition & 1 deletion quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ func main() {

Download a Spark distribution (3.5.0+), unzip the folder, run command:
```
sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.0
sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.2
```

## Run Spark Connect Client Application
Expand Down

0 comments on commit 7070ad0

Please sign in to comment.