From f85b061f2a1ea4b2ee774c4f57398d840638a131 Mon Sep 17 00:00:00 2001 From: Richard Wu Date: Sat, 15 Sep 2018 16:47:00 -0400 Subject: [PATCH] Update instructions for Spark on macOS Setting the `SPARK_HOME` env variable on the latest version of Spark (2.3.1) will cause Spark to crash in a jupyter notebook setting. --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index a0401217..fafa158c 100644 --- a/README.md +++ b/README.md @@ -162,9 +162,13 @@ To install Spark on MacOS run ``` brew install apache-spark ``` +Do NOT add anything to the environment variable `SPARK_HOME` (this will cause the latest version (2.3.1) to crash in the notebook). +Add `/usr/local/Cellar/apache-spark//libexec/python` to the beginning of your path i.e. in your ~/.zshrc or ~/.profile +``` +export PATH="/usr/local/Cellar/apache-spark//libexec/python:$PATH" +``` +where you can find the latest version installed by brew via `brew list apache-spark`. -After installation of spark, add a `SPARK_HOME` environment variable to your shell, and add `/usr/local/Cellar/apache-spark//libexec/python` to -your python path. ### 7. Getting Started