Docker Pyspark With Azure Gen2

Building a docker image with Pyspark and a Azure Gen2 storage connection, so as to enable testing of your data lake with pyspark locally.

Build Docker image

    docker build -t sparklocal .

Run docker image

    docker run -it sparklocal:latest /bin/bash

Run docker image in Visual Studio Code

Install docker

Install extension to run container in another window

Test pyspark locally

Type pyspark into the terminal and you should be promoted with the Spark UI at localhost:4040

    pyspark 

Test docker image

Run python script to test pyspark connection

    python test_spark.py

Run python script to test the connect to Azure Data Lake Storage (ADLS) Gen2

    python test_adls.py

References

Databricks docker deployments

Apache Spark docker deployments

Delta connecting to ADLS Gen2

Connecting to ADLS Gen1 locally

developing inside docker container using visual studio code