Skip to content

Latest commit

 

History

History

sbt-assembly-on-yarn

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 

Example project that uses sbt-assembly to package all dependencies and submit to a YARN cluster.

Set environment variable HADOOP_CONF_DIR or YARN_CONF_DIR so spark-submit can pick up cluster configuration.

It is highly recommended to point spark.yarn.jar to SPARK assembly JAR in spark-defaults.conf. This will avoid uploading the Spark assembly JAR to HDFS every time an application is submitted.