Group project for the course Large Datasets for Scientific Applications, Spring 2016.
- 2+ instaces on an OpenStack-based cloud system (Preferebly running Ubuntu 14.X)
- pysam
- Apache Spark
- python-swiftclient
- python-keystoneclient
- exported username, password, api_url for swift
The commands below assume that you already have set up a spark cluster and installed the prerequisites
git clone https://github.com/adamruul/LDSA2016.git
sudo ./spark-submit --master spark://pmo:7077 --driver-memory 6g --executor-memory 2g ~/LDSA2016/main.py
- Daniel Eliassen
- Octave Mariotti
- Adam Ruul
- Marcus Windmark