Install: BlueGene Q

These are a collection of notes by Aron Ahmadia on building the Proteus stack on the IBM BlueGene/Q environments at the Argonne Leadership Computing Facility.

These notes should be of general use in building and installing scientific Python stacks on semi-cross-compiled environments, but will also contain specific pointers to working on the ALCF machines with Proteus.

Python, NumPy, and SciPy have been partially ported to the BlueGene/Q by IBM. There is some documentation about the ports in:

• /bgsys/drivers/V1R2M2/ppc64/tools/python
• /bgsys/drivers/V1R2M2/ppc64/tools/numpy
• /bgsys/drivers/V1R2M2/ppc64/tools/scipy

Unfortunately, these ports are not quite ready for a variety of reasons. They are slightly out of date, the Python build system has changed considerably on a point release due to the addition of cross-compiler settings to the configure and build scripts, and they hardcode the choice of compiler to a non-system (at least on Vesta) location.

I've gradually ported the builds for each of these packages into HashDist, starting with Python. When working on Vesta, it is important to keep in mind that the front-end nodes (fens) execute a different operating system than the compute-nodes (cns). As a result, code compiled and linked for one architecture tends not to work on the other. To test executables on the back-end, start up an interactive job like this (you may need to choose a different project):

qsub -I -t 1:00:00 -n 4 --project=proteus-hashstack

This will pause until a partition can be allocated for you. Once the partition is allocated, you will be placed back into a shell on the fen, with the ability to submit jobs to the partition:

runjob --block $COBALT_PARTNAME --np 4 -p 16 : /home/aron/hello

More information about runjob, which is particular to ALCF, can be found here.

Non-Interactive Jobs

If you are having difficulty running interactive jobs, you can submit shell scripts instead to the queuing system.

Here's an example script:

#PBS -o out.log
#PBS -n 4
#PBS -t 00:10:00

resoft
date
runjob --block $COBALT_PARTNAME --np 4 -p 16 --envs LD_LIBRARY_PATH=/home/aron/hashstack/proteus.bgq.gnu/lib64 : /home/aron/hashstack/proteus.bgq.gnu/bin/python -c "print 'hello'"
date

You can submit it with the following command:

qsub -t 1:00:00 -n 16 --project=proteus-hashstack --mode script ./test_python.qsub

Note that until the Python has been post-processed with patchelf, it will need an LD_LIBRARY_PATH hack.

References

• Bertrand Brelier's slides on porting the Scientific Python stack to BlueGene/Q
• IBM BlueGene/Q Application Developer's Guide
• Jeff Hammond's BG/Q at ALCF Guide