-
Notifications
You must be signed in to change notification settings - Fork 2
Setting up python3 and mpi4py
The best alternative seems to be to use a python interpreter that is somehow provided by the super computer and create environment on top of that.
Installing python from zero with, for example, conda might be needed in some cases, but always check if there is a recommended version of python that is already preinstalled. The reason to mention this is that in some cases (as you can see in comments below) conda packages conflict with system ones and some errors arise.
If the supercomputer does not have optimized python installation with preinstalled modules, then you can skip one section and try to set up conda. However if python already exist in the super computer, it is beneficial to use it and add only a limited number of extra modules needed.
In Lumi, the following works. Where cray-python is the python module they have configured. We want to create a python environment on top of that to use their mpi4py implementation. Note that for this, we need to pull any preinstalled module into our environment. The instruction are in the script:
# Inputs
python_module_name=cray-python
env_name="craypython"
outer_path=$PWD
# Load the python module
echo "Load the appropiate modules"
module load $python_module_name
# Create the python enviroment
echo "Create the virtual enviroment, added flag to import the system preinstalled modules"
python3 -m venv --system-site-packages $outer_path"/"$env_name
echo "The script finalizes now but you are not done installing modules."
echo "Activate the enviroment you just created with $(echo source $outer_path"/"$env_name"/bin/activate") and install whatever you need"
exit 0
# Activate the enviroment
echo "activate the enviroment"
source $outer_path"/"$env_name/bin/activate
Then you can install any modules on top of that as usual with pip
To use python with other libraries in C/FORTRAN/C++ that have python bindings, you need to make sure that mpi4py and the libraries are built with the same MPI.
This might not be easy to match if you have mpi4py preinstalled in a python module by the supercomputer. So far the solution found is to use the python of the computer to create a new environment but NOT using the preinstalled packages, i.e. editing from the command above:
python3 -m venv $outer_path"/"$env_name
After activating the environment, you can then install mpi4py specifying the mpi wrapper:
env MPICC=$(which cc) python -m pip install --no-cache-dir mpi4py
Afterwards, you just need to make sure that the proper cc wrapper is used for any other application you want to use (like ADIOS2)
In local workstations and some super computers, system wide conda installations are available and can be used. Here we share some scripts we have found useful over the years to set up environments fast and easy. If something does not work scriptwise. It typically does typing the commands one by one.
First you should create an enviroment. It is typically good to change the env and pkgs directory (as done here), since the default is to install enviromentin the home directories, which might get full easily in some cases.
#!/bin/bash -l
## For installing mpi4py in super computers check: https://researchcomputing.princeton.edu/support/knowledge-base/mpi4py
## For fixing the problem of wheel not building in super computers check https://stackoverflow.com/questions/72680287/not-able-to-install-mpi4py-using-pip
# Inputs
outer_path=$PWD
conda_pkgs="miniconda3/.conda/pkgs"
conda_envs="miniconda3/.conda/envs"
CC="$(which cc)"
env_name="python3.10"
python_ver="3.10"
# identify the name of the loader with ls miniconda3/bin/*ld
loader="miniconda3/bin/x86_64-conda-linux-gnu-ld"
#==========================================================================#
# Change the location of the packages and enviroments to not fill up home directories.
mkdir -p $conda_pkgs
mkdir -p $conda_envs
conda config --add pkgs_dirs $conda_pkgs
conda config --add envs_dirs $conda_envs
# Create an enviroment to work
conda create --name $env_name python=$python_ver
echo "use find / -name libstdc++.so.6 2>/dev/null to see where your libstdc++ is. If you get some troubles, consider deleting the one in the enviroment, as might conflict with system"
Read the comments on the following snippet of code. Do not do anything with the linker at first and try does if just the simple commands to install mpi4py result in an error.
If an error in the spirit of: "can not link mpi applications, check your configuration" appears, then try to do the link trick. Apparently rhis was neede din python 3.9. Have not checked for newer versions.
# Activate the enviroment
conda activate $env_name
# ONLY REALLY NEEDED IF MPI4PY CAN NOT LINK
echo "If wheel fix does not work in the script, do it yourself in the shell"
# Just before installing mpi4py add the proper linker
rm -f miniconda3/.conda/envs/$env_name/compiler_compat/ld
ln -s /usr/bin/ld miniconda3/.conda/envs/$env_name/compiler_compat/ld
# Start by installing mpi for python
export MPICC=$(which $CC)
echo $MPICC
pip install mpi4py --no-cache-dir
# ONLY REALLY NEEDED IF MPI4PY CAN NOT LINK
echo "If wheel fix does not work in the script, do it yourself in the sheel"
# After installing mpi4py, go back to the python linker
rm -f miniconda3/.conda/envs/$env_name/compiler_compat/ld
ln -s $loader miniconda3/.conda/envs/$env_name/compiler_compat/ld
echo "If your system is cray, When mpi4py is built locally with cray-mpich, cray-mpich dependencies on system libraries are essentially pulled into the environment. The system libssh is not compatible with some versions of the openssl package that are automatically included in conda environments."
echo "a solution from the internet is to install conda install -c conda-forge libssh. As seen in: http://archive.ambermd.org/202305/0076.html"
echo "proceeding to install. You can just say no after it collect the packages"
conda install -c conda-forge libssh