HPCExecutor configurations

[zmaalick]

Hi,

We are trying to run HPCExecutor.

Here are our configurations:

[executor]
executor = "climate_ref.executor.HPCExecutor"

[executor.config]
address = "eth0"
address_probe_timeout = 30
scheduler = "slurm"
account = "normal" # our account name
username = "my_username"
partition = "cpu"           # or "debug" for testing
#qos = "debug"                   # quality of service
req_nodes = 1                   # number of nodes
walltime = "00:30:00"           # time limit per job
cores_per_worker = 1            # cores per worker
max_workers_per_node = 4       # workers per node
scheduler_options = ""
worker_init = "export PARSL_HTEX_ADVERTISED_ADDRESS=10.138.80.53; export PARSL_HTEX_ADDRESS=10.138.80.53;"

but we are getting connection error:

0:     raise ConnectionError(f"No viable ZMQ url from: {addys}")
0: ConnectionError: No viable ZMQ url from: tcp://127.0.0.1:54201, tcp://10.138.80.53:54201, tcp://151.170.240.197:54201
srun: error: sla-cpu-r-03: task 0: Exited with exit code 1

I above settings we did a bit of work around by adding scheduler_options but didn't work. Anything else we need in configurations?

[lewisjared]

@minxu74 Thoughts?

[minxu74]

@zmaalick Thanks a lot for reporting the issue. The error is from parsl. The HPCExecutor is only a thin layer to call the parsl executor from REF. So, for your configurations, the option address and address_probe_timeout won't be passed to the parsl as I did not put all parsl options to be available to users. Also, I am not aware of the environment variables used in worker_init. The address auto probing from parsl worked very well in DOE HPCs, but it may not work properly in some HPCs as said in parsl FAQ. To pinpoint the reason, could you confirm:

• REF was run in the login node of your HPC
• 10.138.80.53 and 151.170.240.19 is the internal and external IP addresses of the login node that you ran REF.

If so, could you please run REF again following the steps:

• freshly log into your HPC, i.e., exist and relogin
• remove the address options and environment variables in the worker_init
• run ref
• in the directory that you run ref, there will be runinfo directory, could share the log files under it if you have them.

Thanks.

[zmaalick]

Here are the files:

parsl.log

interchange.log

[zmaalick]

Submit Script:

#!/bin/bash

#SBATCH --job-name=parsl.ref_hpc_executor.block-0.1772735870.9763343
#SBATCH --output=/data/scratch/zubair.maalick/runinfo/000/submit_scripts/parsl.ref_hpc_executor.block-0.1772735870.9763343.stdout
#SBATCH --error=/data/scratch/zubair.maalick/runinfo/000/submit_scripts/parsl.ref_hpc_executor.block-0.1772735870.9763343.stderr
#SBATCH --nodes=1
#SBATCH --time=30
#SBATCH --ntasks-per-node=1

#SBATCH --exclusive
#SBATCH --partition=cpu
#SBATCH --account=normal





export JOBNAME="parsl.ref_hpc_executor.block-0.1772735870.9763343"

set -e
export CORES=$SLURM_CPUS_ON_NODE
export NODES=$SLURM_JOB_NUM_NODES

[[ "1" == "1" ]] && echo "Found cores : $CORES"
[[ "1" == "1" ]] && echo "Found nodes : $NODES"
WORKERCOUNT=1

cat << SLURM_EOF > cmd_$SLURM_JOB_NAME.sh
process_worker_pool.py  --max_workers_per_node=4 -a 151.170.240.197,127.0.0.1,10.138.82.232 -p 0 -c 1 -m None --poll 10 --port=54318 --cert_dir /data/scratch/zubair.maalick/runinfo/000/ref_hpc_executor/certificates --logdir=/data/scratch/zubair.maalick/runinfo/000/ref_hpc_executor --block_id=0 --hb_period=30  --hb_threshold=120 --drain_period=None --cpu-affinity none  --mpi-launcher=mpiexec --available-accelerators 
SLURM_EOF
chmod a+x cmd_$SLURM_JOB_NAME.sh

srun --ntasks 1 -l  bash cmd_$SLURM_JOB_NAME.sh

[[ "1" == "1" ]] && echo "Done"

[minxu74]

Thanks. Did you have any logs under runinfo/000/ref_hpc_executor/block-0/*/*.log? I will create a branch in which I will let users use full lists of arguments of parsl functions. Also, I could implement the ThreadPoolExecutor w/o ZeroMQ for users to run ref in a node. I will let you know when it is ready. The quick tryout is as follows if you could modify source code climate-ref/packages/climate-ref/src/climate_ref/executor/hpc.py:

climate-ref/packages/climate-ref/src/climate_ref/executor/hpc.py

Line 328 in 4601968

executor = HighThroughputExecutor(

You could add address in the HighThroughputExecutor function. .