Skip to content

Run HCP pipeline on PNL GPU machines

Jump to bottom
Tashrif Billah edited this page Oct 17, 2024 · 6 revisions

Parallel Execution

(This section is written by Tashrif Billah)

hcp_pnl_topup.lsf was originally written for ERIS GPU cluster. But with dwindling hope of the latter's availability, we have transformed the script to work in PNL GPU machines with minimal modification necessary.

This code block already decides between environments in PNL and ERIS: hcp_pnl_topup.lsf#L73-L79

However, it could not be run parallelly so far. To achieve this step, we are making use of the LSB_JOBINDEX parameter within the script. The idea is that we shall provide this parameter from outside the script so it insinuates LSF/bsub's LSB_JOBINDEX value.

for LSB_JOBINDEX in {1..N/2}
do
  echo $LSB_JOBINDEX
  export LSB_JOBINDEX
  /data/pnl/soft/pnlpipe3/luigi-pnlpipe/workflows/hcp_pnl_topup_goldstein.lsf > /path/to/output/${LSB_JOBINDEX}.txt 2>&1 &
done

A second for-loop can be written with indices {N/2+1..N}. That way we can process two cases in parallel. Care needs to be taken so that N/2+1 is not an odd number like 1. Because we do not want both jobs to route to the same CUDA_VISIBLE_DEVICES. See the script to know how CUDA_VISIBLE_DEVICES is related to LSB_JOBINDEX.

Advanced Parallel Execution using Batching

(This section is written by Ryan Zurrin)

Given the varying computational power of different systems and the need to run multiple tasks in parallel without overwhelming the resources, we have developed a robust parallel execution method. This approach allows you to define a caselist and by default will set a limit on the maximum number of parallel jobs, based on available GPUs or other resource considerations.

Warning

The above for loop contains the trailing & sign after the hcp_pnl_topup_goldstein.lsf command. This will launch each task in the background immediately after starting the previous one.

If you inadvertently set a large value for N, such as 100, and execute the script, it will attempt to run all 100 tasks simultaneously. This can heavily burden or even crash the system. It's imperative to:

  1. Understand the resources each task will consume versus the system's capacity.
  2. Limit the number of parallel tasks to a number safe for your environment.
  3. Always test with a small number of tasks initially to ensure everything runs without issues.

If uncertain, consult a system administrator or someone familiar with the environment before running tasks in parallel.

Implementation

The following script demonstrates a batching approach:

#!/bin/bash

# Define the path for the list of cases.
CASELIST_PATH="/path/to/your/caselist.txt"

# Define the directory where the output logs for each job will be saved.
OUTPUT_DIR="/path/to/your/output/directory/"

# Define the path to the script that will be run for each case.
SCRIPT_PATH="/path/to/your/script.sh"

# Define the maximum number of parallel jobs you want to run at a time.
MAX_PARALLEL_JOBS=$(nvidia-smi -L | wc -l)

# set the start time for time profiling
START_TIME=$(date +%s)

# Export CASELIST_PATH so it can be accessed by any scripts or programs we run.
export CASELIST_PATH

# Count the total number of cases.
# 'wc -l' counts the number of lines in the file.
TOTAL_CASES=$(wc -l < "$CASELIST_PATH")

# This function is responsible for running a "batch" of jobs.
run_batch() {
    # The start and end indices of the batch are passed as arguments.
    local start=$1
    local end=$2

    # Declare an array to keep track of the background job PIDs within this batch.
    declare -a local_jobs

    # Iterate over each index in the batch range.
    for (( j=$start; j<=$end; j++ ))
    do
        # Set the current index as LSB_JOBINDEX, which may be used by the SCRIPT_PATH.
        export LSB_JOBINDEX=$j
        echo "Starting job for LSB_JOBINDEX=$LSB_JOBINDEX"

        # Run the script in the background (& at the end).
        # Any output (both stdout and stderr) will be redirected to a file named after the current index.
        "$SCRIPT_PATH" > "${OUTPUT_DIR}/${LSB_JOBINDEX}.txt" 2>&1 &

        # Save the PID (process ID) of the background job we just started in our array.
        local_jobs+=("$!")
    done

    # This loop waits for each job in the current batch to complete.
    # It ensures we don't start a new batch until the current batch is done.
    for job in "${local_jobs[@]}"
    do
        wait $job
    done
}

# This loop schedules the jobs in batches.
for (( i=1; i<=$TOTAL_CASES; i+=MAX_PARALLEL_JOBS ))
do
    # Calculate the last index of the current batch.
    end=$((i + MAX_PARALLEL_JOBS - 1))

    # If the calculated end exceeds the total number of cases, adjust it.
    if [[ $end -gt $TOTAL_CASES ]]
    then
        end=$TOTAL_CASES
    fi

    # Run the current batch.
    run_batch $i $end
done

# Once all batches have been processed, print a completion message.
echo "All jobs completed!"

# Print the elapsed time.
echo "Elapsed time: $(($ELAPSED_TIME/60)) min $(($ELAPSED_TIME%60)) sec"

Instructions

  1. CASELIST_PATH: Replace /path/to/your/caselist.txt with the actual path to your list of cases or tasks.

  2. OUTPUT_DIR: Replace /path/to/your/output/directory/ with the directory where you want to save the logs for each job.

  3. SCRIPT_PATH: Replace /path/to/your/script.sh with the path to the script that needs to run for each case or task.

  4. MAX_PARALLEL_JOBS: This will not need to be changed by default, as it will set the total parallel jobs to the number of available GPUs. It can be set if this is not desired number of parallel jobs required for your tasks.

  5. Run the script. It will process the tasks in batches, ensuring that no more than MAX_PARALLEL_JOBS tasks are run in parallel.

Adapt Script for Use with Batch Parallel Wrapper

In the original script, you will need to include the following block:

# Check if $caselist is a file
if [ -f $caselist ]; then
    # Extract the ${LSB_JOBINDEX}-th line from the file
    c=`head -${LSB_JOBINDEX} $caselist | tail -1`

    # Determine the number of GPUs and set the GPU to use
    NUM_GPUS=`nvidia-smi -L | wc -l`
    export CUDA_VISIBLE_DEVICES=$(( ${LSB_JOBINDEX} % ${NUM_GPUS} ))
else
    # Assign the value of $caselist to c
    c=$caselist
fi

This will ensure the GPU device is set properly based on the job index.

Benefits

  • Resource Management: By setting the MAX_PARALLEL_JOBS, you can maximize your computational resources without overburdening your system.

  • Flexibility: You can provide a caselist as extensive as you need without manually splitting the list or managing individual processes.

  • Logging: Each job's output is saved in a separate file, making it easy to track and debug.

For further questions or if you encounter any issues during execution, please create an issue in the repository's issues section. Ensure to provide a detailed description of the problem for a faster resolution.

Clone this wiki locally