Skip to content

Tutorial for ProcessND.py

Jump to bottom
jdkio edited this page Jul 15, 2025 · 22 revisions

ProcessND Tutorial (Updated for AL9)

⚠️ Warnings – Please Read First

✅ Use NERSC if Possible

ProcessND.py is now outdated. The official way to generate samples is through NERSC. Contact #nd_production or #nd_reco_sim on Slack for the most up-to-date methods.

🔁 Reuse Existing Samples

Before generating anything new, check if a suitable sample already exists. Coordination leads to better use of resources—talk to others about shared sample needs.

⚠️ Get Approval for Large Jobs

Large jobs cost time and computing resources. Always get permission and notify #nd_production and/or #nd_reco_sim before launching grid jobs.

✅ Always Run a Small Test First

Validate everything before scaling up. Some errors only show up at later stages (e.g., edep files exist even when genie failed). Review log files carefully.

✅ AL9 Compatible (SL7 No Longer Required)

These scripts now run natively in AL9! SL7 containers are no longer required to run the production script. Still needed to compile dune-tms to run on the grid though.

📂 Make Your Output Useful to Others

Save your samples in a persistent, accessible location. Document settings clearly for reproducibility.

❓ Why Does This Tutorial Still Exist?

  1. Sometimes it can take a while to get a NERSC sample
  2. Some features like pileup are still easier through grid jobs.

🔧 Setup Instructions

source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
cd <your local ND_Production clone>
cd toolbox/scripts

☎️ Download instructions

source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
cd /exp/dune/app/users/$USER/<your production subdir>
git clone git@github.com:DUNE/ND_Production.git
cd ND_Production

🛠️ Description of ProcessND.py

ProcessND.py is a wrapper script that creates processnd.sh, which you submit to the grid with jobsub_submit.

🧰 jobsub_submit Key Options

  • -N: Number of output files. If generating, total POT = POT per file × N. If processing existing files, set -N to the number of input files.

  • --memory=4000MB: A good starting point. Double it for overlay.

  • --expected-lifetime=24h: Set as close to actual runtime as possible to improve scheduling.

  • --tar_file_name: Path to your dune-tms tarball. Tag and push your version in Git for reproducibility:

dune-tms should still be compiled in SL7 for compatibility with current worker nodes. Use:

/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
  -B /cvmfs,/exp,/nashome,/pnfs/dune,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf \
  --ipc --pid \
  /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest

# then setup with setup.sh (sl7 setup), and compile
# Tag for reproducability
git tag <tag_name>
git push origin <tag_name>
# then go down one directory with cd .. and do the following
tar -czvf dune-tms.tar.gz dune-tms
mv dune-tms.tar.gz /pnfs/dune/persistent/users/$USER/dune-tms_tarfiles/your_tarfile.tar.gz

🧪 Example: TMS-only Sample

python ProcessND.py --stages tmsreco \
  --indir /pnfs/dune/persistent/users/abooth/.../EDEPSIM_SPILLS/00000/ \
  --outdir /pnfs/dune/scratch/users/$USER/nd_production/my_test

jobsub_submit --group dune --role=Analysis -N 100 \
  --singularity-image /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest \
  --expected-lifetime=24h \
  --append_condor_requirements='(TARGET.HAS_Singularity==true&&TARGET.HAS_CVMFS_dune_opensciencegrid_org==true&&TARGET.HAS_CVMFS_larsoft_opensciencegrid_org==true&&TARGET.CVMFS_dune_opensciencegrid_org_REVISION>=1105)' \
  --memory=4000MB \
  --tar_file_name dropbox:///pnfs/dune/persistent/users/kleykamp/dune-tms_tarfiles/2025-02-13_Kalman_simplest_B_field.tar.gz \
  file://processnd.sh

🧪 Example: Full Chain (gen + g4 + tmsreco)

python ProcessND.py \
  --outdir /pnfs/dune/scratch/users/$USER/nd_production/full_chain \
  --geometry_location /pnfs/dune/persistent/physicsgroups/dunendsim/geometries/TDR_Production_geometry_v_1.0.3/nd_hall_with_lar_tms_sand_TDR_Production_geometry_v_1.0.3.gdml \
  --manual_geometry_override nd_hall_with_lar_tms_sand_TDR_Production_geometry_v_1.0.3.gdml \
  --topvol volDetEnclosure \
  --pot 1e15 \
  --stages gen+g4+tmsreco
  • Use volDetEnclosure to limit to LAr+TMS+Sand. Avoid volWorld unless simulating full rock detector.

🏃 Running Locally

One can run locally with the --test flag. This puts everything in the test directory locally but will otherwise be the same. Make sure to run inside an SL7 container as if you're the grid:

# Start SL7 container
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
  -B /cvmfs,/exp,/nashome,/pnfs/dune,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf \
  --ipc --pid \
  /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
# Run the command created with ProcessND.py, make sure to use the --test flag
./processnd.sh

🔄 Optional Modifications

🔁 Run Only Genie + EDepSim

--stages gen+g4

🧊 LAr-Only Simulation

--topvol volLArBath

Refer to the [geometry viewer](https://dune.github.io/dunendggd/) and Slack channel #nd_geometry for help.

🌊 Overlay (Pileup)

--timing spill --overlay
  • Increase memory to 8000MB at least.

↔️ Off-Axis Simulation

--oa <distance> --dk2nu
  • gsimple flux files do not support off-axis info.

🧲 Magnetic Field Variants

B-field support is possible but not fully validated in this script. Ask around if you need this.

⚠️ Known Gotchas

❌ Wrong Working Directory

Run from inside the scripts/ directory:

cd scripts
python ProcessND.py

Otherwise it won’t find template.sh.

Questions? Ask in #nd_production, #nd_muon_spectrometer, or #nd_muon_spectrometer_code on Slack.

Clone this wiki locally