5. Prune correlated stats

We did steps 4-11 on volumes attached to CyVerse or Jetstream instances.

Working in /mnt/ABC_AJmodels_instant_newreal. (/mnt is the attached volume)

working with files:
input_ABCtoolbox_M1_HPC_OSG_2.txt
input_ABCtoolbox_M2_HPC_OSG_2.txt
input_ABCtoolbox_M3_HPC_OSG_2.txt
which are a combination of simulations with instantaneous growth from HPC and OSG, and have 1,275,807, 1,446,125, 1,343,711 simulations.

---

Use python script subset_stats/main_subset_real.py and subset_stats/main_subset_sim.py to create new file without correlated summary statistics.

/mnt/src/macsswig_simsaj/macss_env/bin/python /mnt/src/macsswig_simsaj/subset_stats/main_subset_sim.py ABC_correlatedstats1_100000.log input_ABCtoolbox_M1_HPC_OSG_203.txt remove

/mnt/src/macsswig_simsaj/macss_env/bin/python /mnt/src/macsswig_simsaj/subset_stats/main_subset_sim.py ABC_correlatedstats1_100000.log input_ABCtoolbox_M2_HPC_OSG.txt remove

/mnt/src/macsswig_simsaj/macss_env/bin/python /mnt/src/macsswig_simsaj/subset_stats/main_subset_sim.py ABC_correlatedstats1_100000.log input_ABCtoolbox_M3_HPC_OSG.txt remove

/mnt/src/macsswig_simsaj/macss_env/bin/python /mnt/src/macsswig_simsaj/subset_stats/main_subset_sim.py ABC_correlatedstats1_1218497.log real_output_M23_IBD.summary remove

Double check the number of stats with

head -1 pruneCorStats_real_output_M23_IBD.summary | tr '\t' '\n' | wc -l
head -1 pruneCorStats_input_ABCtoolbox_M1_HPC_OSG_203.txt | cut -f23- | tr '\t' '\n' | wc -l
head -1 pruneCorStats_input_ABCtoolbox_M2_HPC_OSG.txt | cut -f25- | tr '\t' '\n' | wc -l
head -1 pruneCorStats_input_ABCtoolbox_M3_HPC_OSG.txt | cut -f27- | tr '\t' '\n' | wc -l