Export problems with batch processing

[TheoGervais]

I have analyzed 100 GLs, located at the following location
/scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418-155949/
using 20230418_selected_growthlanes.yaml and mm.properties from /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess.
I have run the export on all my freshly curated data, and I noticed two failure cases :
1) The export procedure just stalls, displaying "Exception in thread "Run$_main" java.lang.IndexOutBoundsException: Index : 0, Size: 0" and a couple other lines in the consol. I can comment this GL in the yaml file and restart the curation, which works alright. This is observed for 6/100 GLs (Pos1_GL4 ; Pos1_GL17 ; Pos8_GL16 ; Pos21_GL3; Pos21_GL5; Pos28_GL2)
2) The export just does not work : when I rerun the export, those get re-exported, while the others display "This has already been exported, it won't be rerun". This is the case for 12/100 GLs (Pos1_GL8 ; Pos1_GL12 ; Pos1_GL15; Pos8_GL6 ; Pos12_GL3 ; Pos16_GL9 ; Pos21_GL12 ; Pos24_GL14; Pos29_GL4; Pos29_GL5; Pos29_GL15; Pos30_GL13). In the corresponding analysis folder, I indeed find no "export" folder. It also does not display any error message. Here's a sample of the content of the export log, when I rerun the export with the already exported YAML file :
2023-05-01 15:09:54,425 - INFO - BATCH RUN STARTED. 2023-05-01 15:09:54,425 - INFO - Batch script version: 0.3.0 2023-05-01 15:09:54,425 - INFO - Run type: EXPORT 2023-05-01 15:09:54,425 - INFO - Command: /scicore/home/nimwegen/GROUP/Moma/MM_Analysis/builds/unstable_20230405__FOR_PREPROCESSING_ONLY__mmpreprocesspy_v0.4.0-5b062368/Preproc/moma_batch_run.py -export /scicore /home/nimwegen/GROUP/MM_Data/Theo/20210507/preprocess/20230418_selected_growthlanes.yaml 2023-05-01 15:09:54,425 - INFO - Any backups created during this run are appended with postfix: __BKP_20230501-150954 2023-05-01 15:09:54,454 - WARNING - Will not perform operation EXPORT for this GL, because it was already exported for this analysis '20230418_analysis' in directory: /scicore/home/nimwegen/gervais/MM_Data/Theo/ 20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418-155949/Pos1/Pos1_GL3/20230418_analysis/export_data__20230418_analysis 2023-05-01 15:09:54,476 - WARNING - Will not perform operation EXPORT for this GL, because it was already exported for this analysis '20230418_analysis' in directory: /scicore/home/nimwegen/gervais/MM_Data/Theo/ 20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418-155949/Pos1/Pos1_GL6/20230418_analysis/export_data__20230418_analysis 2023-05-01 15:10:01,056 - INFO - RUN MOMA: moma -headless -reload /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20 230418-155949/Pos1/Pos1_GL8 -analysis 20230418_analysis -i /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418- 155949/Pos1/Pos1_GL8/20210507_glc_spcm_1_MMStack__Pos1_GL8.tif 2023-05-01 15:10:01,058 - INFO - LOG MOMA: /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418-155949/Pos1/Pos1 _GL8/20230418_analysis/track_data__20230418_analysis/moma.log 2023-05-01 15:10:56,720 - INFO - FINISHED MOMA. 2023-05-01 15:10:56,743 - WARNING - Will not perform operation EXPORT for this GL, because it was already exported for this analysis '20230418_analysis' in directory: /scicore/home/nimwegen/gervais/MM_Data/Theo/ 20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418-155949/Pos1/Pos1_GL11/20230418_analysis/export_data__20230418_analysis 2023-05-01 15:11:03,841 - INFO - RUN MOMA: moma -headless -reload /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20 230418-155949/Pos1/Pos1_GL12 -analysis 20230418_analysis -i /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418 -155949/Pos1/Pos1_GL12/20210507_glc_spcm_1_MMStack__Pos1_GL12.tif 2023-05-01 15:11:03,842 - INFO - LOG MOMA: /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418-155949/Pos1/Pos1 _GL12/20230418_analysis/track_data__20230418_analysis/moma.log 2023-05-01 15:11:55,425 - INFO - FINISHED MOMA. 2023-05-01 15:11:55,451 - WARNING - Will not perform operation EXPORT for this GL, because it was already exported for this analysis '20230418_analysis' in directory: /scicore/home/nimwegen/gervais/MM_Data/Theo/ 20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418-155949/Pos1/Pos1_GL14/20230418_analysis/export_data__20230418_analysis 2023-05-01 15:12:01,875 - INFO - RUN MOMA: moma -headless -reload /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20 230418-155949/Pos1/Pos1_GL15 -analysis 20230418_analysis -i /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418 -155949/Pos1/Pos1_GL15/20210507_glc_spcm_1_MMStack__Pos1_GL15.tif 2023-05-01 15:12:01,876 - INFO - LOG MOMA: /scicore/home/nimwegen/gervais/MM_Data/Theo/20210507/preprocess/20210507_Theo_process_positions_20230418_Analysis_of_missed_positions__output__20230418-155949/Pos1/Pos1 _GL15/20230418_analysis/track_data__20230418_analysis/moma.log 2023-05-01 15:12:55,762 - INFO - FINISHED MOMA.

So it means that in total, I get 18/100 Growth lanes that were curated, but that I cannot export, which is a really substantial amount.

[michaelmell]

Hi Theo.
Thanks for reporting this.

1. For the first issues: For these GLs the ILP became infeasible during tracking and then the export fails. I have not yet observed this issue in other datasets. Perhaps this is related to issues with the settings. I will look into why this happens.
2. As for the second issue: Lis reported this issue with GLs being skipped during export. I created an issue for it here and looking into it.

[michaelmell]

Hi Theo.
As discussed, I removed the prune-roots for these GLs:
Pos1_GL8 ; Pos1_GL12 ; Pos1_GL15; Pos8_GL6 ; Pos12_GL3 ; Pos16_GL9 ; Pos21_GL12 ; Pos24_GL14; Pos29_GL4; Pos29_GL5; Pos29_GL15; Pos30_GL13
You should now be able to reopen them, add the correct pruning, save, and then export.