Aberration mapping

Python based notebooks for analyzing optical aberration modes in confocal light microscopy using Z-stacks of standard fluorescent beads.

See https://en.wikipedia.org/wiki/Optical_aberration

Based on Debayan Saha's Ph.D. thesis [ref].

Requires ssh access and project space on falcon.

Excecute the notebooks in order on a local machine:

1. 01_process_patches.ipynb
2. 02_analyse_zola_results.ipynb

Processing happens as a slurm job on gpu nodes of the cluster.

Analysis happens locally. Currently (2023-11-07), final analyses are not copied back to the remote.

Further details

The 01_process_patches.ipynb selects the beads, processes the patches around the beads, and runs a PSF model on each bead.

Logistics of 01_process_patches.ipynb

The logistics of the 01_process_patches.ipynb notebook are complicated.

It is intended to run on a local computer that has ssh access to the CBG cluster (falcon).

It does the pre-processing locally and generates a folder patches_for_zola with 50 sub-folders, each containing TIFF files of a Z-stack of each selected bead (eg planes_14_z_35_y_1227_x_369.tif) .

The notebook copies these local bead files to the remote (falcon), along with a fiji macro and a bash shell script necessary for processing on the HPC cluster.

To process the beads, the notebook starts a number of batch jobs on gpu nodes of the cluster as follows:

! ssh falcon sbatch {remote_zola_script_path} '{remote_fiji_path} {remote_fiji_macro_path} {remote_imageJ_macro_parameter_string}'

1. SLURM calls a bashscript (currently bashscript_fiji3.sh) to start the processes. The script takes as paramters the location of the ImageJ binary remote_fiji_path, the macro to be run zola_macro_cluster_params_passed.ijm and a string containing the parameters for the PSF modeling plugin (the plugin is ZOLA_-0.2.8-SNAPSHOT.jar).
2. The bashscript starts a headless instance of ImageJ,
3. The headless instance of ImageJ is passed a command line argument that runs a macro to start the Zola analysis.
4. The parameters for the plugin and the filename for the results are passed as a single string which is parsed by the macro into a filepath on the remote for the results, and the parameters needed for the "Calibration: PSF modeling" plugin.

After the SLURM jobs are done running, there is a folder called zola_raw in the patches_for_zola folder on the remote. For each bead, there is a .json file with the amplitudes of each Zernike mode, as a list, along with some metadata.

The 01_process_patches.ipynb notebook concludes at this point, so you can run it top-to bottom and leave the jobs processing on the HPC.

The 02_analyse_zola_results.ipyb notebook copies the zola_raw folder from the remote server for local analysis.

Todo: copy analysis to the remote server (falcon) so that local and remote copies match.

Notes on agreement between my notebooks and Deb's, and the create_abb_map function

I was having trouble reproducing the results Andrea obtained using Deb's notebooks: the aberration maps looked different and the smoothed abberations maps were missing or did not correspond.

Some of the discrepancy is between which beads were selected. The selection is set in the Results.csv file. In Deb's workflow, this file is generated by hand. It is the location of the bead centroids determined by a maximum instensity projection through the stack. Depending on how the threshold is set, different sets of beads are identified.

If I use the Results.csv files from existing analyses, there are still some discrepancies, depending on how beads are selected or rejected. Carefully matching the parameters, I get a close match to the bead selection, but not always the same.

However, corresponding archive and newly analysed beads have the same abberation values for each Zernike mode, indicating that the PSF analysis is functioning properly.

I located a problem with the measurements array. Deb's had shape (Z, 1, 1) where Z is the number of Z slices. For pyclesperanto functions to work properly, it needed to be shape (1,1,Z). Deb had taken the transpose of the array (don't know why). When I left the array un-transposed, the smoothed abberation maps seem reasonable.

The discrepancy in the overall appearance of the aberration maps may be due to which points are accepted and rejected. The voroni expansion will be different depending on the set of points selected.