v1.0.0-beta.9

v1.0.0-beta.9 - Enhancements to Argparse Formatting and Usability

This release introduces significant improvements to argparse help formatting, leveraging the Rich library for enhanced readability and usability in terminal output.

New script help_formatter.py (replaces argparse_utils.py):

• Rich-Enhanced Argparse Help: Implemented RichArgumentParser and CustomHelpAction to integrate richly formatted help messages, including styled docstrings and argument lists.
• SuppressMetavar Class: Updated documentation on nargs configurations to avoid issue with argparse help wrapping to ensure consistent display across different terminal sizes.
• format_argparse_help Function: Applies custom styles to argparse help text, highlighting flags, defaults, and sections.
• format_docstring_for_terminal Function: Formats module docstrings with Rich's styled output for enhanced terminal display, improving readability by highlighting sections, command names, and key elements.

Improvements:

• Refactored get_samples(): Refactored for better flexibility and clearer output, now consolidated with the -e and -d arguments for directory processing.
• New: utils_get_samples Command: For testing --pattern and --dirs args in commands that perform batch processing
• Simplified Sphinx Documentation: Addressed Sphinx warnings and improved compatibility by replacing <asterisk> with * in help_formatter (except when quotes are used)
• Refactored Scripts: Various scripts, such as copy_specific_slices, filter_csv.py, and seg_ilastik, have been refactored for improved functionality and clarity.

Fixes:

• Sphinx Warnings: Resolved multiple warnings during documentation builds, ensuring a cleaner and more reliable doc generation process.

v1.0.0-beta.8

Summary of Changes

New Features:

• New Scripts:
  • filter_csv.py: Allows filtering of a CSV file by a specified column based on a list of values (useful for filtering out data from one hemisphere).
  • gubra_to_CCF_50um.py and gubra_to_CCF.py: For warping from Gubra space to CCF space.
  • points_compressor.py: Compresses coordinate points per region and can output a regional summary.
  • points_to_atlas.py: Warps a point coordinate image to atlas space.
  • resample_points.py: esampling points and saving them as NIfTI images.
  • points_to_img.py: Converts point data into images for visualization.
  • img_to_points.py: Converts images back into point data for further analysis.

Updates and Improvements:

• Script and Function Updates:

  • Converted generators to lists to handle scenarios where no files match, and the script needs to iterate over the list of matched files.
  • Added an option to move files instead of copying them.
  • Refactored functions to work with a DataFrame instead of ndarray for point data.
  • Corrected handling of single float values for current_res and target_res in the resample_and_convert_points function.
  • Fixed logic for saving output and determining whether to use the -o option.
  • Improved the code for updating the affine matrix to avoid affecting the origin and refactored resample_nii() to reduce parameters.

• Documentation:

  • Slicer 3D installation link and functions of dependencies are now mentioned.
  • Updated the installation guide and terminal cheat sheet, adding new video links for beginners.
  • Fixed multiple Sphinx warnings by updating docstrings and removing problematic parameters like *args and **kwargs.
  • Added __doc__ to epilog and improved the clarity of argument descriptions.

• Configuration and Dependency Management:

  • Fixed the default path for the -fri option.
  • Pinned Python version to 3.11 in GitHub Actions due to compatibility issues with distutils in Python 3.12.
  • Updated command names starting with cluster_ to cstats_ for consistency with vstats and rstats commands.

• Bug Fixes:

  • Fixed a warning related to plotting bar graphs.
  • Resolved issues when using small terminal windows with nargs='+' and action=SM.
  • Corrected typos and addressed minor bugs in the handling of command options and argument processing.

These changes introduce new functionality, improve existing scripts, and address bugs.

v1.0.0-beta.7

Release Update Summary

Recent Changes

• Command Renaming:

  • Commands starting with cluster_ have been renamed to cstats_ to ensure consistency with existing vstats and rstats commands.

• Guide Updates:

  • Added a terminal cheat sheet and new video links aimed at beginners.
  • Simplified argument structure to enhance usability.
  • Update the installation guide

• CSV and Logic Updates:

  • Renamed CSV files, updated help documentation, revised arguments, and adjusted logic.
  • Removed specific CSV files: region_ID_side_name_abbr.csv and sunburst_IDPath_Abbrv_wo_root.csv.

• Template and Defaults:

  • Updated information on the iDISCO/LSFM template.
  • Set the defaults to use CCFv3 (30 µm) with the iDISCO/LSFM template. The template has been warped to CCF space with refined alignment.

These changes enhance consistency, simplify usage, and improve documentation, aligning with updates to templates and default settings.

v1.0.0-beta.6

UNRAVEL v1.0.0-beta6 Release Notes

New Features

1. Resampling and Warping:

   • Added resample.py for resampling .nii.gz images.
   • Introduced warp_forward.py to forward warp a moving image (e.g., warping an atlas space image to unpadded tissue space [does not scale to full res like warp_to_native])

2. Support for CCFv3 2020:

   • Added CSV for the 2020 version of CCFv3.
   • Set some defaults for CCFv3 2020 (remaining defaults will be updated in next release).
   • Implemented features to save cell centroids and calculate label densities.

3. Image and Label Processing:

   • New :labels_to_masks.py for converting ilastik segmentation TIF series to masks.
   • New: mean_IF.py and mean_IF_summary.py for plotting data for each cluster.
   • Updated prism.py to work with data from mean_IF.py.

Improvements

1. Documentation and Help:

   • Updated installation guide with improved instructions and added sections on updating UNRAVEL.
   • Added links to the PyPI page for heifetslab-unravel.
   • Improved help messages and error handling across scripts.
   • Addressed Sphinx warnings for better documentation generation.

2. Code Refactoring and Optimization:

   • Refactored code to use decorators for printing commands and times.
   • Enhanced logging and verbosity options.
   • Updated logic for handling image registration and segmentation.
   • Removed redundant dependencies and improved package configurations.

3. Bug Fixes:

   • Fixed issues with file existence checks and path handling.
   • Corrected typos and formatting errors in documentation.
   • Pinned numpy version at 1.24.4 to avoid warnings in Windows.
   • Addressed specific bugs related to argument handling and default values.

4. Miscellaneous:

   • Added sunburst_RGBs.csv customization.
   • Improved error handling for various scripts and functions.
   • Updated arguments and default patterns for input images.

Installation and Configuration

• Added build and twine as optional dependencies.
• Provided an argument for specifying the path to the Ilastik executable, ensuring compatibility across different operating systems.
• Updated installation notes to reflect the latest changes and dependencies.

This release includes significant improvements in functionality, performance, and documentation, making it easier to use and more robust for processing LSFM data.

v1.0.0-beta

What's Changed

Major Refactor and Improvements

• The entire UNRAVEL codebase has been refactored to Python, enhancing readability and maintainability (e.g., reuse of functions).
• Several improvements have been made to streamline the processing pipeline, making it more efficient and user-friendly.

Initial Version and Publication

• The initial version of the code, which was a mixture of Python, Bash, Fiji macros, and XLSX templates, has been significantly upgraded.
• For an in-depth understanding of the initial version, refer to our publication in Neuropsychopharmacology.

New Documentation

• Please checkout our new documentation for the latest guides, etc.

v0.1.2-feature-branch

Release Log: UNRAVEL v0.1.2-feature-branch (Feature Branch Update)

Release Date: August 2024

Overview:

This release is a major update to the UNRAVEL v0.1.0 codebase, incorporating both significant improvements to the original version (v0.1.0) and early drafts of code that were foundational for the v1.0.0-beta release on the main branch. The code in this branch represents a bridge between the initial version described in Rijsketic et al., 2023 and the ongoing development efforts aimed at achieving the full functionality envisioned for v1.0.0-beta. This update focuses on enhancing the speed and accuracy of the registration pipeline, expanding the functionality of statistical analysis, and adding new tools for image processing, cluster validation, and data visualization. Additionally, this release includes numerous bug fixes and optimizations designed to streamline the workflow for voxel-wise analysis of c-Fos immunofluorescence and related tasks.

Major Updates:

• Enhanced Registration Pipeline:

  • Streamlined registration scripts (czi_to_tif3.sh, prep_reg.sh, brain_mask.sh) for faster processing and improved accuracy.
  • Added support for parallel processing and optimized memory usage in image loading and saving functions.
  • Integration of ANTsPy tools for affine initialization and non-linear transformations, enabling more robust alignment to the LSFM atlas.

• Statistical Analysis Enhancements:

  • Introduction of prep_vxw_stats.py for voxel-wise statistical analysis with improved resampling and reorientation functions.
  • New scripts for performing 2×2 ANOVA and other statistical tests with enhanced false discovery rate (FDR) correction methods.
  • Added functionality for calculating effect sizes, performing Tukey's post hoc tests, and generating summary statistics across multiple q-value thresholds.

• Cluster Validation Tools:

  • A comprehensive suite of scripts (validate_clusters.py, validate_clusters_summary_pooled_ttest.py, valid_clusters_2x2_ANOVA_draft.py) for validating voxel-wise clusters using post hoc comparisons, cell density quantification, and statistical summaries.
  • Tools for handling bilateral data, aggregating cluster validation results, and generating publication-ready tables and visualizations.

• Image Processing and Data Handling:

  • New utilities for converting image formats (czi_to_nii.py, h5_to_tifs.py, tif_to_tifs.py) and handling large datasets more efficiently.
  • Updated unravel_utils.py with functions for handling metadata, managing output directories, and processing batch data.
  • Advanced tools for manipulating image volumes, including z-scoring, background subtraction, masking, and warping to atlas space.

• Improved User Experience:

  • Refined help guides and improved command-line interface (CLI) with custom argparse formatting for better readability.
  • Enhanced progress tracking with rich library integration and more detailed error handling.

Bug Fixes:

• Resolved issues related to image orientation, file path handling, and parallel processing in various scripts.
• Fixed inconsistencies in metadata extraction and file naming conventions to ensure compatibility with downstream analysis.
• Addressed bugs in statistical scripts related to argument parsing, file aggregation, and result formatting.

Known Issues:

• Ongoing testing is required for certain new features, particularly in the native_clusters2.py and related validation scripts.
• Some scripts remain in draft form and may require further refinement before use in production environments (please see v1.0.0-beta releases on the main branch for our latest code).

Other changes

• Added initial installation guide (#1)
• Bug fixes, added ABAseg_3dc.sh for single rater regional counts (#2)
• Upgraded miracl_conv_convertTIFFtoNII.py to python3 (#3)
• Updated README.md

This release marks a significant milestone in the development of UNRAVEL, incorporating numerous enhancements that improve both the functionality and user experience. The code in this branch has been utilized in the preparation of several scientific manuscripts, underscoring its robustness and applicability to real-world neuroimaging analysis.

Full Changelog: v0.1.1...v0.1.2-feature

Initial release (w/ README markdown)

We developed UNRAVEL (UN-biased high-Resolution Analysis and Validation of Ensembles using Light sheet images) as a command line tool for voxel-wise analysis of c-Fos immunofluorescence (IF) across the cleared mouse brains and validation of identified clusters of significant voxels using c-Fos+ cell density quantification at cellular resolution.

Please refer to guides regarding necessary dependencies, paths that need to be updated, conventions for file/folder organization and detailed info on running scripts. Key scripts include: find_clusters.sh, glm.sh, and validate_clusters2.sh. Other scripts can be run in a modular fashion (e.g., overview -> prep_tifs.sh -> 488_to_nii.sh -> reg.sh -> rb.sh -> z_brain_template_mask.sh -> glm.sh -> validate_clusters2.sh). Help guides can be found at the beginning of each script or by running: <script>.sh help

If you are unfamiliar with the command line interface, please review Unix tutorials here: https://andysbrainbook.readthedocs.io/en/latest/index.html

Initial release

We developed UNRAVEL (UN-biased high-Resolution Analysis and Validation of Ensembles using Light sheet images) as a command line tool for voxel-wise analysis of c-Fos immunofluorescence (IF) across the cleared mouse brains and validation of identified clusters of significant voxels using c-Fos+ cell density quantification at cellular resolution.

Please refer to guides regarding necessary dependencies, paths that need to be updated, conventions for file/folder organization and detailed info on running scripts. Key scripts include: find_clusters.sh, glm.sh, and validate_clusters2.sh. Other scripts can be run in a modular fashion (e.g., overview -> prep_tifs.sh -> 488_to_nii.sh -> reg.sh -> rb.sh -> z_brain_template_mask.sh -> glm.sh -> validate_clusters2.sh). Help guides can be found at the beginning of each script or by running: <script>.sh help

If you are unfamiliar with the command line interface, please review Unix tutorials here: https://andysbrainbook.readthedocs.io/en/latest/index.html