A Systematic Review of Intermediate Fusion Methods of Multimodal Deep Learning in Biomedical Applications

Citation

If you find our work useful in your research, please consider citing:

Our review paper on arXiv: paper🔥

@article{guarrasi2024systematic,
  title={{A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications}},
  author={Guarrasi, Valerio and Aksu, Fatih and Caruso, Camillo Maria and Di Feola, Francesco and Rofena, Aurora and Ruffini, Filippo and Soda, Paolo},
  journal={arXiv preprint arXiv:2408.02686},
  year={2024}
}

Overview

Multimodal deep learning (MDL) has emerged as an innovative approach in biomedical applications, leveraging the power of deep learning algorithms to interpret and integrate diverse data types. Intermediate fusion techniques stand out for their ability to effectively integrate information at essential stages of the learning process, potentially leading to more accurate and robust models. This systematic review provides an overview of intermediate fusion methods in biomedical applications, covering fundamental concepts, structured analysis, and notation that not only categorizes these methods but also provides a framework that can be extended beyond the biomedical field.

Have a look at a concise overview of our analysis (supplementary material A): Link

In the table below, we make available the detailed analysis of the fusion strategy used by each paper included in the review (supplementary material B).

Title	DOI	Code	Year	Fusion analysis
A Bi-level representation learning model for medical visual question answering	paper	-	2022	Link
A dynamic multi-modal fusion network for ovarian tumor differentiation	paper	-	2022	Link
AATSN: Anatomy Aware Tumor Segmentation Network for PET-CT volumes and images using a lightweight fusion-attention mechanism	paper	-	2023	Link
Attention-like multimodality fusion with data augmentation for diagnosis of mental disorders using MRI	paper	-	2022	Link
AviPer: assisting visually impaired people to perceive the world with visual‑tactile multimodal attention network	paper	-	2022	Link
Comparative assessment of text-image fusion models for medical diagnostics	paper	-	2020	Link
Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data	paper	GitHub	2021	Link
Computer-Aided Hepatocarcinoma Diagnosis Using Multimodal Deep Learning	paper	-	2019	Link
Deep learning approach for predicting lymph node metastasis in non-small cell lung cancer by fusing image–gene data	paper	-	2023	Link
Deep Learning Based Data Fusion Methods for Multimodal Emotion Recognition	paper	-	2022	Link
Deep learning model integrating positron emission tomography and clinical data for prognosis prediction in non-small cell lung cancer patients	paper	-	2023	Link
Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction	paper	-	2023	Link
Deep multimodal fusion for subject-independent stress detection	paper	-	2021	Link
Deep multimodal predictome for studying mental disorders	paper	-	2023	Link
DyHealth: Making Neural Networks Dynamic for Effective Healthcare Analytics	paper	-	2022	Link
End-to-End Learning of Fused Image and Non-Image Features for Improved Breast Cancer Classification from MRI	paper	-	2021	Link
Exploring multimodal fusion for continuous protective behavior detection	paper	GitHub	2022	Link
GMRLNet: A graph-based manifold regularization learning framework for placental insufficiency diagnosis on incomplete multimodal ultrasound data	paper	-	2023	Link
Hierarchical-order multimodal interaction fusion network for grading gliomas	paper	-	2021	Link
Improving detection of prostate cancer foci via information fusion of MRI and temporal enhanced ultrasound	paper	-	2020	Link
Improving knee osteoarthritis classification using multimodal intermediate fusion of X-ray, MRI, and clinical information	paper	-	2023	Link
iTCep: a deep learning framework for identification of T cell epitopes by harnessing fusion features	paper	GitHub	2023	Link
Liver Tumor Detection Via A Multi-Scale Intermediate Multi-Modal Fusion Network on MRI Images	paper	-	2021	Link
Long-term cognitive decline prediction based on multi-modal data using Multimodal3DSiameseNet: transfer learning from Alzheimer’s disease to Parkinson’s disease	paper	GitHub	2023	Link
MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images	paper	GitHub	2022	Link
MIFTP: A Multimodal Multi-Level Independent Fusion Framework with Improved Twin Pyramid for Multilabel Chest X-Ray Image Classification	paper	-	2022	Link
MMHFNet: Multi-modal and multi-layer hybrid fusion network for voice pathology detection	paper	-	2023	Link
Modeling uncertainty in multi-modal fusion for lung cancer survival analysis	paper	-	2021	Link
MS2-GNN: Exploring GNN-Based Multimodal Fusion Network for Depression Detection	paper	-	2022	Link
MSMFM: An Ultrasound Based Multi-Step Modality Fusion Network for Identifying the Histologic Subtypes of Metastatic Cervical Lymphadenopathy	paper	GitHub	2022	Link
Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness	paper	-	2022	Link
Multi-modal fusion model for predicting adverse cardiovascular outcome post percutaneous coronary intervention	paper	-	2022	Link
Multi-objective optimization determines when, which and how to fuse deep networks: An application to predict COVID-19 outcomes	paper	-	2023	Link
Multi-view Deep Neural Networks for multiclass skin lesion diagnosis	paper	-	2022	Link
Multimodal deep learning to predict prognosis in adult and pediatric brain tumors	paper	GitHub	2023	Link
Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification	paper	GitHub	2022	Link
Multimodal fusion models for pulmonary embolism mortality prediction	paper	-	2023	Link
Multimodal fusion of imaging and genomics for lung cancer recurrence prediction	paper	-	2020	Link
Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection	paper	GitHub	2020	Link
Multimodal Hierarchical CNN Feature Fusion for Stress Detection	paper	-	2023	Link
Multimodal Information Fusion for Glaucoma and Diabetic Retinopathy Classification	paper	-	2022	Link
Multimodal Medical Tensor Fusion Network-Based Dl Framework For abnormality Prediction From The Radiology Cxrs And Clinical Text Reports	paper	-	2022	Link
Predicting Brain Degeneration with a Multimodal Siamese Neural Network	paper	-	2020	Link
Predicting heart failure in‐hospital mortality by integrating longitudinal and category data in electronic health records	paper	-	2023	Link
Predicting Successes and Failures of Clinical Trials With Outer Product–Based Convolutional Neural Network	paper	GitHub	2021	Link
Radiopaths: Deep Multimodal Analysis on Chest Radiographs	paper	GitHub	2022	Link
Single Modality vs. Multimodality: What Works Best for Lung Cancer Screening?	paper	-	2023	Link
Stress Detection using CNN Fusion	paper	-	2021	Link
TinyM2Net-V2: A Compact Low Power Sotware Hardware Architecture for Multimodal Deep Neural Networks	paper	-	2023	Link
Toward attention-based learning to predict the risk of brain degeneration with multimodal medical data	paper	-	2023	Link
Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition	paper	-	2023	Link
Trustworthy Deep Neural Network for Inferring Anticancer Synergistic Combinations	paper	-	2023	Link
Two-dimensional attentive fusion for multi-modal learning of neuroimaging and genomics data	paper	-	2022	Link
Weakly supervised multimodal 30-day all-cause mortality prediction for pulmonary embolism patients	paper	-	2022	Link