Finetuning and inference with multiple input modalities

[partha-ghosh]

Hi @theodore-zhao,
I'm working with Biomedparse v2 and have a dataset containing multiple imaging modalities. Does Biomedparse v2 support fusion of multi-modal data at the input level, or are there recommended strategies for combining features from different modalities? Also, what is the recommended approach is for normalizing CT volumes in Biomedparse v2?

[theodore-zhao]

Absolutely! You can mix different modalities in the finetuning, as long as the text prompt correctly describes the ground truth mask. We used a preprocessed data version, and you can check the normalization here:
Following the practice in MedSAM, all images were processed to npz format with an intensity range of [0, 255]. Specifically, for CT images, we normalized Hounsfield units using typical window width and level
values: soft tissues (W:400, L:40), lung (W:1500, L:-160), brain (W:80, L:40), and bone (W:1800, L:400). Subsequently, the intensity values were rescaled to the range of [0, 255]. For other images, we clipped the intensity values in the range between the 0.5th and 99.5th percentiles before rescaling them to the range of [0, 255]. If the original intensity range is already in [0, 255], no preprocessing was
applied.

Let me know if there are any further questions!