Worse performance with Evaluator type "grounding_spatial"

[MaeenBadea]

Questions on Compositionality in Biomedparse

Hi,

Thanks for the incredible work and for releasing the code! We are deeply interested in the compositionality of the model and have a couple of questions regarding its implementation and evaluation.

---

. Evaluation: grounding vs grounding_spatial

When evaluating the model, I noticed a significant performance drop when switching from grounding to grounding_spatial.

• Grounding only

  mIoU: 90.9
  mDice: 94.9
  

• Grounding + Spatial

  mIoU: 40.1
  mDice: 43.8
  

The only change I made was overriding the eval_type in the pipeline file:

XDecoderPipeline.py#L133

eval_type = "grounding_spatial"

From my understanding, no additional modifications should be required, since in the build method the evaluator for grounding_spatial and grounding_refcoco is the same. Could you confirm if this is indeed the case, or if extra steps are needed for proper evaluation?

---

✅ Any clarification on these points would be very helpful!

[theodore-zhao]

Yes this is normal. The model was designed for text prompt segmentation, so it is optimized for the grounding segmentation eval mode. In the spatial mode, additional spatial prompts such as points or scribbles are required.

[MaeenBadea]

Thanks a lot for the fast response!

I’m currently finetuning the model on our in-house dataset. I noticed that the provided finetuning config YAML already enables spatial prompting during training. In the code, the forward_seg function also makes use of spatial prompts (not only text), and the data loaders are preparing these spatial prompts as well.

However, despite this setup, we observe a huge performance drop. From the architecture design, SEEM should benefit from multiple prompts, and compositionality seems to be one of its core properties. I’m trying to reconcile this with our results.

Could you clarify whether the released weights were trained with text prompts only (i.e. task_switch["spatial"] = False), or if spatial prompts were also part of the pretraining?

[theodore-zhao]

The original SEEM architecture is flexible to support multiple types of prompts. In our training, as we aimed for text-only prompting, we only provided the text prompts to the model. In some internal experiments, we also tried text+spatial prompts, but didn't observed significant gain. I would recommend using grounding segmentation unless you want to finetune it with spatial as well