Object Detection with Caption

[hungphongtrn]

First, thank you all for open sourcing this fantastic work.

I want to ask whether the object detection with caption feasible with this model and if yes how can I use it?

Thank you in advance!

[roman-bachmann]

Hi @hungphongtrn,

Thanks for your interest!
I'm not sure I fully understand the question, but 4M is able to both input and output captions and bounding boxes, alongside other modalities. Note that the captions and bounding boxes are not aligned - the captions are at an image-level, while the bounding boxes are labeled with COCO classes.
Please see our Jupyter notebooks for examples.

Best, Roman

[hungphongtrn]

Hi @roman-bachmann ,

Sorry @roman-bachmann for not being clear. My question is that given some labels as "captions", can 4M detect the bbox based on the provided "captions"?

Thank you,
Phong