Working with own data

[horsto]

I want to use XMem on my own data. I got the GUI (interactive_demo.py) to work and it works very well for the detection task I need it to work on, i.e. clicking on the object I want to detect and forward propagation creates the right masks and visualizations.

However, now I am a bit lost: I want XMem to run over all my videos, spitting out masks / annotations for the object I need to detect in each frame. Do I need to initialize every video that I want to analyze with an annotation (i.e. the first frame(s) need to be annotated so it can the forward propagate through all of it) or is there a way to do this automatically?

[hkchengrex]

XMem is class-agnostic and it does not know what to track (there are many possible segmentations of the same scene) unless told. Are you dealing with specific object categories?

[horsto]

I have one very specific object (an animal on a white background) that I need to segment out. It gets detected very well in subsequent frames in the GUI when I point the algorithm to it, but I need it to be segmented from background automatically in all the videos I have without (manual) initialization. Is there a way to do this?

[hkchengrex]

I would imagine that some sort of automatic initialization is possible. If it is a common animal, instance segmentation methods like RCNN might work. If there is only a single foreground object, you can try saliency detection (something like https://github.com/Karel911/TRACER) for initialization.

[horsto]

Great, thanks, I will try TRACER.
Just to give you an idea, I attached two images here from one of the videos. The animal is a cuttlefish - not common at all sadly. It amazes me that the segmentation works that well, but again, I need this to run automatically. I was wondering if there was a way to leverage the huge number of well segmented annotations I get from XMem (via the GUI output) and train / refine an existing algorithm to detect these little guys.