Segment Anything

[ajenez]

Hi!

My team has loved using rectlabel so first off thank you for a great product. I do have a question about a feature. Do you have any thoughts on adding a way to use Segment Anything? https://github.com/facebookresearch/segment-anything

I know my team would find something like that useful.

[ryouchinsa]

Thanks for the feature request.
We will find out how to run the Segment Anything Models on Objective-C/Swift/C++ environments.
facebookresearch/segment-anything#198

[ajenez]

Thank you! Would love to know when this is implemented,

[ryouchinsa]

Thanks for checking in.
We are going to implement MobileSAM on RectLabel.

Currently we are trying to work this c++ wrapper and
building the ONNX Runtime for macOS apps.

[ryouchinsa]

The new version 2023.08.31 was released.

Added "Create polygon using SAM" on Edit menu.
Change the mode to "Create polygon using SAM".
Select a Segment Anything model among "MobileSAM", "ViT-Large SAM", and "ViT-Huge SAM".
After downloading the model, the preprocessing for the image begins. Because of CPU mode, it takes 2 seconds for "MobileSAM", 30 seconds for "ViT-Large SAM", and 60 seconds for "ViT-Huge SAM" on the Apple M1 device.
Downloaded models are saved in the folder "~/Library/Containers/RectLabel/Data/models".
Click to add foreground points.
Click to add background points.
Pressing the option key, you can switch the foreground mode and background mode.
Press enter key to finish labeling.
Press escape key to cancel labeling.

[ajenez]

hi @ryouchinsa, wow! This is really amazing, thank you so much for implementing this. Our team will find it so useful and will be sure to recommend RectLabel.

[ryouchinsa]

Improved "Create polygon using SAM" feature so that you can label pixels using the pixels option. You can label Segment Anything 1 Billion (SA-1B) like dataset by yourself.

[ajenez]

hi @ryouchinsa thank you again for the updates with the SAM mode. One thing I did want to mention to you is that RectLabel seems to crash a bit more often than before. Have you seen this behavior? I'm not sure if it is related to the SAM mode or not.

[ryouchinsa]

Thanks for writing the issue.
As you said, sometimes I had the crash when I was using the SAM mode.
If the crash log includes "onnx runtime" keywords, it because of the SAM mode.

In my case, the crash happened in the sessionPre->Run() function.
I am making the SAM code to be open source.
https://github.com/ryouchinsa/sam-cpp-macos/blob/c6e6d8a4fa808b440e913caebe2fe008b810554a/sam.cpp#L115

The difficult point is I cannot reproduce the problem.
If you could tell me the detailed situation when RectLabel crashed, it would be very helpful for me to fix the problem.

[ajenez]

hi @ryouchinsa it actually seems that the crash may be occurring when adding an object to an image with many labels. I'll try to take a video next time to share.

[ryouchinsa]

If you could share the detailed video, it would be very helpful. I have some amount of test images, but each image includes less than 10 objects.

[ryouchinsa]

In the latest version 2023.12.15, I improved the heap memory management during the SAM mode.
For each image, you need to press the start button to start preprocessing. If you change to another image, the heap memory for SAM is released. If you encountered the crash during the SAM mode with the latest version, please let me know the detailed situation.