About the inference in ScanNet200

[KaKa-101]

Hi, thanks for your great work. I want to know if I conduct instance segmentation evaluation ont the test split of ScanNet dataset, which includes scene0707_00~scene0806_00, what parameters should I change? I just change the test_mode to test in indoor.yaml, but I got the following error:

Thanks very much for your kind reply.

[aycatakmaz]

Hi @KaKa-101

A couple of things to note: It is unfortunately not possible to conduct the instance segmentation evaluation on the test split of the ScanNet dataset using our repository, and by evaluation I mean computing the AP scores. This is because the ground truth instances for the test split of the ScanNet is hidden.

However, it is possible for you to compute the predictions and class assignments on the test scenes. You should use the original indoor.yaml config of our repository - it might be that changing the setting to test_mode in indoor.yaml could cause some errors we are currently not handling in our repository.

One thing to note is that, we are using the eval_on_segments configuration for computing the masks from the Mask3D backbone for the ScanNet validation set. If you do not have access to such segments for the test set (computed from the Segmentator in the original ScanNet repository), you can simply use our alternative script for performing inference on arbitrary scenes. You would need to loop over all scenes from the test set and run this script to export OpenMask3D features and masks. Once you complete this extraction process, you need to run one final script to convert these features to ScanNet class assignments. Our evaluation script does that conversion, but you would need to comment out the following line:

openmask3d/openmask3d/evaluation/run_eval_close_vocab_inst_seg.py

Line 159 in 3bc3fc5

inst_AP = evaluator.evaluate_full(preds, gt_dir, dataset=dataset_type)

as it requires the GT files. Instead, you can simply export the preds dictionary following the ScanNet benchmark submission file structure. Please see this page for details about how to create the submission file, as we do not provide support for that within our codebase at the moment.

Hope this helps!

Best,
Ayca

[KaKa-101]

Thanks for your reply! And I also want to ask if I conduct instance segmentation and mask features computing on the train split of Scannet Dataset, what config/parameters should I change?
Thanks again for your kind help.

[aycatakmaz]

In order to perform instance segmentation and mask feature computation on the train split of the ScanNet dataset, you can simply change the paths in the ScanNet200 config we have in the repository. Instead of valpaths, you can set the train paths. Everything else can stay the same. However, please note that this will not be a fair evaluation with respect to the mask quality, as the instance segmentation checkpoint we provide was trained on the training set of the ScanNet dataset. However, you can use this evaluation to measure how well the CLIP features from our method are.

[KaKa-101]

Thanks for your reply. And I have changed the path in script to the folder contains train split dataset following the right structure as you have mentioned.
But I got the following printing information, it's still about the val split since there are ~1200 scenes in train split of Scannet dataset. And it seems that the masks I get are also about the val split.


So could you tell me what other configures do I need to change for performming on the train split?

[aycatakmaz]

Could you verify whether you have processed all ScanNet scenes including the training scenes (e.g. this step)? Could you please check the label dataset includes also the training scenes, (i.e. SCANNET_LABEL_DB_PATH)?

[KaKa-101]

Yes, I have precessed all ScanNet scenes and the folder is like this:

And I have checked the file label_database.yaml, there are 200 items here like this:

So is the problem that there aren't label for train split in the file label_database.yaml?

[aycatakmaz]

I took a look at the screenshots you sent and you need train_database.yaml to be used instead of validation_database.yaml. Because of the mode parameter in this script, the mask generator uses the validation_database.yaml file. But I wouldn't recommend you to change the mode to train there, because that would result in data augmentations to be applied to the point cloud as if these files are loaded for training. A very hacky way to solve your problem could be to rename the train_database.yaml file temporarily to validation_database.yaml, and validation_database.yaml to validation_database_orig.yaml temporarily while you compute the masks for the training scenes. It's been a while since I worked with this database structure so this is a naive solution I could think of for the time being, but it might not work in the end. Let me know how it goes!

[KaKa-101]

The problem is solved finally and thank you so much! Besides I also want to ask about the model that can be running on an arbitrary scene you provided in this link. How do you get this model and is this model just trainned on Scannet dataset? If so, why can this model be applied to conduct instance segmentation on any scene?
Thanks again.

[aycatakmaz]

Great to hear that the problem is solved!

The model in the link is a Mask3D checkpoint that was trained on the full ScanNet training+validation scenes. The reason why this model is more suitable for the applications with arbitrary indoor scenes is that this model was not trained using the Segmentator segments from ScanNet (see the eval_on_segments argument from the original Mask3D repository). Therefore, it seems to us that this model generalizes better when an arbitrary point cloud is given as input. But to answer your question, yes, it was only trained on the ScanNet dataset.