3D Indoor Instance Segmentation in an Open-World (NeurIPS 2023)

Mohamed El Amine Boudjoghra¹, Salwa K. Al Khatib¹, Jean Lahoud¹, Hisham Cholakkal¹, Rao Muhammad Anwer^1,2, Salman Khan^1,3, Fahad Khan^1,4

¹Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) ²Aalto University ³Australian National University ⁴Linköping University

teaser.mp4

News

• 25 September 2023: 3D-OWIS paper released on arXiv. 📝
• 25 September 2023: Code and splits released. 💻
• 22 September 2023: Paper accepted into NeurIPS2023. 🥳

Abstract

We formulate the problem of 3D point cloud instance segmentation in the open world, where we relax the assumption of a closed setting of classes during the training. We propose an auto-labeling scheme to produce pseudo-labels during training and induce separation to separate known and unknown category labels. We further improve the pseudo-labels quality at inference by adjusting the unknown class probability based on the objectness score distribution. We also introduce carefully curated open-world splits leveraging realistic scenarios based on inherent object distribution, region-based indoor scene exploration and randomness aspect of open-world classes. Extensive experiments reveal the efficacy of the proposed contributions leading to promising open-world 3D instance segmentation performance.

Figure I: Proposed open-world 3D instance segmentation pipeline.

Qualitative results

Figure II: The performance of 3D-OWIS in masking unknown instances is shown in the above figure. The known classes are assigned the green color, while the unknown and background objects are assigned blue and gray respectively.

Figure III: 3D-OWIS is also capable of progressively learning novel classes, as shown in the following figure, once thier labels are introduced in the training process.

Navigate in the scenes

Kindly click on the icon to navigate in the scenes

	Scene 1	Scene 2	Scene 3
3D-OWIS
Ground truth
Input scene

Comparative visualization

we show the progression of learning novel classes throughout the three tasks, and a comparison to the ground truth for our model with and without PC(Probability Correction) and CT(Confidence Threshold).

Installation guide

Kindly check Installation guide on how to setup the Conda environment and how to preprocess ScanNet200 dataset.

Use our splits with another 3D instance segmentation model

You can import the labels of the PREVIOUSLY_KNOWN, CURRENTLY_KNOWN, and UNKNOWN classes from ./datasets/scannet200/owis_splits.py for usage with another codebase.

Example: the following code will load the previously known, currently known, and unknown classes for split A task1

from datasets.scannet200.owis_splits import PREV_KNOWN_CLASSES_LABELS, KNOWN_CLASSES_LABELS, UNKNOWN_CLASSES_LABELS

OW_task = 'task1'
split = 'A'

PREVIOUSLY_KNOWN = PREV_KNOWN_CLASSES_LABELS[split][OW_task]
CURRENTLY_KNOWN = KNOWN_CLASSES_LABELS[split][OW_task]
UNKNOWN = UNKNOWN_CLASSES_LABELS[split][OW_task]

Training and testing 🚆

For training a model <MODEL> with a split <SPLIT> please run the following command:

NB: <SPLIT> takes the values 'A', 'B', or 'C'.

We provide the scripts for reproducing the results for the models 3d_owis, and oracle.

sh scripts/train_<MODEL>/<SPLIT>/train.sh

For testing a task <TASK> for a specific split <SPLIT> please run the following command:

NB: <TASK> takes the values 'task1', 'task2', or 'task3'

Important for inference: please make sure to not forget to specify the <EXPERIMENT_NAME>, <SPLIT>, and <TASK> when running the inference, as the prototypes are loaded from the path saved/<EXPERIMENT_NAME>/<SPLIT>/<TASK>/ to perform the probability correction.

python main_instance_segmentation.py \
general.checkpoint='saved/<EXPERIMENT_NAME>/<SPLIT>/<TASK>/CHECKPOINT_NAME.ckpt' \
general.experiment_name=<EXPERIMENT_NAME> \
general.split=<SPLIT> \
general.OW_task=<TASK> \
general.margin=<CLUSTERING_MARGIN_USED_DURING_TRAINING> \
general.correct_unknown_cls_prob=True \
general.train_mode=false

Trained checkpoints and output predictions

	Checkpoints	Clustering Margin
3D-OWIS		A :1.0, B :3.0, C :1.0
3D-OWIS - PC - CT		A :1.0, B :3.0, C :1.0
Oracle		A :1.0, B :3.0, C :1.0

BibTeX 🙏

@inproceedings{boudjoghra20233d,
 author = {Mohamed El Amine Boudjoghra and Salwa K. Al Khatib and Jean Lahoud and Hisham Cholakkal and Rao Muhammad Anwer and Salman Khan and Fahad Khan},
 booktitle = {Advances in Neural Information Processing Systems},
 title = {3D Indoor Instance Segmentation in an Open-World},
 year = {2023}
}