A Likelihood Ratio-Based Approach to Segmenting Unknown Objects

Nazir Nayal, Youssef Shoeb, Fatma Güney

[arXiv][BibTeX]

Installation

Create the environment

conda create -n uem python=3.10

Install Pytorch:

pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu118

Compile custom cuda kernels

cd modeling/modules/pixel_decoder/ops/
sh make.sh

Install required packages:

pip install -r requirements.txt

Dataset Preparation

We follow the exact same data preparation steps and format as described here

Model Zoo

Segmentation Models

We provide two semantic segmentation models trained only on the 19 classes of cityscapes. One model is trained with a GMM head, and the other with a linear discriminative head.

Backbone	Head	Cityscapes mIoU	config	ckpt
dinov2_vitb14_reg	GMM	82.57	config	model
dinov2_vitb14_reg	Linear	83.11	config	model

OoD Models

		SMIYC-AT		SMIYC-OT		Road Anomaly		FS LaF
Segmentor	UEM Type	AP	FPR95	AP	FPR95	AP	FPR95	AP	FPR95	config	ckpt
GMM	GMM (G-G)	94.10	6.90	88.30	0.40	93.75	6.32	71.34	6.04	config	model
GMM	Discriminative (G-D)	92.50	11.30	92.0	0.20	92.86	8.95	79.02	1.62	config	model
Discriminative	Discriminative (D-D)	95.60	4.70	94.40	0.10	90.94	8.03	72.83	2.60	config	model

Evaluation

The script evaluate_ood.py is provided for evaluating on the OoD datasets Road Anomaly and Fishyscapes LaF. You can run an evaluation as follows:

python evaluate_ood.py --ckpt PATH_TO_CKPT \
     --devices 0 \  # can be 0,1,2,3 for e.g. in case of multi-gpu
     --datasets-folder /PATH-TO-DATASETS/datasets/ \
     --multiple-datasets \
     --dataset ra_and_fslaf \
     --out-path results/ \
     --segmentor-ckpt ckpts/d_d/linear-segmentor.ckpt \

The checkpoints are released on github and can be downloaded from there.

Training

The config files for training are available in the model zoo. There are two stages for training:

1. Train a Segmentor (inlier model)
2. Train a UEM on top of a segmentor (OoD Training)

A segmentor can be trained as follows:

python train_segmentor.py --config configs/segmentation/cityscapes/discriminative.yaml \
SOLVER.BATCH_SIZE 4 \ # this is per gpu
DATA.DATASET_ROOT PATH_TO_CITYSCAPES_DATASET_ROOT

UEM can be trained as follows:

python train_ood.py --config PATH_TO_CONFIG \
 MODEL.SEGMENTOR_CKPT PATH_TO_SEGMENTOR_CKPT \
 SOLVER.BATCH_SIZE 4 \ # this is per gpu
 DATA.DATASET_ROOT datasets/cityscapes \
 DATA.DATASETS_FOLDER datasets/ \
 DATA.COCO_ROOT datsets/coco \

Note that paths to datasets need to be overwritten accordingly.

Citing UEM

@article{nayal2024likelihood,
 title={A Likelihood Ratio-Based Approach to Segmenting Unknown Objects},
 author={Nayal, Nazir and Shoeb, Youssef and G{\"u}ney, Fatma},
 journal={arXiv preprint arXiv:2409.06424},
 year={2024}
}