Inconsistency in Scannet eval results

[jameelhassan]

Hi, I am trying to reproduce the numbers for scannet, and the metrics I get using the same inference script is much lower.
I tried using both NUM_LVL = 1 and 3, strangely, using NUM_LVL=1 gave better metrics. The comparison is given below.

Is there anything missing in the eval for ScanNet that I may have missed? I noticed that the metrics for OpenGaussian are also lower when i reproduce their work, but the trend holds to some extent.
I was able to reproduce the numbers for LERF-OVS without any issues. Thanks.

[Jumpat]

Hi. Do these results come from NUM_LVL = 1? We do set NUM_LVL=1 for scannet evaluation, since the scannet categories only contain instance-level labels. How did you train the segmentation features on the scanner point clouds? Did you keep the point cloud fixed? The evaluation on Scannet is very different from LERF.

[jameelhassan]

Hi,
I train scannet with fixed points for all scenes using the command provided here

LaGa/get_scannet_scene.sh

Line 2 in 9df7586

CUDA_VISIBLE_DEVICES=0 python train_scene.py -s /data_hdd3/cenjiazhong/datasets/nerf_data/scannet/scene0000_00 --densify_until_iter -1

Instance features are trained using

LaGa/get_scannet_features.sh

Line 1 in 9df7586

CUDA_VISIBLE_DEVICES=0 python train_affinity_features.py --model_path ./output/scannet-scene0000_00 --downsample

For eval, I set NUM_LVL = 1 (explicitly), and this requires a minor change in the code given in inference.ipynb when extracting CLIP features (below). This ensures I get only features from the first level, and resolves the error I get with number of masks and features indexing in dynamic K-means. Are there any other changes in the evaluation setup for ScanNet? The rest of the inference.py script worked fine.

            # if NUM_LVL > 1:
            #     interval = lvl_to_mask_id[lvl + 1]
            if NUM_LVL == 1:
                interval = lvl_to_mask_id[lvl]
                f = f[interval[0]:interval[1]+1,:]
            flatten_features.append(f)

        multi_lvl_flatten_features.append(torch.cat(flatten_features, dim = 0))

[Jumpat]

Hi, can you try using different segmentation layers (lvl = 0/1/2)? It seems your scanner feature obtaining is right.