should it be false in stpls3d dataset？

[Rurouni-z]

ISBNet/configs/stpls3d/isbnet_stpls3d.yaml

Line 81 in 44beb83

fp16: True

if i set true, and use pretrained model from isbnet_backbone_stpls3d.yaml then i got error:

AssertionError: No inf checks were recorded for this optimizer.

otherwise i can train the all network

[Rurouni-z]

and i dont know why i always got OOM in training

i use single A100 to train model, but in a multi-GPU environment

and my command set up a specific GPU:

CUDA_VISIBLE_DEVICES=2 python3 tools/train.py configs/stpls3d/isbnet_stpls3d.yaml --trainall --exp_name default

update:

I tried changing fp16=true in backbone.yaml and got the worst results, but if I don't set true I have to set fp16=false in stpls3d.yaml which causes recurring OOM