Training on a custom set with grayscale images

[sadransh]

I have a custom set with grayscale images. images are small ( 224x224). ( I think by default images opens as 3 similar channels) I tried training d1 and d0. after more than 100(or 200) epochs it does not train at all (though loss reaches around 0.5 or less )

Any Idea? I have trained RGB images with the same code w/o any issue.
I have also trained a network using this grayscale set with detecteron2 without any problem.

Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.003
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.003
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.002
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

training command. ( similar command works with rgb dataset)
!./distributed_train.sh 3 ../data --model tf_efficientdet_d1 --dataset weld --mean 0.3286 --std 0.1569 -b 21 --apex-amp --lr .07 --sync-bn --opt fusedmomentum --warmup-epochs 3 --lr-noise 0.3 0.9 --model-ema --model-ema-decay 0.9998 -j 25 --epochs 400

[rwightman]

@sadransh you should add --pretrained to use the pretrained weights if hyou are using the tf_ model, otherwise might as well use the non-tf model as it is slightly faster

The default data pipeline is all RGB based, if you load a grayscale image as RGB it should work fine. Something else must be wrong, possibly with bbox format from the custom dataset?. I have had some inconsistent failures with sync-bn, especially when it is enabled with amp and model-ema. It is worth trying a run with sync-bn disabled, batch size 21 should still be stable.

[sadransh]

I tried removing --syncbn ... no changes. I think the source of the problem is not related to annotations. I trained a dectron2 based model with the same JSON files, and got good results. I have also used similarly generated annotations for RGB and large images with this implementation. This grayscale dataset images are smaller( and also annotations).

boxes in JSON file is in x y w h format. and in parser settings bbox_yxyx: bool = True So I think it is correct since I am using the tf_ model. ( though I tested non-tf too)