Knowledge distillation using Mobilenet on MS1M dataset

[leondgarse]

• Create distillation dataset using data_distiller.py and a MXNet / ONNX / PyTorch / Keras pretrained model. Set use_fp16 to save as float16.

  export TF_USE_LEGACY_KERAS=1  # For compatible with TF>2.16.0
  MODEL=TT_r100_max_pool_E_prelu_dr04_lr_01_l2_5e4_adaface_emb512_sgd_m09_bs512_glint360k_64_only_margin_SG_scale_true_bias_false_random_100_partial4_basic_model_latest.h5
  CUDA_VISIBLE_DEVICES='0' ./data_distiller.py -M $MODEL -D /datasets/ms1m-retinaface-t1_112x112_folders/ --use_fp16 -b 64

• Train Mobilenet, and set add_pointwise_conv=True. In some experiments add_pointwise_conv is better, some others not...

  import losses, train, models
  import tensorflow_addons as tfa
  keras.mixed_precision.set_global_policy("mixed_float16")
  
  data_basic_path = '/datasets/ms1m-retinaface-t1'
  # data_path = data_basic_path + '_112x112_folders'
  eval_paths = [os.path.join(data_basic_path, ii) for ii in ['lfw.bin', 'cfp_fp.bin', 'agedb_30.bin']]
  new_data_path = "ms1m-retinaface-t1_112x112_folders_shuffle_label_embs_512_fp16.tfrecord"
  
  basic_model = models.buildin_models("mobilenet", dropout=0.4, emb_shape=512, output_layer='E', add_pointwise_conv=True, scale=False, use_bias=True)
  basic_model = models.replace_ReLU_with_PReLU(basic_model, target_activation="swish")
  
  tt = train.Train(new_data_path, eval_paths=eval_paths, 
      save_path='TT_mobilenet_swish_pointwise_distill_128_arc_emb512_dr04_wd5e4_bs512_r100_ms1m_fp16_cosine.h5',
      basic_model=basic_model, model=None, 
      lr_base=0.1, lr_decay=0.5, lr_decay_steps=16, lr_min=1e-5,
      batch_size=400, random_status=0)
  
  optimizer = tfa.optimizers.SGDW(learning_rate=0.1, weight_decay=5e-4, momentum=0.9)
  sch = [
      {"loss": losses.ArcfaceLoss(scale=16), "epoch": 5, "optimizer": optimizer, "distill": 128},
      {"loss": losses.ArcfaceLoss(scale=32), "epoch": 5, "distill": 128},
      {"loss": losses.ArcfaceLoss(scale=64), "epoch": 40, "distill": 128},
  ]
  tt.train(sch, 0)

• Result
  

  with pointwise, ms1mv3, cosine lr, swish	1e-06	1e-05	0.0001	0.001	0.01	0.1
  IJBB_11	0.380721	0.85258	0.91889	0.951412	0.972833	0.986465
  IJBC_11	0.851255	0.907808	0.940328	0.963133	0.979751	0.991154

[aidansmyth95ifx]

Where in the repo can I find the pretrained mxnet model?

[leondgarse]

That MXNet model is not required. Any mxnet / pytorch / onnx / h5 model is applicable data_distiller.py#L82-L108. You can pick one from the model table.

export TF_USE_LEGACY_KERAS=1
MODEL=TT_efv2_b0_swish_GDC_arc_emb512_dr0_sgd_l2_5e4_bs512_ms1m_randaug_cutout_bnm09_bne1e4_cos16_batch_float16_E50_arc_sgd_LA_basic_agedb_30_epoch_17_0.977333.h5
python data_distiller.py -M $MODEL  -D test_aligned_112_112 --use_fp16 -b 64

[aidansmyth95ifx]

Thank you :-)