-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_distilbert-base.txt
2595 lines (2595 loc) · 195 KB
/
HCQ_MSRVTT_1kA_distilbert-base.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base
Preparing the dataloaders ...
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base
Preparing the dataloaders ...
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch0.pth ...
Done in 1.159s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch0.pth ...
Done in 2.250s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 503.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 500.523
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.4
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 5.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 503.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.8145
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.3914867641168864
mnt_best : 0.4
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.84274 (QuantReg: 22.65345) QuantErr: 22.65345 batch_time=116.43528
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.52011 (QuantReg: 22.64871) QuantErr: 22.64871 batch_time=0.43033
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.33398 (QuantReg: 22.58366) QuantErr: 22.58366 batch_time=0.46316
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.62988 (QuantReg: 22.63880) QuantErr: 22.63880 batch_time=0.45454
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.44528 (QuantReg: 22.60021) QuantErr: 22.60021 batch_time=0.45580
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.88894 (QuantReg: 22.62334) QuantErr: 22.62334 batch_time=0.43440
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.57971 (QuantReg: 22.64596) QuantErr: 22.64596 batch_time=0.44151
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.23061 (QuantReg: 22.64252) QuantErr: 22.64252 batch_time=0.43693
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.47395 (QuantReg: 22.60229) QuantErr: 22.60229 batch_time=0.43030
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.28618 (QuantReg: 22.61806) QuantErr: 22.61806 batch_time=0.43212
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.20642 (QuantReg: 22.59529) QuantErr: 22.59529 batch_time=0.46599
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.05348 (QuantReg: 22.60873) QuantErr: 22.60873 batch_time=0.43426
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.83758 (QuantReg: 22.63026) QuantErr: 22.63026 batch_time=0.43834
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.88061 (QuantReg: 22.64003) QuantErr: 22.64003 batch_time=0.45195
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.58719 (QuantReg: 22.63640) QuantErr: 22.63640 batch_time=0.43646
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.62138 (QuantReg: 22.66127) QuantErr: 22.66127 batch_time=0.43840
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.33596 (QuantReg: 22.64863) QuantErr: 22.64863 batch_time=0.43476
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.66375 (QuantReg: 22.63329) QuantErr: 22.63329 batch_time=0.44470
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.19037 (QuantReg: 22.63953) QuantErr: 22.63953 batch_time=0.43569
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.21411 (QuantReg: 22.62505) QuantErr: 22.62505 batch_time=0.43600
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.37590 (QuantReg: 22.63910) QuantErr: 22.63910 batch_time=0.46197
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.92797 (QuantReg: 22.61586) QuantErr: 22.61586 batch_time=0.44208
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.19440 (QuantReg: 22.62114) QuantErr: 22.62114 batch_time=0.44209
Train Epoch: 1 codebook_update_time=2.49242
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch1.pth ...
Done in 2.628s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch1.pth ...
Done in 5.171s
epoch : 1
loss : 5.272009886741638
quant_reg : 22.630845268249512
quant_err : 22.630845268249512
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 10.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 32.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 47.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 79.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 12.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 39.919
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.045110021149007
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 34.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 46.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 79.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 12.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 41.402
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.263244835268058
mnt_best : 25.045110021149007
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.20613 (QuantReg: 11.74175) QuantErr: 11.74175 batch_time=193.35993
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.86301 (QuantReg: 12.57694) QuantErr: 12.57694 batch_time=8.02593
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 3.87780 (QuantReg: 12.83769) QuantErr: 12.83769 batch_time=0.44412
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.39610 (QuantReg: 12.62352) QuantErr: 12.62352 batch_time=0.43545
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.11725 (QuantReg: 12.77454) QuantErr: 12.77454 batch_time=0.43801
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.05888 (QuantReg: 13.09196) QuantErr: 13.09196 batch_time=0.45359
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.22102 (QuantReg: 13.02900) QuantErr: 13.02900 batch_time=0.44615
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.81813 (QuantReg: 13.32957) QuantErr: 13.32957 batch_time=0.45163
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.79895 (QuantReg: 13.90870) QuantErr: 13.90870 batch_time=0.44044
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.65588 (QuantReg: 13.33296) QuantErr: 13.33296 batch_time=0.45557
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.79873 (QuantReg: 14.06632) QuantErr: 14.06632 batch_time=0.44923
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.78056 (QuantReg: 13.84675) QuantErr: 13.84675 batch_time=0.45647
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.43409 (QuantReg: 13.43102) QuantErr: 13.43102 batch_time=0.50513
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.76907 (QuantReg: 14.18176) QuantErr: 14.18176 batch_time=0.46554
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.40441 (QuantReg: 14.07955) QuantErr: 14.07955 batch_time=0.45941
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.40190 (QuantReg: 14.03122) QuantErr: 14.03122 batch_time=0.43449
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 4.29587 (QuantReg: 14.14546) QuantErr: 14.14546 batch_time=0.46890
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.78202 (QuantReg: 14.11786) QuantErr: 14.11786 batch_time=0.45905
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.35911 (QuantReg: 14.48715) QuantErr: 14.48715 batch_time=0.45938
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.11534 (QuantReg: 14.55858) QuantErr: 14.55858 batch_time=0.44019
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.59615 (QuantReg: 14.79869) QuantErr: 14.79869 batch_time=0.45074
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.11589 (QuantReg: 14.51616) QuantErr: 14.51616 batch_time=0.44781
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.56883 (QuantReg: 14.61742) QuantErr: 14.61742 batch_time=0.46488
Train Epoch: 2 codebook_update_time=2.36787
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch2.pth ...
Done in 19.367s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch2.pth ...
Done in 21.937s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.6801603622436523
quant_reg : 13.666540561676026
quant_err : 13.666540561676026
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 39.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 53.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.739000000000004
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.155063515442198
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 39.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 54.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.995000000000005
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.26052297671636
mnt_best : 31.155063515442198
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.13098 (QuantReg: 12.34675) QuantErr: 12.34675 batch_time=83.59249
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.58753 (QuantReg: 12.09549) QuantErr: 12.09549 batch_time=0.44640
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.18441 (QuantReg: 12.54706) QuantErr: 12.54706 batch_time=0.43323
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.59964 (QuantReg: 12.27366) QuantErr: 12.27366 batch_time=4.59940
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.03840 (QuantReg: 12.50166) QuantErr: 12.50166 batch_time=0.44856
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.10390 (QuantReg: 12.25545) QuantErr: 12.25545 batch_time=0.43703
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.14015 (QuantReg: 12.58584) QuantErr: 12.58584 batch_time=0.44242
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.38635 (QuantReg: 12.87940) QuantErr: 12.87940 batch_time=8.03988
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.37787 (QuantReg: 12.51465) QuantErr: 12.51465 batch_time=0.47775
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.23847 (QuantReg: 12.82987) QuantErr: 12.82987 batch_time=0.43842
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.14045 (QuantReg: 12.66386) QuantErr: 12.66386 batch_time=0.44939
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 2.56838 (QuantReg: 12.83408) QuantErr: 12.83408 batch_time=0.45234
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.09166 (QuantReg: 13.10688) QuantErr: 13.10688 batch_time=0.43758
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 2.85956 (QuantReg: 12.83318) QuantErr: 12.83318 batch_time=8.75820
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.32294 (QuantReg: 13.05155) QuantErr: 13.05155 batch_time=0.44035
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.76602 (QuantReg: 13.07401) QuantErr: 13.07401 batch_time=0.43738
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.97188 (QuantReg: 13.13722) QuantErr: 13.13722 batch_time=0.44184
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.22633 (QuantReg: 12.95700) QuantErr: 12.95700 batch_time=0.43115
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.62446 (QuantReg: 13.29213) QuantErr: 13.29213 batch_time=0.42894
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.09954 (QuantReg: 13.36422) QuantErr: 13.36422 batch_time=7.32262
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.42462 (QuantReg: 12.90467) QuantErr: 12.90467 batch_time=0.42789
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.14373 (QuantReg: 13.18171) QuantErr: 13.18171 batch_time=0.44876
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.68063 (QuantReg: 13.74211) QuantErr: 13.74211 batch_time=0.47962
Train Epoch: 3 codebook_update_time=2.65372
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch3.pth ...
Done in 3.773s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch3.pth ...
Done in 7.760s
removing stale ckpt [epoch 2] [took 0.09s]
epoch : 3
loss : 3.1390218381881714
quant_reg : 12.82812770462036
quant_err : 12.82812770462036
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 56.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.253
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.476264466958156
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 56.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.2095
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.33415320205652
mnt_best : 33.476264466958156
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.97754 (QuantReg: 12.10056) QuantErr: 12.10056 batch_time=113.53811
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.18093 (QuantReg: 12.49911) QuantErr: 12.49911 batch_time=0.45863
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.81157 (QuantReg: 12.31806) QuantErr: 12.31806 batch_time=6.44276
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.75012 (QuantReg: 12.43679) QuantErr: 12.43679 batch_time=0.44220
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.94561 (QuantReg: 12.51761) QuantErr: 12.51761 batch_time=0.43905
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.70571 (QuantReg: 12.67824) QuantErr: 12.67824 batch_time=0.44731
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.76976 (QuantReg: 12.33684) QuantErr: 12.33684 batch_time=5.09741
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.59797 (QuantReg: 12.82819) QuantErr: 12.82819 batch_time=0.45206
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.77075 (QuantReg: 12.50467) QuantErr: 12.50467 batch_time=0.55016
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.04260 (QuantReg: 12.73257) QuantErr: 12.73257 batch_time=0.45147
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.50710 (QuantReg: 12.74005) QuantErr: 12.74005 batch_time=2.07660
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.86738 (QuantReg: 12.58849) QuantErr: 12.58849 batch_time=0.44464
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.96274 (QuantReg: 12.88575) QuantErr: 12.88575 batch_time=0.43483
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.88365 (QuantReg: 13.10336) QuantErr: 13.10336 batch_time=0.53931
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.80869 (QuantReg: 12.72515) QuantErr: 12.72515 batch_time=0.55320
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.74920 (QuantReg: 12.91890) QuantErr: 12.91890 batch_time=0.44993
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.82033 (QuantReg: 13.12398) QuantErr: 13.12398 batch_time=0.45064
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.68312 (QuantReg: 13.05349) QuantErr: 13.05349 batch_time=8.97208
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.70572 (QuantReg: 13.26083) QuantErr: 13.26083 batch_time=2.07188
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.48810 (QuantReg: 12.89363) QuantErr: 12.89363 batch_time=0.45561
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.37460 (QuantReg: 13.11571) QuantErr: 13.11571 batch_time=7.88054
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.45785 (QuantReg: 13.04725) QuantErr: 13.04725 batch_time=1.82008
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.97003 (QuantReg: 13.33053) QuantErr: 13.33053 batch_time=0.43415
Train Epoch: 4 codebook_update_time=1.91237
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch4.pth ...
Done in 3.637s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch4.pth ...
Done in 7.313s
removing stale ckpt [epoch 3] [took 0.06s]
epoch : 4
loss : 2.8303982601165774
quant_reg : 12.740248588562011
quant_err : 12.740248588562011
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.176
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.38089102873392
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.493
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.0018826175963
mnt_best : 36.38089102873392
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.83931 (QuantReg: 12.13851) QuantErr: 12.13851 batch_time=96.70955
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.00095 (QuantReg: 12.72315) QuantErr: 12.72315 batch_time=0.44041
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 1.98225 (QuantReg: 12.72283) QuantErr: 12.72283 batch_time=0.43718
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.79491 (QuantReg: 12.45291) QuantErr: 12.45291 batch_time=0.43931
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 3.05472 (QuantReg: 12.38039) QuantErr: 12.38039 batch_time=0.45127
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.61253 (QuantReg: 12.44307) QuantErr: 12.44307 batch_time=0.43761
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.71014 (QuantReg: 12.57768) QuantErr: 12.57768 batch_time=0.45151
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.16469 (QuantReg: 12.39293) QuantErr: 12.39293 batch_time=0.45701
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.52687 (QuantReg: 12.46829) QuantErr: 12.46829 batch_time=6.43233
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.71897 (QuantReg: 12.58669) QuantErr: 12.58669 batch_time=0.43187
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.29799 (QuantReg: 12.84130) QuantErr: 12.84130 batch_time=0.43973
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 3.09553 (QuantReg: 12.81386) QuantErr: 12.81386 batch_time=0.42286
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.58797 (QuantReg: 12.96181) QuantErr: 12.96181 batch_time=0.44195
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.37474 (QuantReg: 12.76598) QuantErr: 12.76598 batch_time=0.46014
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.91996 (QuantReg: 12.91977) QuantErr: 12.91977 batch_time=0.44949
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.84801 (QuantReg: 13.01010) QuantErr: 13.01010 batch_time=0.45030
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.44928 (QuantReg: 12.87792) QuantErr: 12.87792 batch_time=0.46763
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.29127 (QuantReg: 12.74856) QuantErr: 12.74856 batch_time=0.45178
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.70013 (QuantReg: 13.17450) QuantErr: 13.17450 batch_time=0.46139
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.42619 (QuantReg: 13.12312) QuantErr: 13.12312 batch_time=0.44799
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.54038 (QuantReg: 12.77630) QuantErr: 12.77630 batch_time=13.66722
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 3.26697 (QuantReg: 13.36591) QuantErr: 13.36591 batch_time=0.45815
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.22347 (QuantReg: 13.00611) QuantErr: 13.00611 batch_time=0.45208
Train Epoch: 5 codebook_update_time=1.75510
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch5.pth ...
Done in 3.217s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch5.pth ...
Done in 6.950s
removing stale ckpt [epoch 4] [took 0.14s]
epoch : 5
loss : 2.596017980098724
quant_reg : 12.784067665100098
quant_err : 12.784067665100098
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.848
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.967048535246825
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.839
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.29652352556085
mnt_best : 37.967048535246825
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.43736 (QuantReg: 12.85634) QuantErr: 12.85634 batch_time=79.65230
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.28919 (QuantReg: 12.33379) QuantErr: 12.33379 batch_time=0.43383
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.51375 (QuantReg: 12.53324) QuantErr: 12.53324 batch_time=0.43684
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.88905 (QuantReg: 12.27685) QuantErr: 12.27685 batch_time=0.43869
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.67955 (QuantReg: 12.36001) QuantErr: 12.36001 batch_time=0.44345
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.69236 (QuantReg: 12.95203) QuantErr: 12.95203 batch_time=0.43720
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.77895 (QuantReg: 12.46198) QuantErr: 12.46198 batch_time=0.44118
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.94819 (QuantReg: 12.85487) QuantErr: 12.85487 batch_time=0.76357
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.51253 (QuantReg: 12.84071) QuantErr: 12.84071 batch_time=0.47514
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.27522 (QuantReg: 12.99007) QuantErr: 12.99007 batch_time=5.11623
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.50007 (QuantReg: 12.68301) QuantErr: 12.68301 batch_time=0.45657
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.42251 (QuantReg: 12.90317) QuantErr: 12.90317 batch_time=0.44829
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.03107 (QuantReg: 12.82031) QuantErr: 12.82031 batch_time=0.45362
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.30742 (QuantReg: 12.68256) QuantErr: 12.68256 batch_time=0.45897
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.36494 (QuantReg: 12.81033) QuantErr: 12.81033 batch_time=0.44316
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.60850 (QuantReg: 13.29075) QuantErr: 13.29075 batch_time=0.48903
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.51511 (QuantReg: 13.08109) QuantErr: 13.08109 batch_time=0.48568
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.53719 (QuantReg: 12.90720) QuantErr: 12.90720 batch_time=0.44709
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.55517 (QuantReg: 12.99070) QuantErr: 12.99070 batch_time=0.42680
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.68555 (QuantReg: 12.95966) QuantErr: 12.95966 batch_time=0.64546
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.38343 (QuantReg: 12.66758) QuantErr: 12.66758 batch_time=0.45303
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.56143 (QuantReg: 13.24298) QuantErr: 13.24298 batch_time=0.43542
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.23884 (QuantReg: 13.20177) QuantErr: 13.20177 batch_time=0.42688
Train Epoch: 6 codebook_update_time=1.79833
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch6.pth ...
Done in 3.479s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch6.pth ...
Done in 6.561s
removing stale ckpt [epoch 5] [took 0.05s]
epoch : 6
loss : 2.4027933650016786
quant_reg : 12.886209732055663
quant_err : 12.886209732055663
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.988
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.29728681182554
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.4395
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.38584412028717
mnt_best : 38.29728681182554
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.16592 (QuantReg: 12.62819) QuantErr: 12.62819 batch_time=84.18184
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.05416 (QuantReg: 12.90144) QuantErr: 12.90144 batch_time=0.43387
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.25560 (QuantReg: 12.56347) QuantErr: 12.56347 batch_time=0.43040
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 1.94165 (QuantReg: 12.98841) QuantErr: 12.98841 batch_time=0.44273
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.48563 (QuantReg: 12.70301) QuantErr: 12.70301 batch_time=0.58714
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.37824 (QuantReg: 12.81924) QuantErr: 12.81924 batch_time=0.44848
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.36601 (QuantReg: 13.03596) QuantErr: 13.03596 batch_time=0.57990
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.21962 (QuantReg: 12.38657) QuantErr: 12.38657 batch_time=0.44142
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.25225 (QuantReg: 12.81373) QuantErr: 12.81373 batch_time=0.43485
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.84233 (QuantReg: 12.98865) QuantErr: 12.98865 batch_time=0.44355
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.20616 (QuantReg: 12.41255) QuantErr: 12.41255 batch_time=0.45771
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.26617 (QuantReg: 13.06769) QuantErr: 13.06769 batch_time=0.44259
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.95081 (QuantReg: 12.94253) QuantErr: 12.94253 batch_time=0.44964
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.44622 (QuantReg: 13.07264) QuantErr: 13.07264 batch_time=8.52896
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.02099 (QuantReg: 13.03953) QuantErr: 13.03953 batch_time=0.44036
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.44134 (QuantReg: 13.25335) QuantErr: 13.25335 batch_time=0.43570
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 2.13932 (QuantReg: 13.22448) QuantErr: 13.22448 batch_time=0.46242
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.22754 (QuantReg: 12.91456) QuantErr: 12.91456 batch_time=0.43656
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.99630 (QuantReg: 13.03073) QuantErr: 13.03073 batch_time=0.45058
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.97812 (QuantReg: 13.19166) QuantErr: 13.19166 batch_time=0.43987
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.20598 (QuantReg: 13.01411) QuantErr: 13.01411 batch_time=0.45060
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.02312 (QuantReg: 13.30625) QuantErr: 13.30625 batch_time=0.45556
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 1.99614 (QuantReg: 13.26652) QuantErr: 13.26652 batch_time=0.44827
Train Epoch: 7 codebook_update_time=1.67517
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch7.pth ...
Done in 3.565s
removing stale ckpt [epoch 6] [took 0.07s]
epoch : 7
loss : 2.23469886302948
quant_reg : 12.928523677825927
quant_err : 12.928523677825927
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.981
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.7830481177112
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.574
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.53293924957601
mnt_best : 38.29728681182554
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.97322 (QuantReg: 12.54423) QuantErr: 12.54423 batch_time=89.59256
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.12365 (QuantReg: 12.99260) QuantErr: 12.99260 batch_time=0.43913
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.20775 (QuantReg: 12.79500) QuantErr: 12.79500 batch_time=0.43907
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.46614 (QuantReg: 12.57037) QuantErr: 12.57037 batch_time=0.46191
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.35225 (QuantReg: 12.82824) QuantErr: 12.82824 batch_time=0.43725
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.97875 (QuantReg: 12.84196) QuantErr: 12.84196 batch_time=0.44469
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.99943 (QuantReg: 12.70513) QuantErr: 12.70513 batch_time=0.45368
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.24386 (QuantReg: 13.09268) QuantErr: 13.09268 batch_time=0.43352
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.05259 (QuantReg: 12.86337) QuantErr: 12.86337 batch_time=4.87491
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.11502 (QuantReg: 12.78467) QuantErr: 12.78467 batch_time=0.47188
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.08211 (QuantReg: 13.38374) QuantErr: 13.38374 batch_time=0.45916
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.12366 (QuantReg: 12.92587) QuantErr: 12.92587 batch_time=0.44605
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.33814 (QuantReg: 12.89537) QuantErr: 12.89537 batch_time=2.47133
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.93174 (QuantReg: 12.62057) QuantErr: 12.62057 batch_time=0.43900
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 1.82272 (QuantReg: 13.13223) QuantErr: 13.13223 batch_time=12.83480
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.83834 (QuantReg: 12.72254) QuantErr: 12.72254 batch_time=0.44061
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 1.90228 (QuantReg: 12.89152) QuantErr: 12.89152 batch_time=0.42839
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.09149 (QuantReg: 12.99404) QuantErr: 12.99404 batch_time=0.43001
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.95687 (QuantReg: 13.33707) QuantErr: 13.33707 batch_time=10.35071
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.07232 (QuantReg: 13.55067) QuantErr: 13.55067 batch_time=0.47648
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.74323 (QuantReg: 13.00340) QuantErr: 13.00340 batch_time=0.43661
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 2.10625 (QuantReg: 13.41444) QuantErr: 13.41444 batch_time=0.43895
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.12032 (QuantReg: 13.25613) QuantErr: 13.25613 batch_time=0.62335
Train Epoch: 8 codebook_update_time=2.39138
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch8.pth ...
Done in 3.520s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch8.pth ...
Done in 7.540s
removing stale ckpt [epoch 7] [took 0.06s]
epoch : 8
loss : 2.107108958721161
quant_reg : 13.001321914672852
quant_err : 13.001321914672852
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.143
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.8509721235103
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.6525
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.220712204785784
mnt_best : 39.8509721235103
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.23154 (QuantReg: 12.60541) QuantErr: 12.60541 batch_time=86.78218
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.79352 (QuantReg: 12.86784) QuantErr: 12.86784 batch_time=0.45545
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.09251 (QuantReg: 12.84211) QuantErr: 12.84211 batch_time=0.47247
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 1.97290 (QuantReg: 13.12009) QuantErr: 13.12009 batch_time=0.44056
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.43343 (QuantReg: 13.01447) QuantErr: 13.01447 batch_time=0.47752
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.44902 (QuantReg: 13.05407) QuantErr: 13.05407 batch_time=0.47600
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.28192 (QuantReg: 13.17143) QuantErr: 13.17143 batch_time=0.58508
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.86442 (QuantReg: 12.78981) QuantErr: 12.78981 batch_time=0.43680
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.23612 (QuantReg: 13.30087) QuantErr: 13.30087 batch_time=0.44343
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.82393 (QuantReg: 13.12640) QuantErr: 13.12640 batch_time=0.44493
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.04196 (QuantReg: 13.34064) QuantErr: 13.34064 batch_time=0.44393
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 2.20685 (QuantReg: 12.88793) QuantErr: 12.88793 batch_time=0.44159
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.96776 (QuantReg: 13.18337) QuantErr: 13.18337 batch_time=3.51249
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.02457 (QuantReg: 13.08485) QuantErr: 13.08485 batch_time=1.41570
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.84212 (QuantReg: 13.31659) QuantErr: 13.31659 batch_time=0.44058
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 2.15487 (QuantReg: 13.20045) QuantErr: 13.20045 batch_time=0.43238
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.91726 (QuantReg: 13.07567) QuantErr: 13.07567 batch_time=0.47768
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.80391 (QuantReg: 13.21221) QuantErr: 13.21221 batch_time=0.47988
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.06918 (QuantReg: 12.88578) QuantErr: 12.88578 batch_time=0.44564
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.91613 (QuantReg: 13.21629) QuantErr: 13.21629 batch_time=1.35598
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.33626 (QuantReg: 13.29532) QuantErr: 13.29532 batch_time=0.43328
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.05479 (QuantReg: 13.17641) QuantErr: 13.17641 batch_time=0.42778
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 2.16629 (QuantReg: 13.39356) QuantErr: 13.39356 batch_time=0.45649
Train Epoch: 9 codebook_update_time=1.94977
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch9.pth ...
Done in 4.100s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch9.pth ...
Done in 8.425s
removing stale ckpt [epoch 8] [took 0.15s]
epoch : 9
loss : 2.0368702211380003
quant_reg : 13.081559982299805
quant_err : 13.081559982299805
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.198
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.0922350704876
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.55
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.03415810811138
mnt_best : 40.0922350704876
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.96111 (QuantReg: 13.05071) QuantErr: 13.05071 batch_time=98.59954
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.00365 (QuantReg: 12.66052) QuantErr: 12.66052 batch_time=0.44136
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.27448 (QuantReg: 12.88852) QuantErr: 12.88852 batch_time=0.44857
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.10704 (QuantReg: 12.74637) QuantErr: 12.74637 batch_time=0.44960
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 2.50097 (QuantReg: 13.17722) QuantErr: 13.17722 batch_time=0.44090
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.86244 (QuantReg: 12.82664) QuantErr: 12.82664 batch_time=0.57130
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.03277 (QuantReg: 13.39884) QuantErr: 13.39884 batch_time=3.58557
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 2.00407 (QuantReg: 13.17100) QuantErr: 13.17100 batch_time=0.96693
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.41414 (QuantReg: 12.74925) QuantErr: 12.74925 batch_time=0.47621
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.05882 (QuantReg: 13.11037) QuantErr: 13.11037 batch_time=0.43975
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.75194 (QuantReg: 13.20733) QuantErr: 13.20733 batch_time=2.30468
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.90814 (QuantReg: 13.36723) QuantErr: 13.36723 batch_time=0.48645
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.73570 (QuantReg: 13.01211) QuantErr: 13.01211 batch_time=0.45201
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.97774 (QuantReg: 13.48272) QuantErr: 13.48272 batch_time=0.44437
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 2.28453 (QuantReg: 13.29398) QuantErr: 13.29398 batch_time=0.43046
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.07353 (QuantReg: 13.08390) QuantErr: 13.08390 batch_time=3.42834
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 1.71121 (QuantReg: 13.48364) QuantErr: 13.48364 batch_time=0.44078
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.26533 (QuantReg: 13.12336) QuantErr: 13.12336 batch_time=0.44312
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.75695 (QuantReg: 13.11042) QuantErr: 13.11042 batch_time=0.79264
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.81529 (QuantReg: 12.96039) QuantErr: 12.96039 batch_time=0.50421
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.95439 (QuantReg: 13.23109) QuantErr: 13.23109 batch_time=0.44845
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.93829 (QuantReg: 12.96974) QuantErr: 12.96974 batch_time=0.44845
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.11132 (QuantReg: 13.20676) QuantErr: 13.20676 batch_time=0.44638
Train Epoch: 10 codebook_update_time=1.78586
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch10.pth ...
Done in 4.528s
removing stale ckpt [epoch 9] [took 0.06s]
epoch : 10
loss : 1.9260354146957397
quant_reg : 13.156722236633302
quant_err : 13.156722236633302
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.159
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.85882554837191
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.583
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.228154497205445
mnt_best : 40.0922350704876
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.02648 (QuantReg: 12.95067) QuantErr: 12.95067 batch_time=110.42600
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.93262 (QuantReg: 12.76720) QuantErr: 12.76720 batch_time=0.44458
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.94748 (QuantReg: 12.98788) QuantErr: 12.98788 batch_time=0.44090
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.52100 (QuantReg: 12.47749) QuantErr: 12.47749 batch_time=0.44289
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.86134 (QuantReg: 13.16830) QuantErr: 13.16830 batch_time=1.14464
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.76273 (QuantReg: 13.13801) QuantErr: 13.13801 batch_time=0.43470
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 2.33270 (QuantReg: 12.78330) QuantErr: 12.78330 batch_time=0.44152
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.32681 (QuantReg: 12.98940) QuantErr: 12.98940 batch_time=1.69335
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.74372 (QuantReg: 13.05216) QuantErr: 13.05216 batch_time=0.43634
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.93533 (QuantReg: 12.66710) QuantErr: 12.66710 batch_time=0.43324
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.53724 (QuantReg: 13.24086) QuantErr: 13.24086 batch_time=0.43813
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.04590 (QuantReg: 12.85056) QuantErr: 12.85056 batch_time=0.43138
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.96091 (QuantReg: 13.35076) QuantErr: 13.35076 batch_time=0.43407
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.62908 (QuantReg: 12.90143) QuantErr: 12.90143 batch_time=0.55549
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.66441 (QuantReg: 13.42794) QuantErr: 13.42794 batch_time=16.33332
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.13077 (QuantReg: 13.18686) QuantErr: 13.18686 batch_time=4.75166
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.44657 (QuantReg: 13.49065) QuantErr: 13.49065 batch_time=0.44511
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.91303 (QuantReg: 13.41787) QuantErr: 13.41787 batch_time=0.44160
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.59059 (QuantReg: 13.61304) QuantErr: 13.61304 batch_time=0.48049
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.38567 (QuantReg: 13.03335) QuantErr: 13.03335 batch_time=0.44461
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.62858 (QuantReg: 13.45970) QuantErr: 13.45970 batch_time=0.44170
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.29007 (QuantReg: 12.96759) QuantErr: 12.96759 batch_time=0.44667
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.62741 (QuantReg: 13.18515) QuantErr: 13.18515 batch_time=0.46997
Train Epoch: 11 codebook_update_time=2.16484
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch11.pth ...
Done in 3.542s
removing stale ckpt [epoch 10] [took 1.22s]
epoch : 11
loss : 1.8745411729812622
quant_reg : 13.170095489501954
quant_err : 13.170095489501954
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.832
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.01783891814232
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.884
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.66409621120723
mnt_best : 40.0922350704876
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.89502 (QuantReg: 13.28428) QuantErr: 13.28428 batch_time=113.11971
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 2.47558 (QuantReg: 12.89919) QuantErr: 12.89919 batch_time=0.44255
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 2.09122 (QuantReg: 13.28958) QuantErr: 13.28958 batch_time=0.47268
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.74207 (QuantReg: 13.37717) QuantErr: 13.37717 batch_time=0.44242
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.94121 (QuantReg: 12.68680) QuantErr: 12.68680 batch_time=0.44091
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.67895 (QuantReg: 13.19343) QuantErr: 13.19343 batch_time=0.43650
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.92704 (QuantReg: 13.12314) QuantErr: 13.12314 batch_time=0.57059
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.56727 (QuantReg: 13.31869) QuantErr: 13.31869 batch_time=0.46477
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.72422 (QuantReg: 13.15746) QuantErr: 13.15746 batch_time=0.44070
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.54431 (QuantReg: 13.25712) QuantErr: 13.25712 batch_time=0.43797
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.24462 (QuantReg: 13.12922) QuantErr: 13.12922 batch_time=2.90870
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.44675 (QuantReg: 13.21403) QuantErr: 13.21403 batch_time=0.45529
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.59735 (QuantReg: 13.51161) QuantErr: 13.51161 batch_time=3.19048
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 2.11030 (QuantReg: 13.22905) QuantErr: 13.22905 batch_time=0.44938
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.61503 (QuantReg: 13.42745) QuantErr: 13.42745 batch_time=0.43959
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.42622 (QuantReg: 13.04123) QuantErr: 13.04123 batch_time=0.44763
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.45997 (QuantReg: 13.19520) QuantErr: 13.19520 batch_time=1.59492
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.66669 (QuantReg: 13.34484) QuantErr: 13.34484 batch_time=0.45246
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.83374 (QuantReg: 13.32657) QuantErr: 13.32657 batch_time=0.43635
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.75166 (QuantReg: 13.49826) QuantErr: 13.49826 batch_time=0.42358
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.50953 (QuantReg: 13.69943) QuantErr: 13.69943 batch_time=0.44626
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.67541 (QuantReg: 13.77893) QuantErr: 13.77893 batch_time=0.44528
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.70391 (QuantReg: 12.95524) QuantErr: 12.95524 batch_time=0.44932
Train Epoch: 12 codebook_update_time=2.23399
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch12.pth ...
Done in 3.488s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch12.pth ...
Done in 7.928s
removing stale ckpt [epoch 11] [took 0.08s]
epoch : 12
loss : 1.7521686344146727
quant_reg : 13.276629028320313
quant_err : 13.276629028320313
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.96
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.19312418640444
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.8245
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.33360525724163
mnt_best : 41.19312418640444
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.56211 (QuantReg: 13.43182) QuantErr: 13.43182 batch_time=110.80927
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.69798 (QuantReg: 13.11252) QuantErr: 13.11252 batch_time=0.43862
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 2.28497 (QuantReg: 12.97978) QuantErr: 12.97978 batch_time=0.44330
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.48225 (QuantReg: 13.26224) QuantErr: 13.26224 batch_time=1.40603
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.35522 (QuantReg: 13.45817) QuantErr: 13.45817 batch_time=0.44930
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.04500 (QuantReg: 12.92591) QuantErr: 12.92591 batch_time=0.47609
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.32356 (QuantReg: 13.03110) QuantErr: 13.03110 batch_time=0.43632
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.76514 (QuantReg: 13.35009) QuantErr: 13.35009 batch_time=8.33870
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.42194 (QuantReg: 13.34566) QuantErr: 13.34566 batch_time=0.44339
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.72091 (QuantReg: 13.55046) QuantErr: 13.55046 batch_time=0.44353
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.63311 (QuantReg: 13.30894) QuantErr: 13.30894 batch_time=0.47401
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.94092 (QuantReg: 13.40281) QuantErr: 13.40281 batch_time=0.43969
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.70263 (QuantReg: 13.20706) QuantErr: 13.20706 batch_time=10.83132
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.57885 (QuantReg: 13.43777) QuantErr: 13.43777 batch_time=0.44795
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.73887 (QuantReg: 13.36778) QuantErr: 13.36778 batch_time=0.42561
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.70637 (QuantReg: 13.02458) QuantErr: 13.02458 batch_time=0.43438
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.69436 (QuantReg: 13.28937) QuantErr: 13.28937 batch_time=0.46249
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.88674 (QuantReg: 13.39034) QuantErr: 13.39034 batch_time=0.47685
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.61235 (QuantReg: 13.34719) QuantErr: 13.34719 batch_time=2.03759
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.32458 (QuantReg: 13.48829) QuantErr: 13.48829 batch_time=0.43812
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 2.06276 (QuantReg: 13.46377) QuantErr: 13.46377 batch_time=0.56877
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.31449 (QuantReg: 13.53864) QuantErr: 13.53864 batch_time=0.44034
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.97021 (QuantReg: 13.11347) QuantErr: 13.11347 batch_time=0.42850
Train Epoch: 13 codebook_update_time=1.62905
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch13.pth ...
Done in 3.634s
removing stale ckpt [epoch 12] [took 0.50s]
epoch : 13
loss : 1.6790368185043334
quant_reg : 13.317774440765382
quant_err : 13.317774440765382
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.644
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.52022888238353
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.2965
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.24769921663741
mnt_best : 41.19312418640444
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 2.12958 (QuantReg: 13.07537) QuantErr: 13.07537 batch_time=119.15509
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.70788 (QuantReg: 13.10187) QuantErr: 13.10187 batch_time=0.48479
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.35267 (QuantReg: 13.39375) QuantErr: 13.39375 batch_time=0.43706
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.41171 (QuantReg: 13.26925) QuantErr: 13.26925 batch_time=0.43436
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.30839 (QuantReg: 13.42309) QuantErr: 13.42309 batch_time=0.43930
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.11342 (QuantReg: 13.52961) QuantErr: 13.52961 batch_time=0.44520
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.53726 (QuantReg: 13.68709) QuantErr: 13.68709 batch_time=0.44017
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.68896 (QuantReg: 13.23769) QuantErr: 13.23769 batch_time=0.43002
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.91501 (QuantReg: 13.05706) QuantErr: 13.05706 batch_time=0.45478
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.60983 (QuantReg: 13.12934) QuantErr: 13.12934 batch_time=2.03053
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.68697 (QuantReg: 13.51930) QuantErr: 13.51930 batch_time=0.44341
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.87158 (QuantReg: 13.42497) QuantErr: 13.42497 batch_time=0.43062
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.52364 (QuantReg: 13.28316) QuantErr: 13.28316 batch_time=0.43780
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.63913 (QuantReg: 13.55263) QuantErr: 13.55263 batch_time=0.44411
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 2.01876 (QuantReg: 13.30379) QuantErr: 13.30379 batch_time=0.46687
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.52221 (QuantReg: 13.38348) QuantErr: 13.38348 batch_time=0.44431
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.73523 (QuantReg: 13.08975) QuantErr: 13.08975 batch_time=0.44769
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.52616 (QuantReg: 13.61855) QuantErr: 13.61855 batch_time=0.43375
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.43834 (QuantReg: 13.34445) QuantErr: 13.34445 batch_time=0.44356
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.68554 (QuantReg: 13.82454) QuantErr: 13.82454 batch_time=18.10450
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.78616 (QuantReg: 13.79523) QuantErr: 13.79523 batch_time=3.24713
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.63985 (QuantReg: 13.64888) QuantErr: 13.64888 batch_time=2.92203
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.89512 (QuantReg: 13.28416) QuantErr: 13.28416 batch_time=0.42698
Train Epoch: 14 codebook_update_time=1.80387
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch14.pth ...
Done in 5.028s
removing stale ckpt [epoch 13] [took 0.10s]
epoch : 14
loss : 1.6350224380493164
quant_reg : 13.353197372436524
quant_err : 13.353197372436524
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.275
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.069148827570324
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.067
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.49647233768151
mnt_best : 41.19312418640444
not_improved_count: 2
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.44404 (QuantReg: 13.25435) QuantErr: 13.25435 batch_time=140.87451
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.86023 (QuantReg: 13.04604) QuantErr: 13.04604 batch_time=0.43442
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.45953 (QuantReg: 13.58148) QuantErr: 13.58148 batch_time=0.44112
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.79754 (QuantReg: 12.98147) QuantErr: 12.98147 batch_time=0.43747
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.61217 (QuantReg: 13.16255) QuantErr: 13.16255 batch_time=0.46027
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.88623 (QuantReg: 13.57858) QuantErr: 13.57858 batch_time=0.43205
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.46066 (QuantReg: 13.39994) QuantErr: 13.39994 batch_time=0.46159
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.72201 (QuantReg: 13.29823) QuantErr: 13.29823 batch_time=0.43932
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.54184 (QuantReg: 13.62122) QuantErr: 13.62122 batch_time=0.43400
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.26423 (QuantReg: 13.67408) QuantErr: 13.67408 batch_time=0.43930
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.66288 (QuantReg: 13.34964) QuantErr: 13.34964 batch_time=0.46235
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.67857 (QuantReg: 13.37129) QuantErr: 13.37129 batch_time=0.43614
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.75138 (QuantReg: 13.38098) QuantErr: 13.38098 batch_time=0.56091
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.45733 (QuantReg: 13.39536) QuantErr: 13.39536 batch_time=2.29414
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.64360 (QuantReg: 13.32799) QuantErr: 13.32799 batch_time=4.74305
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.53816 (QuantReg: 13.06297) QuantErr: 13.06297 batch_time=0.53936
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.44743 (QuantReg: 13.62273) QuantErr: 13.62273 batch_time=0.44451
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.81941 (QuantReg: 13.68212) QuantErr: 13.68212 batch_time=0.55511
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.41115 (QuantReg: 13.45443) QuantErr: 13.45443 batch_time=0.44158
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.91476 (QuantReg: 13.32302) QuantErr: 13.32302 batch_time=0.43872
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.41102 (QuantReg: 13.93954) QuantErr: 13.93954 batch_time=0.88842
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.74241 (QuantReg: 13.62019) QuantErr: 13.62019 batch_time=0.44020
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.08287 (QuantReg: 13.46072) QuantErr: 13.46072 batch_time=0.46155
Train Epoch: 15 codebook_update_time=2.17190
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch15.pth ...
Done in 3.459s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch15.pth ...
Done in 7.340s
removing stale ckpt [epoch 14] [took 0.08s]
epoch : 15
loss : 1.597935359954834
quant_reg : 13.39501524734497
quant_err : 13.39501524734497
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.341
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.5152119198467
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.6275
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.60809156735046
mnt_best : 41.5152119198467
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.48301 (QuantReg: 13.30711) QuantErr: 13.30711 batch_time=103.11811
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.74695 (QuantReg: 13.14978) QuantErr: 13.14978 batch_time=0.46648
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.69030 (QuantReg: 13.45794) QuantErr: 13.45794 batch_time=12.92483
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.56154 (QuantReg: 13.44042) QuantErr: 13.44042 batch_time=0.43224
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.45425 (QuantReg: 13.44639) QuantErr: 13.44639 batch_time=0.45655
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.61571 (QuantReg: 13.55044) QuantErr: 13.55044 batch_time=0.43425
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.27591 (QuantReg: 13.56374) QuantErr: 13.56374 batch_time=0.43420
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.54671 (QuantReg: 13.68697) QuantErr: 13.68697 batch_time=0.42942
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.48423 (QuantReg: 13.51573) QuantErr: 13.51573 batch_time=0.44228
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.22341 (QuantReg: 13.52452) QuantErr: 13.52452 batch_time=0.42706
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.54884 (QuantReg: 13.85678) QuantErr: 13.85678 batch_time=2.33119
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.54156 (QuantReg: 13.36380) QuantErr: 13.36380 batch_time=0.43875
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.21507 (QuantReg: 13.39190) QuantErr: 13.39190 batch_time=0.43923
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.48385 (QuantReg: 13.74293) QuantErr: 13.74293 batch_time=11.17979
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.23407 (QuantReg: 13.55258) QuantErr: 13.55258 batch_time=0.44295
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.44422 (QuantReg: 13.80017) QuantErr: 13.80017 batch_time=0.43923
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.37468 (QuantReg: 13.28562) QuantErr: 13.28562 batch_time=0.44882
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.85835 (QuantReg: 13.31091) QuantErr: 13.31091 batch_time=0.44809
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.87489 (QuantReg: 13.65524) QuantErr: 13.65524 batch_time=0.45228
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.38535 (QuantReg: 13.67370) QuantErr: 13.67370 batch_time=0.43722
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.50098 (QuantReg: 13.49450) QuantErr: 13.49450 batch_time=0.43866
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.42197 (QuantReg: 13.73848) QuantErr: 13.73848 batch_time=0.48754
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.72403 (QuantReg: 13.43318) QuantErr: 13.43318 batch_time=0.44558
Train Epoch: 16 codebook_update_time=1.74334
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch16.pth ...
Done in 4.339s
removing stale ckpt [epoch 15] [took 0.68s]
epoch : 16
loss : 1.5295302834510802
quant_reg : 13.462829563140868
quant_err : 13.462829563140868
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.278
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.758413508610914
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.392
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.92628907874235
mnt_best : 41.5152119198467
not_improved_count: 1
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.81187 (QuantReg: 13.36652) QuantErr: 13.36652 batch_time=117.82736
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.34412 (QuantReg: 13.43064) QuantErr: 13.43064 batch_time=0.55354
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.49105 (QuantReg: 13.22178) QuantErr: 13.22178 batch_time=0.45217
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.63348 (QuantReg: 13.12214) QuantErr: 13.12214 batch_time=0.44810
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.78366 (QuantReg: 13.42763) QuantErr: 13.42763 batch_time=0.48927
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.29208 (QuantReg: 13.40472) QuantErr: 13.40472 batch_time=0.98558
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.77961 (QuantReg: 13.11513) QuantErr: 13.11513 batch_time=5.24663
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.49394 (QuantReg: 13.50937) QuantErr: 13.50937 batch_time=5.16540
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.73202 (QuantReg: 13.76847) QuantErr: 13.76847 batch_time=0.44010
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.43495 (QuantReg: 13.56549) QuantErr: 13.56549 batch_time=0.44678
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.79744 (QuantReg: 13.51089) QuantErr: 13.51089 batch_time=1.08067
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.28372 (QuantReg: 13.58085) QuantErr: 13.58085 batch_time=0.44765
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.63640 (QuantReg: 13.30398) QuantErr: 13.30398 batch_time=0.43513
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.57910 (QuantReg: 13.52230) QuantErr: 13.52230 batch_time=0.43852
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.46250 (QuantReg: 13.58669) QuantErr: 13.58669 batch_time=0.43477
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.58412 (QuantReg: 13.66227) QuantErr: 13.66227 batch_time=0.43756
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.31817 (QuantReg: 13.50098) QuantErr: 13.50098 batch_time=0.45109
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.46104 (QuantReg: 13.50351) QuantErr: 13.50351 batch_time=0.47535
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.45068 (QuantReg: 13.03153) QuantErr: 13.03153 batch_time=0.43716
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.90699 (QuantReg: 13.28501) QuantErr: 13.28501 batch_time=0.47155
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.31602 (QuantReg: 13.47063) QuantErr: 13.47063 batch_time=0.47026
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.56673 (QuantReg: 13.52872) QuantErr: 13.52872 batch_time=0.43603
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.46045 (QuantReg: 12.88967) QuantErr: 12.88967 batch_time=0.47658
Train Epoch: 17 codebook_update_time=2.36828
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch17.pth ...
Done in 3.617s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch17.pth ...
Done in 8.000s
removing stale ckpt [epoch 16] [took 0.11s]
epoch : 17
loss : 1.5189161944389342
quant_reg : 13.507221950531006
quant_err : 13.507221950531006
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 24.9665
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.85735850931893
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.362
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.36072368543717
mnt_best : 41.85735850931893
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.23169 (QuantReg: 13.51557) QuantErr: 13.51557 batch_time=110.12832
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.55758 (QuantReg: 13.36204) QuantErr: 13.36204 batch_time=0.43775
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.77225 (QuantReg: 13.34982) QuantErr: 13.34982 batch_time=0.44939
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.50778 (QuantReg: 13.81217) QuantErr: 13.81217 batch_time=0.43355
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.36569 (QuantReg: 13.83469) QuantErr: 13.83469 batch_time=10.86379
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.34638 (QuantReg: 13.72297) QuantErr: 13.72297 batch_time=2.33527
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.47795 (QuantReg: 13.25615) QuantErr: 13.25615 batch_time=1.47385
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.16673 (QuantReg: 13.61593) QuantErr: 13.61593 batch_time=0.43393
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 0.99477 (QuantReg: 13.62276) QuantErr: 13.62276 batch_time=0.43479
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.50375 (QuantReg: 13.65992) QuantErr: 13.65992 batch_time=0.43171
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.54205 (QuantReg: 13.59214) QuantErr: 13.59214 batch_time=6.48855
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.60647 (QuantReg: 13.31359) QuantErr: 13.31359 batch_time=0.43301
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.30024 (QuantReg: 13.60429) QuantErr: 13.60429 batch_time=11.49131
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.88819 (QuantReg: 13.09666) QuantErr: 13.09666 batch_time=1.38277
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.13175 (QuantReg: 13.58156) QuantErr: 13.58156 batch_time=0.43284
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.50173 (QuantReg: 13.10859) QuantErr: 13.10859 batch_time=0.43681
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.78296 (QuantReg: 13.54596) QuantErr: 13.54596 batch_time=0.45595
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.66916 (QuantReg: 13.42741) QuantErr: 13.42741 batch_time=0.43296
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.60457 (QuantReg: 13.77151) QuantErr: 13.77151 batch_time=0.44148
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.44346 (QuantReg: 13.88443) QuantErr: 13.88443 batch_time=0.42983
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.46015 (QuantReg: 13.63837) QuantErr: 13.63837 batch_time=7.62703
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.42937 (QuantReg: 13.64794) QuantErr: 13.64794 batch_time=0.44339
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.22505 (QuantReg: 13.10571) QuantErr: 13.10571 batch_time=0.43131
Train Epoch: 18 codebook_update_time=1.97879
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch18.pth ...
Done in 6.618s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch18.pth ...
Done in 10.337s
removing stale ckpt [epoch 17] [took 0.05s]
epoch : 18
loss : 1.4820967769622804
quant_reg : 13.528574920654297
quant_err : 13.528574920654297
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.252
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.063961124174654
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.3885
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.575814480316325
mnt_best : 42.063961124174654
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.40087 (QuantReg: 13.81943) QuantErr: 13.81943 batch_time=104.43620
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.49818 (QuantReg: 13.31737) QuantErr: 13.31737 batch_time=0.43232
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.53572 (QuantReg: 13.15177) QuantErr: 13.15177 batch_time=0.42529
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.55907 (QuantReg: 13.28891) QuantErr: 13.28891 batch_time=0.43908
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.28968 (QuantReg: 13.21866) QuantErr: 13.21866 batch_time=0.43134
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.28371 (QuantReg: 13.41754) QuantErr: 13.41754 batch_time=1.27584
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.39547 (QuantReg: 13.47971) QuantErr: 13.47971 batch_time=1.37820
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.60106 (QuantReg: 13.49088) QuantErr: 13.49088 batch_time=0.43002
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.77194 (QuantReg: 13.86724) QuantErr: 13.86724 batch_time=0.43482
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.68067 (QuantReg: 13.37562) QuantErr: 13.37562 batch_time=0.44161
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.36962 (QuantReg: 13.63918) QuantErr: 13.63918 batch_time=0.44200
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.46880 (QuantReg: 13.60437) QuantErr: 13.60437 batch_time=0.47222
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.47663 (QuantReg: 13.49110) QuantErr: 13.49110 batch_time=0.43850
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.61868 (QuantReg: 13.64266) QuantErr: 13.64266 batch_time=0.43164
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.41527 (QuantReg: 13.66497) QuantErr: 13.66497 batch_time=0.43645
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.68356 (QuantReg: 13.60625) QuantErr: 13.60625 batch_time=0.46805
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.42408 (QuantReg: 13.75777) QuantErr: 13.75777 batch_time=0.43668
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.26540 (QuantReg: 13.35986) QuantErr: 13.35986 batch_time=0.43297
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.34799 (QuantReg: 13.71770) QuantErr: 13.71770 batch_time=0.43395
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.63028 (QuantReg: 13.08338) QuantErr: 13.08338 batch_time=0.47826
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.12514 (QuantReg: 13.78447) QuantErr: 13.78447 batch_time=0.44921
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.39463 (QuantReg: 13.52647) QuantErr: 13.52647 batch_time=0.44051
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.48069 (QuantReg: 13.74735) QuantErr: 13.74735 batch_time=0.46271
Train Epoch: 19 codebook_update_time=2.27223
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_distilbert-base/checkpoint-epoch19.pth ...
Done in 3.655s
removing stale ckpt [epoch 18] [took 0.09s]
epoch : 19
loss : 1.4607271556854249
quant_reg : 13.50440510559082
quant_err : 13.50440510559082
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.106
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.05530997212323
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0