-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_L15.txt
2603 lines (2603 loc) · 194 KB
/
HCQ_MSRVTT_1kA_L15.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 1274.2771649360657 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 73.81234693527222 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 48.862210512161255 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch0.pth ...
Done in 1.549s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch0.pth ...
Done in 3.041s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 1.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 5.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 508.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 498.491
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3914867641168864
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 491.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 492.881
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.3914867641168864
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 30.37422 (QuantReg: 22.44950) QuantErr: 22.44950 batch_time=17.04816
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 27.58488 (QuantReg: 22.62272) QuantErr: 22.62272 batch_time=0.64545
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 23.98049 (QuantReg: 22.62040) QuantErr: 22.62040 batch_time=0.69534
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 21.62486 (QuantReg: 22.61346) QuantErr: 22.61346 batch_time=0.63211
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 21.55148 (QuantReg: 22.60448) QuantErr: 22.60448 batch_time=0.66409
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 18.09508 (QuantReg: 22.66546) QuantErr: 22.66546 batch_time=0.67484
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 19.37546 (QuantReg: 22.64121) QuantErr: 22.64121 batch_time=1.83118
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 17.13990 (QuantReg: 22.65406) QuantErr: 22.65406 batch_time=0.69267
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 17.77531 (QuantReg: 22.65351) QuantErr: 22.65351 batch_time=0.66193
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 15.87346 (QuantReg: 22.64672) QuantErr: 22.64672 batch_time=0.68057
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 14.88769 (QuantReg: 22.62619) QuantErr: 22.62619 batch_time=0.71213
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 14.97726 (QuantReg: 22.62910) QuantErr: 22.62910 batch_time=0.65167
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 14.47709 (QuantReg: 22.65334) QuantErr: 22.65334 batch_time=0.73481
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 15.76533 (QuantReg: 22.61733) QuantErr: 22.61733 batch_time=5.02873
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 13.68880 (QuantReg: 22.65315) QuantErr: 22.65315 batch_time=0.65491
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 14.64779 (QuantReg: 22.65578) QuantErr: 22.65578 batch_time=0.65680
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 13.40636 (QuantReg: 22.65479) QuantErr: 22.65479 batch_time=0.67313
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 15.18630 (QuantReg: 22.61470) QuantErr: 22.61470 batch_time=0.65850
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 13.95435 (QuantReg: 22.61641) QuantErr: 22.61641 batch_time=0.65958
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 13.12395 (QuantReg: 22.65346) QuantErr: 22.65346 batch_time=0.69609
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 13.59403 (QuantReg: 22.66649) QuantErr: 22.66649 batch_time=0.73776
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 13.14003 (QuantReg: 22.66279) QuantErr: 22.66279 batch_time=0.64722
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 12.27474 (QuantReg: 22.65307) QuantErr: 22.65307 batch_time=0.63977
Train Epoch: 1 codebook_update_time=4.09348
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch1.pth ...
Done in 4.273s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch1.pth ...
Done in 8.165s
epoch : 1
loss : 17.089199127197265
quant_reg : 22.632978706359864
quant_err : 22.632978706359864
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 8.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 30.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 43.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 79.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 14.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 42.746
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.732806555982336
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 33.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 45.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 79.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 13.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 42.137
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.00925213129053
mnt_best : 22.732806555982336
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 13.15432 (QuantReg: 10.78074) QuantErr: 10.78074 batch_time=24.72859
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 11.12010 (QuantReg: 11.33028) QuantErr: 11.33028 batch_time=0.66315
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 12.48949 (QuantReg: 11.65358) QuantErr: 11.65358 batch_time=0.66809
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 13.07328 (QuantReg: 11.56365) QuantErr: 11.56365 batch_time=0.68514
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 12.80890 (QuantReg: 11.72416) QuantErr: 11.72416 batch_time=0.65689
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 12.12450 (QuantReg: 12.01840) QuantErr: 12.01840 batch_time=0.66067
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 11.09352 (QuantReg: 12.30861) QuantErr: 12.30861 batch_time=0.63562
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 12.20181 (QuantReg: 12.46158) QuantErr: 12.46158 batch_time=0.67232
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 12.19430 (QuantReg: 12.06218) QuantErr: 12.06218 batch_time=0.64727
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 11.93229 (QuantReg: 12.83361) QuantErr: 12.83361 batch_time=0.68733
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 10.35144 (QuantReg: 12.79028) QuantErr: 12.79028 batch_time=0.65232
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 10.81004 (QuantReg: 12.81147) QuantErr: 12.81147 batch_time=0.67300
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 13.11696 (QuantReg: 12.90938) QuantErr: 12.90938 batch_time=0.65284
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 11.43150 (QuantReg: 13.38324) QuantErr: 13.38324 batch_time=0.65364
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 10.47077 (QuantReg: 13.30187) QuantErr: 13.30187 batch_time=0.66440
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 11.94102 (QuantReg: 13.27518) QuantErr: 13.27518 batch_time=0.66307
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 11.87576 (QuantReg: 13.61311) QuantErr: 13.61311 batch_time=0.65057
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 12.76748 (QuantReg: 13.77293) QuantErr: 13.77293 batch_time=0.68189
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 10.63342 (QuantReg: 13.81152) QuantErr: 13.81152 batch_time=0.69821
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 12.01902 (QuantReg: 14.32555) QuantErr: 14.32555 batch_time=0.67386
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 10.54431 (QuantReg: 14.14965) QuantErr: 14.14965 batch_time=0.66428
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 10.83452 (QuantReg: 14.53295) QuantErr: 14.53295 batch_time=0.65798
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 12.51244 (QuantReg: 14.10012) QuantErr: 14.10012 batch_time=0.64673
Train Epoch: 2 codebook_update_time=3.40157
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch2.pth ...
Done in 11.766s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch2.pth ...
Done in 15.597s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.04s]
epoch : 2
loss : 11.752204376220703
quant_reg : 12.8765330619812
quant_err : 12.8765330619812
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 37.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 53.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 83.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.411
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.886899950730633
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 39.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 84.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 33.585
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.673130844107945
mnt_best : 28.886899950730633
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 8.58336 (QuantReg: 11.22417) QuantErr: 11.22417 batch_time=30.90636
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 10.22837 (QuantReg: 11.51076) QuantErr: 11.51076 batch_time=0.64582
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 10.97130 (QuantReg: 11.09208) QuantErr: 11.09208 batch_time=0.64843
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 9.81194 (QuantReg: 11.47024) QuantErr: 11.47024 batch_time=0.64626
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 9.84888 (QuantReg: 11.42272) QuantErr: 11.42272 batch_time=0.68433
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 9.82648 (QuantReg: 11.48941) QuantErr: 11.48941 batch_time=0.65883
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 9.65894 (QuantReg: 11.88322) QuantErr: 11.88322 batch_time=0.65867
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 8.56990 (QuantReg: 11.56408) QuantErr: 11.56408 batch_time=0.79545
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 10.39313 (QuantReg: 11.82426) QuantErr: 11.82426 batch_time=0.64519
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 10.23773 (QuantReg: 12.02767) QuantErr: 12.02767 batch_time=0.64648
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 9.77637 (QuantReg: 12.22674) QuantErr: 12.22674 batch_time=0.67640
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 10.55502 (QuantReg: 12.20712) QuantErr: 12.20712 batch_time=0.66128
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 9.84198 (QuantReg: 12.29491) QuantErr: 12.29491 batch_time=0.65420
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 9.27463 (QuantReg: 12.09862) QuantErr: 12.09862 batch_time=0.64975
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 12.36048 (QuantReg: 12.09363) QuantErr: 12.09363 batch_time=0.66375
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 8.16887 (QuantReg: 12.36328) QuantErr: 12.36328 batch_time=0.65175
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 10.35241 (QuantReg: 12.57216) QuantErr: 12.57216 batch_time=0.65333
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 10.22831 (QuantReg: 12.29282) QuantErr: 12.29282 batch_time=0.64272
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 10.51596 (QuantReg: 12.54054) QuantErr: 12.54054 batch_time=0.70122
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 9.84146 (QuantReg: 12.47631) QuantErr: 12.47631 batch_time=0.65208
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 9.02822 (QuantReg: 12.49811) QuantErr: 12.49811 batch_time=0.65084
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 11.10168 (QuantReg: 12.50915) QuantErr: 12.50915 batch_time=0.64512
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 9.86820 (QuantReg: 12.90735) QuantErr: 12.90735 batch_time=0.69769
Train Epoch: 3 codebook_update_time=3.32568
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch3.pth ...
Done in 3.975s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch3.pth ...
Done in 7.845s
removing stale ckpt [epoch 2] [took 0.02s]
epoch : 3
loss : 10.010972394943238
quant_reg : 12.030701400756836
quant_err : 12.030701400756836
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 40.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.314
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.02870601203549
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 56.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.077
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.701516154855305
mnt_best : 32.02870601203549
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 10.89729 (QuantReg: 11.48768) QuantErr: 11.48768 batch_time=22.60069
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 9.80488 (QuantReg: 11.11931) QuantErr: 11.11931 batch_time=0.72345
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 7.55098 (QuantReg: 11.75534) QuantErr: 11.75534 batch_time=1.12526
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 8.69904 (QuantReg: 11.50587) QuantErr: 11.50587 batch_time=0.72583
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 9.64461 (QuantReg: 11.60166) QuantErr: 11.60166 batch_time=0.67118
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 8.54059 (QuantReg: 12.02299) QuantErr: 12.02299 batch_time=0.64644
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 10.69260 (QuantReg: 12.11474) QuantErr: 12.11474 batch_time=1.09259
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 8.41961 (QuantReg: 12.03121) QuantErr: 12.03121 batch_time=0.63714
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 11.63714 (QuantReg: 12.13436) QuantErr: 12.13436 batch_time=0.73946
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 8.15287 (QuantReg: 11.82309) QuantErr: 11.82309 batch_time=0.71891
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 8.64139 (QuantReg: 11.77757) QuantErr: 11.77757 batch_time=0.70372
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 9.17578 (QuantReg: 11.56710) QuantErr: 11.56710 batch_time=0.65764
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 8.58636 (QuantReg: 12.36001) QuantErr: 12.36001 batch_time=0.68548
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 9.92561 (QuantReg: 12.31571) QuantErr: 12.31571 batch_time=0.76346
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 8.47984 (QuantReg: 12.09343) QuantErr: 12.09343 batch_time=0.63312
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 8.69728 (QuantReg: 11.90518) QuantErr: 11.90518 batch_time=0.65957
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 7.91684 (QuantReg: 11.90867) QuantErr: 11.90867 batch_time=0.65846
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 8.21842 (QuantReg: 11.92921) QuantErr: 11.92921 batch_time=0.65721
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 8.26093 (QuantReg: 12.30556) QuantErr: 12.30556 batch_time=0.66871
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 8.58070 (QuantReg: 12.02023) QuantErr: 12.02023 batch_time=0.67366
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 8.40440 (QuantReg: 12.19776) QuantErr: 12.19776 batch_time=1.65687
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 10.53127 (QuantReg: 12.40705) QuantErr: 12.40705 batch_time=0.64976
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 8.25977 (QuantReg: 12.60664) QuantErr: 12.60664 batch_time=0.64687
Train Epoch: 4 codebook_update_time=3.76442
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch4.pth ...
Done in 4.052s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch4.pth ...
Done in 8.084s
removing stale ckpt [epoch 3] [took 0.02s]
epoch : 4
loss : 9.022286571502686
quant_reg : 12.022587814331054
quant_err : 12.022587814331054
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.452
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.61085117458011
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.553
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.16768480454966
mnt_best : 34.61085117458011
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 9.23338 (QuantReg: 11.74867) QuantErr: 11.74867 batch_time=21.17841
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 8.39725 (QuantReg: 11.36923) QuantErr: 11.36923 batch_time=0.65848
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 9.71361 (QuantReg: 11.42118) QuantErr: 11.42118 batch_time=0.66623
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 7.14012 (QuantReg: 11.66982) QuantErr: 11.66982 batch_time=0.65517
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 8.27874 (QuantReg: 11.91585) QuantErr: 11.91585 batch_time=0.65541
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 8.88265 (QuantReg: 11.98696) QuantErr: 11.98696 batch_time=0.64932
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 8.05416 (QuantReg: 11.67284) QuantErr: 11.67284 batch_time=0.65178
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 9.19829 (QuantReg: 12.32710) QuantErr: 12.32710 batch_time=0.70043
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 8.15560 (QuantReg: 11.55689) QuantErr: 11.55689 batch_time=0.65772
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 8.78342 (QuantReg: 12.17994) QuantErr: 12.17994 batch_time=0.64835
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 8.25491 (QuantReg: 11.96378) QuantErr: 11.96378 batch_time=0.69307
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 8.11652 (QuantReg: 12.18818) QuantErr: 12.18818 batch_time=0.70691
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 8.96559 (QuantReg: 11.98512) QuantErr: 11.98512 batch_time=0.66536
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 9.56629 (QuantReg: 12.41128) QuantErr: 12.41128 batch_time=1.31355
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 8.94101 (QuantReg: 12.13714) QuantErr: 12.13714 batch_time=0.67711
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 7.95875 (QuantReg: 12.50616) QuantErr: 12.50616 batch_time=0.67155
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 8.52824 (QuantReg: 12.35925) QuantErr: 12.35925 batch_time=0.65510
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 7.04833 (QuantReg: 11.84530) QuantErr: 11.84530 batch_time=0.71538
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 7.74957 (QuantReg: 12.01334) QuantErr: 12.01334 batch_time=0.64370
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 8.90256 (QuantReg: 12.34957) QuantErr: 12.34957 batch_time=0.98081
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 8.28686 (QuantReg: 12.22381) QuantErr: 12.22381 batch_time=0.67354
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 8.06692 (QuantReg: 12.52068) QuantErr: 12.52068 batch_time=0.65212
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 8.85611 (QuantReg: 12.17072) QuantErr: 12.17072 batch_time=0.65209
Train Epoch: 5 codebook_update_time=3.68768
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch5.pth ...
Done in 3.950s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch5.pth ...
Done in 7.839s
removing stale ckpt [epoch 4] [took 0.02s]
epoch : 5
loss : 8.153512775421143
quant_reg : 12.075941024780274
quant_err : 12.075941024780274
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.518
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.68957788811878
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.0965
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.894721291721204
mnt_best : 35.68957788811878
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 8.17172 (QuantReg: 12.11576) QuantErr: 12.11576 batch_time=31.06771
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 7.45406 (QuantReg: 11.68830) QuantErr: 11.68830 batch_time=0.64396
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 6.54813 (QuantReg: 12.34054) QuantErr: 12.34054 batch_time=0.64642
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 8.02212 (QuantReg: 12.09752) QuantErr: 12.09752 batch_time=0.67593
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 7.51397 (QuantReg: 11.82585) QuantErr: 11.82585 batch_time=0.67705
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 6.63882 (QuantReg: 11.92197) QuantErr: 11.92197 batch_time=0.64842
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 7.74840 (QuantReg: 11.80544) QuantErr: 11.80544 batch_time=0.65299
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 8.49214 (QuantReg: 12.32666) QuantErr: 12.32666 batch_time=0.72907
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 7.94865 (QuantReg: 12.27289) QuantErr: 12.27289 batch_time=0.65506
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 6.56864 (QuantReg: 12.42014) QuantErr: 12.42014 batch_time=0.75832
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 6.91360 (QuantReg: 12.69785) QuantErr: 12.69785 batch_time=0.65899
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 7.66723 (QuantReg: 12.08221) QuantErr: 12.08221 batch_time=0.65676
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 7.54074 (QuantReg: 12.00823) QuantErr: 12.00823 batch_time=0.65139
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 7.63752 (QuantReg: 12.33620) QuantErr: 12.33620 batch_time=2.49988
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 5.94335 (QuantReg: 12.36413) QuantErr: 12.36413 batch_time=0.66397
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 8.92804 (QuantReg: 11.79990) QuantErr: 11.79990 batch_time=0.66931
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 9.14962 (QuantReg: 11.51341) QuantErr: 11.51341 batch_time=0.65623
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 6.31114 (QuantReg: 12.51645) QuantErr: 12.51645 batch_time=0.63798
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 7.78920 (QuantReg: 12.03398) QuantErr: 12.03398 batch_time=0.66901
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 7.83864 (QuantReg: 12.15661) QuantErr: 12.15661 batch_time=0.64073
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 7.53739 (QuantReg: 12.44497) QuantErr: 12.44497 batch_time=0.64756
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 9.13879 (QuantReg: 12.29389) QuantErr: 12.29389 batch_time=0.90106
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 6.86434 (QuantReg: 12.70745) QuantErr: 12.70745 batch_time=0.67372
Train Epoch: 6 codebook_update_time=3.69056
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch6.pth ...
Done in 3.754s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch6.pth ...
Done in 7.798s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 7.637740818023682
quant_reg : 12.140974784851075
quant_err : 12.140974784851075
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.105
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.7064534877486
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.773
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.87196428449707
mnt_best : 36.7064534877486
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 6.27607 (QuantReg: 11.80010) QuantErr: 11.80010 batch_time=25.83585
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 8.16981 (QuantReg: 12.06318) QuantErr: 12.06318 batch_time=0.64643
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 6.42907 (QuantReg: 11.53149) QuantErr: 11.53149 batch_time=0.65922
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 6.73756 (QuantReg: 12.18402) QuantErr: 12.18402 batch_time=0.64457
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 7.90608 (QuantReg: 12.56892) QuantErr: 12.56892 batch_time=0.64905
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 8.68400 (QuantReg: 12.18457) QuantErr: 12.18457 batch_time=0.64885
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 6.82395 (QuantReg: 12.03070) QuantErr: 12.03070 batch_time=2.40251
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 6.42024 (QuantReg: 12.36347) QuantErr: 12.36347 batch_time=0.66017
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 8.04475 (QuantReg: 11.85946) QuantErr: 11.85946 batch_time=0.66114
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 5.66141 (QuantReg: 12.11873) QuantErr: 12.11873 batch_time=0.65428
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 7.05654 (QuantReg: 12.43981) QuantErr: 12.43981 batch_time=0.74770
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 7.41550 (QuantReg: 12.15632) QuantErr: 12.15632 batch_time=0.65313
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 7.33480 (QuantReg: 11.93823) QuantErr: 11.93823 batch_time=0.83260
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 8.65933 (QuantReg: 12.51482) QuantErr: 12.51482 batch_time=0.65605
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 7.62865 (QuantReg: 11.93725) QuantErr: 11.93725 batch_time=0.64244
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 6.18448 (QuantReg: 12.18502) QuantErr: 12.18502 batch_time=0.65198
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 7.24284 (QuantReg: 12.33527) QuantErr: 12.33527 batch_time=0.65348
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 8.12802 (QuantReg: 12.04543) QuantErr: 12.04543 batch_time=1.03247
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 6.65481 (QuantReg: 12.29900) QuantErr: 12.29900 batch_time=0.65476
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 8.86360 (QuantReg: 12.78018) QuantErr: 12.78018 batch_time=0.64933
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 6.84441 (QuantReg: 12.44091) QuantErr: 12.44091 batch_time=0.69250
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 7.42280 (QuantReg: 12.51799) QuantErr: 12.51799 batch_time=0.67379
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 6.49734 (QuantReg: 12.62439) QuantErr: 12.62439 batch_time=0.67395
Train Epoch: 7 codebook_update_time=3.50680
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch7.pth ...
Done in 3.999s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch7.pth ...
Done in 7.942s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 7.16471842956543
quant_reg : 12.249625999450684
quant_err : 12.249625999450684
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.254
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.64449351343088
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.75
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.208
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.414441133461686
mnt_best : 38.64449351343088
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 5.98260 (QuantReg: 12.26060) QuantErr: 12.26060 batch_time=28.52162
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 6.58390 (QuantReg: 12.09206) QuantErr: 12.09206 batch_time=0.65285
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 5.95027 (QuantReg: 12.32737) QuantErr: 12.32737 batch_time=0.65885
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 5.42573 (QuantReg: 11.61574) QuantErr: 11.61574 batch_time=0.67425
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 6.88022 (QuantReg: 12.07471) QuantErr: 12.07471 batch_time=0.66146
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 7.03395 (QuantReg: 12.11881) QuantErr: 12.11881 batch_time=0.65490
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 5.72098 (QuantReg: 12.04424) QuantErr: 12.04424 batch_time=4.71423
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 7.85007 (QuantReg: 12.54135) QuantErr: 12.54135 batch_time=0.64460
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 8.36551 (QuantReg: 12.24718) QuantErr: 12.24718 batch_time=0.65847
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 8.26829 (QuantReg: 12.26541) QuantErr: 12.26541 batch_time=0.65505
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 6.51942 (QuantReg: 12.56868) QuantErr: 12.56868 batch_time=0.67132
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 5.75685 (QuantReg: 12.28720) QuantErr: 12.28720 batch_time=0.66480
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 6.43182 (QuantReg: 12.50653) QuantErr: 12.50653 batch_time=0.64573
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 7.39522 (QuantReg: 12.34940) QuantErr: 12.34940 batch_time=0.65182
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 5.10882 (QuantReg: 12.35431) QuantErr: 12.35431 batch_time=0.68866
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 7.07553 (QuantReg: 12.38828) QuantErr: 12.38828 batch_time=0.67114
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 7.15489 (QuantReg: 12.48027) QuantErr: 12.48027 batch_time=0.64812
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 6.50436 (QuantReg: 12.33324) QuantErr: 12.33324 batch_time=0.75298
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 5.71753 (QuantReg: 12.66176) QuantErr: 12.66176 batch_time=0.64405
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 7.17936 (QuantReg: 12.47441) QuantErr: 12.47441 batch_time=0.65609
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 9.72456 (QuantReg: 12.64169) QuantErr: 12.64169 batch_time=0.67120
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 6.16525 (QuantReg: 12.33677) QuantErr: 12.33677 batch_time=0.66553
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 6.00973 (QuantReg: 12.34574) QuantErr: 12.34574 batch_time=0.65984
Train Epoch: 8 codebook_update_time=3.69977
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch8.pth ...
Done in 4.903s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch8.pth ...
Done in 9.945s
removing stale ckpt [epoch 7] [took 0.02s]
epoch : 8
loss : 6.683614223480225
quant_reg : 12.315867671966553
quant_err : 12.315867671966553
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.718
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.974421097072884
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.295
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.78826620580687
mnt_best : 38.974421097072884
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 5.34427 (QuantReg: 12.34973) QuantErr: 12.34973 batch_time=25.01956
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 7.00041 (QuantReg: 12.32269) QuantErr: 12.32269 batch_time=0.65754
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 5.69016 (QuantReg: 12.07421) QuantErr: 12.07421 batch_time=0.68276
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 6.17124 (QuantReg: 12.31033) QuantErr: 12.31033 batch_time=0.65528
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 6.16763 (QuantReg: 12.39813) QuantErr: 12.39813 batch_time=0.65845
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 5.69442 (QuantReg: 12.48754) QuantErr: 12.48754 batch_time=0.75474
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 6.11738 (QuantReg: 12.11290) QuantErr: 12.11290 batch_time=0.68110
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 5.86790 (QuantReg: 12.69373) QuantErr: 12.69373 batch_time=0.66162
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 6.37697 (QuantReg: 12.38530) QuantErr: 12.38530 batch_time=0.74088
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 7.96507 (QuantReg: 12.50813) QuantErr: 12.50813 batch_time=0.67293
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 6.19337 (QuantReg: 12.55205) QuantErr: 12.55205 batch_time=0.65606
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 6.68608 (QuantReg: 12.49326) QuantErr: 12.49326 batch_time=0.63781
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 5.67302 (QuantReg: 12.60317) QuantErr: 12.60317 batch_time=0.65554
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 5.82109 (QuantReg: 12.73837) QuantErr: 12.73837 batch_time=0.67105
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 5.00347 (QuantReg: 12.41412) QuantErr: 12.41412 batch_time=0.65288
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 6.11948 (QuantReg: 12.40290) QuantErr: 12.40290 batch_time=0.70820
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 7.45402 (QuantReg: 12.26754) QuantErr: 12.26754 batch_time=0.64997
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 5.35466 (QuantReg: 12.53783) QuantErr: 12.53783 batch_time=0.64755
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 5.01008 (QuantReg: 12.58302) QuantErr: 12.58302 batch_time=0.64768
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 6.94188 (QuantReg: 12.51703) QuantErr: 12.51703 batch_time=0.65895
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 6.29229 (QuantReg: 12.53616) QuantErr: 12.53616 batch_time=0.69374
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 7.36091 (QuantReg: 12.44353) QuantErr: 12.44353 batch_time=0.65730
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 4.98264 (QuantReg: 12.90176) QuantErr: 12.90176 batch_time=0.64805
Train Epoch: 9 codebook_update_time=3.45858
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch9.pth ...
Done in 6.943s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch9.pth ...
Done in 11.910s
removing stale ckpt [epoch 8] [took 0.02s]
epoch : 9
loss : 6.351163307189942
quant_reg : 12.365599281311034
quant_err : 12.365599281311034
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.464
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.46594939815771
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.147
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.8142327656006
mnt_best : 39.46594939815771
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 6.33391 (QuantReg: 12.11600) QuantErr: 12.11600 batch_time=24.03713
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 5.99394 (QuantReg: 12.13708) QuantErr: 12.13708 batch_time=0.70613
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 6.83966 (QuantReg: 12.47377) QuantErr: 12.47377 batch_time=0.65758
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 5.98926 (QuantReg: 12.22284) QuantErr: 12.22284 batch_time=0.65819
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 6.51337 (QuantReg: 12.11650) QuantErr: 12.11650 batch_time=0.65738
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 5.50566 (QuantReg: 12.10428) QuantErr: 12.10428 batch_time=0.65669
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 6.26864 (QuantReg: 12.77548) QuantErr: 12.77548 batch_time=0.66232
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 5.35926 (QuantReg: 12.43559) QuantErr: 12.43559 batch_time=0.64193
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 5.57994 (QuantReg: 12.82792) QuantErr: 12.82792 batch_time=0.67141
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 5.70354 (QuantReg: 12.64724) QuantErr: 12.64724 batch_time=0.65309
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 4.76061 (QuantReg: 12.39523) QuantErr: 12.39523 batch_time=0.65760
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 7.18166 (QuantReg: 12.32443) QuantErr: 12.32443 batch_time=0.65585
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 6.24542 (QuantReg: 12.11804) QuantErr: 12.11804 batch_time=0.64853
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 5.55031 (QuantReg: 12.95999) QuantErr: 12.95999 batch_time=0.64641
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 9.27712 (QuantReg: 12.32192) QuantErr: 12.32192 batch_time=0.64169
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 6.07613 (QuantReg: 12.62377) QuantErr: 12.62377 batch_time=0.65665
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 5.23094 (QuantReg: 12.39244) QuantErr: 12.39244 batch_time=0.65104
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 6.48966 (QuantReg: 12.61741) QuantErr: 12.61741 batch_time=0.66262
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 7.30266 (QuantReg: 12.65367) QuantErr: 12.65367 batch_time=0.66395
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 6.16696 (QuantReg: 12.97818) QuantErr: 12.97818 batch_time=0.67154
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 6.11419 (QuantReg: 12.65093) QuantErr: 12.65093 batch_time=0.86008
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.95063 (QuantReg: 12.77514) QuantErr: 12.77514 batch_time=0.65984
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 5.04131 (QuantReg: 12.68400) QuantErr: 12.68400 batch_time=0.63748
Train Epoch: 10 codebook_update_time=3.47626
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch10.pth ...
Done in 6.064s
removing stale ckpt [epoch 9] [took 0.02s]
epoch : 10
loss : 6.07849605178833
quant_reg : 12.50825626373291
quant_err : 12.50825626373291
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.188
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.425829087456606
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.85
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.929787242997385
mnt_best : 39.46594939815771
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 6.42983 (QuantReg: 12.33393) QuantErr: 12.33393 batch_time=24.84880
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 4.96707 (QuantReg: 12.41557) QuantErr: 12.41557 batch_time=0.65416
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 6.77231 (QuantReg: 12.56819) QuantErr: 12.56819 batch_time=0.69738
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 5.50104 (QuantReg: 12.41241) QuantErr: 12.41241 batch_time=0.67230
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 6.45575 (QuantReg: 12.72278) QuantErr: 12.72278 batch_time=0.68199
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 5.39125 (QuantReg: 12.41873) QuantErr: 12.41873 batch_time=0.65791
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 4.83487 (QuantReg: 12.48208) QuantErr: 12.48208 batch_time=0.65305
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 6.76374 (QuantReg: 12.82609) QuantErr: 12.82609 batch_time=0.66456
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 6.74040 (QuantReg: 12.51501) QuantErr: 12.51501 batch_time=0.65615
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 6.38731 (QuantReg: 12.47728) QuantErr: 12.47728 batch_time=0.65006
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 6.37394 (QuantReg: 12.18820) QuantErr: 12.18820 batch_time=0.64005
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 5.90456 (QuantReg: 12.14911) QuantErr: 12.14911 batch_time=0.66325
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 5.44603 (QuantReg: 12.44290) QuantErr: 12.44290 batch_time=0.71122
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 5.91901 (QuantReg: 12.81184) QuantErr: 12.81184 batch_time=0.66400
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 4.41471 (QuantReg: 12.39616) QuantErr: 12.39616 batch_time=0.67380
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 5.76804 (QuantReg: 12.98603) QuantErr: 12.98603 batch_time=0.69514
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 5.86621 (QuantReg: 12.37442) QuantErr: 12.37442 batch_time=0.68968
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 6.27931 (QuantReg: 12.29980) QuantErr: 12.29980 batch_time=0.64605
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 7.04296 (QuantReg: 12.43708) QuantErr: 12.43708 batch_time=0.67577
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 5.25727 (QuantReg: 12.67173) QuantErr: 12.67173 batch_time=0.63977
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 6.50186 (QuantReg: 12.50514) QuantErr: 12.50514 batch_time=0.96012
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 4.94398 (QuantReg: 12.67206) QuantErr: 12.67206 batch_time=0.71623
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 5.36819 (QuantReg: 12.65341) QuantErr: 12.65341 batch_time=0.65239
Train Epoch: 11 codebook_update_time=3.37729
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch11.pth ...
Done in 5.924s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch11.pth ...
Done in 12.511s
removing stale ckpt [epoch 10] [took 0.02s]
epoch : 11
loss : 5.799474544525147
quant_reg : 12.549648334503173
quant_err : 12.549648334503173
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.054
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.570519477662764
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.0695
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.38813820306814
mnt_best : 41.570519477662764
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 6.81602 (QuantReg: 12.50939) QuantErr: 12.50939 batch_time=26.52352
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 5.77822 (QuantReg: 12.22094) QuantErr: 12.22094 batch_time=0.64545
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 4.86356 (QuantReg: 12.55579) QuantErr: 12.55579 batch_time=0.65385
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 4.66449 (QuantReg: 12.73215) QuantErr: 12.73215 batch_time=0.63919
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 5.01073 (QuantReg: 12.29702) QuantErr: 12.29702 batch_time=0.65325
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 4.83860 (QuantReg: 12.52416) QuantErr: 12.52416 batch_time=0.65103
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 7.50851 (QuantReg: 12.28198) QuantErr: 12.28198 batch_time=0.70480
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 6.09241 (QuantReg: 12.40140) QuantErr: 12.40140 batch_time=0.63808
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 5.80903 (QuantReg: 12.42025) QuantErr: 12.42025 batch_time=0.64935
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 6.72630 (QuantReg: 12.52447) QuantErr: 12.52447 batch_time=0.64112
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 5.91673 (QuantReg: 12.58456) QuantErr: 12.58456 batch_time=1.84887
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 4.85251 (QuantReg: 12.78848) QuantErr: 12.78848 batch_time=0.66813
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 6.70119 (QuantReg: 12.25669) QuantErr: 12.25669 batch_time=0.65863
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 5.18871 (QuantReg: 12.50491) QuantErr: 12.50491 batch_time=3.13314
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 4.88954 (QuantReg: 12.80060) QuantErr: 12.80060 batch_time=0.70633
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 4.74148 (QuantReg: 12.92975) QuantErr: 12.92975 batch_time=0.66604
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 6.23635 (QuantReg: 12.30837) QuantErr: 12.30837 batch_time=0.64984
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 5.48929 (QuantReg: 12.70244) QuantErr: 12.70244 batch_time=0.64685
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 5.92724 (QuantReg: 12.73348) QuantErr: 12.73348 batch_time=1.85866
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 5.80061 (QuantReg: 13.07209) QuantErr: 13.07209 batch_time=0.65552
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 6.23196 (QuantReg: 12.54102) QuantErr: 12.54102 batch_time=0.67459
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 6.55908 (QuantReg: 12.72670) QuantErr: 12.72670 batch_time=0.63861
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 5.21288 (QuantReg: 12.52955) QuantErr: 12.52955 batch_time=0.67915
Train Epoch: 12 codebook_update_time=3.53691
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch12.pth ...
Done in 5.046s
removing stale ckpt [epoch 11] [took 0.14s]
epoch : 12
loss : 5.530421800613404
quant_reg : 12.567639335632323
quant_err : 12.567639335632323
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.89
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.56131064491869
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.211
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.66643032043716
mnt_best : 41.570519477662764
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 5.23040 (QuantReg: 12.31836) QuantErr: 12.31836 batch_time=22.85574
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 6.00412 (QuantReg: 12.30492) QuantErr: 12.30492 batch_time=0.66612
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 5.59727 (QuantReg: 12.61330) QuantErr: 12.61330 batch_time=0.64841
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 5.60109 (QuantReg: 12.32106) QuantErr: 12.32106 batch_time=0.69133
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 5.12091 (QuantReg: 12.65396) QuantErr: 12.65396 batch_time=0.65546
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 5.24139 (QuantReg: 12.61739) QuantErr: 12.61739 batch_time=0.78812
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 4.31649 (QuantReg: 12.51854) QuantErr: 12.51854 batch_time=0.67755
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 4.50686 (QuantReg: 12.94194) QuantErr: 12.94194 batch_time=0.65975
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 4.81054 (QuantReg: 12.78113) QuantErr: 12.78113 batch_time=0.65883
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 4.90136 (QuantReg: 12.71656) QuantErr: 12.71656 batch_time=0.65358
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 6.01907 (QuantReg: 12.45847) QuantErr: 12.45847 batch_time=0.65480
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 6.52129 (QuantReg: 12.33964) QuantErr: 12.33964 batch_time=0.65450
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 6.98154 (QuantReg: 12.76953) QuantErr: 12.76953 batch_time=0.65386
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 5.88274 (QuantReg: 12.50937) QuantErr: 12.50937 batch_time=1.12336
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.97767 (QuantReg: 13.00821) QuantErr: 13.00821 batch_time=0.66891
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 4.80189 (QuantReg: 12.99442) QuantErr: 12.99442 batch_time=0.66090
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 6.29178 (QuantReg: 12.74926) QuantErr: 12.74926 batch_time=0.65644
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 5.19666 (QuantReg: 12.71119) QuantErr: 12.71119 batch_time=0.64727
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 5.22152 (QuantReg: 12.63041) QuantErr: 12.63041 batch_time=0.64321
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 3.98836 (QuantReg: 12.79299) QuantErr: 12.79299 batch_time=0.64549
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 5.16320 (QuantReg: 12.53752) QuantErr: 12.53752 batch_time=0.64792
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 5.32895 (QuantReg: 12.70258) QuantErr: 12.70258 batch_time=0.66696
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 5.30105 (QuantReg: 13.00922) QuantErr: 13.00922 batch_time=0.64711
Train Epoch: 13 codebook_update_time=3.49271
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch13.pth ...
Done in 6.174s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch13.pth ...
Done in 11.001s
removing stale ckpt [epoch 12] [took 0.08s]
epoch : 13
loss : 5.347531148910522
quant_reg : 12.630577388763427
quant_err : 12.630577388763427
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.19
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.65264537999151
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.0135
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.60099695589051
mnt_best : 41.65264537999151
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 4.70201 (QuantReg: 12.63879) QuantErr: 12.63879 batch_time=22.75389
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 5.30919 (QuantReg: 12.36487) QuantErr: 12.36487 batch_time=0.66319
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 5.79933 (QuantReg: 12.29061) QuantErr: 12.29061 batch_time=0.66721
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 4.78775 (QuantReg: 12.46800) QuantErr: 12.46800 batch_time=0.65575
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 5.25464 (QuantReg: 12.58574) QuantErr: 12.58574 batch_time=0.65532
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 5.66950 (QuantReg: 12.50049) QuantErr: 12.50049 batch_time=0.65398
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 4.85198 (QuantReg: 12.78178) QuantErr: 12.78178 batch_time=2.94191
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 5.72514 (QuantReg: 12.44354) QuantErr: 12.44354 batch_time=0.65189
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 4.38246 (QuantReg: 12.97515) QuantErr: 12.97515 batch_time=0.69408
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 5.79058 (QuantReg: 12.48508) QuantErr: 12.48508 batch_time=0.65085
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 4.79178 (QuantReg: 12.42693) QuantErr: 12.42693 batch_time=0.68279
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 5.01429 (QuantReg: 12.50164) QuantErr: 12.50164 batch_time=0.65705
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 4.56000 (QuantReg: 12.74114) QuantErr: 12.74114 batch_time=0.67038
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 5.53823 (QuantReg: 12.36052) QuantErr: 12.36052 batch_time=0.64749
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 4.28752 (QuantReg: 12.67890) QuantErr: 12.67890 batch_time=0.66224
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 5.19646 (QuantReg: 12.20059) QuantErr: 12.20059 batch_time=0.70615
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 3.76350 (QuantReg: 12.60202) QuantErr: 12.60202 batch_time=0.65935
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 5.33493 (QuantReg: 12.47123) QuantErr: 12.47123 batch_time=0.64978
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 4.48306 (QuantReg: 12.68958) QuantErr: 12.68958 batch_time=1.40248
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 4.46323 (QuantReg: 12.80328) QuantErr: 12.80328 batch_time=0.66960
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 5.92631 (QuantReg: 12.17094) QuantErr: 12.17094 batch_time=2.29560
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 5.35498 (QuantReg: 12.58776) QuantErr: 12.58776 batch_time=0.65005
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 5.83346 (QuantReg: 13.00584) QuantErr: 13.00584 batch_time=1.20643
Train Epoch: 14 codebook_update_time=4.32351
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch14.pth ...
Done in 4.200s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch14.pth ...
Done in 9.181s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 5.239548217773438
quant_reg : 12.625655864715576
quant_err : 12.625655864715576
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.364
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.643787952542084
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 22.2145
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.34047323444978
mnt_best : 42.643787952542084
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 4.93917 (QuantReg: 12.70422) QuantErr: 12.70422 batch_time=32.07491
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 5.87304 (QuantReg: 12.62104) QuantErr: 12.62104 batch_time=0.65470
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 4.37655 (QuantReg: 12.90540) QuantErr: 12.90540 batch_time=0.65436
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 4.54113 (QuantReg: 12.58976) QuantErr: 12.58976 batch_time=1.18800
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 4.65605 (QuantReg: 12.56685) QuantErr: 12.56685 batch_time=0.65111
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 5.25071 (QuantReg: 13.04259) QuantErr: 13.04259 batch_time=0.67485
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 4.89034 (QuantReg: 12.77603) QuantErr: 12.77603 batch_time=0.66481
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 4.76442 (QuantReg: 12.71393) QuantErr: 12.71393 batch_time=0.65865
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 4.86903 (QuantReg: 12.76779) QuantErr: 12.76779 batch_time=0.65314
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 4.20178 (QuantReg: 12.75378) QuantErr: 12.75378 batch_time=0.72748
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 5.45140 (QuantReg: 12.73589) QuantErr: 12.73589 batch_time=0.66594
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 5.31833 (QuantReg: 12.58272) QuantErr: 12.58272 batch_time=0.71236
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 4.30191 (QuantReg: 13.16737) QuantErr: 13.16737 batch_time=0.72523
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 4.74535 (QuantReg: 12.87776) QuantErr: 12.87776 batch_time=0.65651
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 5.04069 (QuantReg: 12.79187) QuantErr: 12.79187 batch_time=0.65757
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 4.24194 (QuantReg: 12.91020) QuantErr: 12.91020 batch_time=0.67611
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 4.81840 (QuantReg: 12.73679) QuantErr: 12.73679 batch_time=0.67317
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 4.13336 (QuantReg: 13.13447) QuantErr: 13.13447 batch_time=0.66092
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 5.43324 (QuantReg: 12.93989) QuantErr: 12.93989 batch_time=0.67553
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 3.74256 (QuantReg: 12.52510) QuantErr: 12.52510 batch_time=0.66184
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 5.70015 (QuantReg: 12.88151) QuantErr: 12.88151 batch_time=0.71855
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 5.95117 (QuantReg: 13.04764) QuantErr: 13.04764 batch_time=0.68341
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 4.97877 (QuantReg: 12.92482) QuantErr: 12.92482 batch_time=0.66184
Train Epoch: 15 codebook_update_time=3.65035
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch15.pth ...
Done in 5.802s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 4.999465203285217
quant_reg : 12.733496929168702
quant_err : 12.733496929168702
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.471
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.06112829767913
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.187
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.69501154733098
mnt_best : 42.643787952542084
not_improved_count: 1
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 5.95925 (QuantReg: 11.92506) QuantErr: 11.92506 batch_time=24.62219
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 4.77678 (QuantReg: 12.54807) QuantErr: 12.54807 batch_time=0.63279
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 4.86615 (QuantReg: 12.63967) QuantErr: 12.63967 batch_time=0.71309
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 5.30456 (QuantReg: 12.54910) QuantErr: 12.54910 batch_time=0.66840
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 4.89005 (QuantReg: 12.55607) QuantErr: 12.55607 batch_time=0.64314
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 4.88139 (QuantReg: 12.87098) QuantErr: 12.87098 batch_time=0.65608
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 5.96752 (QuantReg: 12.57647) QuantErr: 12.57647 batch_time=0.66156
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 4.46046 (QuantReg: 12.59565) QuantErr: 12.59565 batch_time=0.64510
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 5.00093 (QuantReg: 12.67394) QuantErr: 12.67394 batch_time=0.64358
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 6.24024 (QuantReg: 12.76975) QuantErr: 12.76975 batch_time=0.64512
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 4.53440 (QuantReg: 12.66610) QuantErr: 12.66610 batch_time=0.68919
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 5.21388 (QuantReg: 12.63367) QuantErr: 12.63367 batch_time=0.73989
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 4.43613 (QuantReg: 12.70297) QuantErr: 12.70297 batch_time=0.86219
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 4.14217 (QuantReg: 13.43477) QuantErr: 13.43477 batch_time=0.66802
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 5.72413 (QuantReg: 12.66120) QuantErr: 12.66120 batch_time=0.65325
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 5.73750 (QuantReg: 12.77211) QuantErr: 12.77211 batch_time=0.63485
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 4.00712 (QuantReg: 12.53422) QuantErr: 12.53422 batch_time=0.64653
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 5.56504 (QuantReg: 12.62402) QuantErr: 12.62402 batch_time=0.65506
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 4.66562 (QuantReg: 12.55698) QuantErr: 12.55698 batch_time=0.70373
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 4.56746 (QuantReg: 12.89698) QuantErr: 12.89698 batch_time=0.64675
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 5.65860 (QuantReg: 12.71257) QuantErr: 12.71257 batch_time=0.64893
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 4.00989 (QuantReg: 12.94517) QuantErr: 12.94517 batch_time=0.64392
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 3.77078 (QuantReg: 12.74943) QuantErr: 12.74943 batch_time=0.67440
Train Epoch: 16 codebook_update_time=3.64412
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch16.pth ...
Done in 4.354s
removing stale ckpt [epoch 15] [took 0.05s]
epoch : 16
loss : 4.839705881118775
quant_reg : 12.757451755523682
quant_err : 12.757451755523682
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.337
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.06634469087434
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.911
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.31583592234888
mnt_best : 42.643787952542084
not_improved_count: 2
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 5.60868 (QuantReg: 12.69976) QuantErr: 12.69976 batch_time=22.93370
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 4.21165 (QuantReg: 12.85749) QuantErr: 12.85749 batch_time=0.64670
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 4.55346 (QuantReg: 12.32388) QuantErr: 12.32388 batch_time=0.70471
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 5.58209 (QuantReg: 12.95356) QuantErr: 12.95356 batch_time=0.68292
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 4.67390 (QuantReg: 12.84426) QuantErr: 12.84426 batch_time=0.66032
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 4.27955 (QuantReg: 12.82731) QuantErr: 12.82731 batch_time=0.67503
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 5.53442 (QuantReg: 12.48807) QuantErr: 12.48807 batch_time=0.67624
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 4.52048 (QuantReg: 12.89100) QuantErr: 12.89100 batch_time=0.65707
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 5.55507 (QuantReg: 12.80412) QuantErr: 12.80412 batch_time=0.66598
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 3.93442 (QuantReg: 12.90441) QuantErr: 12.90441 batch_time=0.66229
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 5.13407 (QuantReg: 12.80118) QuantErr: 12.80118 batch_time=0.65787
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 5.40444 (QuantReg: 12.75002) QuantErr: 12.75002 batch_time=0.69667
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 5.01357 (QuantReg: 12.89533) QuantErr: 12.89533 batch_time=1.41398
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 3.43544 (QuantReg: 13.03856) QuantErr: 13.03856 batch_time=0.64945
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 5.04847 (QuantReg: 12.94853) QuantErr: 12.94853 batch_time=0.70113
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 3.82271 (QuantReg: 12.81199) QuantErr: 12.81199 batch_time=0.72068
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 4.49304 (QuantReg: 12.87594) QuantErr: 12.87594 batch_time=0.64065
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 5.58155 (QuantReg: 12.69623) QuantErr: 12.69623 batch_time=0.73358
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 3.56255 (QuantReg: 13.08766) QuantErr: 13.08766 batch_time=0.65188
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 4.59618 (QuantReg: 12.71552) QuantErr: 12.71552 batch_time=1.10627
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 5.03082 (QuantReg: 13.07200) QuantErr: 13.07200 batch_time=0.96427
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 4.35814 (QuantReg: 12.68474) QuantErr: 12.68474 batch_time=0.65581
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 5.25673 (QuantReg: 12.69268) QuantErr: 12.69268 batch_time=0.65623
Train Epoch: 17 codebook_update_time=3.55337
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch17.pth ...
Done in 5.463s
removing stale ckpt [epoch 16] [took 0.14s]
epoch : 17
loss : 4.74646240234375
quant_reg : 12.798102115631103
quant_err : 12.798102115631103
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.569
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.3524089257731
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.26
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.19828025275583
mnt_best : 42.643787952542084
not_improved_count: 3
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 4.35238 (QuantReg: 12.59269) QuantErr: 12.59269 batch_time=24.26385
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 5.15696 (QuantReg: 12.80813) QuantErr: 12.80813 batch_time=0.65021
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 5.73286 (QuantReg: 12.69025) QuantErr: 12.69025 batch_time=0.66265
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 5.18539 (QuantReg: 12.45813) QuantErr: 12.45813 batch_time=0.69408
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 3.10930 (QuantReg: 12.85783) QuantErr: 12.85783 batch_time=0.67303
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 6.09670 (QuantReg: 12.66057) QuantErr: 12.66057 batch_time=0.67549
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 3.71727 (QuantReg: 12.99767) QuantErr: 12.99767 batch_time=1.27264
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 4.22922 (QuantReg: 12.92379) QuantErr: 12.92379 batch_time=0.67250
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 3.75361 (QuantReg: 12.86456) QuantErr: 12.86456 batch_time=0.65287
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 4.28573 (QuantReg: 12.47279) QuantErr: 12.47279 batch_time=0.66941
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 4.14907 (QuantReg: 12.92087) QuantErr: 12.92087 batch_time=0.65286
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 3.78640 (QuantReg: 12.94731) QuantErr: 12.94731 batch_time=0.68491
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 5.33564 (QuantReg: 12.58181) QuantErr: 12.58181 batch_time=0.66663
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 4.61721 (QuantReg: 12.56091) QuantErr: 12.56091 batch_time=0.66598
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 4.53158 (QuantReg: 12.72247) QuantErr: 12.72247 batch_time=0.84845
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 5.39172 (QuantReg: 13.21880) QuantErr: 13.21880 batch_time=0.73820
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 3.95905 (QuantReg: 13.03467) QuantErr: 13.03467 batch_time=0.73429
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 4.12700 (QuantReg: 12.81861) QuantErr: 12.81861 batch_time=0.67929
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 4.85638 (QuantReg: 12.90088) QuantErr: 12.90088 batch_time=0.69357
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 4.05230 (QuantReg: 13.03630) QuantErr: 13.03630 batch_time=0.66526
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 4.69404 (QuantReg: 12.33483) QuantErr: 12.33483 batch_time=0.66302
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 4.46491 (QuantReg: 13.00667) QuantErr: 13.00667 batch_time=0.65572
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 4.62757 (QuantReg: 12.66239) QuantErr: 12.66239 batch_time=0.67663
Train Epoch: 18 codebook_update_time=3.38188
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch18.pth ...
Done in 4.466s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 4.568733917236328
quant_reg : 12.795279041290284
quant_err : 12.795279041290284
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.954
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.399450420074785
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 22.798
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.4747646267132
mnt_best : 42.643787952542084
not_improved_count: 4
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 3.81795 (QuantReg: 12.78915) QuantErr: 12.78915 batch_time=25.48625
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 4.57512 (QuantReg: 12.65680) QuantErr: 12.65680 batch_time=0.67505
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 4.65180 (QuantReg: 12.64639) QuantErr: 12.64639 batch_time=0.69035
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 4.05678 (QuantReg: 12.72013) QuantErr: 12.72013 batch_time=0.64874
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 4.42007 (QuantReg: 12.57564) QuantErr: 12.57564 batch_time=0.66831
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 4.61253 (QuantReg: 12.87971) QuantErr: 12.87971 batch_time=0.81848
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 4.79054 (QuantReg: 12.46641) QuantErr: 12.46641 batch_time=0.68555
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 4.70159 (QuantReg: 12.57285) QuantErr: 12.57285 batch_time=0.99833
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 4.50808 (QuantReg: 12.67908) QuantErr: 12.67908 batch_time=0.66224
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 4.94567 (QuantReg: 12.91755) QuantErr: 12.91755 batch_time=0.66340
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 3.83953 (QuantReg: 12.75270) QuantErr: 12.75270 batch_time=0.64737
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 4.10428 (QuantReg: 12.98309) QuantErr: 12.98309 batch_time=0.67411
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 4.22687 (QuantReg: 12.64682) QuantErr: 12.64682 batch_time=0.90076
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 4.90697 (QuantReg: 12.72045) QuantErr: 12.72045 batch_time=0.67476
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 3.77074 (QuantReg: 13.00019) QuantErr: 13.00019 batch_time=0.65565
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 4.95605 (QuantReg: 12.89621) QuantErr: 12.89621 batch_time=0.65694
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 3.94718 (QuantReg: 12.65990) QuantErr: 12.65990 batch_time=0.65146
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 5.78308 (QuantReg: 12.41548) QuantErr: 12.41548 batch_time=0.66579
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 4.64186 (QuantReg: 12.86397) QuantErr: 12.86397 batch_time=3.04142
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 4.86030 (QuantReg: 13.16725) QuantErr: 13.16725 batch_time=0.68297
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 4.62013 (QuantReg: 13.18217) QuantErr: 13.18217 batch_time=0.75560
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 5.17764 (QuantReg: 12.88044) QuantErr: 12.88044 batch_time=0.70551
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 4.73986 (QuantReg: 12.98284) QuantErr: 12.98284 batch_time=0.66909
Train Epoch: 19 codebook_update_time=3.43478
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch19.pth ...
Done in 4.112s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L15/checkpoint-epoch19.pth ...
Done in 8.372s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 4.532879644393921
quant_reg : 12.822084655761719
quant_err : 12.822084655761719
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.888