-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full_t0.03.txt
3305 lines (3305 loc) · 235 KB
/
HCQ_MSRVTT_full_t0.03.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 1072.1127078533173 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 50.208431243896484 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 366.5455513000488 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 60.80437231063843 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch0.pth ...
Done in 1.900s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch0.pth ...
Done in 3.262s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
MSRVTT_full_val/t2v_metrics/MedR: 252.0
MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
MSRVTT_full_val/v2t_metrics/MedR: 243.0
MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
MSRVTT_full_test/t2v_metrics/MedR: 1515.0
MSRVTT_full_test/t2v_metrics/MeanR: 1498.5565217391304
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1471.5
MSRVTT_full_test/v2t_metrics/MeanR: 1495.3264214046824
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
mnt_best : 0.12154652794863813
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 10.07163 (QuantReg: 22.44304) QuantErr: 22.44304 batch_time=36.88526
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.14985 (QuantReg: 22.58221) QuantErr: 22.58221 batch_time=0.51423
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.19915 (QuantReg: 22.68928) QuantErr: 22.68928 batch_time=0.55835
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.95077 (QuantReg: 22.65341) QuantErr: 22.65341 batch_time=0.51208
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.51322 (QuantReg: 22.56908) QuantErr: 22.56908 batch_time=0.51408
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.10260 (QuantReg: 22.60261) QuantErr: 22.60261 batch_time=0.86390
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.08555 (QuantReg: 22.56160) QuantErr: 22.56160 batch_time=0.51654
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.88390 (QuantReg: 22.52276) QuantErr: 22.52276 batch_time=0.51298
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.20239 (QuantReg: 22.54940) QuantErr: 22.54940 batch_time=0.51478
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.58778 (QuantReg: 22.57510) QuantErr: 22.57510 batch_time=0.51255
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.10567 (QuantReg: 22.61690) QuantErr: 22.61690 batch_time=0.51746
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.75096 (QuantReg: 22.64125) QuantErr: 22.64125 batch_time=0.51815
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.73739 (QuantReg: 22.60570) QuantErr: 22.60570 batch_time=0.55566
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.63561 (QuantReg: 22.61405) QuantErr: 22.61405 batch_time=0.70746
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.76398 (QuantReg: 22.57742) QuantErr: 22.57742 batch_time=0.51160
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 5.03232 (QuantReg: 22.60584) QuantErr: 22.60584 batch_time=0.53423
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.72348 (QuantReg: 22.60962) QuantErr: 22.60962 batch_time=0.52140
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.36171 (QuantReg: 22.63739) QuantErr: 22.63739 batch_time=0.53385
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.34856 (QuantReg: 22.66554) QuantErr: 22.66554 batch_time=0.51857
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.08942 (QuantReg: 22.61238) QuantErr: 22.61238 batch_time=0.53075
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.53708 (QuantReg: 22.65799) QuantErr: 22.65799 batch_time=0.51599
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.18667 (QuantReg: 22.66867) QuantErr: 22.66867 batch_time=1.99536
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.89483 (QuantReg: 22.63844) QuantErr: 22.63844 batch_time=0.52573
Train Epoch: 1 codebook_update_time=1.83743
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch1.pth ...
Done in 4.152s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch1.pth ...
Done in 8.245s
epoch : 1
loss : 5.322831031799317
quant_reg : 22.60752113342285
quant_err : 22.60752113342285
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 18.91348088531187
MSRVTT_full_val/t2v_metrics/R5: 51.50905432595574
MSRVTT_full_val/t2v_metrics/R10: 65.79476861167002
MSRVTT_full_val/t2v_metrics/R50: 95.57344064386318
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 12.629778672032193
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 40.020465433364784
MSRVTT_full_val/v2t_metrics/R1: 22.334004024144868
MSRVTT_full_val/v2t_metrics/R5: 55.1307847082495
MSRVTT_full_val/v2t_metrics/R10: 72.63581488933602
MSRVTT_full_val/v2t_metrics/R50: 94.96981891348088
MSRVTT_full_val/v2t_metrics/MedR: 5.0
MSRVTT_full_val/v2t_metrics/MeanR: 11.645875251509054
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 44.720212557156664
MSRVTT_full_test/t2v_metrics/R1: 6.120401337792642
MSRVTT_full_test/t2v_metrics/R5: 20.434782608695652
MSRVTT_full_test/t2v_metrics/R10: 31.77257525083612
MSRVTT_full_test/t2v_metrics/R50: 65.91973244147157
MSRVTT_full_test/t2v_metrics/MedR: 24.0
MSRVTT_full_test/t2v_metrics/MeanR: 68.62976588628763
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 15.839231758877157
MSRVTT_full_test/v2t_metrics/R1: 5.351170568561873
MSRVTT_full_test/v2t_metrics/R5: 22.073578595317727
MSRVTT_full_test/v2t_metrics/R10: 34.88294314381271
MSRVTT_full_test/v2t_metrics/R50: 70.90301003344482
MSRVTT_full_test/v2t_metrics/MedR: 20.0
MSRVTT_full_test/v2t_metrics/MeanR: 63.51270903010033
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.0316499149504
mnt_best : 15.839231758877157
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 3.37613 (QuantReg: 9.21660) QuantErr: 9.21660 batch_time=37.14332
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.89424 (QuantReg: 9.48522) QuantErr: 9.48522 batch_time=0.53044
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 3.88970 (QuantReg: 9.46349) QuantErr: 9.46349 batch_time=0.86264
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.71609 (QuantReg: 9.71268) QuantErr: 9.71268 batch_time=0.50104
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.64000 (QuantReg: 10.04311) QuantErr: 10.04311 batch_time=0.51790
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.68747 (QuantReg: 9.95534) QuantErr: 9.95534 batch_time=0.51569
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.90090 (QuantReg: 10.17908) QuantErr: 10.17908 batch_time=7.04421
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.43839 (QuantReg: 10.55268) QuantErr: 10.55268 batch_time=0.52234
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.05509 (QuantReg: 10.58119) QuantErr: 10.58119 batch_time=0.51599
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.39653 (QuantReg: 10.87788) QuantErr: 10.87788 batch_time=0.51841
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.46991 (QuantReg: 10.81184) QuantErr: 10.81184 batch_time=0.50610
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.20299 (QuantReg: 11.34046) QuantErr: 11.34046 batch_time=0.55514
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.15164 (QuantReg: 11.01907) QuantErr: 11.01907 batch_time=0.51325
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.39487 (QuantReg: 11.49088) QuantErr: 11.49088 batch_time=0.50762
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.24435 (QuantReg: 11.44485) QuantErr: 11.44485 batch_time=0.51529
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.13060 (QuantReg: 12.09984) QuantErr: 12.09984 batch_time=0.54018
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.38170 (QuantReg: 12.00167) QuantErr: 12.00167 batch_time=0.50626
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.45304 (QuantReg: 11.91750) QuantErr: 11.91750 batch_time=0.51094
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.06302 (QuantReg: 11.84294) QuantErr: 11.84294 batch_time=0.51343
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 2.93531 (QuantReg: 12.10814) QuantErr: 12.10814 batch_time=0.53597
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 2.65595 (QuantReg: 12.59027) QuantErr: 12.59027 batch_time=0.51031
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.84863 (QuantReg: 12.53828) QuantErr: 12.53828 batch_time=0.51763
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.35970 (QuantReg: 12.73001) QuantErr: 12.73001 batch_time=0.85995
Train Epoch: 2 codebook_update_time=1.77051
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch2.pth ...
Done in 4.107s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch2.pth ...
Done in 8.567s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 3.4692099075317384
quant_reg : 11.069162174224854
quant_err : 11.069162174224854
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 21.730382293762574
MSRVTT_full_val/t2v_metrics/R5: 60.160965794768615
MSRVTT_full_val/t2v_metrics/R10: 75.0503018108652
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.263581488933601
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.1223616140089
MSRVTT_full_val/v2t_metrics/R1: 27.766599597585515
MSRVTT_full_val/v2t_metrics/R5: 66.80080482897384
MSRVTT_full_val/v2t_metrics/R10: 79.67806841046277
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.195171026156942
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 52.87061918640973
MSRVTT_full_test/t2v_metrics/R1: 8.22742474916388
MSRVTT_full_test/t2v_metrics/R5: 25.183946488294314
MSRVTT_full_test/t2v_metrics/R10: 38.12709030100334
MSRVTT_full_test/t2v_metrics/R50: 72.5752508361204
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 57.90133779264214
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.9162295504241
MSRVTT_full_test/v2t_metrics/R1: 9.297658862876254
MSRVTT_full_test/v2t_metrics/R5: 29.130434782608695
MSRVTT_full_test/v2t_metrics/R10: 43.41137123745819
MSRVTT_full_test/v2t_metrics/R50: 78.19397993311037
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 47.21638795986622
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.739174413894624
mnt_best : 19.9162295504241
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 2.86868 (QuantReg: 10.13303) QuantErr: 10.13303 batch_time=46.18404
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.04221 (QuantReg: 10.43336) QuantErr: 10.43336 batch_time=0.50498
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.49498 (QuantReg: 11.00048) QuantErr: 11.00048 batch_time=0.51886
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.54167 (QuantReg: 10.58392) QuantErr: 10.58392 batch_time=0.52454
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.21632 (QuantReg: 10.59972) QuantErr: 10.59972 batch_time=0.52256
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.55063 (QuantReg: 10.78984) QuantErr: 10.78984 batch_time=0.63004
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.74329 (QuantReg: 10.97769) QuantErr: 10.97769 batch_time=1.12899
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.01732 (QuantReg: 11.15390) QuantErr: 11.15390 batch_time=0.50723
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.05169 (QuantReg: 11.24109) QuantErr: 11.24109 batch_time=0.51274
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.64742 (QuantReg: 10.80637) QuantErr: 10.80637 batch_time=0.51305
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.90437 (QuantReg: 10.91076) QuantErr: 10.91076 batch_time=0.50699
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 2.46468 (QuantReg: 11.28045) QuantErr: 11.28045 batch_time=0.51856
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 2.98496 (QuantReg: 11.45219) QuantErr: 11.45219 batch_time=0.52044
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 2.73101 (QuantReg: 11.75332) QuantErr: 11.75332 batch_time=0.51282
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.56095 (QuantReg: 11.12946) QuantErr: 11.12946 batch_time=0.52203
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.20741 (QuantReg: 12.26190) QuantErr: 12.26190 batch_time=0.52147
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.43232 (QuantReg: 11.77834) QuantErr: 11.77834 batch_time=0.50901
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.10274 (QuantReg: 11.67128) QuantErr: 11.67128 batch_time=0.64323
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.52749 (QuantReg: 11.38377) QuantErr: 11.38377 batch_time=0.50866
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.80325 (QuantReg: 12.06316) QuantErr: 12.06316 batch_time=0.50648
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 2.48682 (QuantReg: 12.17610) QuantErr: 12.17610 batch_time=0.59237
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.28751 (QuantReg: 12.50578) QuantErr: 12.50578 batch_time=0.51313
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.57791 (QuantReg: 12.23411) QuantErr: 12.23411 batch_time=0.50834
Train Epoch: 3 codebook_update_time=1.80932
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch3.pth ...
Done in 4.212s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch3.pth ...
Done in 8.730s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 2.839450550079346
quant_reg : 11.386959846496582
quant_err : 11.386959846496582
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 26.358148893360163
MSRVTT_full_val/t2v_metrics/R5: 61.77062374245473
MSRVTT_full_val/t2v_metrics/R10: 76.45875251509054
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.941649899396378
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 49.9315100665697
MSRVTT_full_val/v2t_metrics/R1: 28.772635814889338
MSRVTT_full_val/v2t_metrics/R5: 67.40442655935614
MSRVTT_full_val/v2t_metrics/R10: 81.69014084507042
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.885311871227364
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.11021222699967
MSRVTT_full_test/t2v_metrics/R1: 8.22742474916388
MSRVTT_full_test/t2v_metrics/R5: 27.525083612040135
MSRVTT_full_test/t2v_metrics/R10: 40.36789297658863
MSRVTT_full_test/t2v_metrics/R50: 74.84949832775919
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 52.866889632107025
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.909463150973767
MSRVTT_full_test/v2t_metrics/R1: 10.066889632107024
MSRVTT_full_test/v2t_metrics/R5: 30.13377926421405
MSRVTT_full_test/v2t_metrics/R10: 44.91638795986622
MSRVTT_full_test/v2t_metrics/R50: 78.8628762541806
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 46.751505016722405
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.884596642799266
mnt_best : 20.909463150973767
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.05658 (QuantReg: 10.71497) QuantErr: 10.71497 batch_time=38.55935
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.63842 (QuantReg: 10.91070) QuantErr: 10.91070 batch_time=0.51291
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.02187 (QuantReg: 11.15519) QuantErr: 11.15519 batch_time=0.49759
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.29530 (QuantReg: 11.07648) QuantErr: 11.07648 batch_time=0.49855
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.61993 (QuantReg: 11.06623) QuantErr: 11.06623 batch_time=0.51269
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.89794 (QuantReg: 11.05982) QuantErr: 11.05982 batch_time=0.50358
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.90399 (QuantReg: 11.34473) QuantErr: 11.34473 batch_time=3.21297
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.75123 (QuantReg: 11.57728) QuantErr: 11.57728 batch_time=0.55654
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.52306 (QuantReg: 11.43376) QuantErr: 11.43376 batch_time=0.50019
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.42666 (QuantReg: 11.59499) QuantErr: 11.59499 batch_time=0.50455
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.69995 (QuantReg: 11.95218) QuantErr: 11.95218 batch_time=0.50754
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.66752 (QuantReg: 11.64118) QuantErr: 11.64118 batch_time=0.51486
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.40668 (QuantReg: 11.73919) QuantErr: 11.73919 batch_time=0.50509
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.57764 (QuantReg: 12.02380) QuantErr: 12.02380 batch_time=0.49960
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.80421 (QuantReg: 12.09879) QuantErr: 12.09879 batch_time=0.51952
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.84282 (QuantReg: 12.13315) QuantErr: 12.13315 batch_time=0.53699
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.89860 (QuantReg: 12.01942) QuantErr: 12.01942 batch_time=0.51599
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.60206 (QuantReg: 12.30133) QuantErr: 12.30133 batch_time=0.54410
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.65087 (QuantReg: 12.09037) QuantErr: 12.09037 batch_time=0.50307
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.30069 (QuantReg: 12.19093) QuantErr: 12.19093 batch_time=0.50533
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.30599 (QuantReg: 12.41026) QuantErr: 12.41026 batch_time=0.52402
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.50746 (QuantReg: 11.81514) QuantErr: 11.81514 batch_time=0.54827
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.31071 (QuantReg: 12.11436) QuantErr: 12.11436 batch_time=0.50537
Train Epoch: 4 codebook_update_time=1.68333
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch4.pth ...
Done in 4.938s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch4.pth ...
Done in 9.483s
removing stale ckpt [epoch 3] [took 0.09s]
epoch : 4
loss : 2.530116229534149
quant_reg : 11.727557594299316
quant_err : 11.727557594299316
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 26.156941649899398
MSRVTT_full_val/t2v_metrics/R5: 60.76458752515091
MSRVTT_full_val/t2v_metrics/R10: 75.25150905432595
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.03420523138833
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 49.27019076152064
MSRVTT_full_val/v2t_metrics/R1: 29.979879275653925
MSRVTT_full_val/v2t_metrics/R5: 69.01408450704226
MSRVTT_full_val/v2t_metrics/R10: 80.28169014084507
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.271629778672033
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.97029141612748
MSRVTT_full_test/t2v_metrics/R1: 9.130434782608695
MSRVTT_full_test/t2v_metrics/R5: 27.558528428093645
MSRVTT_full_test/t2v_metrics/R10: 40.668896321070235
MSRVTT_full_test/t2v_metrics/R50: 73.91304347826087
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 55.166220735785956
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.710506534655792
MSRVTT_full_test/v2t_metrics/R1: 10.200668896321071
MSRVTT_full_test/v2t_metrics/R5: 32.04013377926422
MSRVTT_full_test/v2t_metrics/R10: 45.919732441471574
MSRVTT_full_test/v2t_metrics/R50: 80.16722408026756
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 43.69498327759197
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.666494868438544
mnt_best : 21.710506534655792
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.53039 (QuantReg: 11.60401) QuantErr: 11.60401 batch_time=39.48403
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.66982 (QuantReg: 11.51499) QuantErr: 11.51499 batch_time=0.50382
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.68353 (QuantReg: 11.57970) QuantErr: 11.57970 batch_time=0.52426
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 1.97732 (QuantReg: 12.12161) QuantErr: 12.12161 batch_time=0.63261
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.19710 (QuantReg: 11.53810) QuantErr: 11.53810 batch_time=0.49688
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.33807 (QuantReg: 12.56094) QuantErr: 12.56094 batch_time=0.56133
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.24588 (QuantReg: 11.81902) QuantErr: 11.81902 batch_time=0.50529
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.45381 (QuantReg: 11.89145) QuantErr: 11.89145 batch_time=0.51985
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.06593 (QuantReg: 12.07801) QuantErr: 12.07801 batch_time=0.51006
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.49321 (QuantReg: 12.17547) QuantErr: 12.17547 batch_time=0.51864
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.19582 (QuantReg: 12.37942) QuantErr: 12.37942 batch_time=1.36397
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.19402 (QuantReg: 12.48748) QuantErr: 12.48748 batch_time=0.49952
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.48398 (QuantReg: 12.49574) QuantErr: 12.49574 batch_time=1.74359
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 1.64503 (QuantReg: 12.55041) QuantErr: 12.55041 batch_time=0.52035
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.18599 (QuantReg: 12.68144) QuantErr: 12.68144 batch_time=0.54054
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.19244 (QuantReg: 12.58201) QuantErr: 12.58201 batch_time=0.54983
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.01512 (QuantReg: 12.90346) QuantErr: 12.90346 batch_time=0.50612
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.45513 (QuantReg: 12.39349) QuantErr: 12.39349 batch_time=0.50819
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.11731 (QuantReg: 12.57325) QuantErr: 12.57325 batch_time=0.50675
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.11270 (QuantReg: 12.24753) QuantErr: 12.24753 batch_time=0.50474
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.18508 (QuantReg: 12.64841) QuantErr: 12.64841 batch_time=0.50912
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 1.87572 (QuantReg: 12.65288) QuantErr: 12.65288 batch_time=0.73858
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.05006 (QuantReg: 12.59836) QuantErr: 12.59836 batch_time=0.50090
Train Epoch: 5 codebook_update_time=1.73781
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch5.pth ...
Done in 5.663s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch5.pth ...
Done in 10.939s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.22903994846344
quant_reg : 12.258356712341309
quant_err : 12.258356712341309
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
MSRVTT_full_val/t2v_metrics/R5: 64.38631790744466
MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.619718309859156
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.88264578852061
MSRVTT_full_val/v2t_metrics/R1: 32.394366197183096
MSRVTT_full_val/v2t_metrics/R5: 71.83098591549296
MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.682092555331992
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.05866818029883
MSRVTT_full_test/t2v_metrics/R1: 10.033444816053512
MSRVTT_full_test/t2v_metrics/R5: 30.568561872909697
MSRVTT_full_test/t2v_metrics/R10: 42.642140468227424
MSRVTT_full_test/t2v_metrics/R50: 76.75585284280936
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.40367892976589
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.560690994798122
MSRVTT_full_test/v2t_metrics/R1: 11.404682274247492
MSRVTT_full_test/v2t_metrics/R5: 34.247491638795985
MSRVTT_full_test/v2t_metrics/R10: 48.82943143812709
MSRVTT_full_test/v2t_metrics/R50: 82.37458193979933
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 38.3376254180602
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.71762657129169
mnt_best : 23.560690994798122
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.26929 (QuantReg: 12.02267) QuantErr: 12.02267 batch_time=37.63608
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.27975 (QuantReg: 11.64983) QuantErr: 11.64983 batch_time=0.58712
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 1.73164 (QuantReg: 12.35945) QuantErr: 12.35945 batch_time=0.51115
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.21024 (QuantReg: 12.11651) QuantErr: 12.11651 batch_time=0.51060
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 1.51822 (QuantReg: 12.53956) QuantErr: 12.53956 batch_time=0.50472
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 1.79521 (QuantReg: 12.59847) QuantErr: 12.59847 batch_time=0.50375
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.04995 (QuantReg: 12.68151) QuantErr: 12.68151 batch_time=1.04863
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 1.59847 (QuantReg: 12.27142) QuantErr: 12.27142 batch_time=0.51500
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.15196 (QuantReg: 12.41911) QuantErr: 12.41911 batch_time=0.51230
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.01630 (QuantReg: 12.60903) QuantErr: 12.60903 batch_time=0.51596
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 1.95969 (QuantReg: 12.28639) QuantErr: 12.28639 batch_time=0.51189
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.02476 (QuantReg: 12.82800) QuantErr: 12.82800 batch_time=0.52611
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.01150 (QuantReg: 12.36328) QuantErr: 12.36328 batch_time=0.50604
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.14915 (QuantReg: 12.34011) QuantErr: 12.34011 batch_time=3.41047
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 1.71130 (QuantReg: 12.65826) QuantErr: 12.65826 batch_time=0.50943
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 1.74564 (QuantReg: 12.70969) QuantErr: 12.70969 batch_time=0.50652
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.10798 (QuantReg: 13.00653) QuantErr: 13.00653 batch_time=0.50560
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 1.78610 (QuantReg: 13.03165) QuantErr: 13.03165 batch_time=0.51513
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 1.99212 (QuantReg: 12.88212) QuantErr: 12.88212 batch_time=0.50312
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 1.74709 (QuantReg: 12.69743) QuantErr: 12.69743 batch_time=0.67429
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 1.84053 (QuantReg: 12.92586) QuantErr: 12.92586 batch_time=0.52045
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.06794 (QuantReg: 12.72392) QuantErr: 12.72392 batch_time=0.51906
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 1.97267 (QuantReg: 12.94688) QuantErr: 12.94688 batch_time=0.55566
Train Epoch: 6 codebook_update_time=1.71382
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch6.pth ...
Done in 22.866s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch6.pth ...
Done in 26.931s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.0427571654319765
quant_reg : 12.524294567108154
quant_err : 12.524294567108154
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
MSRVTT_full_val/t2v_metrics/R5: 63.38028169014085
MSRVTT_full_val/t2v_metrics/R10: 78.67203219315896
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.164989939637827
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.47191072923643
MSRVTT_full_val/v2t_metrics/R1: 30.58350100603622
MSRVTT_full_val/v2t_metrics/R5: 71.83098591549296
MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
MSRVTT_full_val/v2t_metrics/R50: 98.79275653923541
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.917505030181086
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.9560207902503
MSRVTT_full_test/t2v_metrics/R1: 10.167224080267559
MSRVTT_full_test/t2v_metrics/R5: 30.100334448160535
MSRVTT_full_test/t2v_metrics/R10: 43.54515050167224
MSRVTT_full_test/t2v_metrics/R50: 76.8561872909699
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.51371237458194
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.708523277152377
MSRVTT_full_test/v2t_metrics/R1: 11.605351170568563
MSRVTT_full_test/v2t_metrics/R5: 35.08361204013378
MSRVTT_full_test/v2t_metrics/R10: 48.46153846153846
MSRVTT_full_test/v2t_metrics/R50: 81.40468227424749
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 40.009698996655516
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.02215170518202
mnt_best : 23.708523277152377
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.11845 (QuantReg: 11.95550) QuantErr: 11.95550 batch_time=36.73879
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 1.75488 (QuantReg: 12.86716) QuantErr: 12.86716 batch_time=0.51570
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.20637 (QuantReg: 12.65231) QuantErr: 12.65231 batch_time=0.65486
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.00931 (QuantReg: 12.88451) QuantErr: 12.88451 batch_time=0.53476
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 1.82850 (QuantReg: 12.70363) QuantErr: 12.70363 batch_time=0.53793
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.02988 (QuantReg: 12.43126) QuantErr: 12.43126 batch_time=0.66614
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.09071 (QuantReg: 13.06501) QuantErr: 13.06501 batch_time=1.01563
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.13406 (QuantReg: 12.69611) QuantErr: 12.69611 batch_time=0.52605
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.05106 (QuantReg: 12.71241) QuantErr: 12.71241 batch_time=0.56524
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.82571 (QuantReg: 12.57809) QuantErr: 12.57809 batch_time=0.53090
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.20379 (QuantReg: 12.82801) QuantErr: 12.82801 batch_time=0.99496
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 1.36100 (QuantReg: 13.31719) QuantErr: 13.31719 batch_time=0.50341
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.61685 (QuantReg: 12.80344) QuantErr: 12.80344 batch_time=0.51126
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.09452 (QuantReg: 12.70422) QuantErr: 12.70422 batch_time=0.52421
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.89118 (QuantReg: 12.55594) QuantErr: 12.55594 batch_time=0.52464
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 1.54514 (QuantReg: 12.94401) QuantErr: 12.94401 batch_time=0.51541
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.60751 (QuantReg: 13.25112) QuantErr: 13.25112 batch_time=0.57809
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 1.90999 (QuantReg: 12.86511) QuantErr: 12.86511 batch_time=0.51142
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.37900 (QuantReg: 13.04923) QuantErr: 13.04923 batch_time=0.56873
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.70703 (QuantReg: 13.41481) QuantErr: 13.41481 batch_time=1.23767
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.42377 (QuantReg: 12.96762) QuantErr: 12.96762 batch_time=0.52140
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 1.79393 (QuantReg: 13.06324) QuantErr: 13.06324 batch_time=0.89900
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.06764 (QuantReg: 13.02359) QuantErr: 13.02359 batch_time=0.51105
Train Epoch: 7 codebook_update_time=2.02892
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch7.pth ...
Done in 24.319s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch7.pth ...
Done in 28.416s
removing stale ckpt [epoch 6] [took 0.02s]
epoch : 7
loss : 1.8848004941940308
quant_reg : 12.87858529663086
quant_err : 12.87858529663086
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 30.18108651911469
MSRVTT_full_val/t2v_metrics/R5: 66.39839034205231
MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
MSRVTT_full_val/t2v_metrics/R50: 98.18913480885311
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.231388329979879
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.52395641215783
MSRVTT_full_val/v2t_metrics/R1: 31.99195171026157
MSRVTT_full_val/v2t_metrics/R5: 70.4225352112676
MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.531187122736418
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.4368695035054
MSRVTT_full_test/t2v_metrics/R1: 10.735785953177258
MSRVTT_full_test/t2v_metrics/R5: 32.642140468227424
MSRVTT_full_test/t2v_metrics/R10: 46.42140468227425
MSRVTT_full_test/t2v_metrics/R50: 78.79598662207358
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.33277591973244
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.338267519005676
MSRVTT_full_test/v2t_metrics/R1: 11.605351170568563
MSRVTT_full_test/v2t_metrics/R5: 34.71571906354515
MSRVTT_full_test/v2t_metrics/R10: 48.896321070234116
MSRVTT_full_test/v2t_metrics/R50: 82.04013377926421
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 39.034113712374584
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.00765508800257
mnt_best : 25.338267519005676
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.77664 (QuantReg: 12.71163) QuantErr: 12.71163 batch_time=31.96356
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.05509 (QuantReg: 12.81573) QuantErr: 12.81573 batch_time=0.50872
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.36071 (QuantReg: 12.51158) QuantErr: 12.51158 batch_time=0.94886
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 1.80041 (QuantReg: 13.05244) QuantErr: 13.05244 batch_time=0.51382
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.04531 (QuantReg: 12.77402) QuantErr: 12.77402 batch_time=0.51929
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.81729 (QuantReg: 12.57228) QuantErr: 12.57228 batch_time=0.52997
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.70293 (QuantReg: 13.16150) QuantErr: 13.16150 batch_time=0.54332
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 1.58848 (QuantReg: 13.13439) QuantErr: 13.13439 batch_time=0.52550
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.88265 (QuantReg: 13.02473) QuantErr: 13.02473 batch_time=0.79046
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.78298 (QuantReg: 13.00465) QuantErr: 13.00465 batch_time=0.53236
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 1.66055 (QuantReg: 13.29060) QuantErr: 13.29060 batch_time=0.51348
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.44289 (QuantReg: 13.33665) QuantErr: 13.33665 batch_time=0.49596
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.61513 (QuantReg: 13.07630) QuantErr: 13.07630 batch_time=0.50792
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.64609 (QuantReg: 13.14705) QuantErr: 13.14705 batch_time=0.51516
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.10053 (QuantReg: 13.28791) QuantErr: 13.28791 batch_time=0.51283
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.50396 (QuantReg: 13.11203) QuantErr: 13.11203 batch_time=0.52028
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.11753 (QuantReg: 13.30127) QuantErr: 13.30127 batch_time=0.51628
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 1.87874 (QuantReg: 13.40938) QuantErr: 13.40938 batch_time=0.54811
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.33270 (QuantReg: 13.52777) QuantErr: 13.52777 batch_time=1.81242
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.26971 (QuantReg: 13.46170) QuantErr: 13.46170 batch_time=0.51319
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.34864 (QuantReg: 13.72565) QuantErr: 13.72565 batch_time=0.52091
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.00556 (QuantReg: 13.29458) QuantErr: 13.29458 batch_time=0.53439
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.39702 (QuantReg: 13.30097) QuantErr: 13.30097 batch_time=0.50677
Train Epoch: 8 codebook_update_time=1.72731
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch8.pth ...
Done in 4.611s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 1.770752371788025
quant_reg : 13.045510845184326
quant_err : 13.045510845184326
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 28.772635814889338
MSRVTT_full_val/t2v_metrics/R5: 63.58148893360161
MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.438631790744466
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.80454624918084
MSRVTT_full_val/v2t_metrics/R1: 34.20523138832998
MSRVTT_full_val/v2t_metrics/R5: 71.0261569416499
MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.738430583501006
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.039887164113026
MSRVTT_full_test/t2v_metrics/R1: 10.936454849498327
MSRVTT_full_test/t2v_metrics/R5: 32.441471571906355
MSRVTT_full_test/t2v_metrics/R10: 44.5819397993311
MSRVTT_full_test/t2v_metrics/R50: 78.06020066889631
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.5628762541806
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.102213775137557
MSRVTT_full_test/v2t_metrics/R1: 12.575250836120402
MSRVTT_full_test/v2t_metrics/R5: 36.020066889632105
MSRVTT_full_test/v2t_metrics/R10: 51.103678929765884
MSRVTT_full_test/v2t_metrics/R50: 83.74581939799332
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.287290969899665
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.49953533274866
mnt_best : 25.338267519005676
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.98917 (QuantReg: 12.75271) QuantErr: 12.75271 batch_time=39.71745
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.61426 (QuantReg: 12.96981) QuantErr: 12.96981 batch_time=0.50739
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.05270 (QuantReg: 13.00939) QuantErr: 13.00939 batch_time=0.55240
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.01773 (QuantReg: 12.89952) QuantErr: 12.89952 batch_time=0.51339
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.11309 (QuantReg: 12.83890) QuantErr: 12.83890 batch_time=0.51598
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.83943 (QuantReg: 13.09502) QuantErr: 13.09502 batch_time=0.52760
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.62836 (QuantReg: 12.99808) QuantErr: 12.99808 batch_time=0.54115
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.44199 (QuantReg: 13.15097) QuantErr: 13.15097 batch_time=0.51207
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.93259 (QuantReg: 13.06588) QuantErr: 13.06588 batch_time=0.52286
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.68815 (QuantReg: 13.29938) QuantErr: 13.29938 batch_time=0.51428
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.81522 (QuantReg: 13.31697) QuantErr: 13.31697 batch_time=0.51807
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.50635 (QuantReg: 13.49733) QuantErr: 13.49733 batch_time=0.52945
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.68361 (QuantReg: 13.46684) QuantErr: 13.46684 batch_time=0.51469
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.78861 (QuantReg: 13.61482) QuantErr: 13.61482 batch_time=1.55888
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.86873 (QuantReg: 13.69716) QuantErr: 13.69716 batch_time=0.56656
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.76745 (QuantReg: 13.43184) QuantErr: 13.43184 batch_time=0.51828
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.20505 (QuantReg: 13.71965) QuantErr: 13.71965 batch_time=1.11511
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.42763 (QuantReg: 13.83530) QuantErr: 13.83530 batch_time=0.50252
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.42807 (QuantReg: 13.65457) QuantErr: 13.65457 batch_time=0.53268
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.30568 (QuantReg: 13.40729) QuantErr: 13.40729 batch_time=0.50528
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.54088 (QuantReg: 13.56817) QuantErr: 13.56817 batch_time=0.55060
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.56940 (QuantReg: 13.67107) QuantErr: 13.67107 batch_time=1.83822
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.06578 (QuantReg: 13.99704) QuantErr: 13.99704 batch_time=0.51530
Train Epoch: 9 codebook_update_time=1.79570
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch9.pth ...
Done in 5.949s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch9.pth ...
Done in 10.325s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 1.6411672859191895
quant_reg : 13.367284130096435
quant_err : 13.367284130096435
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 32.796780684104625
MSRVTT_full_val/t2v_metrics/R5: 65.99597585513078
MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
MSRVTT_full_val/t2v_metrics/R50: 98.39034205231388
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.839034205231388
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.80268027960629
MSRVTT_full_val/v2t_metrics/R1: 34.60764587525151
MSRVTT_full_val/v2t_metrics/R5: 73.03822937625755
MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.702213279678069
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.919579606191704
MSRVTT_full_test/t2v_metrics/R1: 11.237458193979933
MSRVTT_full_test/t2v_metrics/R5: 34.080267558528426
MSRVTT_full_test/t2v_metrics/R10: 47.19063545150502
MSRVTT_full_test/t2v_metrics/R50: 79.66555183946488
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.787290969899665
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.242727454052577
MSRVTT_full_test/v2t_metrics/R1: 13.010033444816054
MSRVTT_full_test/v2t_metrics/R5: 37.92642140468227
MSRVTT_full_test/v2t_metrics/R10: 51.67224080267559
MSRVTT_full_test/v2t_metrics/R50: 83.64548494983278
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.607357859531774
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.43241227300731
mnt_best : 26.242727454052577
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.38048 (QuantReg: 13.03663) QuantErr: 13.03663 batch_time=36.97880
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.45071 (QuantReg: 13.45818) QuantErr: 13.45818 batch_time=0.50243
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.47931 (QuantReg: 13.28318) QuantErr: 13.28318 batch_time=0.51352
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.27936 (QuantReg: 13.07540) QuantErr: 13.07540 batch_time=0.50754
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.39493 (QuantReg: 13.81783) QuantErr: 13.81783 batch_time=0.50018
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.29585 (QuantReg: 13.25865) QuantErr: 13.25865 batch_time=0.56576
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.41632 (QuantReg: 13.53897) QuantErr: 13.53897 batch_time=0.54184
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.51856 (QuantReg: 13.27279) QuantErr: 13.27279 batch_time=0.51382
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.59457 (QuantReg: 13.23723) QuantErr: 13.23723 batch_time=0.50092
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.08513 (QuantReg: 13.10910) QuantErr: 13.10910 batch_time=0.50646
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.26913 (QuantReg: 13.67716) QuantErr: 13.67716 batch_time=0.50035
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.69039 (QuantReg: 13.67557) QuantErr: 13.67557 batch_time=0.56816
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.90578 (QuantReg: 13.56019) QuantErr: 13.56019 batch_time=1.98819
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.02950 (QuantReg: 13.48510) QuantErr: 13.48510 batch_time=0.51100
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.26685 (QuantReg: 13.88254) QuantErr: 13.88254 batch_time=0.51878
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.22747 (QuantReg: 13.84581) QuantErr: 13.84581 batch_time=0.49691
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.51161 (QuantReg: 13.40407) QuantErr: 13.40407 batch_time=0.52308
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.33813 (QuantReg: 14.22072) QuantErr: 14.22072 batch_time=0.49429
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.21347 (QuantReg: 14.27179) QuantErr: 14.27179 batch_time=4.92224
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.44381 (QuantReg: 13.82871) QuantErr: 13.82871 batch_time=0.55543
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.42483 (QuantReg: 13.44580) QuantErr: 13.44580 batch_time=0.49423
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.09457 (QuantReg: 13.50370) QuantErr: 13.50370 batch_time=0.50822
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.89563 (QuantReg: 13.47798) QuantErr: 13.47798 batch_time=0.52446
Train Epoch: 10 codebook_update_time=1.68792
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch10.pth ...
Done in 4.703s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 1.5519173178672792
quant_reg : 13.521942432403565
quant_err : 13.521942432403565
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 30.18108651911469
MSRVTT_full_val/t2v_metrics/R5: 65.59356136820925
MSRVTT_full_val/t2v_metrics/R10: 81.69014084507042
MSRVTT_full_val/t2v_metrics/R50: 98.18913480885311
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.235412474849095
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.48227602609188
MSRVTT_full_val/v2t_metrics/R1: 37.625754527162975
MSRVTT_full_val/v2t_metrics/R5: 75.25150905432595
MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.303822937625754
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.57067393676297
MSRVTT_full_test/t2v_metrics/R1: 11.638795986622073
MSRVTT_full_test/t2v_metrics/R5: 32.10702341137124
MSRVTT_full_test/t2v_metrics/R10: 45.18394648829432
MSRVTT_full_test/t2v_metrics/R50: 79.66555183946488
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.921571906354515
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.65453143986109
MSRVTT_full_test/v2t_metrics/R1: 12.240802675585284
MSRVTT_full_test/v2t_metrics/R5: 37.525083612040135
MSRVTT_full_test/v2t_metrics/R10: 53.57859531772575
MSRVTT_full_test/v2t_metrics/R50: 84.51505016722408
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.3566889632107
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.0875826089776
mnt_best : 26.242727454052577
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.30805 (QuantReg: 13.44690) QuantErr: 13.44690 batch_time=40.38943
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.33218 (QuantReg: 13.66147) QuantErr: 13.66147 batch_time=0.54308
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.51671 (QuantReg: 13.30803) QuantErr: 13.30803 batch_time=0.51216
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.51730 (QuantReg: 13.60312) QuantErr: 13.60312 batch_time=0.50746
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.62537 (QuantReg: 13.85503) QuantErr: 13.85503 batch_time=0.50727
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.32904 (QuantReg: 13.77028) QuantErr: 13.77028 batch_time=0.50593
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.81721 (QuantReg: 13.59519) QuantErr: 13.59519 batch_time=0.54132
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.43994 (QuantReg: 13.99275) QuantErr: 13.99275 batch_time=0.50229
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.51552 (QuantReg: 13.39406) QuantErr: 13.39406 batch_time=0.52196
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.65050 (QuantReg: 13.52925) QuantErr: 13.52925 batch_time=0.51756
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.73249 (QuantReg: 13.57458) QuantErr: 13.57458 batch_time=0.52343
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.60901 (QuantReg: 14.00465) QuantErr: 14.00465 batch_time=0.50703
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.87021 (QuantReg: 13.47239) QuantErr: 13.47239 batch_time=0.51501
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.58429 (QuantReg: 13.62381) QuantErr: 13.62381 batch_time=0.61912
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.47941 (QuantReg: 13.65485) QuantErr: 13.65485 batch_time=1.52910
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.53024 (QuantReg: 13.53959) QuantErr: 13.53959 batch_time=0.56802
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.42734 (QuantReg: 13.63213) QuantErr: 13.63213 batch_time=0.52762
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.08461 (QuantReg: 13.80403) QuantErr: 13.80403 batch_time=0.50286
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.22786 (QuantReg: 14.18938) QuantErr: 14.18938 batch_time=0.50690
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.51101 (QuantReg: 14.03917) QuantErr: 14.03917 batch_time=0.58163
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.49444 (QuantReg: 14.20695) QuantErr: 14.20695 batch_time=0.50484
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.20050 (QuantReg: 14.16360) QuantErr: 14.16360 batch_time=0.50876
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.06494 (QuantReg: 14.43571) QuantErr: 14.43571 batch_time=0.50460
Train Epoch: 11 codebook_update_time=1.76664
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch11.pth ...
Done in 14.392s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch11.pth ...
Done in 22.681s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 1.4493475635051727
quant_reg : 13.80793920135498
quant_err : 13.80793920135498
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 30.784708249496983
MSRVTT_full_val/t2v_metrics/R5: 66.39839034205231
MSRVTT_full_val/t2v_metrics/R10: 81.69014084507042
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.6579476861167
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.06649225011874
MSRVTT_full_val/v2t_metrics/R1: 34.20523138832998
MSRVTT_full_val/v2t_metrics/R5: 75.25150905432595
MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.7082494969818915
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.61404554818994
MSRVTT_full_test/t2v_metrics/R1: 11.672240802675585
MSRVTT_full_test/t2v_metrics/R5: 33.54515050167224
MSRVTT_full_test/t2v_metrics/R10: 46.68896321070234
MSRVTT_full_test/t2v_metrics/R50: 79.29765886287625
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.32809364548495
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.343050904901073
MSRVTT_full_test/v2t_metrics/R1: 13.210702341137123
MSRVTT_full_test/v2t_metrics/R5: 37.82608695652174
MSRVTT_full_test/v2t_metrics/R10: 53.57859531772575
MSRVTT_full_test/v2t_metrics/R50: 84.48160535117057
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.31003344481606
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.915955768901217
mnt_best : 26.343050904901073
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.44723 (QuantReg: 13.65227) QuantErr: 13.65227 batch_time=38.45015
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.72914 (QuantReg: 13.67579) QuantErr: 13.67579 batch_time=0.52019
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.25751 (QuantReg: 13.72788) QuantErr: 13.72788 batch_time=0.55637
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.32388 (QuantReg: 13.68119) QuantErr: 13.68119 batch_time=0.51703
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.02860 (QuantReg: 14.09898) QuantErr: 14.09898 batch_time=0.53968
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.28008 (QuantReg: 13.91757) QuantErr: 13.91757 batch_time=0.51727
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.16021 (QuantReg: 13.83839) QuantErr: 13.83839 batch_time=0.51462
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.37663 (QuantReg: 13.74630) QuantErr: 13.74630 batch_time=0.53377
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.19566 (QuantReg: 13.63184) QuantErr: 13.63184 batch_time=0.58786
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.51718 (QuantReg: 13.78066) QuantErr: 13.78066 batch_time=0.52553
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.17387 (QuantReg: 13.83499) QuantErr: 13.83499 batch_time=0.53347
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.55719 (QuantReg: 13.90688) QuantErr: 13.90688 batch_time=0.53811
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.10180 (QuantReg: 14.06961) QuantErr: 14.06961 batch_time=0.52736
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.04491 (QuantReg: 14.58882) QuantErr: 14.58882 batch_time=0.53038
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.64531 (QuantReg: 13.80930) QuantErr: 13.80930 batch_time=0.51833
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.01049 (QuantReg: 14.21872) QuantErr: 14.21872 batch_time=0.54575
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.20116 (QuantReg: 14.25621) QuantErr: 14.25621 batch_time=0.57402
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.69134 (QuantReg: 13.67521) QuantErr: 13.67521 batch_time=0.50762
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.43713 (QuantReg: 13.65826) QuantErr: 13.65826 batch_time=0.50526
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.77693 (QuantReg: 13.93455) QuantErr: 13.93455 batch_time=1.71968
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.45457 (QuantReg: 13.63896) QuantErr: 13.63896 batch_time=0.52871
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.39487 (QuantReg: 14.18503) QuantErr: 14.18503 batch_time=0.53480
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.50285 (QuantReg: 14.33307) QuantErr: 14.33307 batch_time=0.55271
Train Epoch: 12 codebook_update_time=1.66686
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch12.pth ...
Done in 6.613s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch12.pth ...
Done in 12.617s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 1.3786621069908143
quant_reg : 13.926246883392334
quant_err : 13.926246883392334
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 36.21730382293762
MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.138832997987928
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 58.21161466408808
MSRVTT_full_val/v2t_metrics/R1: 37.223340040241446
MSRVTT_full_val/v2t_metrics/R5: 77.66599597585513
MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.189134808853119
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 63.006609124988245
MSRVTT_full_test/t2v_metrics/R1: 11.97324414715719
MSRVTT_full_test/t2v_metrics/R5: 34.94983277591973
MSRVTT_full_test/t2v_metrics/R10: 48.42809364548495
MSRVTT_full_test/t2v_metrics/R50: 81.00334448160535
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 41.30334448160535
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.263698558292255
MSRVTT_full_test/v2t_metrics/R1: 13.244147157190636
MSRVTT_full_test/v2t_metrics/R5: 39.096989966555185
MSRVTT_full_test/v2t_metrics/R10: 55.18394648829432
MSRVTT_full_test/v2t_metrics/R50: 85.11705685618729
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.57759197324415
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.57220000562679
mnt_best : 27.263698558292255
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.22077 (QuantReg: 14.00204) QuantErr: 14.00204 batch_time=37.57890
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.86167 (QuantReg: 13.45687) QuantErr: 13.45687 batch_time=4.84795
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.34994 (QuantReg: 14.12304) QuantErr: 14.12304 batch_time=0.50667
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.14189 (QuantReg: 14.08197) QuantErr: 14.08197 batch_time=0.50863
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.25568 (QuantReg: 14.00688) QuantErr: 14.00688 batch_time=1.35022
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.37488 (QuantReg: 14.35078) QuantErr: 14.35078 batch_time=0.52891
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.37141 (QuantReg: 14.18127) QuantErr: 14.18127 batch_time=0.54103
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.69926 (QuantReg: 14.28839) QuantErr: 14.28839 batch_time=0.50165
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.16105 (QuantReg: 13.77132) QuantErr: 13.77132 batch_time=0.54979
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.54584 (QuantReg: 14.04511) QuantErr: 14.04511 batch_time=0.50516
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.44215 (QuantReg: 13.84797) QuantErr: 13.84797 batch_time=0.52350
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 0.98403 (QuantReg: 14.24296) QuantErr: 14.24296 batch_time=0.53431
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.25480 (QuantReg: 14.38118) QuantErr: 14.38118 batch_time=0.51911
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.32666 (QuantReg: 14.30749) QuantErr: 14.30749 batch_time=0.52971
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.53844 (QuantReg: 13.91843) QuantErr: 13.91843 batch_time=0.50767
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.15471 (QuantReg: 13.98852) QuantErr: 13.98852 batch_time=0.51055
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.02734 (QuantReg: 14.54011) QuantErr: 14.54011 batch_time=0.61834
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.76653 (QuantReg: 13.97441) QuantErr: 13.97441 batch_time=0.60295
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.03933 (QuantReg: 14.69055) QuantErr: 14.69055 batch_time=0.50802
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.72690 (QuantReg: 14.32969) QuantErr: 14.32969 batch_time=0.52545
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.54842 (QuantReg: 14.35689) QuantErr: 14.35689 batch_time=0.51318
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.11829 (QuantReg: 14.24028) QuantErr: 14.24028 batch_time=0.52064
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.34593 (QuantReg: 14.40843) QuantErr: 14.40843 batch_time=0.92388
Train Epoch: 13 codebook_update_time=1.80115
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch13.pth ...
Done in 4.072s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch13.pth ...
Done in 8.021s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 1.317027200460434
quant_reg : 14.150179462432861
quant_err : 14.150179462432861
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 31.58953722334004
MSRVTT_full_val/t2v_metrics/R5: 68.61167002012073
MSRVTT_full_val/t2v_metrics/R10: 82.49496981891348
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.645875251509055
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.33646260996741
MSRVTT_full_val/v2t_metrics/R1: 39.436619718309856
MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
MSRVTT_full_val/v2t_metrics/R10: 86.31790744466801
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.219315895372233
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 63.28225899062843
MSRVTT_full_test/t2v_metrics/R1: 12.040133779264215
MSRVTT_full_test/t2v_metrics/R5: 35.551839464882946
MSRVTT_full_test/t2v_metrics/R10: 48.96321070234114
MSRVTT_full_test/t2v_metrics/R50: 79.49832775919732
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 41.34615384615385
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.57112109266316
MSRVTT_full_test/v2t_metrics/R1: 14.51505016722408
MSRVTT_full_test/v2t_metrics/R5: 40.33444816053512
MSRVTT_full_test/v2t_metrics/R10: 54.71571906354515
MSRVTT_full_test/v2t_metrics/R50: 85.35117056856187
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.16421404682274
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.759153906063084
mnt_best : 27.57112109266316
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.05656 (QuantReg: 14.11441) QuantErr: 14.11441 batch_time=37.88346
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.70573 (QuantReg: 14.28553) QuantErr: 14.28553 batch_time=0.52283
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.33255 (QuantReg: 14.25108) QuantErr: 14.25108 batch_time=0.57378
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.20870 (QuantReg: 14.04397) QuantErr: 14.04397 batch_time=0.55684
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.59684 (QuantReg: 14.15369) QuantErr: 14.15369 batch_time=0.54857
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.28529 (QuantReg: 14.08165) QuantErr: 14.08165 batch_time=0.52432
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.52112 (QuantReg: 14.10736) QuantErr: 14.10736 batch_time=0.55942
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.53429 (QuantReg: 13.88581) QuantErr: 13.88581 batch_time=0.66616
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 0.97121 (QuantReg: 14.08434) QuantErr: 14.08434 batch_time=0.55202
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.23670 (QuantReg: 14.04921) QuantErr: 14.04921 batch_time=0.52231
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.09710 (QuantReg: 14.31238) QuantErr: 14.31238 batch_time=0.58720
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.57524 (QuantReg: 13.80048) QuantErr: 13.80048 batch_time=0.53327
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.06450 (QuantReg: 14.52403) QuantErr: 14.52403 batch_time=0.51890
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 0.95397 (QuantReg: 14.31596) QuantErr: 14.31596 batch_time=2.67340
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.22827 (QuantReg: 14.39271) QuantErr: 14.39271 batch_time=0.50646
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.01029 (QuantReg: 14.39787) QuantErr: 14.39787 batch_time=0.56151
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 0.97429 (QuantReg: 14.20917) QuantErr: 14.20917 batch_time=0.52306
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.58543 (QuantReg: 14.57169) QuantErr: 14.57169 batch_time=0.50811
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.22738 (QuantReg: 14.32255) QuantErr: 14.32255 batch_time=0.51849
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.53740 (QuantReg: 14.27053) QuantErr: 14.27053 batch_time=2.98643
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.29868 (QuantReg: 14.41371) QuantErr: 14.41371 batch_time=0.51388
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.17884 (QuantReg: 14.61811) QuantErr: 14.61811 batch_time=0.51495
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.25525 (QuantReg: 14.29929) QuantErr: 14.29929 batch_time=0.52290
Train Epoch: 14 codebook_update_time=1.68222
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch14.pth ...
Done in 3.751s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 1.271657412290573
quant_reg : 14.25413032913208
quant_err : 14.25413032913208
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 29.77867203219316
MSRVTT_full_val/t2v_metrics/R5: 66.59959758551308
MSRVTT_full_val/t2v_metrics/R10: 80.6841046277666
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.722334004024145
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.29022462298815
MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
MSRVTT_full_val/v2t_metrics/R5: 73.64185110663983
MSRVTT_full_val/v2t_metrics/R10: 84.90945674044265
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.3843058350100605
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.72694834711691
MSRVTT_full_test/t2v_metrics/R1: 11.906354515050166
MSRVTT_full_test/t2v_metrics/R5: 33.74581939799331
MSRVTT_full_test/t2v_metrics/R10: 46.48829431438127
MSRVTT_full_test/t2v_metrics/R50: 77.95986622073579
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.54080267558528
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.532660488316225
MSRVTT_full_test/v2t_metrics/R1: 13.377926421404682
MSRVTT_full_test/v2t_metrics/R5: 38.2943143812709
MSRVTT_full_test/v2t_metrics/R10: 53.41137123745819
MSRVTT_full_test/v2t_metrics/R50: 84.58193979933111
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.04531772575251
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.133687240677588
mnt_best : 27.57112109266316
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.52414 (QuantReg: 13.93040) QuantErr: 13.93040 batch_time=33.16319
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.12583 (QuantReg: 14.23211) QuantErr: 14.23211 batch_time=0.54253
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.07802 (QuantReg: 14.25939) QuantErr: 14.25939 batch_time=0.51411
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.36387 (QuantReg: 14.21986) QuantErr: 14.21986 batch_time=0.52804
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.19470 (QuantReg: 14.36109) QuantErr: 14.36109 batch_time=0.51299
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.52660 (QuantReg: 14.05234) QuantErr: 14.05234 batch_time=0.53382
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.62010 (QuantReg: 14.33631) QuantErr: 14.33631 batch_time=1.14578
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 0.97412 (QuantReg: 14.45049) QuantErr: 14.45049 batch_time=0.50402
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.51047 (QuantReg: 14.25784) QuantErr: 14.25784 batch_time=1.67238
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.30346 (QuantReg: 14.46433) QuantErr: 14.46433 batch_time=0.56749
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.55404 (QuantReg: 14.27022) QuantErr: 14.27022 batch_time=0.51086
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.02299 (QuantReg: 14.13624) QuantErr: 14.13624 batch_time=0.50025
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.19471 (QuantReg: 14.39443) QuantErr: 14.39443 batch_time=0.50966
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.37555 (QuantReg: 14.30704) QuantErr: 14.30704 batch_time=0.94756
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.22512 (QuantReg: 14.09487) QuantErr: 14.09487 batch_time=0.52822
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.37864 (QuantReg: 14.57727) QuantErr: 14.57727 batch_time=0.51377
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.32415 (QuantReg: 14.46058) QuantErr: 14.46058 batch_time=0.52513
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.16502 (QuantReg: 14.50969) QuantErr: 14.50969 batch_time=0.51985
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.06732 (QuantReg: 14.57021) QuantErr: 14.57021 batch_time=0.65586
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.40057 (QuantReg: 14.23043) QuantErr: 14.23043 batch_time=0.52007
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.43621 (QuantReg: 14.58199) QuantErr: 14.58199 batch_time=0.53356
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.39455 (QuantReg: 14.55030) QuantErr: 14.55030 batch_time=0.50741
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.29132 (QuantReg: 14.15588) QuantErr: 14.15588 batch_time=0.51057
Train Epoch: 15 codebook_update_time=1.65914
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch15.pth ...
Done in 4.033s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.03/checkpoint-epoch15.pth ...
Done in 8.300s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 1.2063311808109283
quant_reg : 14.394946002960205