-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full_L3.txt
3309 lines (3309 loc) · 235 KB
/
HCQ_MSRVTT_full_L3.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 779.2634873390198 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 40.29928660392761 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 234.78307271003723 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 189.54125547409058 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch0.pth ...
Done in 2.973s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch0.pth ...
Done in 4.373s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.2012072434607646
MSRVTT_full_val/t2v_metrics/R5: 1.408450704225352
MSRVTT_full_val/t2v_metrics/R10: 2.414486921529175
MSRVTT_full_val/t2v_metrics/R50: 11.87122736418511
MSRVTT_full_val/t2v_metrics/MedR: 260.0
MSRVTT_full_val/t2v_metrics/MeanR: 255.46277665995976
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.8811909738205008
MSRVTT_full_val/v2t_metrics/R1: 0.2012072434607646
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 1.6096579476861168
MSRVTT_full_val/v2t_metrics/R50: 8.048289738430583
MSRVTT_full_val/v2t_metrics/MedR: 256.0
MSRVTT_full_val/v2t_metrics/MeanR: 254.8450704225352
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.6387931798664787
MSRVTT_full_test/t2v_metrics/R1: 0.0
MSRVTT_full_test/t2v_metrics/R5: 0.23411371237458195
MSRVTT_full_test/t2v_metrics/R10: 0.33444816053511706
MSRVTT_full_test/t2v_metrics/R50: 1.37123745819398
MSRVTT_full_test/t2v_metrics/MedR: 1505.5
MSRVTT_full_test/t2v_metrics/MeanR: 1503.0210702341137
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/v2t_metrics/R1: 0.033444816053511704
MSRVTT_full_test/v2t_metrics/R5: 0.13377926421404682
MSRVTT_full_test/v2t_metrics/R10: 0.36789297658862874
MSRVTT_full_test/v2t_metrics/R50: 1.5384615384615385
MSRVTT_full_test/v2t_metrics/MedR: 1517.5
MSRVTT_full_test/v2t_metrics/MeanR: 1510.3939799331104
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.11807185067980144
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 32.85585 (QuantReg: 22.79831) QuantErr: 22.79831 batch_time=27.10723
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 30.24445 (QuantReg: 22.79816) QuantErr: 22.79816 batch_time=0.41417
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 25.95213 (QuantReg: 22.70061) QuantErr: 22.70061 batch_time=0.44227
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 23.20899 (QuantReg: 22.68929) QuantErr: 22.68929 batch_time=0.44028
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 21.87263 (QuantReg: 22.69030) QuantErr: 22.69030 batch_time=0.46217
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 20.50840 (QuantReg: 22.60241) QuantErr: 22.60241 batch_time=0.43017
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 18.98234 (QuantReg: 22.59351) QuantErr: 22.59351 batch_time=0.43487
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 19.21747 (QuantReg: 22.57938) QuantErr: 22.57938 batch_time=0.43519
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 18.31880 (QuantReg: 22.58747) QuantErr: 22.58747 batch_time=0.44302
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 18.24098 (QuantReg: 22.65045) QuantErr: 22.65045 batch_time=0.46865
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 17.85880 (QuantReg: 22.57447) QuantErr: 22.57447 batch_time=0.45083
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 14.92087 (QuantReg: 22.59782) QuantErr: 22.59782 batch_time=0.55086
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 15.72481 (QuantReg: 22.66413) QuantErr: 22.66413 batch_time=8.37643
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 16.75700 (QuantReg: 22.61890) QuantErr: 22.61890 batch_time=0.43899
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 14.59371 (QuantReg: 22.66484) QuantErr: 22.66484 batch_time=0.44155
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 15.56373 (QuantReg: 22.63058) QuantErr: 22.63058 batch_time=0.45319
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 15.68122 (QuantReg: 22.65586) QuantErr: 22.65586 batch_time=0.44552
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 15.27050 (QuantReg: 22.65612) QuantErr: 22.65612 batch_time=0.48328
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 14.09276 (QuantReg: 22.63098) QuantErr: 22.63098 batch_time=0.45889
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 14.03891 (QuantReg: 22.61277) QuantErr: 22.61277 batch_time=0.44215
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 12.75415 (QuantReg: 22.67994) QuantErr: 22.67994 batch_time=0.43350
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 14.71962 (QuantReg: 22.64540) QuantErr: 22.64540 batch_time=0.44640
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 13.93099 (QuantReg: 22.65827) QuantErr: 22.65827 batch_time=0.44139
Train Epoch: 1 codebook_update_time=1.34918
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch1.pth ...
Done in 3.649s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch1.pth ...
Done in 7.374s
epoch : 1
loss : 18.242241424560547
quant_reg : 22.648892669677736
quant_err : 22.648892669677736
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 17.10261569416499
MSRVTT_full_val/t2v_metrics/R5: 50.503018108651915
MSRVTT_full_val/t2v_metrics/R10: 65.79476861167002
MSRVTT_full_val/t2v_metrics/R50: 93.96378269617706
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 13.498993963782697
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 38.446523743195385
MSRVTT_full_val/v2t_metrics/R1: 19.517102615694164
MSRVTT_full_val/v2t_metrics/R5: 57.947686116700204
MSRVTT_full_val/v2t_metrics/R10: 72.63581488933602
MSRVTT_full_val/v2t_metrics/R50: 95.17102615694165
MSRVTT_full_val/v2t_metrics/MedR: 5.0
MSRVTT_full_val/v2t_metrics/MeanR: 11.857142857142858
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 43.4711122904589
MSRVTT_full_test/t2v_metrics/R1: 5.719063545150502
MSRVTT_full_test/t2v_metrics/R5: 20.668896321070235
MSRVTT_full_test/t2v_metrics/R10: 31.304347826086957
MSRVTT_full_test/t2v_metrics/R50: 65.58528428093645
MSRVTT_full_test/t2v_metrics/MedR: 25.5
MSRVTT_full_test/t2v_metrics/MeanR: 74.09749163879599
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 15.467339674929313
MSRVTT_full_test/v2t_metrics/R1: 6.321070234113712
MSRVTT_full_test/v2t_metrics/R5: 23.244147157190636
MSRVTT_full_test/v2t_metrics/R10: 35.752508361204015
MSRVTT_full_test/v2t_metrics/R50: 71.53846153846153
MSRVTT_full_test/v2t_metrics/MedR: 20.0
MSRVTT_full_test/v2t_metrics/MeanR: 63.796321070234114
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 17.38348776826853
mnt_best : 15.467339674929313
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 14.05858 (QuantReg: 11.44015) QuantErr: 11.44015 batch_time=33.85238
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 15.80260 (QuantReg: 11.66120) QuantErr: 11.66120 batch_time=0.48300
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 11.82563 (QuantReg: 11.86171) QuantErr: 11.86171 batch_time=0.48084
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 12.30206 (QuantReg: 12.08834) QuantErr: 12.08834 batch_time=0.43984
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 14.12353 (QuantReg: 12.19362) QuantErr: 12.19362 batch_time=0.46291
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 13.40150 (QuantReg: 12.25146) QuantErr: 12.25146 batch_time=0.45172
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 13.15391 (QuantReg: 12.83502) QuantErr: 12.83502 batch_time=0.44582
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 12.19179 (QuantReg: 13.11477) QuantErr: 13.11477 batch_time=0.43983
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 11.23748 (QuantReg: 12.64319) QuantErr: 12.64319 batch_time=0.50028
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 14.97452 (QuantReg: 13.53232) QuantErr: 13.53232 batch_time=0.44849
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 12.47355 (QuantReg: 12.97190) QuantErr: 12.97190 batch_time=0.43342
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 12.42081 (QuantReg: 13.28956) QuantErr: 13.28956 batch_time=0.46063
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 11.54886 (QuantReg: 13.54886) QuantErr: 13.54886 batch_time=0.43339
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 12.27464 (QuantReg: 13.64693) QuantErr: 13.64693 batch_time=0.45355
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 12.60571 (QuantReg: 14.13509) QuantErr: 14.13509 batch_time=0.43445
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 11.72826 (QuantReg: 13.73311) QuantErr: 13.73311 batch_time=0.44196
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 10.69299 (QuantReg: 13.92267) QuantErr: 13.92267 batch_time=0.79373
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 11.13158 (QuantReg: 13.98775) QuantErr: 13.98775 batch_time=0.42682
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 10.67053 (QuantReg: 14.34211) QuantErr: 14.34211 batch_time=0.48876
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 9.73153 (QuantReg: 13.91892) QuantErr: 13.91892 batch_time=0.42375
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 10.52122 (QuantReg: 14.34412) QuantErr: 14.34412 batch_time=0.44492
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 11.74605 (QuantReg: 13.92942) QuantErr: 13.92942 batch_time=0.44236
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 10.39697 (QuantReg: 14.45884) QuantErr: 14.45884 batch_time=0.42397
Train Epoch: 2 codebook_update_time=1.26764
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch2.pth ...
Done in 4.280s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch2.pth ...
Done in 8.266s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 12.149146018981934
quant_reg : 13.244106685638428
quant_err : 13.244106685638428
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 23.541247484909455
MSRVTT_full_val/t2v_metrics/R5: 57.545271629778675
MSRVTT_full_val/t2v_metrics/R10: 74.64788732394366
MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.058350100603622
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.58923143305815
MSRVTT_full_val/v2t_metrics/R1: 26.156941649899398
MSRVTT_full_val/v2t_metrics/R5: 63.98390342052314
MSRVTT_full_val/v2t_metrics/R10: 79.27565392354124
MSRVTT_full_val/v2t_metrics/R50: 96.579476861167
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.45271629778672
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 51.00340562230017
MSRVTT_full_test/t2v_metrics/R1: 8.093645484949834
MSRVTT_full_test/t2v_metrics/R5: 26.220735785953178
MSRVTT_full_test/t2v_metrics/R10: 37.424749163879596
MSRVTT_full_test/t2v_metrics/R50: 72.37458193979933
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 59.63779264214047
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.951826064649484
MSRVTT_full_test/v2t_metrics/R1: 10.200668896321071
MSRVTT_full_test/v2t_metrics/R5: 29.498327759197323
MSRVTT_full_test/v2t_metrics/R10: 43.61204013377927
MSRVTT_full_test/v2t_metrics/R50: 77.82608695652173
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 49.825752508361205
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.587259423090664
mnt_best : 19.951826064649484
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 10.65837 (QuantReg: 11.61483) QuantErr: 11.61483 batch_time=30.30607
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 10.29684 (QuantReg: 11.42953) QuantErr: 11.42953 batch_time=0.42967
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 10.71062 (QuantReg: 11.76652) QuantErr: 11.76652 batch_time=0.41779
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 9.85394 (QuantReg: 11.94877) QuantErr: 11.94877 batch_time=0.43386
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 11.39816 (QuantReg: 12.01004) QuantErr: 12.01004 batch_time=0.41490
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 10.32315 (QuantReg: 12.21951) QuantErr: 12.21951 batch_time=0.43030
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 8.47892 (QuantReg: 11.80438) QuantErr: 11.80438 batch_time=0.42354
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 9.26555 (QuantReg: 12.39084) QuantErr: 12.39084 batch_time=0.41790
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 10.92874 (QuantReg: 12.26325) QuantErr: 12.26325 batch_time=0.41602
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 10.22449 (QuantReg: 12.64115) QuantErr: 12.64115 batch_time=0.96238
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 10.02801 (QuantReg: 12.10319) QuantErr: 12.10319 batch_time=0.42404
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 9.79375 (QuantReg: 12.41067) QuantErr: 12.41067 batch_time=0.86968
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 12.21120 (QuantReg: 12.67897) QuantErr: 12.67897 batch_time=0.44497
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 10.73918 (QuantReg: 12.80380) QuantErr: 12.80380 batch_time=0.45082
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 10.35908 (QuantReg: 12.80953) QuantErr: 12.80953 batch_time=0.43869
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 9.47240 (QuantReg: 12.62194) QuantErr: 12.62194 batch_time=0.41549
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 9.69389 (QuantReg: 13.03466) QuantErr: 13.03466 batch_time=0.45215
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 10.10056 (QuantReg: 12.48081) QuantErr: 12.48081 batch_time=0.42132
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 9.85667 (QuantReg: 13.01988) QuantErr: 13.01988 batch_time=0.42051
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 10.11460 (QuantReg: 12.95218) QuantErr: 12.95218 batch_time=0.43164
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 11.27119 (QuantReg: 12.78079) QuantErr: 12.78079 batch_time=0.43680
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 9.43257 (QuantReg: 12.88197) QuantErr: 12.88197 batch_time=0.42465
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 9.93975 (QuantReg: 13.19816) QuantErr: 13.19816 batch_time=0.41415
Train Epoch: 3 codebook_update_time=1.04502
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch3.pth ...
Done in 6.373s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch3.pth ...
Done in 11.133s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 10.152871868133545
quant_reg : 12.493335159301758
quant_err : 12.493335159301758
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 25.35211267605634
MSRVTT_full_val/t2v_metrics/R5: 61.3682092555332
MSRVTT_full_val/t2v_metrics/R10: 75.25150905432595
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.967806841046277
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.92050596898273
MSRVTT_full_val/v2t_metrics/R1: 27.16297786720322
MSRVTT_full_val/v2t_metrics/R5: 67.00201207243461
MSRVTT_full_val/v2t_metrics/R10: 79.07444668008048
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.458752515090543
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 52.40432468026335
MSRVTT_full_test/t2v_metrics/R1: 8.762541806020067
MSRVTT_full_test/t2v_metrics/R5: 28.46153846153846
MSRVTT_full_test/t2v_metrics/R10: 40.836120401337794
MSRVTT_full_test/t2v_metrics/R50: 75.15050167224081
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 53.18361204013378
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.67592552242988
MSRVTT_full_test/v2t_metrics/R1: 10.96989966555184
MSRVTT_full_test/v2t_metrics/R5: 32.474916387959865
MSRVTT_full_test/v2t_metrics/R10: 45.48494983277592
MSRVTT_full_test/v2t_metrics/R50: 80.20066889632108
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 44.96020066889632
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.304988192837534
mnt_best : 21.67592552242988
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 10.31193 (QuantReg: 11.66770) QuantErr: 11.66770 batch_time=36.93729
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 8.79738 (QuantReg: 11.96855) QuantErr: 11.96855 batch_time=0.42476
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 9.47104 (QuantReg: 12.23875) QuantErr: 12.23875 batch_time=0.42412
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 7.17666 (QuantReg: 12.47117) QuantErr: 12.47117 batch_time=0.44178
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 9.17817 (QuantReg: 12.14473) QuantErr: 12.14473 batch_time=0.49074
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 8.88997 (QuantReg: 12.22786) QuantErr: 12.22786 batch_time=0.45623
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 9.63442 (QuantReg: 11.93797) QuantErr: 11.93797 batch_time=0.43540
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 8.64141 (QuantReg: 11.63585) QuantErr: 11.63585 batch_time=0.42241
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 8.77465 (QuantReg: 11.97766) QuantErr: 11.97766 batch_time=0.43122
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 9.92321 (QuantReg: 12.26371) QuantErr: 12.26371 batch_time=0.43668
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 9.58786 (QuantReg: 12.07858) QuantErr: 12.07858 batch_time=0.43794
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 10.81381 (QuantReg: 12.13236) QuantErr: 12.13236 batch_time=0.44493
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 9.62987 (QuantReg: 12.43481) QuantErr: 12.43481 batch_time=0.42654
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 8.64977 (QuantReg: 12.24317) QuantErr: 12.24317 batch_time=0.44077
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 10.06771 (QuantReg: 12.29246) QuantErr: 12.29246 batch_time=0.42478
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 7.87638 (QuantReg: 12.59364) QuantErr: 12.59364 batch_time=0.46705
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 9.43943 (QuantReg: 13.11007) QuantErr: 13.11007 batch_time=0.45901
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 7.50602 (QuantReg: 12.77980) QuantErr: 12.77980 batch_time=0.45439
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 9.82948 (QuantReg: 12.78378) QuantErr: 12.78378 batch_time=0.78410
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 8.06569 (QuantReg: 12.84405) QuantErr: 12.84405 batch_time=0.43416
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 8.86313 (QuantReg: 12.66839) QuantErr: 12.66839 batch_time=0.42944
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 8.93842 (QuantReg: 12.82486) QuantErr: 12.82486 batch_time=0.42524
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 7.50736 (QuantReg: 12.37204) QuantErr: 12.37204 batch_time=0.47474
Train Epoch: 4 codebook_update_time=1.42433
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch4.pth ...
Done in 5.043s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch4.pth ...
Done in 10.435s
removing stale ckpt [epoch 3] [took 0.03s]
epoch : 4
loss : 9.002852266311645
quant_reg : 12.361520969390869
quant_err : 12.361520969390869
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 27.56539235412475
MSRVTT_full_val/t2v_metrics/R5: 63.38028169014085
MSRVTT_full_val/t2v_metrics/R10: 77.2635814889336
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.680080482897385
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 51.297680273044755
MSRVTT_full_val/v2t_metrics/R1: 30.58350100603622
MSRVTT_full_val/v2t_metrics/R5: 70.82494969818913
MSRVTT_full_val/v2t_metrics/R10: 81.89134808853119
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.867203219315895
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.18717083334731
MSRVTT_full_test/t2v_metrics/R1: 9.297658862876254
MSRVTT_full_test/t2v_metrics/R5: 28.46153846153846
MSRVTT_full_test/t2v_metrics/R10: 42.240802675585286
MSRVTT_full_test/t2v_metrics/R50: 76.5886287625418
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 51.384949832775916
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.359120364198922
MSRVTT_full_test/v2t_metrics/R1: 11.57190635451505
MSRVTT_full_test/v2t_metrics/R5: 33.34448160535117
MSRVTT_full_test/v2t_metrics/R10: 48.76254180602007
MSRVTT_full_test/v2t_metrics/R50: 82.14046822742475
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 40.63779264214047
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.597352270564038
mnt_best : 22.359120364198922
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 8.90941 (QuantReg: 11.99209) QuantErr: 11.99209 batch_time=28.50364
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 8.49190 (QuantReg: 11.84129) QuantErr: 11.84129 batch_time=0.68130
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 7.95471 (QuantReg: 11.65466) QuantErr: 11.65466 batch_time=0.48135
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 8.00352 (QuantReg: 12.59978) QuantErr: 12.59978 batch_time=0.44437
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 9.16129 (QuantReg: 12.34573) QuantErr: 12.34573 batch_time=0.42934
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 7.88464 (QuantReg: 12.31339) QuantErr: 12.31339 batch_time=0.44284
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 10.82199 (QuantReg: 11.94291) QuantErr: 11.94291 batch_time=1.13375
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 8.68495 (QuantReg: 12.19099) QuantErr: 12.19099 batch_time=0.42721
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 6.87733 (QuantReg: 12.08893) QuantErr: 12.08893 batch_time=0.44007
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 7.11855 (QuantReg: 12.59900) QuantErr: 12.59900 batch_time=0.44615
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 7.55683 (QuantReg: 12.58483) QuantErr: 12.58483 batch_time=0.45264
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 7.17518 (QuantReg: 12.36936) QuantErr: 12.36936 batch_time=0.44525
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 8.41648 (QuantReg: 12.22748) QuantErr: 12.22748 batch_time=0.44294
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 7.60112 (QuantReg: 12.70938) QuantErr: 12.70938 batch_time=0.43896
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 9.04991 (QuantReg: 12.52275) QuantErr: 12.52275 batch_time=0.44535
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 8.52421 (QuantReg: 12.35206) QuantErr: 12.35206 batch_time=0.43573
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 7.33847 (QuantReg: 12.69287) QuantErr: 12.69287 batch_time=0.43115
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 8.00619 (QuantReg: 12.16542) QuantErr: 12.16542 batch_time=0.43239
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 8.85616 (QuantReg: 12.49319) QuantErr: 12.49319 batch_time=0.47686
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 9.71512 (QuantReg: 12.52973) QuantErr: 12.52973 batch_time=0.44321
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 8.47287 (QuantReg: 12.62864) QuantErr: 12.62864 batch_time=0.44384
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 7.73925 (QuantReg: 12.70618) QuantErr: 12.70618 batch_time=0.43179
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 7.30802 (QuantReg: 12.49784) QuantErr: 12.49784 batch_time=0.53626
Train Epoch: 5 codebook_update_time=1.08141
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch5.pth ...
Done in 4.477s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch5.pth ...
Done in 9.547s
removing stale ckpt [epoch 4] [took 0.40s]
epoch : 5
loss : 8.167788110733031
quant_reg : 12.389640781402587
quant_err : 12.389640781402587
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 65.3923541247485
MSRVTT_full_val/t2v_metrics/R10: 78.26961770623743
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.853118712273641
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.174761494211175
MSRVTT_full_val/v2t_metrics/R1: 32.59557344064386
MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.090543259557344
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.3718767685367
MSRVTT_full_test/t2v_metrics/R1: 10.200668896321071
MSRVTT_full_test/t2v_metrics/R5: 30.20066889632107
MSRVTT_full_test/t2v_metrics/R10: 44.64882943143812
MSRVTT_full_test/t2v_metrics/R50: 78.06020066889631
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 47.551839464882946
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.959905277738173
MSRVTT_full_test/v2t_metrics/R1: 12.040133779264215
MSRVTT_full_test/v2t_metrics/R5: 35.618729096989966
MSRVTT_full_test/v2t_metrics/R10: 49.59866220735786
MSRVTT_full_test/v2t_metrics/R50: 83.4113712374582
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 38.31605351170568
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.70723776396443
mnt_best : 23.959905277738173
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 7.88574 (QuantReg: 12.22206) QuantErr: 12.22206 batch_time=32.49376
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 6.80999 (QuantReg: 12.29603) QuantErr: 12.29603 batch_time=0.43975
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 7.19587 (QuantReg: 11.87659) QuantErr: 11.87659 batch_time=0.43327
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 7.09944 (QuantReg: 12.19299) QuantErr: 12.19299 batch_time=0.42493
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 9.06539 (QuantReg: 12.44792) QuantErr: 12.44792 batch_time=0.42979
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 8.11404 (QuantReg: 12.16321) QuantErr: 12.16321 batch_time=0.43498
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 8.31902 (QuantReg: 12.39706) QuantErr: 12.39706 batch_time=0.43243
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 6.64396 (QuantReg: 12.65998) QuantErr: 12.65998 batch_time=0.43590
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 8.74998 (QuantReg: 12.44786) QuantErr: 12.44786 batch_time=0.46531
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 7.93206 (QuantReg: 12.50525) QuantErr: 12.50525 batch_time=0.45195
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 8.68394 (QuantReg: 12.27596) QuantErr: 12.27596 batch_time=0.43842
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 7.24970 (QuantReg: 12.76235) QuantErr: 12.76235 batch_time=0.42977
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 6.64610 (QuantReg: 12.50012) QuantErr: 12.50012 batch_time=0.94689
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 7.06418 (QuantReg: 12.09987) QuantErr: 12.09987 batch_time=0.42391
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 6.89471 (QuantReg: 12.54438) QuantErr: 12.54438 batch_time=0.48992
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 6.42478 (QuantReg: 12.29005) QuantErr: 12.29005 batch_time=0.45833
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 7.10157 (QuantReg: 12.75937) QuantErr: 12.75937 batch_time=0.44784
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 6.92043 (QuantReg: 12.50571) QuantErr: 12.50571 batch_time=0.43505
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 7.27868 (QuantReg: 12.36143) QuantErr: 12.36143 batch_time=0.44087
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 9.07490 (QuantReg: 12.26240) QuantErr: 12.26240 batch_time=0.46277
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 7.28481 (QuantReg: 12.45054) QuantErr: 12.45054 batch_time=0.43387
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 6.77181 (QuantReg: 12.39041) QuantErr: 12.39041 batch_time=0.70905
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 7.56019 (QuantReg: 12.50840) QuantErr: 12.50840 batch_time=0.43126
Train Epoch: 6 codebook_update_time=0.95561
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch6.pth ...
Done in 5.157s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch6.pth ...
Done in 9.926s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 7.497154237747193
quant_reg : 12.412101139068604
quant_err : 12.412101139068604
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 32.19315895372233
MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
MSRVTT_full_val/t2v_metrics/R10: 81.69014084507042
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.167002012072434
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.397369685249735
MSRVTT_full_val/v2t_metrics/R1: 36.21730382293762
MSRVTT_full_val/v2t_metrics/R5: 71.83098591549296
MSRVTT_full_val/v2t_metrics/R10: 85.71428571428571
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.2635814889336014
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.64016644917069
MSRVTT_full_test/t2v_metrics/R1: 11.57190635451505
MSRVTT_full_test/t2v_metrics/R5: 31.97324414715719
MSRVTT_full_test/t2v_metrics/R10: 46.38795986622073
MSRVTT_full_test/t2v_metrics/R50: 79.79933110367892
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 41.96521739130435
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.79480776901778
MSRVTT_full_test/v2t_metrics/R1: 12.876254180602007
MSRVTT_full_test/v2t_metrics/R5: 37.09030100334448
MSRVTT_full_test/v2t_metrics/R10: 51.30434782608695
MSRVTT_full_test/v2t_metrics/R50: 84.34782608695652
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.762040133779266
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.044775464060663
mnt_best : 25.79480776901778
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 6.93888 (QuantReg: 12.28726) QuantErr: 12.28726 batch_time=30.99012
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 7.27485 (QuantReg: 12.50452) QuantErr: 12.50452 batch_time=0.44402
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 7.76974 (QuantReg: 12.29829) QuantErr: 12.29829 batch_time=0.48129
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 6.50671 (QuantReg: 12.15404) QuantErr: 12.15404 batch_time=0.45313
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 7.17130 (QuantReg: 12.08091) QuantErr: 12.08091 batch_time=0.44617
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 6.87438 (QuantReg: 12.32976) QuantErr: 12.32976 batch_time=0.44975
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 5.35414 (QuantReg: 12.56063) QuantErr: 12.56063 batch_time=0.43981
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 7.24873 (QuantReg: 12.57069) QuantErr: 12.57069 batch_time=0.76445
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 9.66635 (QuantReg: 12.47940) QuantErr: 12.47940 batch_time=0.46530
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 6.91855 (QuantReg: 12.49908) QuantErr: 12.49908 batch_time=0.44992
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 7.88489 (QuantReg: 12.74257) QuantErr: 12.74257 batch_time=0.47250
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 7.01074 (QuantReg: 12.47091) QuantErr: 12.47091 batch_time=0.47692
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 6.63992 (QuantReg: 12.37680) QuantErr: 12.37680 batch_time=0.45889
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 6.59783 (QuantReg: 12.44419) QuantErr: 12.44419 batch_time=0.47996
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 6.14962 (QuantReg: 12.67053) QuantErr: 12.67053 batch_time=0.44771
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 6.11798 (QuantReg: 12.57398) QuantErr: 12.57398 batch_time=0.47351
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 8.49627 (QuantReg: 12.55269) QuantErr: 12.55269 batch_time=0.45787
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 5.77290 (QuantReg: 12.74343) QuantErr: 12.74343 batch_time=0.43700
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 7.21919 (QuantReg: 12.34632) QuantErr: 12.34632 batch_time=0.44400
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 6.89077 (QuantReg: 12.61974) QuantErr: 12.61974 batch_time=0.46004
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 8.04173 (QuantReg: 12.59745) QuantErr: 12.59745 batch_time=0.42481
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 6.51762 (QuantReg: 12.68763) QuantErr: 12.68763 batch_time=0.44838
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 5.82662 (QuantReg: 12.62302) QuantErr: 12.62302 batch_time=0.46808
Train Epoch: 7 codebook_update_time=0.99875
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch7.pth ...
Done in 4.478s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 6.896228610992432
quant_reg : 12.54117406463623
quant_err : 12.54117406463623
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 29.979879275653925
MSRVTT_full_val/t2v_metrics/R5: 66.19718309859155
MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.056338028169014
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.21198364268388
MSRVTT_full_val/v2t_metrics/R1: 32.19315895372233
MSRVTT_full_val/v2t_metrics/R5: 73.8430583501006
MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.633802816901408
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.47421328413947
MSRVTT_full_test/t2v_metrics/R1: 10.535117056856187
MSRVTT_full_test/t2v_metrics/R5: 32.0066889632107
MSRVTT_full_test/t2v_metrics/R10: 46.220735785953174
MSRVTT_full_test/t2v_metrics/R50: 79.4648829431438
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.13210702341137
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.978843280619966
MSRVTT_full_test/v2t_metrics/R1: 13.210702341137123
MSRVTT_full_test/v2t_metrics/R5: 36.92307692307692
MSRVTT_full_test/v2t_metrics/R10: 51.63879598662207
MSRVTT_full_test/v2t_metrics/R50: 83.84615384615384
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 36.65919732441471
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.31342963638325
mnt_best : 25.79480776901778
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 7.22019 (QuantReg: 12.15557) QuantErr: 12.15557 batch_time=33.27092
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 6.17785 (QuantReg: 12.43646) QuantErr: 12.43646 batch_time=0.42178
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 7.22998 (QuantReg: 12.38549) QuantErr: 12.38549 batch_time=1.28782
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 5.86753 (QuantReg: 12.86735) QuantErr: 12.86735 batch_time=0.43121
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 7.51360 (QuantReg: 12.53601) QuantErr: 12.53601 batch_time=0.49433
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 7.32752 (QuantReg: 12.49819) QuantErr: 12.49819 batch_time=0.46656
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 6.60464 (QuantReg: 12.70031) QuantErr: 12.70031 batch_time=1.83349
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 7.84239 (QuantReg: 12.41584) QuantErr: 12.41584 batch_time=0.42622
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 6.97365 (QuantReg: 12.61724) QuantErr: 12.61724 batch_time=0.42671
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 5.57214 (QuantReg: 12.60995) QuantErr: 12.60995 batch_time=0.50249
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 6.39321 (QuantReg: 12.73845) QuantErr: 12.73845 batch_time=0.45243
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 5.38301 (QuantReg: 12.52012) QuantErr: 12.52012 batch_time=0.45196
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 7.14389 (QuantReg: 12.43758) QuantErr: 12.43758 batch_time=0.44238
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 6.29462 (QuantReg: 13.01608) QuantErr: 13.01608 batch_time=0.43872
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 7.06885 (QuantReg: 12.57707) QuantErr: 12.57707 batch_time=0.43885
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 5.86382 (QuantReg: 12.51130) QuantErr: 12.51130 batch_time=0.43169
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 6.68648 (QuantReg: 12.49541) QuantErr: 12.49541 batch_time=0.43740
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 5.49521 (QuantReg: 12.60040) QuantErr: 12.60040 batch_time=0.44089
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 7.42338 (QuantReg: 12.94781) QuantErr: 12.94781 batch_time=0.44258
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 7.10308 (QuantReg: 12.67116) QuantErr: 12.67116 batch_time=0.48044
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 6.14804 (QuantReg: 12.75378) QuantErr: 12.75378 batch_time=0.46535
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 5.50959 (QuantReg: 12.79637) QuantErr: 12.79637 batch_time=0.41954
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 4.48604 (QuantReg: 12.61043) QuantErr: 12.61043 batch_time=0.46992
Train Epoch: 8 codebook_update_time=1.23016
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch8.pth ...
Done in 4.480s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 6.4480590171813965
quant_reg : 12.580614009857177
quant_err : 12.580614009857177
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 29.175050301810867
MSRVTT_full_val/t2v_metrics/R5: 66.19718309859155
MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.515090543259557
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.67753972581272
MSRVTT_full_val/v2t_metrics/R1: 31.58953722334004
MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.456740442655936
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.78804335794677
MSRVTT_full_test/t2v_metrics/R1: 10.568561872909699
MSRVTT_full_test/t2v_metrics/R5: 32.240802675585286
MSRVTT_full_test/t2v_metrics/R10: 45.35117056856188
MSRVTT_full_test/t2v_metrics/R50: 78.82943143812709
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.09397993311037
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.90787898458717
MSRVTT_full_test/v2t_metrics/R1: 13.143812709030101
MSRVTT_full_test/v2t_metrics/R5: 36.72240802675585
MSRVTT_full_test/v2t_metrics/R10: 51.67224080267559
MSRVTT_full_test/v2t_metrics/R50: 83.87959866220736
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.841638795986626
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.217066100022908
mnt_best : 25.79480776901778
not_improved_count: 2
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 7.88506 (QuantReg: 12.04969) QuantErr: 12.04969 batch_time=30.95169
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 5.92583 (QuantReg: 12.53598) QuantErr: 12.53598 batch_time=0.44683
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 7.57962 (QuantReg: 12.45527) QuantErr: 12.45527 batch_time=0.44787
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 6.08396 (QuantReg: 12.45638) QuantErr: 12.45638 batch_time=0.43487
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 5.71013 (QuantReg: 12.65966) QuantErr: 12.65966 batch_time=0.43390
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 6.71653 (QuantReg: 12.74411) QuantErr: 12.74411 batch_time=0.43963
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 6.50825 (QuantReg: 12.74573) QuantErr: 12.74573 batch_time=1.49929
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 5.47563 (QuantReg: 12.52899) QuantErr: 12.52899 batch_time=0.79829
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 6.82530 (QuantReg: 12.51484) QuantErr: 12.51484 batch_time=0.44205
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 5.18514 (QuantReg: 12.75487) QuantErr: 12.75487 batch_time=0.44074
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 5.31345 (QuantReg: 12.81301) QuantErr: 12.81301 batch_time=0.51774
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 6.48690 (QuantReg: 12.82190) QuantErr: 12.82190 batch_time=0.45007
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 6.11182 (QuantReg: 12.60709) QuantErr: 12.60709 batch_time=3.81412
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 6.98611 (QuantReg: 12.22885) QuantErr: 12.22885 batch_time=0.45639
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 5.94510 (QuantReg: 12.74417) QuantErr: 12.74417 batch_time=0.42817
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 7.36745 (QuantReg: 12.64758) QuantErr: 12.64758 batch_time=0.43158
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 5.24982 (QuantReg: 12.86215) QuantErr: 12.86215 batch_time=0.43290
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 6.62141 (QuantReg: 12.91276) QuantErr: 12.91276 batch_time=0.42894
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 4.72670 (QuantReg: 12.94839) QuantErr: 12.94839 batch_time=1.86866
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 5.63188 (QuantReg: 12.97427) QuantErr: 12.97427 batch_time=0.50367
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 6.56272 (QuantReg: 12.90164) QuantErr: 12.90164 batch_time=0.43615
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 5.82022 (QuantReg: 12.81933) QuantErr: 12.81933 batch_time=0.42890
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 5.46131 (QuantReg: 13.34725) QuantErr: 13.34725 batch_time=0.45245
Train Epoch: 9 codebook_update_time=1.27577
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch9.pth ...
Done in 4.899s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch9.pth ...
Done in 28.125s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 6.178337562561035
quant_reg : 12.69364302444458
quant_err : 12.69364302444458
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 29.577464788732396
MSRVTT_full_val/t2v_metrics/R5: 68.00804828973843
MSRVTT_full_val/t2v_metrics/R10: 78.87323943661971
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.396378269617706
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.13568610119686
MSRVTT_full_val/v2t_metrics/R1: 33.40040241448692
MSRVTT_full_val/v2t_metrics/R5: 74.04426559356136
MSRVTT_full_val/v2t_metrics/R10: 85.91549295774648
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.617706237424548
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.67212975630632
MSRVTT_full_test/t2v_metrics/R1: 12.006688963210703
MSRVTT_full_test/t2v_metrics/R5: 32.474916387959865
MSRVTT_full_test/t2v_metrics/R10: 45.38461538461539
MSRVTT_full_test/t2v_metrics/R50: 78.99665551839465
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 47.509698996655516
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.059134477812105
MSRVTT_full_test/v2t_metrics/R1: 13.979933110367893
MSRVTT_full_test/v2t_metrics/R5: 38.26086956521739
MSRVTT_full_test/v2t_metrics/R10: 53.47826086956522
MSRVTT_full_test/v2t_metrics/R50: 84.54849498327759
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.438294314381274
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.58292842602312
mnt_best : 26.059134477812105
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 5.80324 (QuantReg: 13.01300) QuantErr: 13.01300 batch_time=37.00398
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 4.98373 (QuantReg: 12.76316) QuantErr: 12.76316 batch_time=0.42678
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 6.11358 (QuantReg: 12.88920) QuantErr: 12.88920 batch_time=0.42729
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 6.00877 (QuantReg: 12.67776) QuantErr: 12.67776 batch_time=0.43582
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 6.94251 (QuantReg: 12.26119) QuantErr: 12.26119 batch_time=0.58164
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 5.68771 (QuantReg: 12.63270) QuantErr: 12.63270 batch_time=0.43372
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 6.42622 (QuantReg: 12.74110) QuantErr: 12.74110 batch_time=0.48654
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 5.46889 (QuantReg: 12.84187) QuantErr: 12.84187 batch_time=0.45065
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 7.34552 (QuantReg: 12.48493) QuantErr: 12.48493 batch_time=0.43240
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 7.87407 (QuantReg: 12.50287) QuantErr: 12.50287 batch_time=0.46578
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 5.30487 (QuantReg: 12.67660) QuantErr: 12.67660 batch_time=0.43724
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 5.91518 (QuantReg: 12.70086) QuantErr: 12.70086 batch_time=0.43650
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 6.34737 (QuantReg: 12.66086) QuantErr: 12.66086 batch_time=0.44332
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 5.11401 (QuantReg: 12.81498) QuantErr: 12.81498 batch_time=0.44242
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 4.69636 (QuantReg: 12.87346) QuantErr: 12.87346 batch_time=0.43283
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 5.71119 (QuantReg: 12.97631) QuantErr: 12.97631 batch_time=0.64298
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 5.27334 (QuantReg: 12.91722) QuantErr: 12.91722 batch_time=0.42847
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 6.33470 (QuantReg: 13.03710) QuantErr: 13.03710 batch_time=0.44713
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 5.89766 (QuantReg: 12.79606) QuantErr: 12.79606 batch_time=0.46550
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 5.59353 (QuantReg: 12.88054) QuantErr: 12.88054 batch_time=0.45343
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 6.35207 (QuantReg: 12.81186) QuantErr: 12.81186 batch_time=0.43033
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.80781 (QuantReg: 12.57023) QuantErr: 12.57023 batch_time=0.43694
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 4.45508 (QuantReg: 12.80535) QuantErr: 12.80535 batch_time=0.43534
Train Epoch: 10 codebook_update_time=1.12655
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch10.pth ...
Done in 5.002s
removing stale ckpt [epoch 9] [took 0.13s]
epoch : 10
loss : 5.773538927078247
quant_reg : 12.686797508239746
quant_err : 12.686797508239746
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 28.37022132796781
MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.181086519114688
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.48328288926259
MSRVTT_full_val/v2t_metrics/R1: 35.61368209255533
MSRVTT_full_val/v2t_metrics/R5: 71.62977867203219
MSRVTT_full_val/v2t_metrics/R10: 86.31790744466801
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.869215291750503
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.38612061200483
MSRVTT_full_test/t2v_metrics/R1: 11.605351170568563
MSRVTT_full_test/t2v_metrics/R5: 32.441471571906355
MSRVTT_full_test/t2v_metrics/R10: 45.25083612040134
MSRVTT_full_test/t2v_metrics/R50: 78.66220735785953
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.84280936454849
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.73130505478652
MSRVTT_full_test/v2t_metrics/R1: 13.010033444816054
MSRVTT_full_test/v2t_metrics/R5: 38.66220735785953
MSRVTT_full_test/v2t_metrics/R10: 54.180602006688964
MSRVTT_full_test/v2t_metrics/R50: 84.38127090301003
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.461371237458195
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.093287090384013
mnt_best : 26.059134477812105
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 5.80956 (QuantReg: 12.62844) QuantErr: 12.62844 batch_time=34.41549
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 4.24614 (QuantReg: 12.92875) QuantErr: 12.92875 batch_time=0.43612
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 6.10636 (QuantReg: 12.95179) QuantErr: 12.95179 batch_time=0.42159
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 6.01269 (QuantReg: 12.50086) QuantErr: 12.50086 batch_time=0.45845
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 5.75457 (QuantReg: 12.54735) QuantErr: 12.54735 batch_time=0.45393
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 5.16062 (QuantReg: 12.77864) QuantErr: 12.77864 batch_time=0.46421
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 5.51560 (QuantReg: 12.75253) QuantErr: 12.75253 batch_time=0.67832
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 5.25266 (QuantReg: 12.52953) QuantErr: 12.52953 batch_time=0.42482
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 6.25288 (QuantReg: 12.85794) QuantErr: 12.85794 batch_time=0.42795
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 5.46716 (QuantReg: 12.56282) QuantErr: 12.56282 batch_time=0.42981
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 6.04858 (QuantReg: 12.96050) QuantErr: 12.96050 batch_time=0.42355
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 6.53729 (QuantReg: 12.47635) QuantErr: 12.47635 batch_time=0.55226
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 5.53903 (QuantReg: 12.61777) QuantErr: 12.61777 batch_time=0.93085
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 5.42825 (QuantReg: 13.04634) QuantErr: 13.04634 batch_time=0.45273
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 5.60686 (QuantReg: 12.60651) QuantErr: 12.60651 batch_time=1.81795
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 5.66431 (QuantReg: 12.93976) QuantErr: 12.93976 batch_time=0.44208
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 6.29899 (QuantReg: 12.64462) QuantErr: 12.64462 batch_time=0.46505
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 5.36502 (QuantReg: 12.86254) QuantErr: 12.86254 batch_time=0.47346
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 6.37710 (QuantReg: 12.53551) QuantErr: 12.53551 batch_time=0.43628
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 5.71089 (QuantReg: 12.49591) QuantErr: 12.49591 batch_time=0.42000
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 5.81933 (QuantReg: 13.10112) QuantErr: 13.10112 batch_time=0.42504
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 6.09075 (QuantReg: 12.74083) QuantErr: 12.74083 batch_time=0.47600
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 5.49151 (QuantReg: 13.18763) QuantErr: 13.18763 batch_time=0.48625
Train Epoch: 11 codebook_update_time=1.12809
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch11.pth ...
Done in 14.436s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 5.532399850845337
quant_reg : 12.74368141555786
quant_err : 12.74368141555786
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 28.169014084507044
MSRVTT_full_val/t2v_metrics/R5: 65.79476861167002
MSRVTT_full_val/t2v_metrics/R10: 76.65995975855131
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.995975855130784
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.180775908139246
MSRVTT_full_val/v2t_metrics/R1: 33.40040241448692
MSRVTT_full_val/v2t_metrics/R5: 73.03822937625755
MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.056338028169014
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.53942929636464
MSRVTT_full_test/t2v_metrics/R1: 10.903010033444817
MSRVTT_full_test/t2v_metrics/R5: 31.73913043478261
MSRVTT_full_test/t2v_metrics/R10: 44.74916387959866
MSRVTT_full_test/t2v_metrics/R50: 77.49163879598662
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.59849498327759
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.925399068489554
MSRVTT_full_test/v2t_metrics/R1: 12.909698996655518
MSRVTT_full_test/v2t_metrics/R5: 37.19063545150502
MSRVTT_full_test/v2t_metrics/R10: 53.07692307692308
MSRVTT_full_test/v2t_metrics/R50: 84.24749163879599
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.63026755852843
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.42739494105359
mnt_best : 26.059134477812105
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 4.85163 (QuantReg: 12.86046) QuantErr: 12.86046 batch_time=39.47857
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 5.51196 (QuantReg: 12.81888) QuantErr: 12.81888 batch_time=0.41452
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 5.46875 (QuantReg: 12.70908) QuantErr: 12.70908 batch_time=0.43919
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 6.28325 (QuantReg: 12.51942) QuantErr: 12.51942 batch_time=0.43692
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 4.89320 (QuantReg: 12.67175) QuantErr: 12.67175 batch_time=0.42015
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 4.99552 (QuantReg: 12.80762) QuantErr: 12.80762 batch_time=0.41829
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 5.70627 (QuantReg: 12.72831) QuantErr: 12.72831 batch_time=0.44094
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 5.90601 (QuantReg: 12.95348) QuantErr: 12.95348 batch_time=0.43599
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 5.70583 (QuantReg: 12.50048) QuantErr: 12.50048 batch_time=0.42042
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 5.03456 (QuantReg: 12.83111) QuantErr: 12.83111 batch_time=0.44584
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 4.21780 (QuantReg: 12.41409) QuantErr: 12.41409 batch_time=0.43739
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 4.23376 (QuantReg: 12.78825) QuantErr: 12.78825 batch_time=0.64242
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 3.23299 (QuantReg: 12.94927) QuantErr: 12.94927 batch_time=0.44874
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 6.57112 (QuantReg: 12.62480) QuantErr: 12.62480 batch_time=0.41910
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 4.96964 (QuantReg: 13.04479) QuantErr: 13.04479 batch_time=0.44657
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 4.65857 (QuantReg: 13.11139) QuantErr: 13.11139 batch_time=0.44319
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 5.37976 (QuantReg: 13.22591) QuantErr: 13.22591 batch_time=1.23823
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 4.65683 (QuantReg: 13.01100) QuantErr: 13.01100 batch_time=0.42047
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 5.01269 (QuantReg: 12.93755) QuantErr: 12.93755 batch_time=0.41694
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 4.35198 (QuantReg: 12.77905) QuantErr: 12.77905 batch_time=0.41926
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 5.69386 (QuantReg: 12.89334) QuantErr: 12.89334 batch_time=0.43055
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 5.17093 (QuantReg: 13.19380) QuantErr: 13.19380 batch_time=0.79115
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 5.11430 (QuantReg: 12.72889) QuantErr: 12.72889 batch_time=0.42895
Train Epoch: 12 codebook_update_time=1.01701
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch12.pth ...
Done in 23.294s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch12.pth ...
Done in 28.096s
removing stale ckpt [epoch 11] [took 0.02s]
epoch : 12
loss : 5.208314287185669
quant_reg : 12.785920459747315
quant_err : 12.785920459747315
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 31.388329979879277
MSRVTT_full_val/t2v_metrics/R5: 70.62374245472837
MSRVTT_full_val/t2v_metrics/R10: 82.69617706237425
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.806841046277666
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.80694796812396
MSRVTT_full_val/v2t_metrics/R1: 37.82696177062374
MSRVTT_full_val/v2t_metrics/R5: 76.05633802816901
MSRVTT_full_val/v2t_metrics/R10: 87.72635814889335
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.388329979879276
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 63.19591273567722
MSRVTT_full_test/t2v_metrics/R1: 11.872909698996656
MSRVTT_full_test/t2v_metrics/R5: 34.91638795986622
MSRVTT_full_test/t2v_metrics/R10: 49.531772575250834
MSRVTT_full_test/t2v_metrics/R50: 80.76923076923077
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 42.332441471571904
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.38357306223415
MSRVTT_full_test/v2t_metrics/R1: 14.749163879598662
MSRVTT_full_test/v2t_metrics/R5: 40.836120401337794
MSRVTT_full_test/v2t_metrics/R10: 55.852842809364546
MSRVTT_full_test/v2t_metrics/R50: 85.65217391304348
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.8061872909699
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.28140188168761
mnt_best : 27.38357306223415
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 5.49318 (QuantReg: 12.92590) QuantErr: 12.92590 batch_time=33.74323
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 5.53372 (QuantReg: 12.80486) QuantErr: 12.80486 batch_time=0.44357
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 4.21354 (QuantReg: 12.73759) QuantErr: 12.73759 batch_time=0.43085
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 5.83326 (QuantReg: 12.72560) QuantErr: 12.72560 batch_time=0.43599
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 3.99604 (QuantReg: 12.67945) QuantErr: 12.67945 batch_time=0.43612
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 3.71657 (QuantReg: 13.27030) QuantErr: 13.27030 batch_time=0.45471
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 4.54245 (QuantReg: 12.72006) QuantErr: 12.72006 batch_time=0.43870
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 4.48123 (QuantReg: 12.82285) QuantErr: 12.82285 batch_time=0.43077
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 5.07666 (QuantReg: 12.51180) QuantErr: 12.51180 batch_time=0.42638
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 4.78694 (QuantReg: 12.86067) QuantErr: 12.86067 batch_time=0.42282
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 4.98178 (QuantReg: 12.76111) QuantErr: 12.76111 batch_time=0.43454
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 5.24371 (QuantReg: 12.78134) QuantErr: 12.78134 batch_time=0.45948
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 5.21429 (QuantReg: 12.59617) QuantErr: 12.59617 batch_time=0.44129
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 4.90764 (QuantReg: 12.73357) QuantErr: 12.73357 batch_time=0.42249
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.75816 (QuantReg: 12.79078) QuantErr: 12.79078 batch_time=0.48660
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 5.34505 (QuantReg: 12.59012) QuantErr: 12.59012 batch_time=0.44014
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 4.75594 (QuantReg: 13.03182) QuantErr: 13.03182 batch_time=0.81261
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 5.06536 (QuantReg: 12.75699) QuantErr: 12.75699 batch_time=0.43014
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 4.93186 (QuantReg: 12.94472) QuantErr: 12.94472 batch_time=0.42413
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 5.24353 (QuantReg: 12.72422) QuantErr: 12.72422 batch_time=0.42343
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 4.58857 (QuantReg: 13.25368) QuantErr: 13.25368 batch_time=0.43810
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 6.26059 (QuantReg: 12.86953) QuantErr: 12.86953 batch_time=0.45878
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 4.39982 (QuantReg: 12.85880) QuantErr: 12.85880 batch_time=0.43417
Train Epoch: 13 codebook_update_time=1.17270
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch13.pth ...
Done in 3.972s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 5.072538291931152
quant_reg : 12.848708503723145
quant_err : 12.848708503723145
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 34.60764587525151
MSRVTT_full_val/t2v_metrics/R5: 69.21529175050301
MSRVTT_full_val/t2v_metrics/R10: 82.09255533199195
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.305835010060363
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 58.15128224821894
MSRVTT_full_val/v2t_metrics/R1: 37.02213279678068
MSRVTT_full_val/v2t_metrics/R5: 75.85513078470825
MSRVTT_full_val/v2t_metrics/R10: 87.92756539235413
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.633802816901408
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.737009705923704
MSRVTT_full_test/t2v_metrics/R1: 11.672240802675585
MSRVTT_full_test/t2v_metrics/R5: 33.67892976588629
MSRVTT_full_test/t2v_metrics/R10: 46.82274247491639
MSRVTT_full_test/t2v_metrics/R50: 79.29765886287625
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.02742474916388
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.403193351241686
MSRVTT_full_test/v2t_metrics/R1: 14.414715719063546
MSRVTT_full_test/v2t_metrics/R5: 40.5685618729097
MSRVTT_full_test/v2t_metrics/R10: 56.08695652173913
MSRVTT_full_test/v2t_metrics/R50: 85.35117056856187
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.19364548494983
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.010013420043464
mnt_best : 27.38357306223415
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 5.81442 (QuantReg: 12.71445) QuantErr: 12.71445 batch_time=34.18989
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 3.98032 (QuantReg: 13.13276) QuantErr: 13.13276 batch_time=0.42828
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 4.25002 (QuantReg: 12.95084) QuantErr: 12.95084 batch_time=0.43563
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 3.97096 (QuantReg: 12.75461) QuantErr: 12.75461 batch_time=0.42741
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 5.43122 (QuantReg: 12.95874) QuantErr: 12.95874 batch_time=0.44316
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 4.78748 (QuantReg: 12.86394) QuantErr: 12.86394 batch_time=0.43130
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 4.96865 (QuantReg: 12.60582) QuantErr: 12.60582 batch_time=0.44384
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 5.10833 (QuantReg: 12.41806) QuantErr: 12.41806 batch_time=0.44754
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 4.16652 (QuantReg: 13.04499) QuantErr: 13.04499 batch_time=0.46960
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 3.80807 (QuantReg: 13.31240) QuantErr: 13.31240 batch_time=0.42260
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 3.78466 (QuantReg: 13.02935) QuantErr: 13.02935 batch_time=0.44390
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 5.60844 (QuantReg: 13.23104) QuantErr: 13.23104 batch_time=0.43768
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 4.90777 (QuantReg: 12.92118) QuantErr: 12.92118 batch_time=2.83851
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 5.55328 (QuantReg: 12.65683) QuantErr: 12.65683 batch_time=0.41612
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 5.03727 (QuantReg: 13.01292) QuantErr: 13.01292 batch_time=0.41383
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 5.45262 (QuantReg: 12.88829) QuantErr: 12.88829 batch_time=0.41756
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 5.33051 (QuantReg: 12.57675) QuantErr: 12.57675 batch_time=0.42330
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 5.96692 (QuantReg: 12.74470) QuantErr: 12.74470 batch_time=0.42790
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 5.40900 (QuantReg: 13.14830) QuantErr: 13.14830 batch_time=0.43348
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 5.50296 (QuantReg: 12.94911) QuantErr: 12.94911 batch_time=0.41804
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 4.63731 (QuantReg: 13.01787) QuantErr: 13.01787 batch_time=0.45087
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 4.56155 (QuantReg: 13.21176) QuantErr: 13.21176 batch_time=0.44149
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 4.98631 (QuantReg: 13.08484) QuantErr: 13.08484 batch_time=0.43788
Train Epoch: 14 codebook_update_time=0.97665
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch14.pth ...
Done in 3.852s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 4.858015194892883
quant_reg : 12.87810097503662
quant_err : 12.87810097503662
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 32.19315895372233
MSRVTT_full_val/t2v_metrics/R5: 68.00804828973843
MSRVTT_full_val/t2v_metrics/R10: 81.69014084507042
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.020120724346077
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.341860427251135
MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
MSRVTT_full_val/v2t_metrics/R5: 76.45875251509054
MSRVTT_full_val/v2t_metrics/R10: 88.12877263581488
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.494969818913481
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.259101346122165
MSRVTT_full_test/t2v_metrics/R1: 12.006688963210703
MSRVTT_full_test/t2v_metrics/R5: 34.88294314381271
MSRVTT_full_test/t2v_metrics/R10: 48.32775919732441
MSRVTT_full_test/t2v_metrics/R50: 80.2675585284281
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.76254180602007
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.252792718717174
MSRVTT_full_test/v2t_metrics/R1: 14.68227424749164
MSRVTT_full_test/v2t_metrics/R5: 41.03678929765886
MSRVTT_full_test/v2t_metrics/R10: 56.48829431438127
MSRVTT_full_test/v2t_metrics/R50: 86.38795986622074
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 30.851003344481605
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.40721590551757
mnt_best : 27.38357306223415
not_improved_count: 2
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 5.08349 (QuantReg: 12.76031) QuantErr: 12.76031 batch_time=38.16022
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 4.10501 (QuantReg: 13.14082) QuantErr: 13.14082 batch_time=0.43143
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 5.41930 (QuantReg: 12.90268) QuantErr: 12.90268 batch_time=0.43860
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 5.42652 (QuantReg: 12.57979) QuantErr: 12.57979 batch_time=0.67289
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 4.80126 (QuantReg: 12.82047) QuantErr: 12.82047 batch_time=0.45392
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 5.18442 (QuantReg: 12.86601) QuantErr: 12.86601 batch_time=0.42380
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 5.60370 (QuantReg: 12.65144) QuantErr: 12.65144 batch_time=0.43655
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 5.13353 (QuantReg: 12.82859) QuantErr: 12.82859 batch_time=0.42893
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 4.07025 (QuantReg: 13.11568) QuantErr: 13.11568 batch_time=0.47268
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 4.92883 (QuantReg: 12.62067) QuantErr: 12.62067 batch_time=0.45028
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 3.84878 (QuantReg: 12.94265) QuantErr: 12.94265 batch_time=0.44842
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 4.50434 (QuantReg: 12.78616) QuantErr: 12.78616 batch_time=0.44453
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 4.15734 (QuantReg: 13.06217) QuantErr: 13.06217 batch_time=0.43088
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 4.92104 (QuantReg: 12.91008) QuantErr: 12.91008 batch_time=1.49783
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 4.21350 (QuantReg: 12.82538) QuantErr: 12.82538 batch_time=0.43078
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 5.09935 (QuantReg: 12.79439) QuantErr: 12.79439 batch_time=0.43652
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 4.52236 (QuantReg: 13.02978) QuantErr: 13.02978 batch_time=0.42684
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 4.90993 (QuantReg: 12.65930) QuantErr: 12.65930 batch_time=0.55415
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 4.70365 (QuantReg: 13.07147) QuantErr: 13.07147 batch_time=0.43399
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 4.22555 (QuantReg: 13.07132) QuantErr: 13.07132 batch_time=0.44017
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 3.23033 (QuantReg: 13.07854) QuantErr: 13.07854 batch_time=0.44366
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 4.04429 (QuantReg: 12.84018) QuantErr: 12.84018 batch_time=0.44236
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 4.87335 (QuantReg: 12.87399) QuantErr: 12.87399 batch_time=0.42713
Train Epoch: 15 codebook_update_time=1.05126
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_L3/checkpoint-epoch15.pth ...
Done in 4.403s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 4.706735206604004
quant_reg : 12.881005752563476
quant_err : 12.881005752563476
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
MSRVTT_full_val/t2v_metrics/R5: 66.19718309859155
MSRVTT_full_val/t2v_metrics/R10: 82.49496981891348
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005