-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full_t0.1.txt
3309 lines (3309 loc) · 233 KB
/
HCQ_MSRVTT_full_t0.1.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 825.5565114021301 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 56.00535583496094 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 280.3361213207245 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 166.00615072250366 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch0.pth ...
Done in 2.689s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch0.pth ...
Done in 4.603s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
MSRVTT_full_val/t2v_metrics/MedR: 252.0
MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
MSRVTT_full_val/v2t_metrics/MedR: 243.0
MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
MSRVTT_full_test/t2v_metrics/MedR: 1515.0
MSRVTT_full_test/t2v_metrics/MeanR: 1498.5565217391304
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1471.5
MSRVTT_full_test/v2t_metrics/MeanR: 1495.3264214046824
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
mnt_best : 0.12154652794863813
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.73770 (QuantReg: 22.44554) QuantErr: 22.44554 batch_time=29.43825
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.71574 (QuantReg: 22.52802) QuantErr: 22.52802 batch_time=1.04319
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.12892 (QuantReg: 22.61871) QuantErr: 22.61871 batch_time=0.49833
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 7.01682 (QuantReg: 22.65707) QuantErr: 22.65707 batch_time=1.98183
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.45240 (QuantReg: 22.64140) QuantErr: 22.64140 batch_time=0.52782
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.23941 (QuantReg: 22.62227) QuantErr: 22.62227 batch_time=0.50463
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.21231 (QuantReg: 22.62445) QuantErr: 22.62445 batch_time=0.51893
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.91538 (QuantReg: 22.62173) QuantErr: 22.62173 batch_time=2.18201
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.59422 (QuantReg: 22.66101) QuantErr: 22.66101 batch_time=0.49413
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.74538 (QuantReg: 22.64662) QuantErr: 22.64662 batch_time=0.49244
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.42443 (QuantReg: 22.67731) QuantErr: 22.67731 batch_time=0.51193
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.13228 (QuantReg: 22.66573) QuantErr: 22.66573 batch_time=0.49358
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.11523 (QuantReg: 22.64967) QuantErr: 22.64967 batch_time=0.51159
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.96043 (QuantReg: 22.66985) QuantErr: 22.66985 batch_time=0.49095
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 5.00170 (QuantReg: 22.66203) QuantErr: 22.66203 batch_time=0.50808
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 5.21806 (QuantReg: 22.64056) QuantErr: 22.64056 batch_time=0.48666
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 5.00095 (QuantReg: 22.63998) QuantErr: 22.63998 batch_time=0.50429
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.74884 (QuantReg: 22.61730) QuantErr: 22.61730 batch_time=0.48899
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.80275 (QuantReg: 22.60291) QuantErr: 22.60291 batch_time=0.50597
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.51238 (QuantReg: 22.61337) QuantErr: 22.61337 batch_time=0.48762
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.74374 (QuantReg: 22.64421) QuantErr: 22.64421 batch_time=0.49285
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.59383 (QuantReg: 22.65733) QuantErr: 22.65733 batch_time=1.13534
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.27732 (QuantReg: 22.61850) QuantErr: 22.61850 batch_time=0.52516
Train Epoch: 1 codebook_update_time=2.06193
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch1.pth ...
Done in 4.082s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch1.pth ...
Done in 8.051s
epoch : 1
loss : 5.572731321334839
quant_reg : 22.626732887268066
quant_err : 22.626732887268066
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 17.10261569416499
MSRVTT_full_val/t2v_metrics/R5: 47.88732394366197
MSRVTT_full_val/t2v_metrics/R10: 64.98993963782696
MSRVTT_full_val/t2v_metrics/R50: 94.56740442655935
MSRVTT_full_val/t2v_metrics/MedR: 6.0
MSRVTT_full_val/t2v_metrics/MeanR: 14.891348088531187
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 37.61632923755842
MSRVTT_full_val/v2t_metrics/R1: 19.114688128772634
MSRVTT_full_val/v2t_metrics/R5: 54.12474849094568
MSRVTT_full_val/v2t_metrics/R10: 68.61167002012073
MSRVTT_full_val/v2t_metrics/R50: 94.36619718309859
MSRVTT_full_val/v2t_metrics/MedR: 5.0
MSRVTT_full_val/v2t_metrics/MeanR: 13.03420523138833
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 41.40508679467761
MSRVTT_full_test/t2v_metrics/R1: 5.48494983277592
MSRVTT_full_test/t2v_metrics/R5: 18.327759197324415
MSRVTT_full_test/t2v_metrics/R10: 28.896321070234112
MSRVTT_full_test/t2v_metrics/R50: 64.31438127090301
MSRVTT_full_test/t2v_metrics/MedR: 27.0
MSRVTT_full_test/t2v_metrics/MeanR: 78.23979933110368
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 14.268386382178925
MSRVTT_full_test/v2t_metrics/R1: 6.454849498327759
MSRVTT_full_test/v2t_metrics/R5: 21.505016722408026
MSRVTT_full_test/v2t_metrics/R10: 32.374581939799334
MSRVTT_full_test/v2t_metrics/R50: 68.19397993311037
MSRVTT_full_test/v2t_metrics/MedR: 23.0
MSRVTT_full_test/v2t_metrics/MeanR: 69.4685618729097
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.5022574452145
mnt_best : 14.268386382178925
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.09026 (QuantReg: 11.12673) QuantErr: 11.12673 batch_time=39.91298
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.29285 (QuantReg: 11.10795) QuantErr: 11.10795 batch_time=0.53410
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.21585 (QuantReg: 11.39646) QuantErr: 11.39646 batch_time=0.52495
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.09998 (QuantReg: 11.52046) QuantErr: 11.52046 batch_time=0.50162
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.15817 (QuantReg: 11.67440) QuantErr: 11.67440 batch_time=0.49016
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.08800 (QuantReg: 11.34377) QuantErr: 11.34377 batch_time=0.50613
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.40115 (QuantReg: 11.81781) QuantErr: 11.81781 batch_time=0.51506
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.01411 (QuantReg: 11.80146) QuantErr: 11.80146 batch_time=0.51150
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.47492 (QuantReg: 11.98538) QuantErr: 11.98538 batch_time=0.48719
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.94307 (QuantReg: 11.95199) QuantErr: 11.95199 batch_time=0.49832
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.94128 (QuantReg: 11.86184) QuantErr: 11.86184 batch_time=0.48847
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.81122 (QuantReg: 11.91259) QuantErr: 11.91259 batch_time=0.71582
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.63881 (QuantReg: 11.97691) QuantErr: 11.97691 batch_time=0.53155
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.85640 (QuantReg: 12.16196) QuantErr: 12.16196 batch_time=0.54048
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.68790 (QuantReg: 12.32705) QuantErr: 12.32705 batch_time=0.53406
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.77296 (QuantReg: 12.71255) QuantErr: 12.71255 batch_time=0.55122
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.86028 (QuantReg: 12.35546) QuantErr: 12.35546 batch_time=0.52024
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.89819 (QuantReg: 12.63291) QuantErr: 12.63291 batch_time=0.52697
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.50343 (QuantReg: 12.29199) QuantErr: 12.29199 batch_time=0.49445
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.61492 (QuantReg: 12.78368) QuantErr: 12.78368 batch_time=0.50620
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.34665 (QuantReg: 12.64008) QuantErr: 12.64008 batch_time=0.54852
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 4.23856 (QuantReg: 12.53615) QuantErr: 12.53615 batch_time=0.49521
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.88702 (QuantReg: 13.02143) QuantErr: 13.02143 batch_time=0.50886
Train Epoch: 2 codebook_update_time=1.77663
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch2.pth ...
Done in 21.386s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch2.pth ...
Done in 26.864s
removing stale ckpt [epoch 1] [took 0.36s]
removing stale ckpt [epoch 0] [took 0.12s]
epoch : 2
loss : 3.952354695320129
quant_reg : 12.031916660308838
quant_err : 12.031916660308838
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 20.120724346076457
MSRVTT_full_val/t2v_metrics/R5: 54.32595573440644
MSRVTT_full_val/t2v_metrics/R10: 70.82494969818913
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.736418511066399
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 42.61989942043538
MSRVTT_full_val/v2t_metrics/R1: 25.35211267605634
MSRVTT_full_val/v2t_metrics/R5: 60.160965794768615
MSRVTT_full_val/v2t_metrics/R10: 78.47082494969818
MSRVTT_full_val/v2t_metrics/R50: 95.97585513078471
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 10.362173038229376
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 49.28094850891742
MSRVTT_full_test/t2v_metrics/R1: 7.3244147157190636
MSRVTT_full_test/t2v_metrics/R5: 23.244147157190636
MSRVTT_full_test/t2v_metrics/R10: 36.32107023411371
MSRVTT_full_test/t2v_metrics/R50: 71.00334448160535
MSRVTT_full_test/t2v_metrics/MedR: 20.0
MSRVTT_full_test/t2v_metrics/MeanR: 61.6494983277592
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.354746603869867
MSRVTT_full_test/v2t_metrics/R1: 8.729096989966555
MSRVTT_full_test/v2t_metrics/R5: 26.923076923076923
MSRVTT_full_test/v2t_metrics/R10: 40.0
MSRVTT_full_test/v2t_metrics/R50: 75.01672240802675
MSRVTT_full_test/v2t_metrics/MedR: 16.0
MSRVTT_full_test/v2t_metrics/MeanR: 52.44046822742475
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.10496651594879
mnt_best : 18.354746603869867
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.37737 (QuantReg: 10.06279) QuantErr: 10.06279 batch_time=30.63429
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.68980 (QuantReg: 10.37116) QuantErr: 10.37116 batch_time=0.52137
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.97268 (QuantReg: 10.54015) QuantErr: 10.54015 batch_time=0.49641
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.29520 (QuantReg: 9.73119) QuantErr: 9.73119 batch_time=0.57759
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.64717 (QuantReg: 10.25964) QuantErr: 10.25964 batch_time=0.51855
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.17138 (QuantReg: 10.34797) QuantErr: 10.34797 batch_time=0.52937
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.32736 (QuantReg: 10.43727) QuantErr: 10.43727 batch_time=1.74391
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.55277 (QuantReg: 10.26670) QuantErr: 10.26670 batch_time=0.50670
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.41212 (QuantReg: 10.19809) QuantErr: 10.19809 batch_time=0.52065
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.27094 (QuantReg: 10.26163) QuantErr: 10.26163 batch_time=0.50218
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.44696 (QuantReg: 10.30338) QuantErr: 10.30338 batch_time=0.53134
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.04033 (QuantReg: 10.47595) QuantErr: 10.47595 batch_time=0.51319
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.55742 (QuantReg: 10.77750) QuantErr: 10.77750 batch_time=0.50156
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.39430 (QuantReg: 10.77383) QuantErr: 10.77383 batch_time=0.50735
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.01198 (QuantReg: 10.57118) QuantErr: 10.57118 batch_time=0.53221
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.93611 (QuantReg: 11.03334) QuantErr: 11.03334 batch_time=0.49669
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.97524 (QuantReg: 10.60830) QuantErr: 10.60830 batch_time=0.49548
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.59445 (QuantReg: 10.55796) QuantErr: 10.55796 batch_time=0.52496
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.09915 (QuantReg: 10.54907) QuantErr: 10.54907 batch_time=0.55437
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.41932 (QuantReg: 10.78624) QuantErr: 10.78624 batch_time=0.49309
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.08494 (QuantReg: 10.87619) QuantErr: 10.87619 batch_time=0.51717
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.91045 (QuantReg: 10.87408) QuantErr: 10.87408 batch_time=0.52304
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.16005 (QuantReg: 10.90548) QuantErr: 10.90548 batch_time=0.51729
Train Epoch: 3 codebook_update_time=1.66114
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch3.pth ...
Done in 5.387s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch3.pth ...
Done in 10.381s
removing stale ckpt [epoch 2] [took 0.06s]
epoch : 3
loss : 3.4093528327941893
quant_reg : 10.546623703002929
quant_err : 10.546623703002929
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 23.541247484909455
MSRVTT_full_val/t2v_metrics/R5: 58.95372233400403
MSRVTT_full_val/t2v_metrics/R10: 73.8430583501006
MSRVTT_full_val/t2v_metrics/R50: 95.97585513078471
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.79476861167002
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.79686860402423
MSRVTT_full_val/v2t_metrics/R1: 25.150905432595575
MSRVTT_full_val/v2t_metrics/R5: 65.59356136820925
MSRVTT_full_val/v2t_metrics/R10: 80.28169014084507
MSRVTT_full_val/v2t_metrics/R50: 96.78068410462777
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.112676056338028
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 50.973420918388776
MSRVTT_full_test/t2v_metrics/R1: 8.82943143812709
MSRVTT_full_test/t2v_metrics/R5: 26.287625418060202
MSRVTT_full_test/t2v_metrics/R10: 38.896321070234116
MSRVTT_full_test/t2v_metrics/R50: 73.87959866220736
MSRVTT_full_test/t2v_metrics/MedR: 17.0
MSRVTT_full_test/t2v_metrics/MeanR: 56.42809364548495
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.822404275177053
MSRVTT_full_test/v2t_metrics/R1: 9.698996655518394
MSRVTT_full_test/v2t_metrics/R5: 29.39799331103679
MSRVTT_full_test/v2t_metrics/R10: 44.147157190635454
MSRVTT_full_test/v2t_metrics/R50: 78.69565217391305
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 49.760200668896324
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.262108643917717
mnt_best : 20.822404275177053
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.68439 (QuantReg: 9.69773) QuantErr: 9.69773 batch_time=32.54195
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.26771 (QuantReg: 9.77025) QuantErr: 9.77025 batch_time=0.51274
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.89415 (QuantReg: 9.81610) QuantErr: 9.81610 batch_time=0.49554
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.08339 (QuantReg: 9.75113) QuantErr: 9.75113 batch_time=0.72861
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.17119 (QuantReg: 9.83648) QuantErr: 9.83648 batch_time=0.49895
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.44674 (QuantReg: 9.95036) QuantErr: 9.95036 batch_time=0.64911
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.36930 (QuantReg: 9.99620) QuantErr: 9.99620 batch_time=0.48922
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 3.24072 (QuantReg: 10.01899) QuantErr: 10.01899 batch_time=1.82407
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.99840 (QuantReg: 9.85213) QuantErr: 9.85213 batch_time=0.49726
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.01096 (QuantReg: 9.85825) QuantErr: 9.85825 batch_time=1.74516
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.19425 (QuantReg: 10.04464) QuantErr: 10.04464 batch_time=0.48531
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.16859 (QuantReg: 9.86044) QuantErr: 9.86044 batch_time=0.52387
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.01676 (QuantReg: 9.85894) QuantErr: 9.85894 batch_time=0.53163
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 3.25720 (QuantReg: 10.24677) QuantErr: 10.24677 batch_time=0.55261
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.18467 (QuantReg: 10.33797) QuantErr: 10.33797 batch_time=0.51991
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 3.37325 (QuantReg: 10.50442) QuantErr: 10.50442 batch_time=0.50148
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.34100 (QuantReg: 10.18457) QuantErr: 10.18457 batch_time=0.51430
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 3.17873 (QuantReg: 10.46721) QuantErr: 10.46721 batch_time=0.50082
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 3.11876 (QuantReg: 10.16031) QuantErr: 10.16031 batch_time=0.51672
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.90681 (QuantReg: 10.15817) QuantErr: 10.15817 batch_time=0.48763
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.87286 (QuantReg: 10.54223) QuantErr: 10.54223 batch_time=1.40970
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 3.12201 (QuantReg: 10.02069) QuantErr: 10.02069 batch_time=0.50345
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.94202 (QuantReg: 10.15910) QuantErr: 10.15910 batch_time=0.53462
Train Epoch: 4 codebook_update_time=1.97003
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch4.pth ...
Done in 4.858s
removing stale ckpt [epoch 3] [took 0.05s]
epoch : 4
loss : 3.1139764575958253
quant_reg : 10.087905838012695
quant_err : 10.087905838012695
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 22.736418511066397
MSRVTT_full_val/t2v_metrics/R5: 59.95975855130785
MSRVTT_full_val/t2v_metrics/R10: 73.44064386317908
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.774647887323944
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.434360482866055
MSRVTT_full_val/v2t_metrics/R1: 28.571428571428573
MSRVTT_full_val/v2t_metrics/R5: 66.59959758551308
MSRVTT_full_val/v2t_metrics/R10: 79.47686116700201
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.589537223340042
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 53.27802155244179
MSRVTT_full_test/t2v_metrics/R1: 8.060200668896321
MSRVTT_full_test/t2v_metrics/R5: 26.02006688963211
MSRVTT_full_test/t2v_metrics/R10: 37.95986622073578
MSRVTT_full_test/t2v_metrics/R50: 73.17725752508362
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 58.610033444816054
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.9676204108203
MSRVTT_full_test/v2t_metrics/R1: 9.498327759197325
MSRVTT_full_test/v2t_metrics/R5: 30.334448160535118
MSRVTT_full_test/v2t_metrics/R10: 44.381270903010034
MSRVTT_full_test/v2t_metrics/R50: 79.66555183946488
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 48.73177257525084
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.38447702281215
mnt_best : 20.822404275177053
not_improved_count: 1
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.13990 (QuantReg: 9.88596) QuantErr: 9.88596 batch_time=32.15676
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.10803 (QuantReg: 9.98583) QuantErr: 9.98583 batch_time=0.52725
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 3.30469 (QuantReg: 9.74473) QuantErr: 9.74473 batch_time=0.50052
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.57423 (QuantReg: 9.97383) QuantErr: 9.97383 batch_time=0.53020
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.93654 (QuantReg: 9.62316) QuantErr: 9.62316 batch_time=0.90989
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 3.03321 (QuantReg: 10.19886) QuantErr: 10.19886 batch_time=0.50889
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.85704 (QuantReg: 9.70779) QuantErr: 9.70779 batch_time=1.23671
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.09554 (QuantReg: 9.61354) QuantErr: 9.61354 batch_time=0.48864
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.70750 (QuantReg: 9.91764) QuantErr: 9.91764 batch_time=0.50003
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 3.09703 (QuantReg: 9.80161) QuantErr: 9.80161 batch_time=0.50581
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.86978 (QuantReg: 9.90972) QuantErr: 9.90972 batch_time=0.53417
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.87661 (QuantReg: 9.86893) QuantErr: 9.86893 batch_time=0.49897
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.91789 (QuantReg: 10.18225) QuantErr: 10.18225 batch_time=0.66242
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.36128 (QuantReg: 10.02870) QuantErr: 10.02870 batch_time=0.82336
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.84728 (QuantReg: 10.15547) QuantErr: 10.15547 batch_time=0.50563
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.81316 (QuantReg: 10.07198) QuantErr: 10.07198 batch_time=0.51897
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.81374 (QuantReg: 10.36751) QuantErr: 10.36751 batch_time=0.49567
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 3.03981 (QuantReg: 9.84819) QuantErr: 9.84819 batch_time=0.54878
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.67214 (QuantReg: 10.04933) QuantErr: 10.04933 batch_time=0.50429
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.96338 (QuantReg: 9.80186) QuantErr: 9.80186 batch_time=0.61354
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.83187 (QuantReg: 10.02945) QuantErr: 10.02945 batch_time=0.49724
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.47673 (QuantReg: 10.10045) QuantErr: 10.10045 batch_time=0.49132
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.74514 (QuantReg: 9.80353) QuantErr: 9.80353 batch_time=0.67664
Train Epoch: 5 codebook_update_time=1.66572
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch5.pth ...
Done in 4.582s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch5.pth ...
Done in 10.574s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 2.8581816635131836
quant_reg : 9.902129875183105
quant_err : 9.902129875183105
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 24.346076458752513
MSRVTT_full_val/t2v_metrics/R5: 61.77062374245473
MSRVTT_full_val/t2v_metrics/R10: 76.86116700201207
MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.269617706237424
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.71237609943349
MSRVTT_full_val/v2t_metrics/R1: 30.18108651911469
MSRVTT_full_val/v2t_metrics/R5: 67.20321931589537
MSRVTT_full_val/v2t_metrics/R10: 81.89134808853119
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.327967806841047
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.969399118376025
MSRVTT_full_test/t2v_metrics/R1: 9.063545150501673
MSRVTT_full_test/t2v_metrics/R5: 28.42809364548495
MSRVTT_full_test/t2v_metrics/R10: 41.070234113712374
MSRVTT_full_test/t2v_metrics/R50: 76.08695652173913
MSRVTT_full_test/t2v_metrics/MedR: 15.5
MSRVTT_full_test/t2v_metrics/MeanR: 51.94531772575251
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.954539870762147
MSRVTT_full_test/v2t_metrics/R1: 10.167224080267559
MSRVTT_full_test/v2t_metrics/R5: 33.17725752508361
MSRVTT_full_test/v2t_metrics/R10: 47.22408026755853
MSRVTT_full_test/v2t_metrics/R50: 80.73578595317726
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 44.60200668896321
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.16143827105787
mnt_best : 21.954539870762147
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.99657 (QuantReg: 9.47920) QuantErr: 9.47920 batch_time=38.54350
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.89840 (QuantReg: 9.21438) QuantErr: 9.21438 batch_time=0.48793
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.52199 (QuantReg: 9.54584) QuantErr: 9.54584 batch_time=1.16960
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.63163 (QuantReg: 9.62455) QuantErr: 9.62455 batch_time=0.58165
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.32571 (QuantReg: 9.48413) QuantErr: 9.48413 batch_time=0.52311
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.46689 (QuantReg: 9.85377) QuantErr: 9.85377 batch_time=0.49950
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.50733 (QuantReg: 9.97149) QuantErr: 9.97149 batch_time=0.52004
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.34401 (QuantReg: 9.63919) QuantErr: 9.63919 batch_time=0.52736
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 3.00869 (QuantReg: 9.82076) QuantErr: 9.82076 batch_time=0.49763
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.57704 (QuantReg: 9.75625) QuantErr: 9.75625 batch_time=0.54543
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.64799 (QuantReg: 9.21709) QuantErr: 9.21709 batch_time=0.50032
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.53533 (QuantReg: 9.83111) QuantErr: 9.83111 batch_time=0.49501
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.49441 (QuantReg: 9.65105) QuantErr: 9.65105 batch_time=0.51058
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.73912 (QuantReg: 9.49064) QuantErr: 9.49064 batch_time=0.51246
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.39400 (QuantReg: 9.98860) QuantErr: 9.98860 batch_time=0.74026
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.43133 (QuantReg: 9.76527) QuantErr: 9.76527 batch_time=0.51600
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.81697 (QuantReg: 10.02618) QuantErr: 10.02618 batch_time=0.49831
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.58350 (QuantReg: 10.29776) QuantErr: 10.29776 batch_time=0.49505
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.59959 (QuantReg: 9.70841) QuantErr: 9.70841 batch_time=0.49433
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.48245 (QuantReg: 9.88706) QuantErr: 9.88706 batch_time=1.29983
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.45225 (QuantReg: 9.72032) QuantErr: 9.72032 batch_time=0.49378
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.69947 (QuantReg: 9.79434) QuantErr: 9.79434 batch_time=0.48545
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.70364 (QuantReg: 9.82782) QuantErr: 9.82782 batch_time=0.53104
Train Epoch: 6 codebook_update_time=1.76751
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch6.pth ...
Done in 5.788s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch6.pth ...
Done in 10.404s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.6766875925064086
quant_reg : 9.727334255218507
quant_err : 9.727334255218507
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 26.156941649899398
MSRVTT_full_val/t2v_metrics/R5: 62.57545271629779
MSRVTT_full_val/t2v_metrics/R10: 76.65995975855131
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.537223340040242
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.06334344258597
MSRVTT_full_val/v2t_metrics/R1: 32.99798792756539
MSRVTT_full_val/v2t_metrics/R5: 69.01408450704226
MSRVTT_full_val/v2t_metrics/R10: 81.69014084507042
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.804828973843058
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.08626271287999
MSRVTT_full_test/t2v_metrics/R1: 9.966555183946488
MSRVTT_full_test/t2v_metrics/R5: 29.464882943143813
MSRVTT_full_test/t2v_metrics/R10: 42.408026755852845
MSRVTT_full_test/t2v_metrics/R50: 76.88963210702342
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 50.54648829431438
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.17924505911893
MSRVTT_full_test/v2t_metrics/R1: 11.638795986622073
MSRVTT_full_test/v2t_metrics/R5: 33.84615384615385
MSRVTT_full_test/v2t_metrics/R10: 47.92642140468227
MSRVTT_full_test/v2t_metrics/R50: 81.53846153846153
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 42.76454849498328
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.627524561701808
mnt_best : 23.17924505911893
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.59162 (QuantReg: 9.07832) QuantErr: 9.07832 batch_time=31.68241
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.37440 (QuantReg: 9.81546) QuantErr: 9.81546 batch_time=0.50044
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.71263 (QuantReg: 9.83595) QuantErr: 9.83595 batch_time=0.49382
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.54326 (QuantReg: 9.66266) QuantErr: 9.66266 batch_time=0.49237
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.43478 (QuantReg: 9.30049) QuantErr: 9.30049 batch_time=0.50629
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.65263 (QuantReg: 9.40894) QuantErr: 9.40894 batch_time=0.50007
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.64846 (QuantReg: 9.66359) QuantErr: 9.66359 batch_time=0.49264
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.89611 (QuantReg: 9.65736) QuantErr: 9.65736 batch_time=0.55162
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.66627 (QuantReg: 9.72351) QuantErr: 9.72351 batch_time=1.24448
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.52217 (QuantReg: 9.71109) QuantErr: 9.71109 batch_time=0.54137
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.74051 (QuantReg: 9.65954) QuantErr: 9.65954 batch_time=0.49147
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.15028 (QuantReg: 9.95643) QuantErr: 9.95643 batch_time=0.50506
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.36845 (QuantReg: 9.54147) QuantErr: 9.54147 batch_time=0.49604
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.59382 (QuantReg: 9.58740) QuantErr: 9.58740 batch_time=0.52728
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.54196 (QuantReg: 9.35888) QuantErr: 9.35888 batch_time=0.52925
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.44788 (QuantReg: 9.62339) QuantErr: 9.62339 batch_time=0.50715
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 2.24542 (QuantReg: 9.75147) QuantErr: 9.75147 batch_time=0.73058
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.54757 (QuantReg: 9.32264) QuantErr: 9.32264 batch_time=0.52059
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.05029 (QuantReg: 9.72981) QuantErr: 9.72981 batch_time=0.71408
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.32787 (QuantReg: 9.66348) QuantErr: 9.66348 batch_time=0.51769
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.93216 (QuantReg: 9.72155) QuantErr: 9.72155 batch_time=0.49941
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.49091 (QuantReg: 9.55352) QuantErr: 9.55352 batch_time=0.54838
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.58999 (QuantReg: 9.56586) QuantErr: 9.56586 batch_time=0.49734
Train Epoch: 7 codebook_update_time=1.74190
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch7.pth ...
Done in 5.485s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.5298948454856873
quant_reg : 9.687481285095215
quant_err : 9.687481285095215
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 29.175050301810867
MSRVTT_full_val/t2v_metrics/R5: 63.78269617706238
MSRVTT_full_val/t2v_metrics/R10: 78.67203219315896
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.179074446680081
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.70416679036259
MSRVTT_full_val/v2t_metrics/R1: 29.979879275653925
MSRVTT_full_val/v2t_metrics/R5: 71.42857142857143
MSRVTT_full_val/v2t_metrics/R10: 82.29376257545272
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.698189134808853
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.06471099313647
MSRVTT_full_test/t2v_metrics/R1: 9.130434782608695
MSRVTT_full_test/t2v_metrics/R5: 29.096989966555185
MSRVTT_full_test/t2v_metrics/R10: 42.74247491638796
MSRVTT_full_test/t2v_metrics/R50: 77.72575250836121
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 50.48227424749164
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.476726592132223
MSRVTT_full_test/v2t_metrics/R1: 12.240802675585284
MSRVTT_full_test/v2t_metrics/R5: 35.986622073578594
MSRVTT_full_test/v2t_metrics/R10: 49.93311036789298
MSRVTT_full_test/v2t_metrics/R50: 82.10702341137124
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 41.38010033444816
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.018606578993825
mnt_best : 23.17924505911893
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.46202 (QuantReg: 9.71058) QuantErr: 9.71058 batch_time=29.70550
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.64801 (QuantReg: 9.81532) QuantErr: 9.81532 batch_time=0.49961
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.83141 (QuantReg: 9.47721) QuantErr: 9.47721 batch_time=0.49697
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.42982 (QuantReg: 9.79024) QuantErr: 9.79024 batch_time=0.71522
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.71418 (QuantReg: 9.73182) QuantErr: 9.73182 batch_time=0.49351
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.59098 (QuantReg: 9.63990) QuantErr: 9.63990 batch_time=0.49915
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.37611 (QuantReg: 9.99559) QuantErr: 9.99559 batch_time=0.51581
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.32934 (QuantReg: 9.91954) QuantErr: 9.91954 batch_time=0.49457
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.71578 (QuantReg: 9.76595) QuantErr: 9.76595 batch_time=1.84108
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.29398 (QuantReg: 9.40602) QuantErr: 9.40602 batch_time=0.55429
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.40632 (QuantReg: 9.84028) QuantErr: 9.84028 batch_time=0.51900
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.75755 (QuantReg: 9.68016) QuantErr: 9.68016 batch_time=0.50176
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.27802 (QuantReg: 9.69058) QuantErr: 9.69058 batch_time=0.49031
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.13260 (QuantReg: 9.57693) QuantErr: 9.57693 batch_time=0.74940
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.73441 (QuantReg: 9.83162) QuantErr: 9.83162 batch_time=0.51683
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.26931 (QuantReg: 9.55881) QuantErr: 9.55881 batch_time=0.50479
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.62212 (QuantReg: 9.59399) QuantErr: 9.59399 batch_time=0.49800
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.66893 (QuantReg: 9.85495) QuantErr: 9.85495 batch_time=0.53638
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.01026 (QuantReg: 9.73357) QuantErr: 9.73357 batch_time=0.71179
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.04058 (QuantReg: 9.52713) QuantErr: 9.52713 batch_time=1.36102
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.05333 (QuantReg: 10.03412) QuantErr: 10.03412 batch_time=0.49152
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.76531 (QuantReg: 9.52021) QuantErr: 9.52021 batch_time=0.49237
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.99917 (QuantReg: 9.65403) QuantErr: 9.65403 batch_time=0.50284
Train Epoch: 8 codebook_update_time=1.67370
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch8.pth ...
Done in 5.718s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch8.pth ...
Done in 10.341s
removing stale ckpt [epoch 7] [took 0.03s]
epoch : 8
loss : 2.4371680397987365
quant_reg : 9.656676395416259
quant_err : 9.656676395416259
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 23.541247484909455
MSRVTT_full_val/t2v_metrics/R5: 61.77062374245473
MSRVTT_full_val/t2v_metrics/R10: 78.47082494969818
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.460764587525151
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.503516496344744
MSRVTT_full_val/v2t_metrics/R1: 31.58953722334004
MSRVTT_full_val/v2t_metrics/R5: 68.81287726358148
MSRVTT_full_val/v2t_metrics/R10: 82.29376257545272
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.605633802816901
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.34559466667467
MSRVTT_full_test/t2v_metrics/R1: 10.267558528428093
MSRVTT_full_test/t2v_metrics/R5: 29.632107023411372
MSRVTT_full_test/t2v_metrics/R10: 43.74581939799331
MSRVTT_full_test/t2v_metrics/R50: 77.59197324414716
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.053177257525086
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.698567335614317
MSRVTT_full_test/v2t_metrics/R1: 11.605351170568563
MSRVTT_full_test/v2t_metrics/R5: 34.31438127090301
MSRVTT_full_test/v2t_metrics/R10: 49.264214046822744
MSRVTT_full_test/v2t_metrics/R50: 82.876254180602
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 39.991638795986624
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.970479796031082
mnt_best : 23.698567335614317
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.51206 (QuantReg: 9.17986) QuantErr: 9.17986 batch_time=33.97367
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.22183 (QuantReg: 9.33017) QuantErr: 9.33017 batch_time=0.51623
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.56076 (QuantReg: 9.47258) QuantErr: 9.47258 batch_time=0.48837
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.57506 (QuantReg: 9.56305) QuantErr: 9.56305 batch_time=0.51932
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.77858 (QuantReg: 9.45335) QuantErr: 9.45335 batch_time=0.61899
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.60535 (QuantReg: 9.68318) QuantErr: 9.68318 batch_time=0.50933
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.31978 (QuantReg: 9.60558) QuantErr: 9.60558 batch_time=1.32197
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 2.04680 (QuantReg: 9.53398) QuantErr: 9.53398 batch_time=0.51296
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.54745 (QuantReg: 9.45408) QuantErr: 9.45408 batch_time=0.49749
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.37399 (QuantReg: 9.60892) QuantErr: 9.60892 batch_time=0.52215
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.34031 (QuantReg: 9.51572) QuantErr: 9.51572 batch_time=0.73944
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 2.19973 (QuantReg: 9.72665) QuantErr: 9.72665 batch_time=0.55105
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.37502 (QuantReg: 9.75116) QuantErr: 9.75116 batch_time=0.52153
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.45866 (QuantReg: 9.74775) QuantErr: 9.74775 batch_time=1.91571
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.41220 (QuantReg: 9.99392) QuantErr: 9.99392 batch_time=0.54061
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 2.49264 (QuantReg: 9.54187) QuantErr: 9.54187 batch_time=0.49666
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.96751 (QuantReg: 9.72149) QuantErr: 9.72149 batch_time=0.51746
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.09095 (QuantReg: 9.64550) QuantErr: 9.64550 batch_time=0.51599
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.05409 (QuantReg: 9.94329) QuantErr: 9.94329 batch_time=0.85304
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.21573 (QuantReg: 9.44708) QuantErr: 9.44708 batch_time=0.94558
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.26482 (QuantReg: 9.59301) QuantErr: 9.59301 batch_time=0.51745
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.30976 (QuantReg: 9.66478) QuantErr: 9.66478 batch_time=0.55678
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.79804 (QuantReg: 9.63644) QuantErr: 9.63644 batch_time=0.49144
Train Epoch: 9 codebook_update_time=1.63111
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch9.pth ...
Done in 5.032s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch9.pth ...
Done in 9.885s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 2.3159452924728394
quant_reg : 9.607510822296142
quant_err : 9.607510822296142
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
MSRVTT_full_val/t2v_metrics/R10: 78.67203219315896
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.054325955734406
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.91240907730674
MSRVTT_full_val/v2t_metrics/R1: 34.40643863179074
MSRVTT_full_val/v2t_metrics/R5: 70.82494969818913
MSRVTT_full_val/v2t_metrics/R10: 83.09859154929578
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.637826961770624
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.72277266369747
MSRVTT_full_test/t2v_metrics/R1: 11.37123745819398
MSRVTT_full_test/t2v_metrics/R5: 30.668896321070235
MSRVTT_full_test/t2v_metrics/R10: 44.78260869565217
MSRVTT_full_test/t2v_metrics/R50: 78.66220735785953
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 47.3685618729097
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.996071306752615
MSRVTT_full_test/v2t_metrics/R1: 12.675585284280936
MSRVTT_full_test/v2t_metrics/R5: 36.08695652173913
MSRVTT_full_test/v2t_metrics/R10: 51.20401337792642
MSRVTT_full_test/v2t_metrics/R50: 82.47491638795987
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 41.1752508361204
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.611508774992938
mnt_best : 24.996071306752615
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.96644 (QuantReg: 9.16974) QuantErr: 9.16974 batch_time=29.97284
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.18232 (QuantReg: 9.59371) QuantErr: 9.59371 batch_time=0.55908
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.99576 (QuantReg: 9.22052) QuantErr: 9.22052 batch_time=0.51684
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.00756 (QuantReg: 9.27909) QuantErr: 9.27909 batch_time=0.49169
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 2.07798 (QuantReg: 9.88936) QuantErr: 9.88936 batch_time=0.55410
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.60426 (QuantReg: 9.57611) QuantErr: 9.57611 batch_time=0.55188
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.98219 (QuantReg: 9.61676) QuantErr: 9.61676 batch_time=0.57520
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 2.11472 (QuantReg: 9.35296) QuantErr: 9.35296 batch_time=0.49493
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.25513 (QuantReg: 9.55895) QuantErr: 9.55895 batch_time=0.50978
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.70581 (QuantReg: 9.32677) QuantErr: 9.32677 batch_time=0.55462
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.01430 (QuantReg: 9.45224) QuantErr: 9.45224 batch_time=0.49704
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.30217 (QuantReg: 9.66920) QuantErr: 9.66920 batch_time=0.53874
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.39563 (QuantReg: 9.56509) QuantErr: 9.56509 batch_time=0.51522
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.64699 (QuantReg: 9.64866) QuantErr: 9.64866 batch_time=0.49500
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.97913 (QuantReg: 9.77444) QuantErr: 9.77444 batch_time=0.52196
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.83510 (QuantReg: 9.59297) QuantErr: 9.59297 batch_time=0.48997
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.16964 (QuantReg: 9.64781) QuantErr: 9.64781 batch_time=0.48649
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.99683 (QuantReg: 9.80363) QuantErr: 9.80363 batch_time=0.48100
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.07168 (QuantReg: 9.89488) QuantErr: 9.89488 batch_time=0.98274
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.00081 (QuantReg: 9.48889) QuantErr: 9.48889 batch_time=0.51529
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.02262 (QuantReg: 9.31100) QuantErr: 9.31100 batch_time=0.54454
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.71328 (QuantReg: 9.67448) QuantErr: 9.67448 batch_time=0.50020
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.28821 (QuantReg: 9.37965) QuantErr: 9.37965 batch_time=0.50599
Train Epoch: 10 codebook_update_time=1.63599
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch10.pth ...
Done in 14.933s
removing stale ckpt [epoch 9] [took 0.02s]
epoch : 10
loss : 2.217851896762848
quant_reg : 9.5602430267334
quant_err : 9.5602430267334
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 28.772635814889338
MSRVTT_full_val/t2v_metrics/R5: 63.17907444668008
MSRVTT_full_val/t2v_metrics/R10: 78.47082494969818
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.509054325955734
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.25008991242564
MSRVTT_full_val/v2t_metrics/R1: 33.40040241448692
MSRVTT_full_val/v2t_metrics/R5: 69.61770623742454
MSRVTT_full_val/v2t_metrics/R10: 82.09255533199195
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.682092555331992
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.57824118704483
MSRVTT_full_test/t2v_metrics/R1: 9.498327759197325
MSRVTT_full_test/t2v_metrics/R5: 31.337792642140467
MSRVTT_full_test/t2v_metrics/R10: 43.64548494983278
MSRVTT_full_test/t2v_metrics/R50: 78.52842809364549
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.300334448160534
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.50814130478168
MSRVTT_full_test/v2t_metrics/R1: 11.97324414715719
MSRVTT_full_test/v2t_metrics/R5: 34.68227424749164
MSRVTT_full_test/v2t_metrics/R10: 50.06688963210702
MSRVTT_full_test/v2t_metrics/R50: 82.77591973244147
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 41.23344481605351
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.49729710139384
mnt_best : 24.996071306752615
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.06770 (QuantReg: 9.47787) QuantErr: 9.47787 batch_time=32.66722
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.94592 (QuantReg: 9.58068) QuantErr: 9.58068 batch_time=0.50407
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.15909 (QuantReg: 9.19869) QuantErr: 9.19869 batch_time=0.51872
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.23761 (QuantReg: 9.70990) QuantErr: 9.70990 batch_time=0.50641
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.25193 (QuantReg: 9.85109) QuantErr: 9.85109 batch_time=0.51024
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.08097 (QuantReg: 9.53024) QuantErr: 9.53024 batch_time=0.50658
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 2.28118 (QuantReg: 9.21422) QuantErr: 9.21422 batch_time=0.49643
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.93962 (QuantReg: 9.68282) QuantErr: 9.68282 batch_time=0.49809
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.10501 (QuantReg: 9.49878) QuantErr: 9.49878 batch_time=0.49178
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.57227 (QuantReg: 9.42266) QuantErr: 9.42266 batch_time=1.01204
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.35502 (QuantReg: 9.27045) QuantErr: 9.27045 batch_time=0.49666
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.17776 (QuantReg: 9.77425) QuantErr: 9.77425 batch_time=0.51091
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.62638 (QuantReg: 9.33208) QuantErr: 9.33208 batch_time=0.51514
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.35438 (QuantReg: 9.47332) QuantErr: 9.47332 batch_time=0.50555
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 2.06156 (QuantReg: 9.23397) QuantErr: 9.23397 batch_time=0.49327
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.09214 (QuantReg: 9.24219) QuantErr: 9.24219 batch_time=0.50331
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.00233 (QuantReg: 9.27843) QuantErr: 9.27843 batch_time=0.49831
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.88464 (QuantReg: 9.43280) QuantErr: 9.43280 batch_time=0.49637
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 2.01603 (QuantReg: 9.95047) QuantErr: 9.95047 batch_time=0.49687
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 2.35759 (QuantReg: 9.72434) QuantErr: 9.72434 batch_time=0.49294
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 2.28620 (QuantReg: 9.90346) QuantErr: 9.90346 batch_time=0.49594
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.99707 (QuantReg: 9.67170) QuantErr: 9.67170 batch_time=0.51454
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.90150 (QuantReg: 10.02171) QuantErr: 10.02171 batch_time=0.51025
Train Epoch: 11 codebook_update_time=1.81401
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch11.pth ...
Done in 4.668s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 2.139794557094574
quant_reg : 9.572949630737305
quant_err : 9.572949630737305
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
MSRVTT_full_val/t2v_metrics/R5: 64.38631790744466
MSRVTT_full_val/t2v_metrics/R10: 78.26961770623743
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.619718309859154
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.65799292794433
MSRVTT_full_val/v2t_metrics/R1: 33.601609657947684
MSRVTT_full_val/v2t_metrics/R5: 70.22132796780684
MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.617706237424548
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.188917259404526
MSRVTT_full_test/t2v_metrics/R1: 10.668896321070234
MSRVTT_full_test/t2v_metrics/R5: 32.34113712374582
MSRVTT_full_test/t2v_metrics/R10: 44.381270903010034
MSRVTT_full_test/t2v_metrics/R50: 78.69565217391305
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.34615384615385
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.83275125698871
MSRVTT_full_test/v2t_metrics/R1: 11.806020066889632
MSRVTT_full_test/v2t_metrics/R5: 36.82274247491639
MSRVTT_full_test/v2t_metrics/R10: 50.468227424749166
MSRVTT_full_test/v2t_metrics/R50: 83.01003344481606
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 39.92775919732441
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.99492014746828
mnt_best : 24.996071306752615
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 2.10512 (QuantReg: 9.47118) QuantErr: 9.47118 batch_time=31.88058
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 2.25439 (QuantReg: 9.61046) QuantErr: 9.61046 batch_time=1.24664
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.90563 (QuantReg: 9.32929) QuantErr: 9.32929 batch_time=0.49180
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.96778 (QuantReg: 9.44614) QuantErr: 9.44614 batch_time=0.49880
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.77077 (QuantReg: 9.57475) QuantErr: 9.57475 batch_time=0.49278
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 2.11504 (QuantReg: 9.34071) QuantErr: 9.34071 batch_time=0.49764
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.79708 (QuantReg: 9.51548) QuantErr: 9.51548 batch_time=0.50828
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.98440 (QuantReg: 9.48151) QuantErr: 9.48151 batch_time=0.49767
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.92475 (QuantReg: 9.48985) QuantErr: 9.48985 batch_time=0.50758
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 2.36933 (QuantReg: 9.49559) QuantErr: 9.49559 batch_time=0.49533
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.75533 (QuantReg: 9.45672) QuantErr: 9.45672 batch_time=0.49122
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 2.22941 (QuantReg: 9.46260) QuantErr: 9.46260 batch_time=0.49892
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.88048 (QuantReg: 9.60844) QuantErr: 9.60844 batch_time=0.50459
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.83956 (QuantReg: 9.83655) QuantErr: 9.83655 batch_time=2.87316
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.12114 (QuantReg: 9.49096) QuantErr: 9.49096 batch_time=0.49490
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.68622 (QuantReg: 9.54963) QuantErr: 9.54963 batch_time=1.46688
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.81448 (QuantReg: 9.46148) QuantErr: 9.46148 batch_time=0.49026
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 2.44194 (QuantReg: 9.31620) QuantErr: 9.31620 batch_time=0.50052
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.30803 (QuantReg: 9.45235) QuantErr: 9.45235 batch_time=0.49535
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 2.27734 (QuantReg: 9.52746) QuantErr: 9.52746 batch_time=0.49055
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 2.06120 (QuantReg: 9.09724) QuantErr: 9.09724 batch_time=0.49435
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.00586 (QuantReg: 9.64898) QuantErr: 9.64898 batch_time=0.50630
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 2.16629 (QuantReg: 9.91232) QuantErr: 9.91232 batch_time=0.49167
Train Epoch: 12 codebook_update_time=1.73578
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch12.pth ...
Done in 5.707s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 2.072114281654358
quant_reg : 9.519542114257813
quant_err : 9.519542114257813
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 27.96780684104628
MSRVTT_full_val/t2v_metrics/R5: 65.59356136820925
MSRVTT_full_val/t2v_metrics/R10: 78.06841046277665
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.716297786720322
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.3196688564011
MSRVTT_full_val/v2t_metrics/R1: 32.99798792756539
MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
MSRVTT_full_val/v2t_metrics/R50: 96.579476861167
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.7364185110663986
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.331337557881824
MSRVTT_full_test/t2v_metrics/R1: 10.668896321070234
MSRVTT_full_test/t2v_metrics/R5: 31.17056856187291
MSRVTT_full_test/t2v_metrics/R10: 43.91304347826087
MSRVTT_full_test/t2v_metrics/R50: 77.9933110367893
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.94648829431438
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.442891811447314
MSRVTT_full_test/v2t_metrics/R1: 12.073578595317725
MSRVTT_full_test/v2t_metrics/R5: 36.25418060200669
MSRVTT_full_test/v2t_metrics/R10: 51.53846153846154
MSRVTT_full_test/v2t_metrics/R50: 82.876254180602
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 39.45535117056856
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.255859296754053
mnt_best : 24.996071306752615
not_improved_count: 3
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.88787 (QuantReg: 9.42522) QuantErr: 9.42522 batch_time=40.06229
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 2.58840 (QuantReg: 9.02827) QuantErr: 9.02827 batch_time=0.48536
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 2.26518 (QuantReg: 9.55122) QuantErr: 9.55122 batch_time=0.48332
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.86520 (QuantReg: 9.52706) QuantErr: 9.52706 batch_time=0.48673
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.06043 (QuantReg: 9.55785) QuantErr: 9.55785 batch_time=0.48600
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.02428 (QuantReg: 9.76405) QuantErr: 9.76405 batch_time=0.49660
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 2.04610 (QuantReg: 9.83495) QuantErr: 9.83495 batch_time=0.49687
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 2.44406 (QuantReg: 9.75840) QuantErr: 9.75840 batch_time=0.48772
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.75288 (QuantReg: 9.10587) QuantErr: 9.10587 batch_time=0.48741
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 2.12261 (QuantReg: 9.53263) QuantErr: 9.53263 batch_time=0.51636
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 2.06262 (QuantReg: 9.41004) QuantErr: 9.41004 batch_time=0.48536
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.66601 (QuantReg: 9.45994) QuantErr: 9.45994 batch_time=0.49041
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.92650 (QuantReg: 9.53346) QuantErr: 9.53346 batch_time=0.49639
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.96565 (QuantReg: 9.76510) QuantErr: 9.76510 batch_time=1.92621
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 2.17498 (QuantReg: 9.33095) QuantErr: 9.33095 batch_time=0.51195
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.85902 (QuantReg: 9.43775) QuantErr: 9.43775 batch_time=0.49742
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.69002 (QuantReg: 9.68123) QuantErr: 9.68123 batch_time=0.51605
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 2.27813 (QuantReg: 9.40952) QuantErr: 9.40952 batch_time=0.48867
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.73845 (QuantReg: 9.82372) QuantErr: 9.82372 batch_time=0.51271
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.36710 (QuantReg: 9.60610) QuantErr: 9.60610 batch_time=0.70397
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 2.32362 (QuantReg: 9.67703) QuantErr: 9.67703 batch_time=0.51331
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.89714 (QuantReg: 9.63847) QuantErr: 9.63847 batch_time=0.50799
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.97510 (QuantReg: 9.67541) QuantErr: 9.67541 batch_time=0.49075
Train Epoch: 13 codebook_update_time=1.59810
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch13.pth ...
Done in 4.548s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch13.pth ...
Done in 8.607s
removing stale ckpt [epoch 12] [took 0.65s]
epoch : 13
loss : 2.0058848280906676
quant_reg : 9.545579822540283
quant_err : 9.545579822540283
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 29.77867203219316
MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.788732394366196
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.08095285315748
MSRVTT_full_val/v2t_metrics/R1: 34.20523138832998
MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.4607645875251505
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.0815713597829
MSRVTT_full_test/t2v_metrics/R1: 11.538461538461538
MSRVTT_full_test/t2v_metrics/R5: 30.869565217391305
MSRVTT_full_test/t2v_metrics/R10: 45.852842809364546
MSRVTT_full_test/t2v_metrics/R50: 79.1638795986622
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.51270903010033
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.37162177896157
MSRVTT_full_test/v2t_metrics/R1: 11.839464882943144
MSRVTT_full_test/v2t_metrics/R5: 36.92307692307692
MSRVTT_full_test/v2t_metrics/R10: 52.0066889632107
MSRVTT_full_test/v2t_metrics/R50: 83.1438127090301
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 39.60033444816054
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.32890064609451
mnt_best : 25.37162177896157
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.82533 (QuantReg: 9.67546) QuantErr: 9.67546 batch_time=31.06962
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.42672 (QuantReg: 9.75568) QuantErr: 9.75568 batch_time=0.48927
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.87704 (QuantReg: 9.68481) QuantErr: 9.68481 batch_time=0.48556
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.89257 (QuantReg: 9.53909) QuantErr: 9.53909 batch_time=0.50941
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.21130 (QuantReg: 9.47822) QuantErr: 9.47822 batch_time=0.47662
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.94275 (QuantReg: 9.35827) QuantErr: 9.35827 batch_time=0.48947
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.23821 (QuantReg: 9.54591) QuantErr: 9.54591 batch_time=1.93385
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 2.11043 (QuantReg: 9.08395) QuantErr: 9.08395 batch_time=0.51096
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.71229 (QuantReg: 9.20695) QuantErr: 9.20695 batch_time=0.50121
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.85489 (QuantReg: 9.25117) QuantErr: 9.25117 batch_time=0.48751
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.87429 (QuantReg: 9.54137) QuantErr: 9.54137 batch_time=0.48229
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 2.12534 (QuantReg: 9.27463) QuantErr: 9.27463 batch_time=0.48722
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.85656 (QuantReg: 9.88797) QuantErr: 9.88797 batch_time=1.31964
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.69104 (QuantReg: 9.54278) QuantErr: 9.54278 batch_time=1.81757
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.79037 (QuantReg: 9.48609) QuantErr: 9.48609 batch_time=0.49646
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.72930 (QuantReg: 9.56291) QuantErr: 9.56291 batch_time=0.48718
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.73860 (QuantReg: 9.28780) QuantErr: 9.28780 batch_time=0.49636
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 2.09216 (QuantReg: 9.79860) QuantErr: 9.79860 batch_time=0.50285
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.78240 (QuantReg: 9.30983) QuantErr: 9.30983 batch_time=0.48246
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 2.19040 (QuantReg: 9.24648) QuantErr: 9.24648 batch_time=0.49347
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.07599 (QuantReg: 9.64071) QuantErr: 9.64071 batch_time=0.48438
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.89952 (QuantReg: 9.76194) QuantErr: 9.76194 batch_time=0.49498
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.85480 (QuantReg: 9.53834) QuantErr: 9.53834 batch_time=0.51759
Train Epoch: 14 codebook_update_time=1.72912
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch14.pth ...
Done in 23.000s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 1.960813542842865
quant_reg : 9.510387943267823
quant_err : 9.510387943267823
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 28.571428571428573
MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
MSRVTT_full_val/t2v_metrics/R10: 77.06237424547284
MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.37625754527163
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.304632877580566
MSRVTT_full_val/v2t_metrics/R1: 34.60764587525151
MSRVTT_full_val/v2t_metrics/R5: 70.02012072434607
MSRVTT_full_val/v2t_metrics/R10: 81.89134808853119
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.458752515090543
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.32807513025551
MSRVTT_full_test/t2v_metrics/R1: 10.167224080267559
MSRVTT_full_test/t2v_metrics/R5: 30.80267558528428
MSRVTT_full_test/t2v_metrics/R10: 44.71571906354515
MSRVTT_full_test/t2v_metrics/R50: 77.45819397993311
MSRVTT_full_test/t2v_metrics/MedR: 13.5
MSRVTT_full_test/t2v_metrics/MeanR: 53.709030100334445
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.103698445496494
MSRVTT_full_test/v2t_metrics/R1: 12.073578595317725
MSRVTT_full_test/v2t_metrics/R5: 36.38795986622073
MSRVTT_full_test/v2t_metrics/R10: 50.36789297658863
MSRVTT_full_test/v2t_metrics/R50: 82.17391304347827
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 41.602341137123744
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.074745932180612
mnt_best : 25.37162177896157
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.36010 (QuantReg: 9.21051) QuantErr: 9.21051 batch_time=37.28978
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.93706 (QuantReg: 9.29362) QuantErr: 9.29362 batch_time=1.70647
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.01339 (QuantReg: 9.29539) QuantErr: 9.29539 batch_time=0.49440
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.02460 (QuantReg: 9.30553) QuantErr: 9.30553 batch_time=0.49966
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.03756 (QuantReg: 9.73160) QuantErr: 9.73160 batch_time=0.50255
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.97864 (QuantReg: 9.17741) QuantErr: 9.17741 batch_time=0.49869
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.25368 (QuantReg: 9.32194) QuantErr: 9.32194 batch_time=0.49896
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.78992 (QuantReg: 9.48733) QuantErr: 9.48733 batch_time=2.14365
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.15175 (QuantReg: 9.48558) QuantErr: 9.48558 batch_time=0.53302
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.83628 (QuantReg: 9.54686) QuantErr: 9.54686 batch_time=0.49695
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.09321 (QuantReg: 9.36822) QuantErr: 9.36822 batch_time=0.74092
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.67429 (QuantReg: 9.24765) QuantErr: 9.24765 batch_time=0.49484
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.78697 (QuantReg: 9.43836) QuantErr: 9.43836 batch_time=0.51219
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 2.30752 (QuantReg: 9.67488) QuantErr: 9.67488 batch_time=0.51052
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.89413 (QuantReg: 9.29821) QuantErr: 9.29821 batch_time=0.49340
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.02241 (QuantReg: 9.72334) QuantErr: 9.72334 batch_time=0.50351
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 2.28120 (QuantReg: 9.72381) QuantErr: 9.72381 batch_time=0.51449
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.81703 (QuantReg: 9.48864) QuantErr: 9.48864 batch_time=0.52049
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.72446 (QuantReg: 9.63053) QuantErr: 9.63053 batch_time=0.50604
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.09094 (QuantReg: 9.21898) QuantErr: 9.21898 batch_time=0.49425
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.14681 (QuantReg: 9.67914) QuantErr: 9.67914 batch_time=0.53563
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.09959 (QuantReg: 9.73994) QuantErr: 9.73994 batch_time=0.49978
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.07152 (QuantReg: 9.38213) QuantErr: 9.38213 batch_time=0.51385
Train Epoch: 15 codebook_update_time=1.66108
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch15.pth ...
Done in 20.338s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch15.pth ...
Done in 24.186s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 1.9088825998306274
quant_reg : 9.507624305725098
quant_err : 9.507624305725098
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_full_val/t2v_metrics/R1: 30.382293762575454
MSRVTT_full_val/t2v_metrics/R5: 69.01408450704226