-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA.txt
2602 lines (2601 loc) · 194 KB
/
HCQ_MSRVTT_1kA.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 1399.2535359859467 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 83.5559434890747 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 61.18525171279907 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch0.pth ...
Done in 3.281s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch0.pth ...
Done in 5.054s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 486.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 496.278
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 6.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 509.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.537
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.5192494101851104
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.81458 (QuantReg: 22.49638) QuantErr: 22.49638 batch_time=40.98641
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.64904 (QuantReg: 22.50508) QuantErr: 22.50508 batch_time=0.54485
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.56301 (QuantReg: 22.53892) QuantErr: 22.53892 batch_time=0.51973
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.38077 (QuantReg: 22.57972) QuantErr: 22.57972 batch_time=0.54121
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.49565 (QuantReg: 22.64288) QuantErr: 22.64288 batch_time=0.50928
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.81904 (QuantReg: 22.60749) QuantErr: 22.60749 batch_time=0.51402
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.80113 (QuantReg: 22.57744) QuantErr: 22.57744 batch_time=0.55247
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.03843 (QuantReg: 22.62452) QuantErr: 22.62452 batch_time=0.54264
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.44199 (QuantReg: 22.60307) QuantErr: 22.60307 batch_time=0.51067
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.45282 (QuantReg: 22.62855) QuantErr: 22.62855 batch_time=0.51000
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 4.69521 (QuantReg: 22.58858) QuantErr: 22.58858 batch_time=0.52496
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.70020 (QuantReg: 22.61821) QuantErr: 22.61821 batch_time=0.52932
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.18677 (QuantReg: 22.63303) QuantErr: 22.63303 batch_time=0.55527
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.36814 (QuantReg: 22.64074) QuantErr: 22.64074 batch_time=0.51266
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.76453 (QuantReg: 22.65092) QuantErr: 22.65092 batch_time=0.58619
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.14957 (QuantReg: 22.68394) QuantErr: 22.68394 batch_time=0.51902
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.88900 (QuantReg: 22.67934) QuantErr: 22.67934 batch_time=0.77711
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.55495 (QuantReg: 22.66296) QuantErr: 22.66296 batch_time=0.51214
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.07463 (QuantReg: 22.65596) QuantErr: 22.65596 batch_time=0.55729
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.50904 (QuantReg: 22.63895) QuantErr: 22.63895 batch_time=0.49454
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 3.64325 (QuantReg: 22.69133) QuantErr: 22.69133 batch_time=0.49577
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.24828 (QuantReg: 22.66160) QuantErr: 22.66160 batch_time=0.50824
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.72175 (QuantReg: 22.68702) QuantErr: 22.68702 batch_time=0.82706
Train Epoch: 1 codebook_update_time=1.99262
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch1.pth ...
Done in 3.863s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch1.pth ...
Done in 8.040s
epoch : 1
loss : 5.387369966506958
quant_reg : 22.620893562316894
quant_err : 22.620893562316894
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 11.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 33.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 46.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 78.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 13.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 42.634
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.050338866040132
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 32.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 44.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 79.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 13.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 40.342
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.649340267538104
mnt_best : 26.050338866040132
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.19658 (QuantReg: 11.69272) QuantErr: 11.69272 batch_time=34.68535
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.08692 (QuantReg: 11.80013) QuantErr: 11.80013 batch_time=0.52338
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.10345 (QuantReg: 12.24447) QuantErr: 12.24447 batch_time=0.50852
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.88845 (QuantReg: 12.18617) QuantErr: 12.18617 batch_time=0.52599
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.90425 (QuantReg: 12.89029) QuantErr: 12.89029 batch_time=0.52091
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.98323 (QuantReg: 12.80228) QuantErr: 12.80228 batch_time=0.50537
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.79906 (QuantReg: 12.67745) QuantErr: 12.67745 batch_time=0.53147
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.79444 (QuantReg: 12.96687) QuantErr: 12.96687 batch_time=0.50427
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.69514 (QuantReg: 13.12283) QuantErr: 13.12283 batch_time=0.60399
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.19967 (QuantReg: 13.11597) QuantErr: 13.11597 batch_time=0.51678
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.76746 (QuantReg: 13.01554) QuantErr: 13.01554 batch_time=0.51986
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.22834 (QuantReg: 13.54878) QuantErr: 13.54878 batch_time=0.54412
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.69089 (QuantReg: 13.49959) QuantErr: 13.49959 batch_time=0.52241
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.14207 (QuantReg: 13.48767) QuantErr: 13.48767 batch_time=0.50457
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.99794 (QuantReg: 13.62113) QuantErr: 13.62113 batch_time=0.54836
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.50833 (QuantReg: 13.92481) QuantErr: 13.92481 batch_time=0.50225
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.88538 (QuantReg: 13.81016) QuantErr: 13.81016 batch_time=0.50137
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.65736 (QuantReg: 14.21573) QuantErr: 14.21573 batch_time=0.52261
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.36101 (QuantReg: 14.32304) QuantErr: 14.32304 batch_time=0.50739
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.37064 (QuantReg: 14.35914) QuantErr: 14.35914 batch_time=0.51500
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.51937 (QuantReg: 14.32075) QuantErr: 14.32075 batch_time=0.49866
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.59926 (QuantReg: 14.26386) QuantErr: 14.26386 batch_time=0.51978
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.73343 (QuantReg: 14.39434) QuantErr: 14.39434 batch_time=0.52182
Train Epoch: 2 codebook_update_time=1.72824
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch2.pth ...
Done in 22.141s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch2.pth ...
Done in 26.327s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.718916591644287
quant_reg : 13.33253591156006
quant_err : 13.33253591156006
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 37.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 53.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 83.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.867
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.622781957921966
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 40.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 55.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 83.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.832499999999996
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.57993734634045
mnt_best : 30.622781957921966
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.37596 (QuantReg: 11.77480) QuantErr: 11.77480 batch_time=33.83422
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.23580 (QuantReg: 12.03973) QuantErr: 12.03973 batch_time=0.62434
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.32750 (QuantReg: 12.19064) QuantErr: 12.19064 batch_time=0.51522
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.93345 (QuantReg: 12.11860) QuantErr: 12.11860 batch_time=0.52485
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 2.86843 (QuantReg: 12.29593) QuantErr: 12.29593 batch_time=0.50227
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.37298 (QuantReg: 12.23400) QuantErr: 12.23400 batch_time=0.53396
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.76295 (QuantReg: 12.56815) QuantErr: 12.56815 batch_time=0.76102
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.46972 (QuantReg: 12.46091) QuantErr: 12.46091 batch_time=0.56047
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.37315 (QuantReg: 12.39373) QuantErr: 12.39373 batch_time=0.52684
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.67958 (QuantReg: 12.73173) QuantErr: 12.73173 batch_time=0.52088
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.45570 (QuantReg: 12.54875) QuantErr: 12.54875 batch_time=0.49698
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.17721 (QuantReg: 12.36061) QuantErr: 12.36061 batch_time=0.59451
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 2.89164 (QuantReg: 12.41958) QuantErr: 12.41958 batch_time=0.55876
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.39695 (QuantReg: 12.33929) QuantErr: 12.33929 batch_time=6.25142
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.56052 (QuantReg: 12.17668) QuantErr: 12.17668 batch_time=0.49652
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.88025 (QuantReg: 12.49620) QuantErr: 12.49620 batch_time=0.56279
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.23894 (QuantReg: 13.00611) QuantErr: 13.00611 batch_time=0.51808
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 2.86648 (QuantReg: 12.89405) QuantErr: 12.89405 batch_time=0.54912
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.96565 (QuantReg: 12.80971) QuantErr: 12.80971 batch_time=0.52220
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.03003 (QuantReg: 12.87893) QuantErr: 12.87893 batch_time=0.51312
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.13053 (QuantReg: 12.89266) QuantErr: 12.89266 batch_time=0.56557
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.10683 (QuantReg: 13.08313) QuantErr: 13.08313 batch_time=0.49616
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.14659 (QuantReg: 13.10741) QuantErr: 13.10741 batch_time=0.50104
Train Epoch: 3 codebook_update_time=2.43907
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch3.pth ...
Done in 6.756s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch3.pth ...
Done in 11.010s
removing stale ckpt [epoch 2] [took 0.03s]
epoch : 3
loss : 3.155210781097412
quant_reg : 12.529800701141358
quant_err : 12.529800701141358
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 56.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.938
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.823669213496345
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.176
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.268330744522984
mnt_best : 32.823669213496345
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.03607 (QuantReg: 12.06786) QuantErr: 12.06786 batch_time=41.24024
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.79767 (QuantReg: 12.17486) QuantErr: 12.17486 batch_time=0.60257
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.89486 (QuantReg: 12.09010) QuantErr: 12.09010 batch_time=1.21130
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.76069 (QuantReg: 12.33063) QuantErr: 12.33063 batch_time=0.54612
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.80003 (QuantReg: 12.38696) QuantErr: 12.38696 batch_time=0.60028
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.94900 (QuantReg: 12.18429) QuantErr: 12.18429 batch_time=0.54614
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.73061 (QuantReg: 12.15209) QuantErr: 12.15209 batch_time=0.52881
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.85024 (QuantReg: 12.31035) QuantErr: 12.31035 batch_time=0.51099
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.12169 (QuantReg: 12.09410) QuantErr: 12.09410 batch_time=0.58615
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.47755 (QuantReg: 12.46868) QuantErr: 12.46868 batch_time=0.68268
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.82683 (QuantReg: 12.99284) QuantErr: 12.99284 batch_time=0.53642
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.53026 (QuantReg: 12.95136) QuantErr: 12.95136 batch_time=0.51298
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.02232 (QuantReg: 12.59953) QuantErr: 12.59953 batch_time=0.50035
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.58850 (QuantReg: 12.67277) QuantErr: 12.67277 batch_time=0.50990
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.71110 (QuantReg: 12.69187) QuantErr: 12.69187 batch_time=0.50513
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.81785 (QuantReg: 12.68870) QuantErr: 12.68870 batch_time=0.51365
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.83058 (QuantReg: 12.52363) QuantErr: 12.52363 batch_time=0.60428
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.84934 (QuantReg: 12.80585) QuantErr: 12.80585 batch_time=0.50648
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.62535 (QuantReg: 13.07942) QuantErr: 13.07942 batch_time=0.49043
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.45501 (QuantReg: 12.70068) QuantErr: 12.70068 batch_time=1.16511
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.36078 (QuantReg: 13.22988) QuantErr: 13.22988 batch_time=0.51655
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.19325 (QuantReg: 13.33150) QuantErr: 13.33150 batch_time=0.56919
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.81677 (QuantReg: 12.83376) QuantErr: 12.83376 batch_time=0.50384
Train Epoch: 4 codebook_update_time=1.73079
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch4.pth ...
Done in 4.012s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch4.pth ...
Done in 8.038s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 2.866916563987732
quant_reg : 12.485093025207519
quant_err : 12.485093025207519
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.796
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.48854990116877
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.794
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.079917350058416
mnt_best : 35.48854990116877
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.82537 (QuantReg: 11.96408) QuantErr: 11.96408 batch_time=33.83733
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.11318 (QuantReg: 12.27521) QuantErr: 12.27521 batch_time=0.53577
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.28027 (QuantReg: 12.16227) QuantErr: 12.16227 batch_time=0.51489
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.49882 (QuantReg: 12.44869) QuantErr: 12.44869 batch_time=0.50700
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.79789 (QuantReg: 12.29080) QuantErr: 12.29080 batch_time=0.52406
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.31733 (QuantReg: 12.52358) QuantErr: 12.52358 batch_time=0.57910
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.72152 (QuantReg: 12.21903) QuantErr: 12.21903 batch_time=0.69052
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.10786 (QuantReg: 11.92643) QuantErr: 11.92643 batch_time=1.17099
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.54501 (QuantReg: 12.46864) QuantErr: 12.46864 batch_time=0.55696
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.55883 (QuantReg: 12.26512) QuantErr: 12.26512 batch_time=0.51544
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.67278 (QuantReg: 12.57944) QuantErr: 12.57944 batch_time=0.57013
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.64775 (QuantReg: 12.36477) QuantErr: 12.36477 batch_time=0.51175
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.50785 (QuantReg: 12.76540) QuantErr: 12.76540 batch_time=0.51734
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.15512 (QuantReg: 12.20837) QuantErr: 12.20837 batch_time=0.52803
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.94461 (QuantReg: 12.56945) QuantErr: 12.56945 batch_time=0.53352
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.38922 (QuantReg: 12.82874) QuantErr: 12.82874 batch_time=0.73363
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.73729 (QuantReg: 12.53836) QuantErr: 12.53836 batch_time=0.50853
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.60717 (QuantReg: 12.50694) QuantErr: 12.50694 batch_time=0.52248
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.75713 (QuantReg: 12.85807) QuantErr: 12.85807 batch_time=0.50809
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.53191 (QuantReg: 12.55009) QuantErr: 12.55009 batch_time=0.54240
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.91462 (QuantReg: 12.79526) QuantErr: 12.79526 batch_time=0.49465
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.37830 (QuantReg: 12.66305) QuantErr: 12.66305 batch_time=0.61701
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.81025 (QuantReg: 12.90572) QuantErr: 12.90572 batch_time=0.57941
Train Epoch: 5 codebook_update_time=1.77200
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch5.pth ...
Done in 6.035s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch5.pth ...
Done in 10.442s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 2.588942636489868
quant_reg : 12.527422508239747
quant_err : 12.527422508239747
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.199
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.72792739157029
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.729
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.20867651929111
mnt_best : 37.72792739157029
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.64731 (QuantReg: 12.32998) QuantErr: 12.32998 batch_time=30.75415
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.77840 (QuantReg: 11.87980) QuantErr: 11.87980 batch_time=6.01338
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.48231 (QuantReg: 12.17228) QuantErr: 12.17228 batch_time=0.49526
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.41979 (QuantReg: 12.20150) QuantErr: 12.20150 batch_time=0.73608
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.30403 (QuantReg: 12.28919) QuantErr: 12.28919 batch_time=0.50220
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.28398 (QuantReg: 12.23595) QuantErr: 12.23595 batch_time=0.52644
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.68087 (QuantReg: 12.61078) QuantErr: 12.61078 batch_time=1.74055
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.32833 (QuantReg: 12.60358) QuantErr: 12.60358 batch_time=0.54129
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.39458 (QuantReg: 12.55585) QuantErr: 12.55585 batch_time=0.50488
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.25170 (QuantReg: 12.48586) QuantErr: 12.48586 batch_time=0.51651
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.39836 (QuantReg: 12.94824) QuantErr: 12.94824 batch_time=0.49918
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.33754 (QuantReg: 12.49436) QuantErr: 12.49436 batch_time=0.52215
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.25037 (QuantReg: 12.62291) QuantErr: 12.62291 batch_time=0.53254
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.27850 (QuantReg: 12.42533) QuantErr: 12.42533 batch_time=0.49630
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.74347 (QuantReg: 12.86352) QuantErr: 12.86352 batch_time=0.51824
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.42150 (QuantReg: 12.64217) QuantErr: 12.64217 batch_time=0.57578
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.14150 (QuantReg: 12.86792) QuantErr: 12.86792 batch_time=0.51411
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.51212 (QuantReg: 12.70982) QuantErr: 12.70982 batch_time=0.49700
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.57361 (QuantReg: 12.79829) QuantErr: 12.79829 batch_time=0.57286
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.39796 (QuantReg: 12.67519) QuantErr: 12.67519 batch_time=0.50309
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.21239 (QuantReg: 12.88302) QuantErr: 12.88302 batch_time=0.84431
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.12788 (QuantReg: 12.57896) QuantErr: 12.57896 batch_time=0.50831
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.11004 (QuantReg: 12.72260) QuantErr: 12.72260 batch_time=0.49867
Train Epoch: 6 codebook_update_time=2.14175
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch6.pth ...
Done in 4.362s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 2.4067695913314817
quant_reg : 12.605543056488036
quant_err : 12.605543056488036
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.621
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.161529003105684
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.65
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.18496914197272
mnt_best : 37.72792739157029
not_improved_count: 1
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.47082 (QuantReg: 11.97619) QuantErr: 11.97619 batch_time=30.32288
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.43290 (QuantReg: 12.26441) QuantErr: 12.26441 batch_time=0.51194
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.18147 (QuantReg: 12.00715) QuantErr: 12.00715 batch_time=0.52136
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 1.99112 (QuantReg: 12.34341) QuantErr: 12.34341 batch_time=0.79836
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 1.99277 (QuantReg: 12.38013) QuantErr: 12.38013 batch_time=0.50708
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.03983 (QuantReg: 12.84535) QuantErr: 12.84535 batch_time=0.56860
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.29332 (QuantReg: 12.73936) QuantErr: 12.73936 batch_time=3.95737
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.26983 (QuantReg: 12.61518) QuantErr: 12.61518 batch_time=3.25310
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.27202 (QuantReg: 12.77270) QuantErr: 12.77270 batch_time=0.99757
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.80869 (QuantReg: 12.59822) QuantErr: 12.59822 batch_time=0.50026
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.68588 (QuantReg: 12.62483) QuantErr: 12.62483 batch_time=0.49954
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.29057 (QuantReg: 12.68415) QuantErr: 12.68415 batch_time=0.50469
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.06912 (QuantReg: 12.85257) QuantErr: 12.85257 batch_time=0.56780
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.31588 (QuantReg: 12.47264) QuantErr: 12.47264 batch_time=0.50639
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.69449 (QuantReg: 12.52145) QuantErr: 12.52145 batch_time=0.51150
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.25441 (QuantReg: 12.76181) QuantErr: 12.76181 batch_time=0.53487
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.72349 (QuantReg: 13.01724) QuantErr: 13.01724 batch_time=0.50631
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.25830 (QuantReg: 12.63743) QuantErr: 12.63743 batch_time=0.60682
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.73332 (QuantReg: 12.85003) QuantErr: 12.85003 batch_time=0.51054
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.58362 (QuantReg: 13.06465) QuantErr: 13.06465 batch_time=0.61514
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.17005 (QuantReg: 13.06131) QuantErr: 13.06131 batch_time=0.53218
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.16754 (QuantReg: 13.35697) QuantErr: 13.35697 batch_time=0.51544
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.20638 (QuantReg: 12.96198) QuantErr: 12.96198 batch_time=0.50546
Train Epoch: 7 codebook_update_time=2.75125
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch7.pth ...
Done in 5.462s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch7.pth ...
Done in 9.780s
removing stale ckpt [epoch 6] [took 0.35s]
epoch : 7
loss : 2.2300485882759093
quant_reg : 12.68372780227661
quant_err : 12.68372780227661
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.942
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.75221827473922
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.258
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.20736480495823
mnt_best : 39.75221827473922
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.53389 (QuantReg: 12.53685) QuantErr: 12.53685 batch_time=33.19570
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.47009 (QuantReg: 12.39980) QuantErr: 12.39980 batch_time=0.96457
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.67688 (QuantReg: 12.69475) QuantErr: 12.69475 batch_time=0.49622
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.22794 (QuantReg: 12.40773) QuantErr: 12.40773 batch_time=0.49417
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.27063 (QuantReg: 12.87241) QuantErr: 12.87241 batch_time=0.50775
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.04921 (QuantReg: 12.51608) QuantErr: 12.51608 batch_time=0.49513
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.17923 (QuantReg: 12.80977) QuantErr: 12.80977 batch_time=0.50707
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.32581 (QuantReg: 12.90642) QuantErr: 12.90642 batch_time=0.54992
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.36457 (QuantReg: 12.76330) QuantErr: 12.76330 batch_time=0.52031
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.49935 (QuantReg: 12.79101) QuantErr: 12.79101 batch_time=0.50896
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.31545 (QuantReg: 12.91959) QuantErr: 12.91959 batch_time=0.49841
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.09882 (QuantReg: 12.64058) QuantErr: 12.64058 batch_time=0.49948
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.22117 (QuantReg: 12.66529) QuantErr: 12.66529 batch_time=0.51257
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.40001 (QuantReg: 13.04595) QuantErr: 13.04595 batch_time=0.50379
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.14566 (QuantReg: 13.02846) QuantErr: 13.02846 batch_time=0.50670
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.95753 (QuantReg: 12.59322) QuantErr: 12.59322 batch_time=0.56547
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.00679 (QuantReg: 12.63303) QuantErr: 12.63303 batch_time=0.50748
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.69475 (QuantReg: 12.75607) QuantErr: 12.75607 batch_time=0.49847
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.29155 (QuantReg: 12.89427) QuantErr: 12.89427 batch_time=0.50734
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.23891 (QuantReg: 12.43404) QuantErr: 12.43404 batch_time=1.42005
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.23447 (QuantReg: 12.81945) QuantErr: 12.81945 batch_time=0.49647
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.73028 (QuantReg: 12.95338) QuantErr: 12.95338 batch_time=0.50102
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.38933 (QuantReg: 12.99076) QuantErr: 12.99076 batch_time=0.74051
Train Epoch: 8 codebook_update_time=1.77842
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch8.pth ...
Done in 4.315s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch8.pth ...
Done in 9.263s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 2.1322562732696535
quant_reg : 12.752060535430909
quant_err : 12.752060535430909
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.043
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.9862619488556
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.892
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.84922719474521
mnt_best : 39.9862619488556
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.82339 (QuantReg: 12.93335) QuantErr: 12.93335 batch_time=30.38642
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.23885 (QuantReg: 12.76204) QuantErr: 12.76204 batch_time=0.74528
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.73723 (QuantReg: 12.53764) QuantErr: 12.53764 batch_time=0.52847
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.13993 (QuantReg: 12.76345) QuantErr: 12.76345 batch_time=0.50059
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.03680 (QuantReg: 12.94506) QuantErr: 12.94506 batch_time=0.52549
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.70624 (QuantReg: 13.06028) QuantErr: 13.06028 batch_time=0.58768
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.79052 (QuantReg: 13.06783) QuantErr: 13.06783 batch_time=0.50035
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 2.03294 (QuantReg: 13.04399) QuantErr: 13.04399 batch_time=0.50023
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.19338 (QuantReg: 12.48172) QuantErr: 12.48172 batch_time=0.49916
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.87888 (QuantReg: 12.78055) QuantErr: 12.78055 batch_time=0.50723
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.03764 (QuantReg: 12.77623) QuantErr: 12.77623 batch_time=0.63670
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.62678 (QuantReg: 12.78890) QuantErr: 12.78890 batch_time=0.50435
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.04734 (QuantReg: 12.78833) QuantErr: 12.78833 batch_time=0.49647
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.84069 (QuantReg: 12.78615) QuantErr: 12.78615 batch_time=0.50432
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.08001 (QuantReg: 12.53236) QuantErr: 12.53236 batch_time=0.54031
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.70368 (QuantReg: 13.33237) QuantErr: 13.33237 batch_time=0.52942
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.09938 (QuantReg: 13.22252) QuantErr: 13.22252 batch_time=0.50290
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.53831 (QuantReg: 13.02651) QuantErr: 13.02651 batch_time=0.58628
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.98562 (QuantReg: 12.98660) QuantErr: 12.98660 batch_time=0.50340
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.97043 (QuantReg: 13.06014) QuantErr: 13.06014 batch_time=0.49683
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.81939 (QuantReg: 13.27875) QuantErr: 13.27875 batch_time=0.50781
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.79601 (QuantReg: 12.62325) QuantErr: 12.62325 batch_time=0.52728
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.89790 (QuantReg: 12.73822) QuantErr: 12.73822 batch_time=0.63443
Train Epoch: 9 codebook_update_time=1.69831
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch9.pth ...
Done in 6.812s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch9.pth ...
Done in 11.511s
removing stale ckpt [epoch 8] [took 0.02s]
epoch : 9
loss : 1.995203962802887
quant_reg : 12.837140064239502
quant_err : 12.837140064239502
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.316
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.05332554473255
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.6165
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.70806505672913
mnt_best : 40.05332554473255
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.87910 (QuantReg: 12.87447) QuantErr: 12.87447 batch_time=28.37710
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.00996 (QuantReg: 12.72718) QuantErr: 12.72718 batch_time=0.50234
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.30265 (QuantReg: 13.04019) QuantErr: 13.04019 batch_time=0.50285
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.96246 (QuantReg: 12.46269) QuantErr: 12.46269 batch_time=0.50478
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.71743 (QuantReg: 12.66632) QuantErr: 12.66632 batch_time=0.53056
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.93186 (QuantReg: 12.95106) QuantErr: 12.95106 batch_time=0.52373
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.37833 (QuantReg: 12.58035) QuantErr: 12.58035 batch_time=6.67884
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.91559 (QuantReg: 12.90808) QuantErr: 12.90808 batch_time=0.52282
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.94528 (QuantReg: 12.68948) QuantErr: 12.68948 batch_time=0.53117
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.92994 (QuantReg: 12.84000) QuantErr: 12.84000 batch_time=0.53358
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.74436 (QuantReg: 12.78447) QuantErr: 12.78447 batch_time=0.51308
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.01395 (QuantReg: 13.02710) QuantErr: 13.02710 batch_time=0.52369
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.97468 (QuantReg: 12.63715) QuantErr: 12.63715 batch_time=0.51723
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.57597 (QuantReg: 13.29128) QuantErr: 13.29128 batch_time=0.63372
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.92282 (QuantReg: 12.70790) QuantErr: 12.70790 batch_time=0.72756
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.19231 (QuantReg: 13.13998) QuantErr: 13.13998 batch_time=0.50262
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.04979 (QuantReg: 12.91085) QuantErr: 12.91085 batch_time=0.51369
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.84842 (QuantReg: 13.18928) QuantErr: 13.18928 batch_time=0.51452
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.60941 (QuantReg: 13.14783) QuantErr: 13.14783 batch_time=0.54568
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.56005 (QuantReg: 13.34696) QuantErr: 13.34696 batch_time=0.52365
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.00270 (QuantReg: 13.00635) QuantErr: 13.00635 batch_time=0.52458
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.21499 (QuantReg: 12.86511) QuantErr: 12.86511 batch_time=0.51325
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.08929 (QuantReg: 13.19392) QuantErr: 13.19392 batch_time=0.54891
Train Epoch: 10 codebook_update_time=1.72298
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch10.pth ...
Done in 19.264s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch10.pth ...
Done in 24.302s
removing stale ckpt [epoch 9] [took 0.05s]
epoch : 10
loss : 1.8956627707481384
quant_reg : 12.942074924468994
quant_err : 12.942074924468994
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.7
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.51607834924003
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.4055
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.365891638799816
mnt_best : 41.51607834924003
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.86138 (QuantReg: 12.62964) QuantErr: 12.62964 batch_time=32.98681
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.05196 (QuantReg: 12.47084) QuantErr: 12.47084 batch_time=0.62538
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.12654 (QuantReg: 12.80230) QuantErr: 12.80230 batch_time=0.52633
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.87351 (QuantReg: 12.99411) QuantErr: 12.99411 batch_time=0.58182
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.03000 (QuantReg: 12.76749) QuantErr: 12.76749 batch_time=0.72576
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.26246 (QuantReg: 13.43856) QuantErr: 13.43856 batch_time=0.51544
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.57301 (QuantReg: 12.96789) QuantErr: 12.96789 batch_time=1.26942
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.02185 (QuantReg: 12.57992) QuantErr: 12.57992 batch_time=1.12497
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.85354 (QuantReg: 12.81263) QuantErr: 12.81263 batch_time=0.59828
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.75443 (QuantReg: 13.05260) QuantErr: 13.05260 batch_time=0.49396
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.53761 (QuantReg: 13.09379) QuantErr: 13.09379 batch_time=0.49670
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.61454 (QuantReg: 12.96810) QuantErr: 12.96810 batch_time=0.56001
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.94963 (QuantReg: 12.81832) QuantErr: 12.81832 batch_time=0.51398
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.73258 (QuantReg: 12.82299) QuantErr: 12.82299 batch_time=0.50496
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.48138 (QuantReg: 13.04627) QuantErr: 13.04627 batch_time=0.54805
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.25940 (QuantReg: 12.57858) QuantErr: 12.57858 batch_time=0.54038
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.68288 (QuantReg: 12.98174) QuantErr: 12.98174 batch_time=0.51589
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.79031 (QuantReg: 13.21716) QuantErr: 13.21716 batch_time=0.51272
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.85951 (QuantReg: 12.79563) QuantErr: 12.79563 batch_time=0.50137
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.64026 (QuantReg: 13.10490) QuantErr: 13.10490 batch_time=1.95829
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.82683 (QuantReg: 12.84767) QuantErr: 12.84767 batch_time=0.64714
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.86949 (QuantReg: 13.17120) QuantErr: 13.17120 batch_time=0.50487
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.72755 (QuantReg: 13.35550) QuantErr: 13.35550 batch_time=0.50751
Train Epoch: 11 codebook_update_time=1.62366
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch11.pth ...
Done in 5.174s
removing stale ckpt [epoch 10] [took 0.14s]
epoch : 11
loss : 1.811429819583893
quant_reg : 12.947463226318359
quant_err : 12.947463226318359
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.286
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.263266843765344
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.2365
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.88753585626039
mnt_best : 41.51607834924003
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.63459 (QuantReg: 12.74696) QuantErr: 12.74696 batch_time=39.83725
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.78996 (QuantReg: 12.86919) QuantErr: 12.86919 batch_time=0.49634
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.84005 (QuantReg: 12.93579) QuantErr: 12.93579 batch_time=0.57913
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.03123 (QuantReg: 12.95473) QuantErr: 12.95473 batch_time=0.49227
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.70738 (QuantReg: 12.86690) QuantErr: 12.86690 batch_time=0.51011
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.78375 (QuantReg: 12.99863) QuantErr: 12.99863 batch_time=0.51716
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.97031 (QuantReg: 13.24503) QuantErr: 13.24503 batch_time=0.57142
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.89689 (QuantReg: 12.61271) QuantErr: 12.61271 batch_time=0.53565
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.75875 (QuantReg: 12.94433) QuantErr: 12.94433 batch_time=0.50797
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.62599 (QuantReg: 12.99584) QuantErr: 12.99584 batch_time=0.51605
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.60447 (QuantReg: 13.03265) QuantErr: 13.03265 batch_time=0.49388
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.52329 (QuantReg: 13.23173) QuantErr: 13.23173 batch_time=0.51400
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.61105 (QuantReg: 12.85604) QuantErr: 12.85604 batch_time=0.54188
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.77008 (QuantReg: 12.69160) QuantErr: 12.69160 batch_time=0.54665
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.96272 (QuantReg: 13.38324) QuantErr: 13.38324 batch_time=0.49976
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.76983 (QuantReg: 13.08876) QuantErr: 13.08876 batch_time=0.51099
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.86110 (QuantReg: 13.23419) QuantErr: 13.23419 batch_time=0.53491
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.46604 (QuantReg: 13.07193) QuantErr: 13.07193 batch_time=0.50845
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.91896 (QuantReg: 13.06465) QuantErr: 13.06465 batch_time=0.49904
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.52780 (QuantReg: 13.24373) QuantErr: 13.24373 batch_time=0.51291
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.32368 (QuantReg: 13.44944) QuantErr: 13.44944 batch_time=0.52984
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.77890 (QuantReg: 13.33694) QuantErr: 13.33694 batch_time=0.50127
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.60484 (QuantReg: 13.09478) QuantErr: 13.09478 batch_time=0.53044
Train Epoch: 12 codebook_update_time=1.63596
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch12.pth ...
Done in 5.244s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch12.pth ...
Done in 10.747s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 1.7439429616928102
quant_reg : 12.995950000762939
quant_err : 12.995950000762939
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.684
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.36375655617999
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.988
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.58649247108604
mnt_best : 42.36375655617999
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.82434 (QuantReg: 12.68026) QuantErr: 12.68026 batch_time=31.92564
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.27408 (QuantReg: 13.29690) QuantErr: 13.29690 batch_time=0.51605
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.76175 (QuantReg: 12.81292) QuantErr: 12.81292 batch_time=0.50191
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.36757 (QuantReg: 13.17158) QuantErr: 13.17158 batch_time=0.58721
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.88309 (QuantReg: 13.12911) QuantErr: 13.12911 batch_time=0.53157
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.90915 (QuantReg: 12.66866) QuantErr: 12.66866 batch_time=0.56042
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.67356 (QuantReg: 13.18871) QuantErr: 13.18871 batch_time=1.61532
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.78581 (QuantReg: 13.19065) QuantErr: 13.19065 batch_time=0.48897
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.62998 (QuantReg: 13.21118) QuantErr: 13.21118 batch_time=0.52410
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.58014 (QuantReg: 13.05722) QuantErr: 13.05722 batch_time=0.51456
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.61380 (QuantReg: 13.19659) QuantErr: 13.19659 batch_time=0.54141
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.52323 (QuantReg: 13.49472) QuantErr: 13.49472 batch_time=0.52137
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.67205 (QuantReg: 13.02525) QuantErr: 13.02525 batch_time=0.52000
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.64906 (QuantReg: 13.14489) QuantErr: 13.14489 batch_time=2.52048
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.44369 (QuantReg: 13.18837) QuantErr: 13.18837 batch_time=0.52496
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.98255 (QuantReg: 13.12223) QuantErr: 13.12223 batch_time=0.51748
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.65386 (QuantReg: 13.46721) QuantErr: 13.46721 batch_time=0.51705
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.95480 (QuantReg: 13.11663) QuantErr: 13.11663 batch_time=0.53962
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.49160 (QuantReg: 13.32869) QuantErr: 13.32869 batch_time=0.52462
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.79606 (QuantReg: 13.08666) QuantErr: 13.08666 batch_time=1.55560
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.24853 (QuantReg: 13.40336) QuantErr: 13.40336 batch_time=0.49416
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.73606 (QuantReg: 13.25838) QuantErr: 13.25838 batch_time=0.55262
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.99832 (QuantReg: 13.17021) QuantErr: 13.17021 batch_time=0.50937
Train Epoch: 13 codebook_update_time=1.96136
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch13.pth ...
Done in 5.974s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 1.6922753291130066
quant_reg : 13.110749290466309
quant_err : 13.110749290466309
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.82
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.809422263354584
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.273
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.32970600839254
mnt_best : 42.36375655617999
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.64319 (QuantReg: 13.20434) QuantErr: 13.20434 batch_time=35.33237
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.46260 (QuantReg: 12.91609) QuantErr: 12.91609 batch_time=0.51640
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.62972 (QuantReg: 12.97278) QuantErr: 12.97278 batch_time=0.54454
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.49006 (QuantReg: 13.31707) QuantErr: 13.31707 batch_time=0.50038
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.49693 (QuantReg: 13.11675) QuantErr: 13.11675 batch_time=0.52010
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.95173 (QuantReg: 12.71265) QuantErr: 12.71265 batch_time=0.78497
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.89828 (QuantReg: 13.03141) QuantErr: 13.03141 batch_time=0.50034
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.60909 (QuantReg: 13.00791) QuantErr: 13.00791 batch_time=0.50083
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.52758 (QuantReg: 12.67600) QuantErr: 12.67600 batch_time=0.51988
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.38736 (QuantReg: 13.09623) QuantErr: 13.09623 batch_time=0.54416
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.10081 (QuantReg: 12.99480) QuantErr: 12.99480 batch_time=0.50651
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.40130 (QuantReg: 12.99684) QuantErr: 12.99684 batch_time=0.50820
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.49203 (QuantReg: 13.27675) QuantErr: 13.27675 batch_time=0.54188
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.83428 (QuantReg: 12.86754) QuantErr: 12.86754 batch_time=0.52364
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.52097 (QuantReg: 13.01545) QuantErr: 13.01545 batch_time=0.51456
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.91072 (QuantReg: 12.98574) QuantErr: 12.98574 batch_time=0.51402
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.50682 (QuantReg: 13.36252) QuantErr: 13.36252 batch_time=0.53199
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.79663 (QuantReg: 13.01803) QuantErr: 13.01803 batch_time=0.50876
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.58587 (QuantReg: 13.40602) QuantErr: 13.40602 batch_time=0.53923
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.71300 (QuantReg: 13.00551) QuantErr: 13.00551 batch_time=0.49646
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.78043 (QuantReg: 12.98787) QuantErr: 12.98787 batch_time=0.51022
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.39349 (QuantReg: 13.63028) QuantErr: 13.63028 batch_time=0.57180
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.23144 (QuantReg: 13.28108) QuantErr: 13.28108 batch_time=0.52248
Train Epoch: 14 codebook_update_time=1.94754
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch14.pth ...
Done in 6.679s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch14.pth ...
Done in 11.758s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 1.6245474772453308
quant_reg : 13.09201619720459
quant_err : 13.09201619720459
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.365
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.46161683320754
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.922
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.55312310225605
mnt_best : 42.46161683320754
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.49312 (QuantReg: 13.04050) QuantErr: 13.04050 batch_time=32.72462
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.12232 (QuantReg: 13.15921) QuantErr: 13.15921 batch_time=0.50573
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.58191 (QuantReg: 13.11031) QuantErr: 13.11031 batch_time=5.71491
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.34619 (QuantReg: 12.78996) QuantErr: 12.78996 batch_time=0.50041
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.76038 (QuantReg: 13.14167) QuantErr: 13.14167 batch_time=0.50856
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.70350 (QuantReg: 13.03809) QuantErr: 13.03809 batch_time=0.50951
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.77842 (QuantReg: 13.41546) QuantErr: 13.41546 batch_time=0.58950
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.25502 (QuantReg: 12.96999) QuantErr: 12.96999 batch_time=0.52590
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.87502 (QuantReg: 13.29750) QuantErr: 13.29750 batch_time=0.50935
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.44259 (QuantReg: 13.24115) QuantErr: 13.24115 batch_time=0.51260
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.83708 (QuantReg: 13.25389) QuantErr: 13.25389 batch_time=0.50855
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.30827 (QuantReg: 13.52037) QuantErr: 13.52037 batch_time=0.50458
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.59541 (QuantReg: 13.27620) QuantErr: 13.27620 batch_time=0.51809
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.45120 (QuantReg: 13.10831) QuantErr: 13.10831 batch_time=0.55253
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.65641 (QuantReg: 13.25088) QuantErr: 13.25088 batch_time=3.29773
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.40417 (QuantReg: 13.04260) QuantErr: 13.04260 batch_time=0.52250
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.38191 (QuantReg: 13.34822) QuantErr: 13.34822 batch_time=0.52983
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.63103 (QuantReg: 13.21704) QuantErr: 13.21704 batch_time=0.52396
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.87798 (QuantReg: 12.94954) QuantErr: 12.94954 batch_time=0.53485
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.40997 (QuantReg: 13.36344) QuantErr: 13.36344 batch_time=0.51920
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.28958 (QuantReg: 13.19466) QuantErr: 13.19466 batch_time=0.50506
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.61242 (QuantReg: 13.46070) QuantErr: 13.46070 batch_time=0.51074
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.44614 (QuantReg: 13.14462) QuantErr: 13.14462 batch_time=0.52270
Train Epoch: 15 codebook_update_time=4.23576
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch15.pth ...
Done in 6.766s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch15.pth ...
Done in 11.753s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 1.5585461993217469
quant_reg : 13.161257621765136
quant_err : 13.161257621765136
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.0
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.092600154720905
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.0965
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.468949869588805
mnt_best : 43.092600154720905
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.76745 (QuantReg: 12.92055) QuantErr: 12.92055 batch_time=33.67066
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.55195 (QuantReg: 12.71341) QuantErr: 12.71341 batch_time=0.49832
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.46570 (QuantReg: 13.01558) QuantErr: 13.01558 batch_time=0.49451
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.13783 (QuantReg: 12.58011) QuantErr: 12.58011 batch_time=0.50101
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.55003 (QuantReg: 13.32420) QuantErr: 13.32420 batch_time=0.54155
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.71898 (QuantReg: 13.36809) QuantErr: 13.36809 batch_time=0.63714
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.27222 (QuantReg: 13.31476) QuantErr: 13.31476 batch_time=1.26919
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.50812 (QuantReg: 12.89217) QuantErr: 12.89217 batch_time=0.50871
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.45471 (QuantReg: 13.06726) QuantErr: 13.06726 batch_time=0.50517
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.38398 (QuantReg: 13.40417) QuantErr: 13.40417 batch_time=0.54770
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.31187 (QuantReg: 13.42413) QuantErr: 13.42413 batch_time=0.50480
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.45777 (QuantReg: 13.30160) QuantErr: 13.30160 batch_time=0.49137
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.42767 (QuantReg: 13.21803) QuantErr: 13.21803 batch_time=1.40514
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.92268 (QuantReg: 13.20474) QuantErr: 13.20474 batch_time=0.51220
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.79577 (QuantReg: 13.40750) QuantErr: 13.40750 batch_time=0.51108
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.69654 (QuantReg: 13.33849) QuantErr: 13.33849 batch_time=0.51541
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.29746 (QuantReg: 13.58767) QuantErr: 13.58767 batch_time=0.50074
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.74167 (QuantReg: 13.22700) QuantErr: 13.22700 batch_time=0.50155
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.47839 (QuantReg: 13.33908) QuantErr: 13.33908 batch_time=0.52758
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.34898 (QuantReg: 13.45551) QuantErr: 13.45551 batch_time=2.26507
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.56185 (QuantReg: 13.21165) QuantErr: 13.21165 batch_time=0.51745
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.70120 (QuantReg: 13.06703) QuantErr: 13.06703 batch_time=0.50741
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.56317 (QuantReg: 13.13631) QuantErr: 13.13631 batch_time=0.61929
Train Epoch: 16 codebook_update_time=1.71108
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch16.pth ...
Done in 7.006s
removing stale ckpt [epoch 15] [took 0.04s]
epoch : 16
loss : 1.5521403017044066
quant_reg : 13.176825359344482
quant_err : 13.176825359344482
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.678
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.779626079069274
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.978
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.23745893639233
mnt_best : 43.092600154720905
not_improved_count: 1
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.28528 (QuantReg: 13.32855) QuantErr: 13.32855 batch_time=33.28939
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.60200 (QuantReg: 12.86671) QuantErr: 12.86671 batch_time=0.50271
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.39617 (QuantReg: 12.94753) QuantErr: 12.94753 batch_time=0.53685
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.97128 (QuantReg: 13.38254) QuantErr: 13.38254 batch_time=0.51378
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.60953 (QuantReg: 12.78701) QuantErr: 12.78701 batch_time=0.53504
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.45637 (QuantReg: 13.25560) QuantErr: 13.25560 batch_time=0.50495
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.68254 (QuantReg: 13.26466) QuantErr: 13.26466 batch_time=1.57467
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.33985 (QuantReg: 13.16271) QuantErr: 13.16271 batch_time=0.56060
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.46258 (QuantReg: 12.99123) QuantErr: 12.99123 batch_time=0.57802
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.61564 (QuantReg: 13.22685) QuantErr: 13.22685 batch_time=0.50514
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.51236 (QuantReg: 13.32696) QuantErr: 13.32696 batch_time=0.52707
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.48458 (QuantReg: 13.13571) QuantErr: 13.13571 batch_time=0.49749
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.66280 (QuantReg: 13.10326) QuantErr: 13.10326 batch_time=0.99052
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.49411 (QuantReg: 13.34410) QuantErr: 13.34410 batch_time=0.52218
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.85567 (QuantReg: 12.75261) QuantErr: 12.75261 batch_time=0.50890
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.39774 (QuantReg: 13.17993) QuantErr: 13.17993 batch_time=0.62233
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.61816 (QuantReg: 13.45013) QuantErr: 13.45013 batch_time=0.50168
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.29776 (QuantReg: 13.51090) QuantErr: 13.51090 batch_time=0.49678
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.45260 (QuantReg: 12.79092) QuantErr: 12.79092 batch_time=0.49769
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.29048 (QuantReg: 13.39479) QuantErr: 13.39479 batch_time=0.53237
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.49098 (QuantReg: 13.03592) QuantErr: 13.03592 batch_time=0.69488
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.56229 (QuantReg: 13.59987) QuantErr: 13.59987 batch_time=0.50388
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.22309 (QuantReg: 13.19243) QuantErr: 13.19243 batch_time=0.53171
Train Epoch: 17 codebook_update_time=1.73962
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch17.pth ...
Done in 8.939s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch17.pth ...
Done in 14.270s
removing stale ckpt [epoch 16] [took 0.30s]
epoch : 17
loss : 1.493693069934845
quant_reg : 13.231121612548828
quant_err : 13.231121612548828
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.127
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.652531861063295
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.4535
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.31800466282661
mnt_best : 43.652531861063295
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.56048 (QuantReg: 13.02396) QuantErr: 13.02396 batch_time=33.07595
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.41757 (QuantReg: 13.10549) QuantErr: 13.10549 batch_time=0.52066
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.33457 (QuantReg: 13.31039) QuantErr: 13.31039 batch_time=0.53672
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.21595 (QuantReg: 13.40996) QuantErr: 13.40996 batch_time=0.54469
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.38985 (QuantReg: 13.01509) QuantErr: 13.01509 batch_time=0.50585
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.24819 (QuantReg: 13.50473) QuantErr: 13.50473 batch_time=0.51286
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.06654 (QuantReg: 13.61776) QuantErr: 13.61776 batch_time=0.52579
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.26837 (QuantReg: 13.41543) QuantErr: 13.41543 batch_time=0.50736
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.69281 (QuantReg: 13.42345) QuantErr: 13.42345 batch_time=1.42209
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.39458 (QuantReg: 13.01768) QuantErr: 13.01768 batch_time=0.50871
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.79054 (QuantReg: 13.22392) QuantErr: 13.22392 batch_time=0.51251
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.54860 (QuantReg: 13.31817) QuantErr: 13.31817 batch_time=0.54129
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.54745 (QuantReg: 13.12117) QuantErr: 13.12117 batch_time=0.53399
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.68308 (QuantReg: 13.07404) QuantErr: 13.07404 batch_time=2.09911
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.54057 (QuantReg: 13.02049) QuantErr: 13.02049 batch_time=0.49897
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.47556 (QuantReg: 13.63540) QuantErr: 13.63540 batch_time=0.54850
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.32337 (QuantReg: 13.18001) QuantErr: 13.18001 batch_time=0.59974
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.21012 (QuantReg: 13.36845) QuantErr: 13.36845 batch_time=0.52037
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.32662 (QuantReg: 13.45258) QuantErr: 13.45258 batch_time=0.49835
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.04337 (QuantReg: 13.46272) QuantErr: 13.46272 batch_time=0.56057
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.82702 (QuantReg: 13.05875) QuantErr: 13.05875 batch_time=0.49827
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.30668 (QuantReg: 13.31885) QuantErr: 13.31885 batch_time=0.50353
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.39367 (QuantReg: 13.42600) QuantErr: 13.42600 batch_time=0.54164
Train Epoch: 18 codebook_update_time=1.70092
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch18.pth ...
Done in 4.799s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch18.pth ...
Done in 9.270s
removing stale ckpt [epoch 17] [took 0.31s]
epoch : 18
loss : 1.434643602371216
quant_reg : 13.281886882781983
quant_err : 13.281886882781983
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 24.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.008
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 44.28350273282908
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.544
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.74175040130827
mnt_best : 44.28350273282908
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.20656 (QuantReg: 13.33627) QuantErr: 13.33627 batch_time=31.80370
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.34155 (QuantReg: 13.27732) QuantErr: 13.27732 batch_time=0.56856
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.58979 (QuantReg: 12.98961) QuantErr: 12.98961 batch_time=0.49279
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.73395 (QuantReg: 13.19019) QuantErr: 13.19019 batch_time=0.50870
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.28592 (QuantReg: 13.14472) QuantErr: 13.14472 batch_time=0.59577
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.05359 (QuantReg: 13.46574) QuantErr: 13.46574 batch_time=0.50907
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.09192 (QuantReg: 13.38394) QuantErr: 13.38394 batch_time=0.51480
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.31405 (QuantReg: 13.09302) QuantErr: 13.09302 batch_time=0.50098
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.25222 (QuantReg: 13.28266) QuantErr: 13.28266 batch_time=0.51451
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.61876 (QuantReg: 13.01996) QuantErr: 13.01996 batch_time=0.49546
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.41761 (QuantReg: 13.08382) QuantErr: 13.08382 batch_time=0.70011
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.29076 (QuantReg: 13.50783) QuantErr: 13.50783 batch_time=0.51046
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.44972 (QuantReg: 13.59852) QuantErr: 13.59852 batch_time=0.51112
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.39768 (QuantReg: 13.36952) QuantErr: 13.36952 batch_time=4.90578
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.71348 (QuantReg: 13.08947) QuantErr: 13.08947 batch_time=0.48582
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.01028 (QuantReg: 13.28108) QuantErr: 13.28108 batch_time=0.79188
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.25263 (QuantReg: 13.29260) QuantErr: 13.29260 batch_time=0.50502
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.44840 (QuantReg: 13.23565) QuantErr: 13.23565 batch_time=0.51327
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.49570 (QuantReg: 13.02288) QuantErr: 13.02288 batch_time=0.49986
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.41359 (QuantReg: 13.32381) QuantErr: 13.32381 batch_time=0.52422
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.46004 (QuantReg: 13.47262) QuantErr: 13.47262 batch_time=0.49561
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.74302 (QuantReg: 13.17885) QuantErr: 13.17885 batch_time=0.50101
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.46851 (QuantReg: 13.67100) QuantErr: 13.67100 batch_time=0.51716
Train Epoch: 19 codebook_update_time=1.75533
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch19.pth ...
Done in 4.828s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA/checkpoint-epoch19.pth ...
Done in 10.383s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 1.4106206090450286
quant_reg : 13.308876224517823
quant_err : 13.308876224517823
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 24.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.2