-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_t0.1.txt
2607 lines (2607 loc) · 193 KB
/
HCQ_MSRVTT_1kA_t0.1.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 580.3060283660889 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 65.0928795337677 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 49.843714475631714 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch0.pth ...
Done in 1.351s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch0.pth ...
Done in 2.713s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 486.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 496.278
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 6.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 509.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.537
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.5192494101851104
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.72864 (QuantReg: 22.50035) QuantErr: 22.50035 batch_time=22.11962
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.86061 (QuantReg: 22.54588) QuantErr: 22.54588 batch_time=2.04117
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.64189 (QuantReg: 22.61770) QuantErr: 22.61770 batch_time=0.52358
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.60854 (QuantReg: 22.64102) QuantErr: 22.64102 batch_time=0.49055
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.68131 (QuantReg: 22.60759) QuantErr: 22.60759 batch_time=0.53055
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.95207 (QuantReg: 22.64045) QuantErr: 22.64045 batch_time=0.50141
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.95502 (QuantReg: 22.56469) QuantErr: 22.56469 batch_time=0.48731
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.41967 (QuantReg: 22.62769) QuantErr: 22.62769 batch_time=4.00723
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.83567 (QuantReg: 22.62117) QuantErr: 22.62117 batch_time=0.52938
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.71617 (QuantReg: 22.64704) QuantErr: 22.64704 batch_time=0.49819
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 4.95678 (QuantReg: 22.59793) QuantErr: 22.59793 batch_time=0.49593
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.11035 (QuantReg: 22.62016) QuantErr: 22.62016 batch_time=0.48839
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.43605 (QuantReg: 22.58220) QuantErr: 22.58220 batch_time=0.51601
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.77678 (QuantReg: 22.63476) QuantErr: 22.63476 batch_time=0.50135
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 5.05562 (QuantReg: 22.59694) QuantErr: 22.59694 batch_time=0.48688
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.56300 (QuantReg: 22.59225) QuantErr: 22.59225 batch_time=0.49077
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 5.10718 (QuantReg: 22.60695) QuantErr: 22.60695 batch_time=0.49167
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.85250 (QuantReg: 22.63480) QuantErr: 22.63480 batch_time=0.56775
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.41987 (QuantReg: 22.60013) QuantErr: 22.60013 batch_time=0.49231
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.74180 (QuantReg: 22.60040) QuantErr: 22.60040 batch_time=0.49168
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.04404 (QuantReg: 22.64661) QuantErr: 22.64661 batch_time=0.48757
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.60436 (QuantReg: 22.61704) QuantErr: 22.61704 batch_time=0.50754
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.06083 (QuantReg: 22.62713) QuantErr: 22.62713 batch_time=0.50419
Train Epoch: 1 codebook_update_time=1.96918
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch1.pth ...
Done in 3.953s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch1.pth ...
Done in 7.723s
epoch : 1
loss : 5.663895118713379
quant_reg : 22.609122581481934
quant_err : 22.609122581481934
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 9.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 29.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 43.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 76.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 14.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 45.172
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.051837083783767
MSRVTT_jsfusion_test/v2t_metrics/R1: 9.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 31.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 43.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 78.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 14.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 44.163
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.71693141099112
mnt_best : 23.051837083783767
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.46940 (QuantReg: 10.86223) QuantErr: 10.86223 batch_time=38.72787
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.27772 (QuantReg: 10.91354) QuantErr: 10.91354 batch_time=0.50085
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.36496 (QuantReg: 11.35099) QuantErr: 11.35099 batch_time=0.51473
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.24285 (QuantReg: 11.16632) QuantErr: 11.16632 batch_time=0.49120
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.33462 (QuantReg: 11.80266) QuantErr: 11.80266 batch_time=0.48644
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.34793 (QuantReg: 11.64189) QuantErr: 11.64189 batch_time=0.48990
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.11213 (QuantReg: 11.64567) QuantErr: 11.64567 batch_time=0.49327
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.32143 (QuantReg: 11.84380) QuantErr: 11.84380 batch_time=0.49913
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.02476 (QuantReg: 12.07098) QuantErr: 12.07098 batch_time=0.50578
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.61000 (QuantReg: 11.94986) QuantErr: 11.94986 batch_time=0.49530
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 4.06172 (QuantReg: 11.73707) QuantErr: 11.73707 batch_time=0.50625
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.81086 (QuantReg: 12.27945) QuantErr: 12.27945 batch_time=0.48795
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 4.04186 (QuantReg: 11.94534) QuantErr: 11.94534 batch_time=0.49434
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.46046 (QuantReg: 12.17510) QuantErr: 12.17510 batch_time=0.49717
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 4.34639 (QuantReg: 12.02823) QuantErr: 12.02823 batch_time=0.51608
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.96196 (QuantReg: 12.57843) QuantErr: 12.57843 batch_time=0.49779
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 4.26900 (QuantReg: 12.45613) QuantErr: 12.45613 batch_time=0.49432
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 4.02385 (QuantReg: 12.51158) QuantErr: 12.51158 batch_time=0.48752
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.76392 (QuantReg: 12.69954) QuantErr: 12.69954 batch_time=0.49539
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.76611 (QuantReg: 12.39642) QuantErr: 12.39642 batch_time=0.49920
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.85370 (QuantReg: 12.56629) QuantErr: 12.56629 batch_time=0.49037
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 4.01635 (QuantReg: 12.64751) QuantErr: 12.64751 batch_time=0.53676
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 4.11317 (QuantReg: 12.73294) QuantErr: 12.73294 batch_time=0.48627
Train Epoch: 2 codebook_update_time=1.65689
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch2.pth ...
Done in 3.829s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch2.pth ...
Done in 7.560s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 4.116101954460144
quant_reg : 12.002482055664062
quant_err : 12.002482055664062
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 36.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 52.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 82.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 10.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 37.039
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.46320274117325
MSRVTT_jsfusion_test/v2t_metrics/R1: 12.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 37.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 52.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 81.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 36.162
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.238325386929816
mnt_best : 28.46320274117325
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.83240 (QuantReg: 10.22933) QuantErr: 10.22933 batch_time=30.13282
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.70056 (QuantReg: 10.24414) QuantErr: 10.24414 batch_time=0.49743
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.65039 (QuantReg: 10.31859) QuantErr: 10.31859 batch_time=0.49127
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.50884 (QuantReg: 10.16239) QuantErr: 10.16239 batch_time=0.49027
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.47312 (QuantReg: 10.34955) QuantErr: 10.34955 batch_time=0.49189
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.94839 (QuantReg: 9.89073) QuantErr: 9.89073 batch_time=0.49423
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 4.16334 (QuantReg: 10.57747) QuantErr: 10.57747 batch_time=0.51448
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.94964 (QuantReg: 10.36439) QuantErr: 10.36439 batch_time=0.49239
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.89070 (QuantReg: 10.44915) QuantErr: 10.44915 batch_time=0.48424
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.22798 (QuantReg: 10.56716) QuantErr: 10.56716 batch_time=0.47793
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.67764 (QuantReg: 10.28881) QuantErr: 10.28881 batch_time=0.47769
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.60479 (QuantReg: 10.28554) QuantErr: 10.28554 batch_time=0.49121
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.35686 (QuantReg: 10.13151) QuantErr: 10.13151 batch_time=0.49282
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.63934 (QuantReg: 10.32685) QuantErr: 10.32685 batch_time=0.82457
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.97062 (QuantReg: 9.97324) QuantErr: 9.97324 batch_time=0.48843
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.35433 (QuantReg: 10.23594) QuantErr: 10.23594 batch_time=0.50344
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.72913 (QuantReg: 10.59612) QuantErr: 10.59612 batch_time=0.81510
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.62170 (QuantReg: 10.68999) QuantErr: 10.68999 batch_time=0.62583
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.36511 (QuantReg: 10.55196) QuantErr: 10.55196 batch_time=0.48970
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.46933 (QuantReg: 10.64839) QuantErr: 10.64839 batch_time=2.68237
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.41722 (QuantReg: 10.34787) QuantErr: 10.34787 batch_time=0.49211
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.50849 (QuantReg: 10.89046) QuantErr: 10.89046 batch_time=0.48387
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.59092 (QuantReg: 10.92802) QuantErr: 10.92802 batch_time=0.48653
Train Epoch: 3 codebook_update_time=2.04356
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch3.pth ...
Done in 3.974s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch3.pth ...
Done in 7.679s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.6068576641082766
quant_reg : 10.399535919189454
quant_err : 10.399535919189454
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 38.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 54.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 35.911
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.950283124694202
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 40.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 54.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.4475
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.10901362090149
mnt_best : 30.950283124694202
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.50324 (QuantReg: 9.94679) QuantErr: 9.94679 batch_time=30.70904
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.33671 (QuantReg: 9.67413) QuantErr: 9.67413 batch_time=0.49112
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 3.43165 (QuantReg: 9.87329) QuantErr: 9.87329 batch_time=0.49221
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.30565 (QuantReg: 9.95428) QuantErr: 9.95428 batch_time=0.50202
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.24870 (QuantReg: 9.99899) QuantErr: 9.99899 batch_time=0.50269
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.41778 (QuantReg: 9.78777) QuantErr: 9.78777 batch_time=0.52077
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.13041 (QuantReg: 9.69898) QuantErr: 9.69898 batch_time=0.60303
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 3.34425 (QuantReg: 10.06032) QuantErr: 10.06032 batch_time=0.49697
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.46784 (QuantReg: 9.82000) QuantErr: 9.82000 batch_time=0.51102
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.93857 (QuantReg: 9.94440) QuantErr: 9.94440 batch_time=0.53095
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.30972 (QuantReg: 10.38466) QuantErr: 10.38466 batch_time=0.52912
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.04727 (QuantReg: 10.50649) QuantErr: 10.50649 batch_time=0.76014
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.50061 (QuantReg: 9.94403) QuantErr: 9.94403 batch_time=0.54207
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 3.16112 (QuantReg: 10.19647) QuantErr: 10.19647 batch_time=0.49334
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.24569 (QuantReg: 10.03594) QuantErr: 10.03594 batch_time=0.49811
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 3.30750 (QuantReg: 9.87351) QuantErr: 9.87351 batch_time=0.49647
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.20334 (QuantReg: 9.90902) QuantErr: 9.90902 batch_time=0.49277
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 3.29682 (QuantReg: 10.04535) QuantErr: 10.04535 batch_time=0.50072
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 3.11456 (QuantReg: 10.51437) QuantErr: 10.51437 batch_time=0.59659
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.91948 (QuantReg: 9.88520) QuantErr: 9.88520 batch_time=0.48843
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.77324 (QuantReg: 10.32974) QuantErr: 10.32974 batch_time=0.50303
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.77930 (QuantReg: 10.36792) QuantErr: 10.36792 batch_time=0.49099
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 3.35477 (QuantReg: 10.03122) QuantErr: 10.03122 batch_time=0.50103
Train Epoch: 4 codebook_update_time=1.62712
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch4.pth ...
Done in 3.492s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch4.pth ...
Done in 6.939s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 3.3383344812393188
quant_reg : 9.977731983184814
quant_err : 9.977731983184814
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 13.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 56.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.846
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.639069338644518
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.5875
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.740157642114326
mnt_best : 31.639069338644518
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.14830 (QuantReg: 9.41833) QuantErr: 9.41833 batch_time=29.80333
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.52173 (QuantReg: 9.85665) QuantErr: 9.85665 batch_time=0.51369
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.78275 (QuantReg: 9.28660) QuantErr: 9.28660 batch_time=0.51251
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 3.11033 (QuantReg: 9.93257) QuantErr: 9.93257 batch_time=0.49030
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 3.25154 (QuantReg: 9.48491) QuantErr: 9.48491 batch_time=0.48996
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.96880 (QuantReg: 9.94692) QuantErr: 9.94692 batch_time=0.52163
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 3.12894 (QuantReg: 9.42271) QuantErr: 9.42271 batch_time=0.51311
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.58075 (QuantReg: 9.43358) QuantErr: 9.43358 batch_time=0.51356
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 3.11928 (QuantReg: 10.07358) QuantErr: 10.07358 batch_time=2.19666
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 3.09499 (QuantReg: 9.68175) QuantErr: 9.68175 batch_time=0.49090
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 3.07181 (QuantReg: 9.80957) QuantErr: 9.80957 batch_time=0.53847
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 3.06838 (QuantReg: 9.57728) QuantErr: 9.57728 batch_time=0.49265
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.94772 (QuantReg: 10.10006) QuantErr: 10.10006 batch_time=1.49301
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.67988 (QuantReg: 9.62407) QuantErr: 9.62407 batch_time=0.50160
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 3.45801 (QuantReg: 10.03938) QuantErr: 10.03938 batch_time=0.50913
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.88268 (QuantReg: 9.95325) QuantErr: 9.95325 batch_time=0.51005
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 3.26959 (QuantReg: 9.80315) QuantErr: 9.80315 batch_time=0.54445
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 3.17817 (QuantReg: 9.78887) QuantErr: 9.78887 batch_time=0.50211
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 3.05656 (QuantReg: 10.03403) QuantErr: 10.03403 batch_time=0.48499
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 3.02377 (QuantReg: 9.56311) QuantErr: 9.56311 batch_time=0.49600
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 3.28042 (QuantReg: 9.98679) QuantErr: 9.98679 batch_time=0.49576
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.91465 (QuantReg: 9.58417) QuantErr: 9.58417 batch_time=0.49492
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.27439 (QuantReg: 10.06102) QuantErr: 10.06102 batch_time=0.50187
Train Epoch: 5 codebook_update_time=1.95443
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch5.pth ...
Done in 3.814s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch5.pth ...
Done in 7.650s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 3.0971610994338987
quant_reg : 9.785357196807862
quant_err : 9.785357196807862
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.451
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.81473810936049
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.6115
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.71628849720981
mnt_best : 34.81473810936049
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 3.19242 (QuantReg: 9.63910) QuantErr: 9.63910 batch_time=30.58430
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 3.03627 (QuantReg: 9.18069) QuantErr: 9.18069 batch_time=0.48443
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 3.10491 (QuantReg: 9.42623) QuantErr: 9.42623 batch_time=0.48509
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 3.02023 (QuantReg: 9.59474) QuantErr: 9.59474 batch_time=0.49440
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 3.04169 (QuantReg: 9.44382) QuantErr: 9.44382 batch_time=0.49112
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.76882 (QuantReg: 9.43145) QuantErr: 9.43145 batch_time=0.47997
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 3.11898 (QuantReg: 9.85089) QuantErr: 9.85089 batch_time=2.08841
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.89750 (QuantReg: 9.67994) QuantErr: 9.67994 batch_time=0.52669
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.94661 (QuantReg: 9.67858) QuantErr: 9.67858 batch_time=1.01307
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.88180 (QuantReg: 9.38548) QuantErr: 9.38548 batch_time=0.51711
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.78970 (QuantReg: 10.10207) QuantErr: 10.10207 batch_time=0.48739
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.89566 (QuantReg: 9.58099) QuantErr: 9.58099 batch_time=0.50003
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.76434 (QuantReg: 9.60081) QuantErr: 9.60081 batch_time=0.49116
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.80948 (QuantReg: 9.66283) QuantErr: 9.66283 batch_time=0.49524
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 3.27386 (QuantReg: 9.88485) QuantErr: 9.88485 batch_time=0.49933
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.98981 (QuantReg: 9.91644) QuantErr: 9.91644 batch_time=0.75296
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.64511 (QuantReg: 9.94577) QuantErr: 9.94577 batch_time=0.49283
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 3.06818 (QuantReg: 9.61636) QuantErr: 9.61636 batch_time=0.49318
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 3.10139 (QuantReg: 9.88909) QuantErr: 9.88909 batch_time=0.49112
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.93314 (QuantReg: 9.78315) QuantErr: 9.78315 batch_time=1.02790
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.69122 (QuantReg: 9.72777) QuantErr: 9.72777 batch_time=0.49884
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.65570 (QuantReg: 9.47333) QuantErr: 9.47333 batch_time=0.49540
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.60243 (QuantReg: 9.76058) QuantErr: 9.76058 batch_time=0.49203
Train Epoch: 6 codebook_update_time=1.73493
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch6.pth ...
Done in 3.791s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch6.pth ...
Done in 7.532s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.9326532077789307
quant_reg : 9.70142077255249
quant_err : 9.70142077255249
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.763000000000005
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.38471788686317
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.713
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.37631041204469
mnt_best : 35.38471788686317
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.83880 (QuantReg: 9.18692) QuantErr: 9.18692 batch_time=29.86332
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.90276 (QuantReg: 9.47926) QuantErr: 9.47926 batch_time=0.52197
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.84201 (QuantReg: 9.21509) QuantErr: 9.21509 batch_time=0.49985
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.57911 (QuantReg: 9.51920) QuantErr: 9.51920 batch_time=0.48829
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.53867 (QuantReg: 9.45364) QuantErr: 9.45364 batch_time=1.24061
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.58127 (QuantReg: 9.65558) QuantErr: 9.65558 batch_time=0.49448
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.85877 (QuantReg: 9.88630) QuantErr: 9.88630 batch_time=1.16467
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.76005 (QuantReg: 9.74204) QuantErr: 9.74204 batch_time=0.50279
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.80723 (QuantReg: 9.90124) QuantErr: 9.90124 batch_time=0.48816
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.44116 (QuantReg: 9.59467) QuantErr: 9.59467 batch_time=0.49076
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 3.21275 (QuantReg: 9.84635) QuantErr: 9.84635 batch_time=0.50068
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.85899 (QuantReg: 9.65301) QuantErr: 9.65301 batch_time=0.49457
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.64719 (QuantReg: 9.60212) QuantErr: 9.60212 batch_time=0.48299
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.78005 (QuantReg: 9.43339) QuantErr: 9.43339 batch_time=1.61390
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.38550 (QuantReg: 9.44075) QuantErr: 9.44075 batch_time=1.41016
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.95449 (QuantReg: 9.60253) QuantErr: 9.60253 batch_time=0.48273
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 2.36094 (QuantReg: 9.72150) QuantErr: 9.72150 batch_time=0.61092
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.70890 (QuantReg: 9.33549) QuantErr: 9.33549 batch_time=0.48530
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.26723 (QuantReg: 9.70781) QuantErr: 9.70781 batch_time=0.48618
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.93541 (QuantReg: 9.81332) QuantErr: 9.81332 batch_time=0.48061
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.78007 (QuantReg: 9.85630) QuantErr: 9.85630 batch_time=0.54903
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.65252 (QuantReg: 10.34734) QuantErr: 10.34734 batch_time=0.50183
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.72946 (QuantReg: 9.86432) QuantErr: 9.86432 batch_time=0.50082
Train Epoch: 7 codebook_update_time=1.64240
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch7.pth ...
Done in 3.535s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch7.pth ...
Done in 7.100s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 2.7711694765090944
quant_reg : 9.641656204223633
quant_err : 9.641656204223633
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.565
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.47067137970474
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.25
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.4675
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.5744582834143
mnt_best : 36.47067137970474
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.81027 (QuantReg: 9.31973) QuantErr: 9.31973 batch_time=33.75843
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.01110 (QuantReg: 9.41481) QuantErr: 9.41481 batch_time=0.48920
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 3.27391 (QuantReg: 9.70032) QuantErr: 9.70032 batch_time=0.48062
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.72711 (QuantReg: 9.34221) QuantErr: 9.34221 batch_time=0.48887
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.90792 (QuantReg: 9.86984) QuantErr: 9.86984 batch_time=0.49438
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.53216 (QuantReg: 9.14048) QuantErr: 9.14048 batch_time=0.48563
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.72600 (QuantReg: 9.71461) QuantErr: 9.71461 batch_time=0.49060
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.91482 (QuantReg: 9.71495) QuantErr: 9.71495 batch_time=0.49295
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.83563 (QuantReg: 9.48641) QuantErr: 9.48641 batch_time=0.48799
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.97491 (QuantReg: 9.68197) QuantErr: 9.68197 batch_time=0.48575
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 3.01283 (QuantReg: 9.90831) QuantErr: 9.90831 batch_time=0.52762
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.63054 (QuantReg: 9.42866) QuantErr: 9.42866 batch_time=0.54909
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.71033 (QuantReg: 9.44931) QuantErr: 9.44931 batch_time=0.49327
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.87027 (QuantReg: 9.98133) QuantErr: 9.98133 batch_time=0.49827
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.67672 (QuantReg: 9.81962) QuantErr: 9.81962 batch_time=0.48880
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.52597 (QuantReg: 9.49983) QuantErr: 9.49983 batch_time=0.48696
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.47114 (QuantReg: 9.37856) QuantErr: 9.37856 batch_time=0.48682
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 3.05070 (QuantReg: 9.64623) QuantErr: 9.64623 batch_time=0.49604
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.83044 (QuantReg: 9.58219) QuantErr: 9.58219 batch_time=0.47348
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.91903 (QuantReg: 9.39868) QuantErr: 9.39868 batch_time=0.50604
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.73754 (QuantReg: 9.61070) QuantErr: 9.61070 batch_time=0.71522
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 2.23604 (QuantReg: 9.61161) QuantErr: 9.61161 batch_time=0.48959
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.94983 (QuantReg: 9.63801) QuantErr: 9.63801 batch_time=0.49282
Train Epoch: 8 codebook_update_time=1.63170
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch8.pth ...
Done in 3.856s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch8.pth ...
Done in 7.492s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 2.6848056564331055
quant_reg : 9.5786618309021
quant_err : 9.5786618309021
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.65
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.51330372417762
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.319
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.07619554756409
mnt_best : 37.51330372417762
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.29839 (QuantReg: 9.73059) QuantErr: 9.73059 batch_time=28.06571
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.85137 (QuantReg: 9.61797) QuantErr: 9.61797 batch_time=0.52087
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.24530 (QuantReg: 9.19103) QuantErr: 9.19103 batch_time=0.48336
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.79197 (QuantReg: 9.56480) QuantErr: 9.56480 batch_time=0.48811
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.58608 (QuantReg: 9.67307) QuantErr: 9.67307 batch_time=0.48747
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.28615 (QuantReg: 9.67513) QuantErr: 9.67513 batch_time=0.50463
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.36056 (QuantReg: 9.64466) QuantErr: 9.64466 batch_time=0.53814
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 2.56422 (QuantReg: 9.80729) QuantErr: 9.80729 batch_time=0.48612
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.82083 (QuantReg: 9.36161) QuantErr: 9.36161 batch_time=0.52272
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.36693 (QuantReg: 9.61005) QuantErr: 9.61005 batch_time=0.50500
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.53910 (QuantReg: 9.73296) QuantErr: 9.73296 batch_time=0.49701
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 2.25059 (QuantReg: 9.49848) QuantErr: 9.49848 batch_time=0.48783
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.60499 (QuantReg: 9.60397) QuantErr: 9.60397 batch_time=0.49648
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.42434 (QuantReg: 9.29066) QuantErr: 9.29066 batch_time=0.51271
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.55375 (QuantReg: 9.13209) QuantErr: 9.13209 batch_time=0.49068
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 2.28059 (QuantReg: 9.83150) QuantErr: 9.83150 batch_time=0.85509
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.78970 (QuantReg: 9.74045) QuantErr: 9.74045 batch_time=0.48938
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.19370 (QuantReg: 9.78735) QuantErr: 9.78735 batch_time=0.48432
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.70947 (QuantReg: 9.63132) QuantErr: 9.63132 batch_time=1.93840
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.38146 (QuantReg: 9.75740) QuantErr: 9.75740 batch_time=0.49558
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.43722 (QuantReg: 9.87451) QuantErr: 9.87451 batch_time=0.49004
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.30521 (QuantReg: 9.10001) QuantErr: 9.10001 batch_time=0.52780
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 2.50077 (QuantReg: 9.33168) QuantErr: 9.33168 batch_time=0.48591
Train Epoch: 9 codebook_update_time=2.14416
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch9.pth ...
Done in 3.674s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch9.pth ...
Done in 7.984s
removing stale ckpt [epoch 8] [took 0.12s]
epoch : 9
loss : 2.5567427854537965
quant_reg : 9.542099376678467
quant_err : 9.542099376678467
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.221
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.59147799923033
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.1705
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.738190186156004
mnt_best : 37.59147799923033
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 2.38536 (QuantReg: 9.67764) QuantErr: 9.67764 batch_time=30.48762
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.56422 (QuantReg: 9.45977) QuantErr: 9.45977 batch_time=0.51807
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.90116 (QuantReg: 9.69552) QuantErr: 9.69552 batch_time=0.48710
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.47518 (QuantReg: 9.30733) QuantErr: 9.30733 batch_time=0.50133
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 2.25215 (QuantReg: 9.46809) QuantErr: 9.46809 batch_time=0.48898
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.43461 (QuantReg: 9.43931) QuantErr: 9.43931 batch_time=0.50053
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.77915 (QuantReg: 9.33525) QuantErr: 9.33525 batch_time=4.14909
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 2.61005 (QuantReg: 9.61377) QuantErr: 9.61377 batch_time=0.49732
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.63868 (QuantReg: 9.47823) QuantErr: 9.47823 batch_time=0.50273
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.48730 (QuantReg: 9.59982) QuantErr: 9.59982 batch_time=0.48974
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.33769 (QuantReg: 9.39418) QuantErr: 9.39418 batch_time=0.49459
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.64018 (QuantReg: 9.64541) QuantErr: 9.64541 batch_time=0.49581
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.53596 (QuantReg: 9.22797) QuantErr: 9.22797 batch_time=0.48364
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.22587 (QuantReg: 9.73843) QuantErr: 9.73843 batch_time=0.48809
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 2.45415 (QuantReg: 9.22719) QuantErr: 9.22719 batch_time=0.49005
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.76764 (QuantReg: 9.75758) QuantErr: 9.75758 batch_time=0.49151
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.55755 (QuantReg: 9.50246) QuantErr: 9.50246 batch_time=0.50408
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.37079 (QuantReg: 9.54652) QuantErr: 9.54652 batch_time=0.49466
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.25566 (QuantReg: 9.67558) QuantErr: 9.67558 batch_time=0.49350
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.03592 (QuantReg: 9.92904) QuantErr: 9.92904 batch_time=0.50396
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.58172 (QuantReg: 9.41388) QuantErr: 9.41388 batch_time=0.50041
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.84317 (QuantReg: 9.64842) QuantErr: 9.64842 batch_time=0.51022
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.54267 (QuantReg: 9.78411) QuantErr: 9.78411 batch_time=1.17927
Train Epoch: 10 codebook_update_time=1.65519
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch10.pth ...
Done in 3.785s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 2.4656791424751283
quant_reg : 9.525885345458985
quant_err : 9.525885345458985
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.984
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.31868671490555
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.885999999999996
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.599735926921575
mnt_best : 37.59147799923033
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.40934 (QuantReg: 9.30815) QuantErr: 9.30815 batch_time=31.18437
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.73987 (QuantReg: 8.96557) QuantErr: 8.96557 batch_time=0.74532
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.60405 (QuantReg: 9.38265) QuantErr: 9.38265 batch_time=0.48804
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.54224 (QuantReg: 9.50514) QuantErr: 9.50514 batch_time=0.84103
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.51277 (QuantReg: 9.55641) QuantErr: 9.55641 batch_time=0.49259
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.89642 (QuantReg: 10.03425) QuantErr: 10.03425 batch_time=0.48720
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 2.14247 (QuantReg: 9.46838) QuantErr: 9.46838 batch_time=0.54095
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.50295 (QuantReg: 9.18902) QuantErr: 9.18902 batch_time=0.49979
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.43287 (QuantReg: 9.37777) QuantErr: 9.37777 batch_time=0.50168
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.49761 (QuantReg: 9.62738) QuantErr: 9.62738 batch_time=0.49599
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.16467 (QuantReg: 9.65242) QuantErr: 9.65242 batch_time=0.49375
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.20811 (QuantReg: 9.62554) QuantErr: 9.62554 batch_time=0.85216
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.40849 (QuantReg: 9.31787) QuantErr: 9.31787 batch_time=0.50651
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.26398 (QuantReg: 9.33727) QuantErr: 9.33727 batch_time=2.99346
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 2.25032 (QuantReg: 9.61778) QuantErr: 9.61778 batch_time=0.49616
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.64244 (QuantReg: 9.08008) QuantErr: 9.08008 batch_time=0.62551
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.29360 (QuantReg: 9.31682) QuantErr: 9.31682 batch_time=0.49683
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 2.40968 (QuantReg: 9.71679) QuantErr: 9.71679 batch_time=0.48735
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 2.37158 (QuantReg: 9.28170) QuantErr: 9.28170 batch_time=0.49609
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 2.14555 (QuantReg: 9.52020) QuantErr: 9.52020 batch_time=0.53417
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 2.27736 (QuantReg: 9.44239) QuantErr: 9.44239 batch_time=0.49734
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.32141 (QuantReg: 9.65203) QuantErr: 9.65203 batch_time=0.49788
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 2.29963 (QuantReg: 9.75886) QuantErr: 9.75886 batch_time=0.49351
Train Epoch: 11 codebook_update_time=1.69097
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch11.pth ...
Done in 3.708s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch11.pth ...
Done in 7.362s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 2.384103579521179
quant_reg : 9.48477638244629
quant_err : 9.48477638244629
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.384
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.252522709432625
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.3045
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.88933717790115
mnt_best : 38.252522709432625
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 2.11449 (QuantReg: 9.27649) QuantErr: 9.27649 batch_time=30.78999
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 2.25001 (QuantReg: 9.34333) QuantErr: 9.34333 batch_time=0.52104
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 2.47443 (QuantReg: 9.64352) QuantErr: 9.64352 batch_time=0.50813
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.50630 (QuantReg: 9.45511) QuantErr: 9.45511 batch_time=0.49679
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.21741 (QuantReg: 9.38172) QuantErr: 9.38172 batch_time=0.48484
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 2.37237 (QuantReg: 9.46223) QuantErr: 9.46223 batch_time=0.48659
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 2.69746 (QuantReg: 9.84826) QuantErr: 9.84826 batch_time=0.71062
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 2.43956 (QuantReg: 9.05841) QuantErr: 9.05841 batch_time=0.49862
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 2.30345 (QuantReg: 9.44781) QuantErr: 9.44781 batch_time=0.48836
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 2.12838 (QuantReg: 9.48652) QuantErr: 9.48652 batch_time=0.50522
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 2.16181 (QuantReg: 9.60023) QuantErr: 9.60023 batch_time=0.51380
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 2.16175 (QuantReg: 9.61017) QuantErr: 9.61017 batch_time=0.51581
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 2.20014 (QuantReg: 9.31535) QuantErr: 9.31535 batch_time=0.49197
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 2.22725 (QuantReg: 9.21107) QuantErr: 9.21107 batch_time=0.49896
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.47065 (QuantReg: 9.90779) QuantErr: 9.90779 batch_time=0.49610
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 2.35254 (QuantReg: 9.59808) QuantErr: 9.59808 batch_time=0.49744
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.43036 (QuantReg: 9.62940) QuantErr: 9.62940 batch_time=0.49085
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 2.10412 (QuantReg: 9.58029) QuantErr: 9.58029 batch_time=0.57937
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.43190 (QuantReg: 9.39917) QuantErr: 9.39917 batch_time=1.76646
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 2.11855 (QuantReg: 9.59662) QuantErr: 9.59662 batch_time=0.49682
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 2.24742 (QuantReg: 9.69092) QuantErr: 9.69092 batch_time=0.49410
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.40626 (QuantReg: 9.74696) QuantErr: 9.74696 batch_time=0.49646
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 2.18498 (QuantReg: 9.62266) QuantErr: 9.62266 batch_time=0.48589
Train Epoch: 12 codebook_update_time=1.64617
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch12.pth ...
Done in 4.551s
removing stale ckpt [epoch 11] [took 0.14s]
epoch : 12
loss : 2.331424171447754
quant_reg : 9.486841495513916
quant_err : 9.486841495513916
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.74
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.127640952949946
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.04
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.02531426884725
mnt_best : 38.252522709432625
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.53322 (QuantReg: 9.27503) QuantErr: 9.27503 batch_time=29.79680
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.91094 (QuantReg: 9.88780) QuantErr: 9.88780 batch_time=0.49981
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 2.36599 (QuantReg: 9.18738) QuantErr: 9.18738 batch_time=0.49843
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 2.04824 (QuantReg: 9.56615) QuantErr: 9.56615 batch_time=0.55694
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.36082 (QuantReg: 9.64798) QuantErr: 9.64798 batch_time=0.51492
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.44866 (QuantReg: 9.24005) QuantErr: 9.24005 batch_time=0.49525
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 2.22392 (QuantReg: 9.64757) QuantErr: 9.64757 batch_time=0.49521
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 2.44642 (QuantReg: 9.65150) QuantErr: 9.65150 batch_time=0.49092
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 2.17700 (QuantReg: 9.80140) QuantErr: 9.80140 batch_time=0.52906
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 2.08176 (QuantReg: 9.42358) QuantErr: 9.42358 batch_time=0.49890
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 2.10235 (QuantReg: 9.74873) QuantErr: 9.74873 batch_time=0.48889
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 2.18536 (QuantReg: 9.88563) QuantErr: 9.88563 batch_time=0.50800
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 2.27006 (QuantReg: 9.52752) QuantErr: 9.52752 batch_time=0.49232
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 2.30005 (QuantReg: 9.55157) QuantErr: 9.55157 batch_time=0.49338
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 2.07592 (QuantReg: 9.52597) QuantErr: 9.52597 batch_time=0.49616
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 2.51921 (QuantReg: 9.69378) QuantErr: 9.69378 batch_time=0.53395
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 2.30076 (QuantReg: 9.66043) QuantErr: 9.66043 batch_time=0.49613
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 2.44190 (QuantReg: 9.38868) QuantErr: 9.38868 batch_time=0.50388
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 2.05023 (QuantReg: 9.64091) QuantErr: 9.64091 batch_time=0.49461
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.42604 (QuantReg: 9.64409) QuantErr: 9.64409 batch_time=0.49675
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.87210 (QuantReg: 9.54976) QuantErr: 9.54976 batch_time=0.49919
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 2.23656 (QuantReg: 9.40701) QuantErr: 9.40701 batch_time=0.51592
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.39542 (QuantReg: 9.44969) QuantErr: 9.44969 batch_time=0.50727
Train Epoch: 13 codebook_update_time=1.77833
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch13.pth ...
Done in 4.854s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch13.pth ...
Done in 9.376s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 2.2748138079643248
quant_reg : 9.52313892364502
quant_err : 9.52313892364502
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.326
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.46663892564602
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.472
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.284249849161974
mnt_best : 39.46663892564602
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 2.14093 (QuantReg: 9.54477) QuantErr: 9.54477 batch_time=30.30536
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.00832 (QuantReg: 9.31634) QuantErr: 9.31634 batch_time=0.48724
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 2.24190 (QuantReg: 9.31850) QuantErr: 9.31850 batch_time=0.49035
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.15006 (QuantReg: 9.74110) QuantErr: 9.74110 batch_time=0.48642
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.92766 (QuantReg: 9.53401) QuantErr: 9.53401 batch_time=0.50626
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.57036 (QuantReg: 9.23355) QuantErr: 9.23355 batch_time=0.51831
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.31130 (QuantReg: 9.23649) QuantErr: 9.23649 batch_time=0.86958
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 2.22358 (QuantReg: 9.49637) QuantErr: 9.49637 batch_time=0.51594
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.97408 (QuantReg: 9.14165) QuantErr: 9.14165 batch_time=0.50223
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.96732 (QuantReg: 9.47742) QuantErr: 9.47742 batch_time=0.49345
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.55068 (QuantReg: 9.39376) QuantErr: 9.39376 batch_time=0.49380
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.97406 (QuantReg: 9.32196) QuantErr: 9.32196 batch_time=0.49529
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 2.10132 (QuantReg: 9.80006) QuantErr: 9.80006 batch_time=0.91310
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 2.41854 (QuantReg: 9.20525) QuantErr: 9.20525 batch_time=2.77479
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 2.24475 (QuantReg: 9.51583) QuantErr: 9.51583 batch_time=0.50153
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 2.34531 (QuantReg: 9.36305) QuantErr: 9.36305 batch_time=0.49557
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 2.14856 (QuantReg: 9.65460) QuantErr: 9.65460 batch_time=0.49186
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 2.36306 (QuantReg: 9.46800) QuantErr: 9.46800 batch_time=0.48726
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 2.27717 (QuantReg: 9.63980) QuantErr: 9.63980 batch_time=0.50036
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 2.21962 (QuantReg: 9.51023) QuantErr: 9.51023 batch_time=1.62546
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.41130 (QuantReg: 9.47942) QuantErr: 9.47942 batch_time=0.49960
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.05001 (QuantReg: 9.82123) QuantErr: 9.82123 batch_time=1.92620
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.90021 (QuantReg: 9.53992) QuantErr: 9.53992 batch_time=0.61971
Train Epoch: 14 codebook_update_time=1.68330
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch14.pth ...
Done in 4.870s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch14.pth ...
Done in 10.037s
removing stale ckpt [epoch 13] [took 0.18s]
epoch : 14
loss : 2.205806387901306
quant_reg : 9.485227931976318
quant_err : 9.485227931976318
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.269
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.797119392872006
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.6965
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.30940225219971
mnt_best : 39.797119392872006
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.02397 (QuantReg: 9.46259) QuantErr: 9.46259 batch_time=33.43825
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.70135 (QuantReg: 9.58187) QuantErr: 9.58187 batch_time=0.49253
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.19857 (QuantReg: 9.48112) QuantErr: 9.48112 batch_time=0.50931
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.89863 (QuantReg: 9.17630) QuantErr: 9.17630 batch_time=0.69466
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.30843 (QuantReg: 9.40145) QuantErr: 9.40145 batch_time=0.54421
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 2.18355 (QuantReg: 9.54176) QuantErr: 9.54176 batch_time=0.50502
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.19086 (QuantReg: 9.86026) QuantErr: 9.86026 batch_time=0.49174
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.94932 (QuantReg: 9.21495) QuantErr: 9.21495 batch_time=0.49121
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.39852 (QuantReg: 9.42834) QuantErr: 9.42834 batch_time=0.48774
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 2.13463 (QuantReg: 9.34700) QuantErr: 9.34700 batch_time=0.48868
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.46101 (QuantReg: 9.71591) QuantErr: 9.71591 batch_time=1.54644
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.00486 (QuantReg: 9.80306) QuantErr: 9.80306 batch_time=0.48997
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.20157 (QuantReg: 9.52336) QuantErr: 9.52336 batch_time=0.53900
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.89040 (QuantReg: 9.48224) QuantErr: 9.48224 batch_time=0.53125
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.07387 (QuantReg: 9.55949) QuantErr: 9.55949 batch_time=1.19777
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.00111 (QuantReg: 9.40024) QuantErr: 9.40024 batch_time=0.59826
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.99941 (QuantReg: 9.52767) QuantErr: 9.52767 batch_time=0.48583
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 2.05889 (QuantReg: 9.40068) QuantErr: 9.40068 batch_time=0.48759
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 2.50238 (QuantReg: 9.41088) QuantErr: 9.41088 batch_time=0.50027
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.01197 (QuantReg: 9.64679) QuantErr: 9.64679 batch_time=0.49679
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.98682 (QuantReg: 9.52711) QuantErr: 9.52711 batch_time=0.49581
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.17119 (QuantReg: 9.73941) QuantErr: 9.73941 batch_time=0.50142
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.97192 (QuantReg: 9.49454) QuantErr: 9.49454 batch_time=0.51135
Train Epoch: 15 codebook_update_time=2.13022
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch15.pth ...
Done in 6.144s
removing stale ckpt [epoch 14] [took 0.22s]
epoch : 15
loss : 2.1482748951911925
quant_reg : 9.47901425933838
quant_err : 9.47901425933838
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.546
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.63505723589466
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.696
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.000677363387794
mnt_best : 39.797119392872006
not_improved_count: 1
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 2.44750 (QuantReg: 9.31842) QuantErr: 9.31842 batch_time=37.84227
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 2.23178 (QuantReg: 9.22139) QuantErr: 9.22139 batch_time=0.49032
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 2.10834 (QuantReg: 9.50517) QuantErr: 9.50517 batch_time=0.49695
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.64229 (QuantReg: 9.09835) QuantErr: 9.09835 batch_time=0.49586
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 2.10152 (QuantReg: 9.48261) QuantErr: 9.48261 batch_time=0.95199
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 2.34849 (QuantReg: 9.75940) QuantErr: 9.75940 batch_time=0.50179
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.92237 (QuantReg: 9.58540) QuantErr: 9.58540 batch_time=1.98563
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 2.17433 (QuantReg: 9.22745) QuantErr: 9.22745 batch_time=0.50053
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 2.18947 (QuantReg: 9.42910) QuantErr: 9.42910 batch_time=0.56263
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 2.12650 (QuantReg: 9.64673) QuantErr: 9.64673 batch_time=0.50710
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.94268 (QuantReg: 9.53347) QuantErr: 9.53347 batch_time=0.49295
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.95502 (QuantReg: 9.45826) QuantErr: 9.45826 batch_time=0.50716
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.01072 (QuantReg: 9.46668) QuantErr: 9.46668 batch_time=0.50822
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 2.51658 (QuantReg: 9.65252) QuantErr: 9.65252 batch_time=0.53732
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.46942 (QuantReg: 9.51564) QuantErr: 9.51564 batch_time=0.54738
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 2.14655 (QuantReg: 9.57120) QuantErr: 9.57120 batch_time=0.49841
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.89608 (QuantReg: 9.72315) QuantErr: 9.72315 batch_time=0.49623
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 2.28223 (QuantReg: 9.36240) QuantErr: 9.36240 batch_time=0.49851
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.07459 (QuantReg: 9.36920) QuantErr: 9.36920 batch_time=0.49308
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.93003 (QuantReg: 9.74138) QuantErr: 9.74138 batch_time=0.50459
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.12055 (QuantReg: 9.56170) QuantErr: 9.56170 batch_time=0.49991
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 2.26580 (QuantReg: 9.35172) QuantErr: 9.35172 batch_time=0.50967
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 2.17557 (QuantReg: 9.55688) QuantErr: 9.55688 batch_time=0.50385
Train Epoch: 16 codebook_update_time=1.65593
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch16.pth ...
Done in 5.005s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 2.127503707408905
quant_reg : 9.473912899017334
quant_err : 9.473912899017334
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.598
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.965081170632054
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.9165
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.66359677026227
mnt_best : 39.797119392872006
not_improved_count: 2
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.88900 (QuantReg: 9.62059) QuantErr: 9.62059 batch_time=33.19253
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 2.19476 (QuantReg: 9.24748) QuantErr: 9.24748 batch_time=0.49178
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.95346 (QuantReg: 9.24429) QuantErr: 9.24429 batch_time=0.48510
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.43746 (QuantReg: 9.51454) QuantErr: 9.51454 batch_time=0.49398
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 2.13720 (QuantReg: 9.12741) QuantErr: 9.12741 batch_time=0.49333
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 2.24557 (QuantReg: 9.60202) QuantErr: 9.60202 batch_time=0.49173
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 2.23497 (QuantReg: 9.60245) QuantErr: 9.60245 batch_time=1.83264
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.86802 (QuantReg: 9.41389) QuantErr: 9.41389 batch_time=0.48955
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 2.08113 (QuantReg: 9.17775) QuantErr: 9.17775 batch_time=0.50133
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 2.21117 (QuantReg: 9.62494) QuantErr: 9.62494 batch_time=0.49688
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 2.24049 (QuantReg: 9.50612) QuantErr: 9.50612 batch_time=0.86306
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 2.02219 (QuantReg: 9.42718) QuantErr: 9.42718 batch_time=0.48525
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 2.28850 (QuantReg: 9.35152) QuantErr: 9.35152 batch_time=0.50459
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.16225 (QuantReg: 9.71611) QuantErr: 9.71611 batch_time=0.54698
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 2.39480 (QuantReg: 9.09637) QuantErr: 9.09637 batch_time=0.51261
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.04353 (QuantReg: 9.41478) QuantErr: 9.41478 batch_time=0.49347
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 2.37721 (QuantReg: 9.61967) QuantErr: 9.61967 batch_time=0.52710
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.87382 (QuantReg: 9.76468) QuantErr: 9.76468 batch_time=0.51114
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.15212 (QuantReg: 9.12536) QuantErr: 9.12536 batch_time=0.49487
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.86842 (QuantReg: 9.61090) QuantErr: 9.61090 batch_time=0.53287
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 2.06366 (QuantReg: 9.25966) QuantErr: 9.25966 batch_time=0.50542
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 2.18535 (QuantReg: 9.75673) QuantErr: 9.75673 batch_time=0.49644
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.91888 (QuantReg: 9.32607) QuantErr: 9.32607 batch_time=0.48644
Train Epoch: 17 codebook_update_time=1.63388
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch17.pth ...
Done in 5.023s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch17.pth ...
Done in 9.491s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 2.079463959693909
quant_reg : 9.487652618408204
quant_err : 9.487652618408204
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.648
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.92841362787737
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.469
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.427390137881574
mnt_best : 39.92841362787737
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.18817 (QuantReg: 9.27655) QuantErr: 9.27655 batch_time=30.79454
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 2.09474 (QuantReg: 9.33127) QuantErr: 9.33127 batch_time=0.50976
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.89985 (QuantReg: 9.37823) QuantErr: 9.37823 batch_time=0.49531
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.81998 (QuantReg: 9.59665) QuantErr: 9.59665 batch_time=0.50920
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 2.06309 (QuantReg: 9.35103) QuantErr: 9.35103 batch_time=0.48677
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.95872 (QuantReg: 9.69836) QuantErr: 9.69836 batch_time=0.48628
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.70346 (QuantReg: 9.75180) QuantErr: 9.75180 batch_time=0.48810
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.90102 (QuantReg: 9.50720) QuantErr: 9.50720 batch_time=0.48890
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 2.26679 (QuantReg: 9.67892) QuantErr: 9.67892 batch_time=0.51207
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.89577 (QuantReg: 9.31644) QuantErr: 9.31644 batch_time=0.48638
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 2.23298 (QuantReg: 9.32967) QuantErr: 9.32967 batch_time=0.48857
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 2.22017 (QuantReg: 9.55166) QuantErr: 9.55166 batch_time=0.47840
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.89814 (QuantReg: 9.39357) QuantErr: 9.39357 batch_time=0.51812
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 2.24644 (QuantReg: 9.40077) QuantErr: 9.40077 batch_time=0.51818
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 2.02180 (QuantReg: 9.07625) QuantErr: 9.07625 batch_time=0.51660
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 2.05266 (QuantReg: 9.72806) QuantErr: 9.72806 batch_time=0.50239
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.92145 (QuantReg: 9.21167) QuantErr: 9.21167 batch_time=0.50027
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.90495 (QuantReg: 9.38070) QuantErr: 9.38070 batch_time=0.51938
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.07466 (QuantReg: 9.54251) QuantErr: 9.54251 batch_time=0.49664
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.77834 (QuantReg: 9.51788) QuantErr: 9.51788 batch_time=0.48003
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 2.36863 (QuantReg: 9.21364) QuantErr: 9.21364 batch_time=0.48803
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.05046 (QuantReg: 9.60699) QuantErr: 9.60699 batch_time=0.49744
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 2.05010 (QuantReg: 9.49549) QuantErr: 9.49549 batch_time=0.49906
Train Epoch: 18 codebook_update_time=1.70037
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch18.pth ...
Done in 4.776s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch18.pth ...
Done in 9.919s
removing stale ckpt [epoch 17] [took 0.02s]
epoch : 18
loss : 2.034042736053467
quant_reg : 9.484231956481933
quant_err : 9.484231956481933
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.32
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.604534101760294
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.546
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.035774447244314
mnt_best : 40.604534101760294
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.97753 (QuantReg: 9.57644) QuantErr: 9.57644 batch_time=30.80254
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.85638 (QuantReg: 9.44366) QuantErr: 9.44366 batch_time=0.48385
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 2.23271 (QuantReg: 9.29142) QuantErr: 9.29142 batch_time=0.53151
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 2.25988 (QuantReg: 9.57191) QuantErr: 9.57191 batch_time=0.53448
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.92459 (QuantReg: 9.39207) QuantErr: 9.39207 batch_time=0.49019
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.77733 (QuantReg: 9.66562) QuantErr: 9.66562 batch_time=0.48691
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.73160 (QuantReg: 9.48920) QuantErr: 9.48920 batch_time=0.49881
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.93183 (QuantReg: 9.32518) QuantErr: 9.32518 batch_time=0.49891
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.78948 (QuantReg: 9.35692) QuantErr: 9.35692 batch_time=0.48650
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 2.12482 (QuantReg: 8.96240) QuantErr: 8.96240 batch_time=0.48659
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 2.01671 (QuantReg: 9.31042) QuantErr: 9.31042 batch_time=0.48571
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.94538 (QuantReg: 9.62839) QuantErr: 9.62839 batch_time=0.49131
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 2.11849 (QuantReg: 9.80778) QuantErr: 9.80778 batch_time=0.49908
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 2.02845 (QuantReg: 9.56951) QuantErr: 9.56951 batch_time=0.51334
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 2.33820 (QuantReg: 9.36170) QuantErr: 9.36170 batch_time=0.50369
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.55194 (QuantReg: 9.37651) QuantErr: 9.37651 batch_time=0.52312
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.91818 (QuantReg: 9.45245) QuantErr: 9.45245 batch_time=0.50144
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 2.01264 (QuantReg: 9.30678) QuantErr: 9.30678 batch_time=0.48849
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.93326 (QuantReg: 9.18018) QuantErr: 9.18018 batch_time=1.06320
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.83884 (QuantReg: 9.33594) QuantErr: 9.33594 batch_time=0.50763
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 2.15902 (QuantReg: 9.78810) QuantErr: 9.78810 batch_time=0.49448
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 2.19399 (QuantReg: 9.38041) QuantErr: 9.38041 batch_time=0.47834
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.99786 (QuantReg: 9.70623) QuantErr: 9.70623 batch_time=0.53011
Train Epoch: 19 codebook_update_time=1.61517
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch19.pth ...
Done in 5.609s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.1/checkpoint-epoch19.pth ...
Done in 10.541s
removing stale ckpt [epoch 18] [took 0.02s]
epoch : 19
loss : 2.006883062839508
quant_reg : 9.462685665130616
quant_err : 9.462685665130616
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.0