-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_L31.txt
2607 lines (2607 loc) · 195 KB
/
HCQ_MSRVTT_1kA_L31.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 758.2213871479034 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 79.00582242012024 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 51.26963424682617 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch0.pth ...
Done in 1.518s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch0.pth ...
Done in 3.061s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 1.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 5.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 493.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 497.737
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.48202845283504603
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 5.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 498.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 505.3195
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.4641588833612779
mnt_best : 0.48202845283504603
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 29.89663 (QuantReg: 22.59459) QuantErr: 22.59459 batch_time=30.78541
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 27.79935 (QuantReg: 22.60187) QuantErr: 22.60187 batch_time=0.99034
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 24.80278 (QuantReg: 22.65713) QuantErr: 22.65713 batch_time=0.92532
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 21.40261 (QuantReg: 22.65371) QuantErr: 22.65371 batch_time=2.34367
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 19.69526 (QuantReg: 22.66428) QuantErr: 22.66428 batch_time=0.94328
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 19.55890 (QuantReg: 22.66284) QuantErr: 22.66284 batch_time=0.97457
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 18.00274 (QuantReg: 22.66956) QuantErr: 22.66956 batch_time=1.00565
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 17.55100 (QuantReg: 22.65466) QuantErr: 22.65466 batch_time=1.02917
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 17.55083 (QuantReg: 22.67546) QuantErr: 22.67546 batch_time=1.28303
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 15.37063 (QuantReg: 22.67007) QuantErr: 22.67007 batch_time=0.97952
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 17.11647 (QuantReg: 22.67727) QuantErr: 22.67727 batch_time=0.95425
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 16.70647 (QuantReg: 22.64173) QuantErr: 22.64173 batch_time=0.96146
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 15.36116 (QuantReg: 22.61941) QuantErr: 22.61941 batch_time=0.96741
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 16.66200 (QuantReg: 22.63877) QuantErr: 22.63877 batch_time=0.97336
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 14.85268 (QuantReg: 22.65720) QuantErr: 22.65720 batch_time=1.06641
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 13.63951 (QuantReg: 22.64886) QuantErr: 22.64886 batch_time=0.93828
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 14.76163 (QuantReg: 22.64895) QuantErr: 22.64895 batch_time=0.91645
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 14.14204 (QuantReg: 22.64174) QuantErr: 22.64174 batch_time=0.94333
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 14.44629 (QuantReg: 22.62582) QuantErr: 22.62582 batch_time=0.93147
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 14.17005 (QuantReg: 22.61928) QuantErr: 22.61928 batch_time=0.94994
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 15.47453 (QuantReg: 22.63895) QuantErr: 22.63895 batch_time=0.92971
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 13.17490 (QuantReg: 22.64299) QuantErr: 22.64299 batch_time=0.97028
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 13.37563 (QuantReg: 22.63142) QuantErr: 22.63142 batch_time=1.04302
Train Epoch: 1 codebook_update_time=9.06706
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch1.pth ...
Done in 3.911s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch1.pth ...
Done in 7.860s
epoch : 1
loss : 17.15393857574463
quant_reg : 22.643875175476076
quant_err : 22.643875175476076
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 9.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 29.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 44.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 77.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 14.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 44.959
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.04281878610671
MSRVTT_jsfusion_test/v2t_metrics/R1: 10.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 30.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 44.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 78.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 13.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 45.4105
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.02088979818282
mnt_best : 23.04281878610671
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 11.96206 (QuantReg: 10.71960) QuantErr: 10.71960 batch_time=31.38387
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 11.20891 (QuantReg: 10.91537) QuantErr: 10.91537 batch_time=0.90360
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 13.36488 (QuantReg: 11.30145) QuantErr: 11.30145 batch_time=1.01445
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 13.74975 (QuantReg: 11.39367) QuantErr: 11.39367 batch_time=0.92499
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 11.57125 (QuantReg: 11.94589) QuantErr: 11.94589 batch_time=1.06594
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 12.73718 (QuantReg: 11.78610) QuantErr: 11.78610 batch_time=1.02020
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 12.91330 (QuantReg: 12.04603) QuantErr: 12.04603 batch_time=0.92175
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 12.24374 (QuantReg: 12.31382) QuantErr: 12.31382 batch_time=0.96735
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 10.92780 (QuantReg: 12.37266) QuantErr: 12.37266 batch_time=0.91331
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 12.75985 (QuantReg: 12.51236) QuantErr: 12.51236 batch_time=0.92098
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 11.55750 (QuantReg: 12.09157) QuantErr: 12.09157 batch_time=0.98177
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 12.40373 (QuantReg: 12.77722) QuantErr: 12.77722 batch_time=0.89583
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 11.48190 (QuantReg: 12.99397) QuantErr: 12.99397 batch_time=1.08634
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 12.47346 (QuantReg: 12.94310) QuantErr: 12.94310 batch_time=1.11984
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 10.96754 (QuantReg: 12.74644) QuantErr: 12.74644 batch_time=0.92053
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 10.50140 (QuantReg: 13.27167) QuantErr: 13.27167 batch_time=0.94573
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 10.90514 (QuantReg: 13.08710) QuantErr: 13.08710 batch_time=0.95792
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 10.48556 (QuantReg: 13.85788) QuantErr: 13.85788 batch_time=0.91899
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 11.78537 (QuantReg: 13.14997) QuantErr: 13.14997 batch_time=0.91749
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 11.29182 (QuantReg: 13.70763) QuantErr: 13.70763 batch_time=0.97704
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 11.84098 (QuantReg: 13.56371) QuantErr: 13.56371 batch_time=0.94195
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 11.39963 (QuantReg: 13.31595) QuantErr: 13.31595 batch_time=1.01952
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 10.78001 (QuantReg: 14.17490) QuantErr: 14.17490 batch_time=0.94436
Train Epoch: 2 codebook_update_time=7.87210
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch2.pth ...
Done in 3.962s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch2.pth ...
Done in 7.805s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 11.7922179107666
quant_reg : 12.609162910461425
quant_err : 12.609162910461425
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 13.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 38.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 52.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 83.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 35.246
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.03937939726272
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 54.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 84.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.482
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.2342805831229
mnt_best : 30.03937939726272
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 10.05675 (QuantReg: 11.06346) QuantErr: 11.06346 batch_time=31.62554
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 9.89226 (QuantReg: 11.23579) QuantErr: 11.23579 batch_time=0.98339
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 12.29387 (QuantReg: 11.31264) QuantErr: 11.31264 batch_time=0.90123
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 9.87854 (QuantReg: 11.44449) QuantErr: 11.44449 batch_time=0.93215
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 9.67014 (QuantReg: 11.27541) QuantErr: 11.27541 batch_time=0.88623
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 8.94147 (QuantReg: 11.78840) QuantErr: 11.78840 batch_time=0.97009
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 9.25067 (QuantReg: 11.66095) QuantErr: 11.66095 batch_time=1.10417
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 10.42632 (QuantReg: 11.64065) QuantErr: 11.64065 batch_time=1.39603
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 10.39649 (QuantReg: 11.44969) QuantErr: 11.44969 batch_time=0.90346
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 10.87288 (QuantReg: 11.73318) QuantErr: 11.73318 batch_time=0.90002
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 9.10459 (QuantReg: 11.84946) QuantErr: 11.84946 batch_time=0.92381
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 9.69590 (QuantReg: 11.28630) QuantErr: 11.28630 batch_time=0.94042
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 10.27170 (QuantReg: 12.06752) QuantErr: 12.06752 batch_time=0.92411
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 9.66831 (QuantReg: 12.02289) QuantErr: 12.02289 batch_time=0.95652
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 11.47026 (QuantReg: 12.53286) QuantErr: 12.53286 batch_time=0.91148
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 8.13538 (QuantReg: 11.97316) QuantErr: 11.97316 batch_time=0.94703
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 10.28497 (QuantReg: 12.07623) QuantErr: 12.07623 batch_time=0.99789
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 8.93641 (QuantReg: 12.16033) QuantErr: 12.16033 batch_time=1.08799
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 9.73000 (QuantReg: 12.04339) QuantErr: 12.04339 batch_time=0.96547
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 9.11312 (QuantReg: 11.86605) QuantErr: 11.86605 batch_time=1.00852
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 10.25916 (QuantReg: 12.14403) QuantErr: 12.14403 batch_time=1.15069
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 8.65078 (QuantReg: 12.62565) QuantErr: 12.62565 batch_time=0.92940
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 8.92398 (QuantReg: 12.89548) QuantErr: 12.89548 batch_time=1.06786
Train Epoch: 3 codebook_update_time=8.29778
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch3.pth ...
Done in 4.090s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch3.pth ...
Done in 8.017s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 10.021158624649049
quant_reg : 11.845990913391113
quant_err : 11.845990913391113
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.701
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.08396737267789
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.6885
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.73546853157946
mnt_best : 33.08396737267789
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 8.33785 (QuantReg: 11.35355) QuantErr: 11.35355 batch_time=35.23021
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 8.66937 (QuantReg: 11.14832) QuantErr: 11.14832 batch_time=0.90819
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 8.97486 (QuantReg: 11.02062) QuantErr: 11.02062 batch_time=0.99545
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 9.08324 (QuantReg: 11.56318) QuantErr: 11.56318 batch_time=0.91705
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 8.19890 (QuantReg: 11.69799) QuantErr: 11.69799 batch_time=0.91260
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 10.53128 (QuantReg: 11.33715) QuantErr: 11.33715 batch_time=1.00389
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 8.83362 (QuantReg: 11.43836) QuantErr: 11.43836 batch_time=0.91980
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 9.30530 (QuantReg: 11.36424) QuantErr: 11.36424 batch_time=1.07547
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 8.98974 (QuantReg: 11.60433) QuantErr: 11.60433 batch_time=0.95317
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 7.47298 (QuantReg: 11.86235) QuantErr: 11.86235 batch_time=0.92475
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 8.75361 (QuantReg: 11.83891) QuantErr: 11.83891 batch_time=0.89723
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 9.62312 (QuantReg: 12.00655) QuantErr: 12.00655 batch_time=1.04381
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 8.65762 (QuantReg: 12.16595) QuantErr: 12.16595 batch_time=0.91905
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 7.88925 (QuantReg: 11.99639) QuantErr: 11.99639 batch_time=0.96546
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 8.61724 (QuantReg: 12.01461) QuantErr: 12.01461 batch_time=0.98742
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 8.20235 (QuantReg: 12.33181) QuantErr: 12.33181 batch_time=1.77863
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 8.03303 (QuantReg: 11.86526) QuantErr: 11.86526 batch_time=0.93650
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 8.54440 (QuantReg: 12.91282) QuantErr: 12.91282 batch_time=1.05449
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 8.45618 (QuantReg: 11.90481) QuantErr: 11.90481 batch_time=1.00377
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 8.47740 (QuantReg: 12.08376) QuantErr: 12.08376 batch_time=1.09401
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 9.54697 (QuantReg: 11.73556) QuantErr: 11.73556 batch_time=1.02689
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 8.43944 (QuantReg: 12.02864) QuantErr: 12.02864 batch_time=0.93305
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 8.02739 (QuantReg: 12.35726) QuantErr: 12.35726 batch_time=1.02819
Train Epoch: 4 codebook_update_time=7.52439
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch4.pth ...
Done in 3.737s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch4.pth ...
Done in 7.461s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 9.017835218429566
quant_reg : 11.804735935211182
quant_err : 11.804735935211182
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 56.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.669
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.10347608855931
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 44.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.3035
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.79696821073893
mnt_best : 35.10347608855931
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 9.22158 (QuantReg: 11.42153) QuantErr: 11.42153 batch_time=34.44043
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 8.58214 (QuantReg: 11.65588) QuantErr: 11.65588 batch_time=0.94217
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 7.99061 (QuantReg: 11.39879) QuantErr: 11.39879 batch_time=0.93521
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 7.79143 (QuantReg: 11.61077) QuantErr: 11.61077 batch_time=0.90177
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 8.27966 (QuantReg: 11.37678) QuantErr: 11.37678 batch_time=0.98652
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 7.90541 (QuantReg: 11.63148) QuantErr: 11.63148 batch_time=0.93518
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 7.44440 (QuantReg: 11.81653) QuantErr: 11.81653 batch_time=0.90109
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 8.48808 (QuantReg: 12.11825) QuantErr: 12.11825 batch_time=0.96461
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 8.80060 (QuantReg: 11.57908) QuantErr: 11.57908 batch_time=1.08354
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 8.07593 (QuantReg: 12.08099) QuantErr: 12.08099 batch_time=0.99750
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 7.39989 (QuantReg: 11.76292) QuantErr: 11.76292 batch_time=0.92819
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 7.43649 (QuantReg: 11.87479) QuantErr: 11.87479 batch_time=0.99809
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 7.64015 (QuantReg: 12.22478) QuantErr: 12.22478 batch_time=0.92036
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 10.30635 (QuantReg: 11.68318) QuantErr: 11.68318 batch_time=0.88589
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 7.78902 (QuantReg: 12.06333) QuantErr: 12.06333 batch_time=0.91892
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 7.29179 (QuantReg: 12.08873) QuantErr: 12.08873 batch_time=0.97913
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 8.25657 (QuantReg: 12.16709) QuantErr: 12.16709 batch_time=1.02546
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 10.35471 (QuantReg: 12.09931) QuantErr: 12.09931 batch_time=0.92534
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 9.36801 (QuantReg: 11.98592) QuantErr: 11.98592 batch_time=1.08109
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 6.52090 (QuantReg: 12.27213) QuantErr: 12.27213 batch_time=0.97228
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 8.29219 (QuantReg: 12.21975) QuantErr: 12.21975 batch_time=1.12171
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 8.53941 (QuantReg: 12.42756) QuantErr: 12.42756 batch_time=1.03812
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 8.00555 (QuantReg: 12.50358) QuantErr: 12.50358 batch_time=1.06316
Train Epoch: 5 codebook_update_time=7.43942
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch5.pth ...
Done in 11.514s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch5.pth ...
Done in 26.626s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 8.239827648162843
quant_reg : 11.93704672241211
quant_err : 11.93704672241211
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.531
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.46138657495352
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.75
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.006
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.503121019167125
mnt_best : 36.46138657495352
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 8.17666 (QuantReg: 11.91494) QuantErr: 11.91494 batch_time=27.12494
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 8.65153 (QuantReg: 11.41498) QuantErr: 11.41498 batch_time=0.92710
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 8.08615 (QuantReg: 11.90158) QuantErr: 11.90158 batch_time=0.90656
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 8.36471 (QuantReg: 11.76171) QuantErr: 11.76171 batch_time=0.90913
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 8.22078 (QuantReg: 12.16668) QuantErr: 12.16668 batch_time=1.21503
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 7.57943 (QuantReg: 11.75274) QuantErr: 11.75274 batch_time=0.92900
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 7.59143 (QuantReg: 12.30954) QuantErr: 12.30954 batch_time=3.12774
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 8.11985 (QuantReg: 12.23140) QuantErr: 12.23140 batch_time=2.29333
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 6.73448 (QuantReg: 11.69144) QuantErr: 11.69144 batch_time=0.92124
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 7.65239 (QuantReg: 11.88413) QuantErr: 11.88413 batch_time=0.93343
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 7.34239 (QuantReg: 11.99796) QuantErr: 11.99796 batch_time=1.00850
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 7.49199 (QuantReg: 12.14928) QuantErr: 12.14928 batch_time=1.03189
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 7.80537 (QuantReg: 12.05897) QuantErr: 12.05897 batch_time=0.90203
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 7.88249 (QuantReg: 11.97816) QuantErr: 11.97816 batch_time=0.95022
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 8.42248 (QuantReg: 11.74190) QuantErr: 11.74190 batch_time=1.02504
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 6.82735 (QuantReg: 12.36847) QuantErr: 12.36847 batch_time=0.99383
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 6.69309 (QuantReg: 12.37868) QuantErr: 12.37868 batch_time=0.92366
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 6.54963 (QuantReg: 12.08106) QuantErr: 12.08106 batch_time=0.98931
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 8.98012 (QuantReg: 11.98508) QuantErr: 11.98508 batch_time=0.99792
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 7.04940 (QuantReg: 12.30404) QuantErr: 12.30404 batch_time=1.05763
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 8.10459 (QuantReg: 12.07385) QuantErr: 12.07385 batch_time=1.20169
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 6.88451 (QuantReg: 12.33691) QuantErr: 12.33691 batch_time=1.18100
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 5.94051 (QuantReg: 12.40258) QuantErr: 12.40258 batch_time=1.31314
Train Epoch: 6 codebook_update_time=7.97058
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch6.pth ...
Done in 4.269s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch6.pth ...
Done in 8.039s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 7.547654710769653
quant_reg : 12.01076031112671
quant_err : 12.01076031112671
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.322
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.88231686272776
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.527
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.16453127983747
mnt_best : 36.88231686272776
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 8.48895 (QuantReg: 11.51937) QuantErr: 11.51937 batch_time=29.69923
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 7.37632 (QuantReg: 11.90444) QuantErr: 11.90444 batch_time=0.93598
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 7.89413 (QuantReg: 11.84307) QuantErr: 11.84307 batch_time=0.95112
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 6.44785 (QuantReg: 11.48405) QuantErr: 11.48405 batch_time=1.03876
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 9.42924 (QuantReg: 11.83582) QuantErr: 11.83582 batch_time=1.05023
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 7.68100 (QuantReg: 12.16512) QuantErr: 12.16512 batch_time=0.93392
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 6.94985 (QuantReg: 12.40033) QuantErr: 12.40033 batch_time=0.97006
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 7.95409 (QuantReg: 11.86034) QuantErr: 11.86034 batch_time=1.06155
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 7.30388 (QuantReg: 12.08392) QuantErr: 12.08392 batch_time=1.08066
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 7.96718 (QuantReg: 11.90397) QuantErr: 11.90397 batch_time=1.39453
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 8.05359 (QuantReg: 12.22099) QuantErr: 12.22099 batch_time=0.92865
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 6.81834 (QuantReg: 12.14985) QuantErr: 12.14985 batch_time=0.97151
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 7.63637 (QuantReg: 12.09117) QuantErr: 12.09117 batch_time=1.19584
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 7.38614 (QuantReg: 11.99434) QuantErr: 11.99434 batch_time=0.90936
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 5.78378 (QuantReg: 12.31098) QuantErr: 12.31098 batch_time=1.06329
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 6.29795 (QuantReg: 12.62754) QuantErr: 12.62754 batch_time=1.05515
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 6.48530 (QuantReg: 12.46906) QuantErr: 12.46906 batch_time=0.99500
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 7.73289 (QuantReg: 12.59218) QuantErr: 12.59218 batch_time=1.07689
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 6.95243 (QuantReg: 12.51854) QuantErr: 12.51854 batch_time=0.96348
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 7.26615 (QuantReg: 12.36139) QuantErr: 12.36139 batch_time=1.41002
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 7.30516 (QuantReg: 12.39910) QuantErr: 12.39910 batch_time=1.12272
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 6.57745 (QuantReg: 12.20697) QuantErr: 12.20697 batch_time=1.00151
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 7.36373 (QuantReg: 11.80540) QuantErr: 11.80540 batch_time=1.15966
Train Epoch: 7 codebook_update_time=7.34115
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch7.pth ...
Done in 3.833s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch7.pth ...
Done in 8.018s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 7.113863626480103
quant_reg : 12.131113471984863
quant_err : 12.131113471984863
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.908
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.953837758490266
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.383
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.885297704246504
mnt_best : 37.953837758490266
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 6.41432 (QuantReg: 11.69509) QuantErr: 11.69509 batch_time=30.38549
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 7.10320 (QuantReg: 12.24871) QuantErr: 12.24871 batch_time=0.90819
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 7.09839 (QuantReg: 12.23728) QuantErr: 12.23728 batch_time=0.90330
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 6.34128 (QuantReg: 12.08358) QuantErr: 12.08358 batch_time=1.71839
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 5.61586 (QuantReg: 11.84954) QuantErr: 11.84954 batch_time=1.00358
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 6.70553 (QuantReg: 12.43398) QuantErr: 12.43398 batch_time=0.93559
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 7.22322 (QuantReg: 12.29445) QuantErr: 12.29445 batch_time=0.96017
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 7.09875 (QuantReg: 12.28846) QuantErr: 12.28846 batch_time=0.91366
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 6.62183 (QuantReg: 12.02832) QuantErr: 12.02832 batch_time=1.10359
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 5.15735 (QuantReg: 12.04876) QuantErr: 12.04876 batch_time=1.04629
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 6.31785 (QuantReg: 11.77727) QuantErr: 11.77727 batch_time=1.07045
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 7.48994 (QuantReg: 11.89030) QuantErr: 11.89030 batch_time=0.95713
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 6.17583 (QuantReg: 12.10913) QuantErr: 12.10913 batch_time=1.08092
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 7.28262 (QuantReg: 12.06775) QuantErr: 12.06775 batch_time=0.94872
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 6.23295 (QuantReg: 12.16010) QuantErr: 12.16010 batch_time=1.54345
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 7.22641 (QuantReg: 12.22238) QuantErr: 12.22238 batch_time=1.14010
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 5.75925 (QuantReg: 12.51381) QuantErr: 12.51381 batch_time=1.01661
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 7.54344 (QuantReg: 12.20480) QuantErr: 12.20480 batch_time=0.97614
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 7.43318 (QuantReg: 12.56181) QuantErr: 12.56181 batch_time=1.00886
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 6.33734 (QuantReg: 12.40749) QuantErr: 12.40749 batch_time=1.12113
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 7.07535 (QuantReg: 12.41529) QuantErr: 12.41529 batch_time=0.95434
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 6.46835 (QuantReg: 12.61271) QuantErr: 12.61271 batch_time=1.05415
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 6.77307 (QuantReg: 12.15791) QuantErr: 12.15791 batch_time=1.13828
Train Epoch: 8 codebook_update_time=8.25736
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch8.pth ...
Done in 4.047s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch8.pth ...
Done in 10.035s
removing stale ckpt [epoch 7] [took 0.03s]
epoch : 8
loss : 6.690860202789307
quant_reg : 12.189224243164062
quant_err : 12.189224243164062
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.027
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.85788910712561
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 49.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.468
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.9428350096816
mnt_best : 38.85788910712561
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 7.14910 (QuantReg: 11.59703) QuantErr: 11.59703 batch_time=29.55181
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 5.23460 (QuantReg: 12.16928) QuantErr: 12.16928 batch_time=0.93048
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 6.90129 (QuantReg: 12.04012) QuantErr: 12.04012 batch_time=0.94399
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 7.92804 (QuantReg: 11.64655) QuantErr: 11.64655 batch_time=1.46300
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 5.12842 (QuantReg: 11.84109) QuantErr: 11.84109 batch_time=0.93826
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 6.76341 (QuantReg: 12.21298) QuantErr: 12.21298 batch_time=0.92406
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 6.30157 (QuantReg: 12.14713) QuantErr: 12.14713 batch_time=0.91918
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 6.47254 (QuantReg: 11.88835) QuantErr: 11.88835 batch_time=0.91026
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 6.88298 (QuantReg: 12.25117) QuantErr: 12.25117 batch_time=0.96648
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 6.55942 (QuantReg: 12.17188) QuantErr: 12.17188 batch_time=1.07633
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 7.12662 (QuantReg: 12.32297) QuantErr: 12.32297 batch_time=0.94582
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 6.11418 (QuantReg: 12.18241) QuantErr: 12.18241 batch_time=1.06500
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 6.55718 (QuantReg: 12.00848) QuantErr: 12.00848 batch_time=0.90624
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 6.52577 (QuantReg: 12.53968) QuantErr: 12.53968 batch_time=0.92025
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 6.58682 (QuantReg: 12.38054) QuantErr: 12.38054 batch_time=0.94259
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 5.87319 (QuantReg: 12.46218) QuantErr: 12.46218 batch_time=0.92593
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 5.82963 (QuantReg: 12.76253) QuantErr: 12.76253 batch_time=0.92317
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 6.06493 (QuantReg: 12.03833) QuantErr: 12.03833 batch_time=1.15599
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 5.85999 (QuantReg: 12.42848) QuantErr: 12.42848 batch_time=0.96139
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 6.65382 (QuantReg: 12.51868) QuantErr: 12.51868 batch_time=1.80233
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 6.59486 (QuantReg: 12.79606) QuantErr: 12.79606 batch_time=0.96475
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 6.30577 (QuantReg: 12.67943) QuantErr: 12.67943 batch_time=0.91039
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 6.17443 (QuantReg: 12.23636) QuantErr: 12.23636 batch_time=1.04583
Train Epoch: 9 codebook_update_time=8.05266
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch9.pth ...
Done in 4.914s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch9.pth ...
Done in 9.530s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 6.257290893554687
quant_reg : 12.268948093414307
quant_err : 12.268948093414307
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.177
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.36689911451742
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.6935
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.532045698882804
mnt_best : 39.36689911451742
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 5.84402 (QuantReg: 12.12847) QuantErr: 12.12847 batch_time=35.48949
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 5.62168 (QuantReg: 12.34149) QuantErr: 12.34149 batch_time=0.89203
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 6.39787 (QuantReg: 12.27483) QuantErr: 12.27483 batch_time=0.98425
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 6.27447 (QuantReg: 11.81720) QuantErr: 11.81720 batch_time=0.90150
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 5.35615 (QuantReg: 12.03295) QuantErr: 12.03295 batch_time=0.97227
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 6.89126 (QuantReg: 12.42489) QuantErr: 12.42489 batch_time=0.94835
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 5.85166 (QuantReg: 12.20419) QuantErr: 12.20419 batch_time=0.95519
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 5.58666 (QuantReg: 12.19190) QuantErr: 12.19190 batch_time=1.05072
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 5.70173 (QuantReg: 12.44352) QuantErr: 12.44352 batch_time=0.90295
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 8.35999 (QuantReg: 12.40791) QuantErr: 12.40791 batch_time=0.94859
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 6.90363 (QuantReg: 12.05187) QuantErr: 12.05187 batch_time=0.91304
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 5.70661 (QuantReg: 11.99230) QuantErr: 11.99230 batch_time=0.97803
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 7.10211 (QuantReg: 12.17511) QuantErr: 12.17511 batch_time=1.14004
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 6.76533 (QuantReg: 12.21097) QuantErr: 12.21097 batch_time=2.18834
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 6.64756 (QuantReg: 12.37872) QuantErr: 12.37872 batch_time=0.90778
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 6.50409 (QuantReg: 12.35318) QuantErr: 12.35318 batch_time=0.95673
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 5.89025 (QuantReg: 12.36266) QuantErr: 12.36266 batch_time=0.95328
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 6.48579 (QuantReg: 12.15399) QuantErr: 12.15399 batch_time=1.06218
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 6.57148 (QuantReg: 12.27868) QuantErr: 12.27868 batch_time=0.96723
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 5.71843 (QuantReg: 12.74081) QuantErr: 12.74081 batch_time=1.00230
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 6.80639 (QuantReg: 12.30921) QuantErr: 12.30921 batch_time=1.74305
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 7.17360 (QuantReg: 11.98293) QuantErr: 11.98293 batch_time=1.19528
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 5.04670 (QuantReg: 12.53939) QuantErr: 12.53939 batch_time=1.21244
Train Epoch: 10 codebook_update_time=8.16343
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch10.pth ...
Done in 5.079s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch10.pth ...
Done in 10.208s
removing stale ckpt [epoch 9] [took 0.07s]
epoch : 10
loss : 6.01626043510437
quant_reg : 12.31114507293701
quant_err : 12.31114507293701
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.891
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.59814295954342
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.676
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.05562053970779
mnt_best : 39.59814295954342
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 5.80107 (QuantReg: 12.20724) QuantErr: 12.20724 batch_time=26.80875
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 6.57766 (QuantReg: 12.09975) QuantErr: 12.09975 batch_time=0.98844
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 6.84067 (QuantReg: 12.34076) QuantErr: 12.34076 batch_time=0.91009
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 5.98382 (QuantReg: 12.52073) QuantErr: 12.52073 batch_time=0.91316
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 5.56190 (QuantReg: 12.15871) QuantErr: 12.15871 batch_time=0.91106
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 6.04195 (QuantReg: 12.35335) QuantErr: 12.35335 batch_time=0.96597
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 5.60648 (QuantReg: 12.83893) QuantErr: 12.83893 batch_time=3.29065
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 5.48284 (QuantReg: 12.15624) QuantErr: 12.15624 batch_time=1.00978
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 5.07867 (QuantReg: 12.56170) QuantErr: 12.56170 batch_time=0.91906
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 5.43068 (QuantReg: 12.44787) QuantErr: 12.44787 batch_time=0.95018
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 6.35623 (QuantReg: 12.42851) QuantErr: 12.42851 batch_time=0.91465
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 6.24707 (QuantReg: 12.33964) QuantErr: 12.33964 batch_time=0.92937
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 6.33618 (QuantReg: 12.12516) QuantErr: 12.12516 batch_time=0.90249
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 5.55167 (QuantReg: 12.82127) QuantErr: 12.82127 batch_time=6.02549
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 4.43285 (QuantReg: 12.71914) QuantErr: 12.71914 batch_time=0.99313
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 5.12720 (QuantReg: 12.59423) QuantErr: 12.59423 batch_time=0.96292
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 5.91594 (QuantReg: 12.37014) QuantErr: 12.37014 batch_time=0.98037
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 5.55114 (QuantReg: 12.51161) QuantErr: 12.51161 batch_time=0.88859
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 7.42899 (QuantReg: 12.40521) QuantErr: 12.40521 batch_time=0.96299
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 3.98865 (QuantReg: 12.25915) QuantErr: 12.25915 batch_time=1.13666
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 5.24488 (QuantReg: 12.52248) QuantErr: 12.52248 batch_time=0.95575
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 5.62450 (QuantReg: 12.69919) QuantErr: 12.69919 batch_time=0.96372
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 6.72807 (QuantReg: 12.22840) QuantErr: 12.22840 batch_time=0.97499
Train Epoch: 11 codebook_update_time=7.79724
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch11.pth ...
Done in 5.007s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch11.pth ...
Done in 10.382s
removing stale ckpt [epoch 10] [took 0.02s]
epoch : 11
loss : 5.7063061113357545
quant_reg : 12.392695747375488
quant_err : 12.392695747375488
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.157
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.76398205380343
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.3155
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.180263585608856
mnt_best : 40.76398205380343
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 5.67519 (QuantReg: 12.34134) QuantErr: 12.34134 batch_time=28.22425
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 5.92739 (QuantReg: 12.14830) QuantErr: 12.14830 batch_time=1.10268
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 6.33580 (QuantReg: 12.48585) QuantErr: 12.48585 batch_time=1.05294
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 5.84091 (QuantReg: 12.19869) QuantErr: 12.19869 batch_time=0.96614
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 4.85371 (QuantReg: 12.36919) QuantErr: 12.36919 batch_time=1.00119
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 4.90984 (QuantReg: 12.33763) QuantErr: 12.33763 batch_time=0.96121
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 5.05103 (QuantReg: 12.49275) QuantErr: 12.49275 batch_time=1.04434
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 4.96498 (QuantReg: 12.70373) QuantErr: 12.70373 batch_time=0.91731
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 5.75939 (QuantReg: 12.26521) QuantErr: 12.26521 batch_time=0.93911
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 7.01282 (QuantReg: 12.35601) QuantErr: 12.35601 batch_time=0.92567
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 6.01044 (QuantReg: 12.41616) QuantErr: 12.41616 batch_time=0.97140
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 4.88792 (QuantReg: 12.49844) QuantErr: 12.49844 batch_time=0.93619
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 4.83667 (QuantReg: 12.80782) QuantErr: 12.80782 batch_time=2.54245
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 5.36068 (QuantReg: 12.49973) QuantErr: 12.49973 batch_time=4.82695
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 6.37823 (QuantReg: 12.41814) QuantErr: 12.41814 batch_time=0.95449
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 4.48098 (QuantReg: 12.62485) QuantErr: 12.62485 batch_time=0.99235
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 6.05470 (QuantReg: 12.85495) QuantErr: 12.85495 batch_time=0.97087
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 5.78635 (QuantReg: 12.65171) QuantErr: 12.65171 batch_time=0.93078
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 5.25459 (QuantReg: 12.46487) QuantErr: 12.46487 batch_time=0.96831
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 5.30220 (QuantReg: 12.18705) QuantErr: 12.18705 batch_time=0.98066
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 6.12178 (QuantReg: 12.39992) QuantErr: 12.39992 batch_time=1.06051
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 4.94881 (QuantReg: 12.72706) QuantErr: 12.72706 batch_time=1.12947
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 5.81857 (QuantReg: 12.82779) QuantErr: 12.82779 batch_time=1.05608
Train Epoch: 12 codebook_update_time=7.99697
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch12.pth ...
Done in 4.907s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch12.pth ...
Done in 10.198s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 5.480696293830872
quant_reg : 12.478915390014649
quant_err : 12.478915390014649
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.878
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.22617025846929
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.1555
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.85192755860147
mnt_best : 41.22617025846929
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 4.64774 (QuantReg: 12.31171) QuantErr: 12.31171 batch_time=28.78817
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 6.15952 (QuantReg: 12.36440) QuantErr: 12.36440 batch_time=0.91212
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 5.60586 (QuantReg: 12.32911) QuantErr: 12.32911 batch_time=0.93250
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 5.62928 (QuantReg: 12.20570) QuantErr: 12.20570 batch_time=0.96188
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 6.73336 (QuantReg: 12.02742) QuantErr: 12.02742 batch_time=1.00712
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 4.33454 (QuantReg: 12.71368) QuantErr: 12.71368 batch_time=0.92658
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 5.54450 (QuantReg: 11.91213) QuantErr: 11.91213 batch_time=1.21842
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 5.21436 (QuantReg: 12.13471) QuantErr: 12.13471 batch_time=0.92400
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 5.93322 (QuantReg: 12.29018) QuantErr: 12.29018 batch_time=0.90081
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 6.01535 (QuantReg: 12.07633) QuantErr: 12.07633 batch_time=0.92383
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 5.07092 (QuantReg: 12.46831) QuantErr: 12.46831 batch_time=0.89117
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 5.56783 (QuantReg: 12.73764) QuantErr: 12.73764 batch_time=0.92166
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 6.57917 (QuantReg: 12.31355) QuantErr: 12.31355 batch_time=0.90579
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 5.77894 (QuantReg: 12.28590) QuantErr: 12.28590 batch_time=0.91800
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 4.85973 (QuantReg: 12.48315) QuantErr: 12.48315 batch_time=0.90605
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 5.22497 (QuantReg: 12.58038) QuantErr: 12.58038 batch_time=0.96542
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 5.20830 (QuantReg: 12.39988) QuantErr: 12.39988 batch_time=1.16735
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 4.58949 (QuantReg: 12.78255) QuantErr: 12.78255 batch_time=1.04090
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 6.09244 (QuantReg: 12.46771) QuantErr: 12.46771 batch_time=0.95736
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 5.33509 (QuantReg: 12.56738) QuantErr: 12.56738 batch_time=1.03881
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 5.11963 (QuantReg: 12.63902) QuantErr: 12.63902 batch_time=0.93305
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 5.30067 (QuantReg: 12.31384) QuantErr: 12.31384 batch_time=0.93312
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 5.05392 (QuantReg: 12.24374) QuantErr: 12.24374 batch_time=1.04466
Train Epoch: 13 codebook_update_time=7.54448
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch13.pth ...
Done in 4.310s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch13.pth ...
Done in 9.257s
removing stale ckpt [epoch 12] [took 0.04s]
epoch : 13
loss : 5.299593955993652
quant_reg : 12.43711763381958
quant_err : 12.43711763381958
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.58
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.65601178937618
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.981
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.67910628358642
mnt_best : 41.65601178937618
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 4.36849 (QuantReg: 12.44402) QuantErr: 12.44402 batch_time=30.09784
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 5.13260 (QuantReg: 12.23795) QuantErr: 12.23795 batch_time=0.89997
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 5.44059 (QuantReg: 12.48708) QuantErr: 12.48708 batch_time=0.90829
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 4.56893 (QuantReg: 12.60854) QuantErr: 12.60854 batch_time=0.93406
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 5.67000 (QuantReg: 12.27391) QuantErr: 12.27391 batch_time=0.95374
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 5.78825 (QuantReg: 12.25562) QuantErr: 12.25562 batch_time=0.95793
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 6.31358 (QuantReg: 12.41081) QuantErr: 12.41081 batch_time=0.98586
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 4.20808 (QuantReg: 12.54713) QuantErr: 12.54713 batch_time=0.93551
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 6.14180 (QuantReg: 12.28298) QuantErr: 12.28298 batch_time=1.06404
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 5.33063 (QuantReg: 12.57296) QuantErr: 12.57296 batch_time=0.92878
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 4.39136 (QuantReg: 12.49556) QuantErr: 12.49556 batch_time=0.90043
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 4.99062 (QuantReg: 12.53273) QuantErr: 12.53273 batch_time=0.91475
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 4.86525 (QuantReg: 12.62713) QuantErr: 12.62713 batch_time=0.89576
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 5.47418 (QuantReg: 12.12033) QuantErr: 12.12033 batch_time=1.40902
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 5.21623 (QuantReg: 12.57976) QuantErr: 12.57976 batch_time=0.91211
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 6.26264 (QuantReg: 12.43276) QuantErr: 12.43276 batch_time=0.93975
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 4.27640 (QuantReg: 12.74977) QuantErr: 12.74977 batch_time=0.95357
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 5.08075 (QuantReg: 12.77315) QuantErr: 12.77315 batch_time=0.93882
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 4.07978 (QuantReg: 12.50399) QuantErr: 12.50399 batch_time=1.89799
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 6.65376 (QuantReg: 12.52885) QuantErr: 12.52885 batch_time=3.32612
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 4.68737 (QuantReg: 12.45402) QuantErr: 12.45402 batch_time=1.06577
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 5.83468 (QuantReg: 12.64823) QuantErr: 12.64823 batch_time=1.03965
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 7.13540 (QuantReg: 12.59863) QuantErr: 12.59863 batch_time=0.96920
Train Epoch: 14 codebook_update_time=8.16268
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch14.pth ...
Done in 4.508s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch14.pth ...
Done in 9.433s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 5.1926002111434935
quant_reg : 12.531077583312989
quant_err : 12.531077583312989
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.488
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.66131533052406
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.5005
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.140195023501256
mnt_best : 41.66131533052406
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 5.02535 (QuantReg: 12.21950) QuantErr: 12.21950 batch_time=32.05150
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 5.77758 (QuantReg: 12.47419) QuantErr: 12.47419 batch_time=0.92048
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 3.63670 (QuantReg: 12.92006) QuantErr: 12.92006 batch_time=0.90281
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 4.39463 (QuantReg: 12.22848) QuantErr: 12.22848 batch_time=1.13100
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 5.42169 (QuantReg: 12.48432) QuantErr: 12.48432 batch_time=0.94261
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 4.63011 (QuantReg: 12.50243) QuantErr: 12.50243 batch_time=0.94148
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 6.76070 (QuantReg: 12.88929) QuantErr: 12.88929 batch_time=1.53801
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 4.55496 (QuantReg: 12.80331) QuantErr: 12.80331 batch_time=0.96448
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 5.17812 (QuantReg: 12.22725) QuantErr: 12.22725 batch_time=1.03425
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 5.21073 (QuantReg: 12.68780) QuantErr: 12.68780 batch_time=0.92484
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 5.32046 (QuantReg: 12.47110) QuantErr: 12.47110 batch_time=0.96982
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 6.02831 (QuantReg: 12.78886) QuantErr: 12.78886 batch_time=1.04562
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 5.13874 (QuantReg: 12.58686) QuantErr: 12.58686 batch_time=2.99403
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 5.88022 (QuantReg: 12.38744) QuantErr: 12.38744 batch_time=0.90846
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 6.01669 (QuantReg: 12.63366) QuantErr: 12.63366 batch_time=0.95750
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 4.22392 (QuantReg: 12.13203) QuantErr: 12.13203 batch_time=0.95189
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 5.83346 (QuantReg: 12.26209) QuantErr: 12.26209 batch_time=0.92698
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 5.10558 (QuantReg: 12.79511) QuantErr: 12.79511 batch_time=0.96527
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 5.64138 (QuantReg: 12.37844) QuantErr: 12.37844 batch_time=0.93118
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 6.19367 (QuantReg: 12.68220) QuantErr: 12.68220 batch_time=0.97436
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 5.05901 (QuantReg: 12.76260) QuantErr: 12.76260 batch_time=1.03745
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 4.79343 (QuantReg: 12.38033) QuantErr: 12.38033 batch_time=1.01396
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 5.83931 (QuantReg: 12.33023) QuantErr: 12.33023 batch_time=1.40932
Train Epoch: 15 codebook_update_time=8.11101
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch15.pth ...
Done in 12.933s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch15.pth ...
Done in 28.805s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 5.072844372749328
quant_reg : 12.551437675476075
quant_err : 12.551437675476075
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.166
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.90693680089957
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.288
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.85054286120769
mnt_best : 41.90693680089957
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 5.00583 (QuantReg: 12.26865) QuantErr: 12.26865 batch_time=29.52793
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 4.90659 (QuantReg: 12.69276) QuantErr: 12.69276 batch_time=0.92451
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 5.33241 (QuantReg: 12.46820) QuantErr: 12.46820 batch_time=0.93070
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 5.37750 (QuantReg: 12.44134) QuantErr: 12.44134 batch_time=0.93284
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 4.36388 (QuantReg: 12.45918) QuantErr: 12.45918 batch_time=1.02987
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 5.68187 (QuantReg: 12.41495) QuantErr: 12.41495 batch_time=0.89336
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 4.88490 (QuantReg: 12.56320) QuantErr: 12.56320 batch_time=3.24097
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 6.25846 (QuantReg: 12.39051) QuantErr: 12.39051 batch_time=0.93107
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 4.08241 (QuantReg: 12.55400) QuantErr: 12.55400 batch_time=0.95414
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 4.06927 (QuantReg: 12.45288) QuantErr: 12.45288 batch_time=0.92779
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 4.26819 (QuantReg: 12.60897) QuantErr: 12.60897 batch_time=0.92763
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 5.62388 (QuantReg: 12.56394) QuantErr: 12.56394 batch_time=0.92658
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 4.75392 (QuantReg: 12.39842) QuantErr: 12.39842 batch_time=0.95369
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 4.52142 (QuantReg: 12.35541) QuantErr: 12.35541 batch_time=3.36222
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 4.74821 (QuantReg: 12.71277) QuantErr: 12.71277 batch_time=0.91394
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 5.09080 (QuantReg: 12.30803) QuantErr: 12.30803 batch_time=1.31149
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 5.16471 (QuantReg: 12.66429) QuantErr: 12.66429 batch_time=0.90742
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 5.10919 (QuantReg: 12.74024) QuantErr: 12.74024 batch_time=0.92268
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 3.23033 (QuantReg: 12.46298) QuantErr: 12.46298 batch_time=0.94351
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 4.57399 (QuantReg: 12.60537) QuantErr: 12.60537 batch_time=0.95124
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 5.48557 (QuantReg: 12.32385) QuantErr: 12.32385 batch_time=0.92172
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 5.61222 (QuantReg: 12.78159) QuantErr: 12.78159 batch_time=1.05744
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 3.89466 (QuantReg: 12.95083) QuantErr: 12.95083 batch_time=1.08065
Train Epoch: 16 codebook_update_time=7.48383
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch16.pth ...
Done in 6.398s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch16.pth ...
Done in 11.065s
removing stale ckpt [epoch 15] [took 0.11s]
epoch : 16
loss : 4.798682402610779
quant_reg : 12.548522205352784
quant_err : 12.548522205352784
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.91
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.6977053066136
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.479
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.29651397256349
mnt_best : 42.6977053066136
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 3.55123 (QuantReg: 12.46644) QuantErr: 12.46644 batch_time=34.63673
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 3.67390 (QuantReg: 12.81275) QuantErr: 12.81275 batch_time=0.93391
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 4.07139 (QuantReg: 12.56474) QuantErr: 12.56474 batch_time=0.92509
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 4.31028 (QuantReg: 12.74636) QuantErr: 12.74636 batch_time=0.91384
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 4.27723 (QuantReg: 12.41350) QuantErr: 12.41350 batch_time=0.92404
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 4.86805 (QuantReg: 12.52683) QuantErr: 12.52683 batch_time=0.91916
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 4.83726 (QuantReg: 12.21334) QuantErr: 12.21334 batch_time=0.91216
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 4.83089 (QuantReg: 12.69393) QuantErr: 12.69393 batch_time=1.01434
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 4.74836 (QuantReg: 12.50063) QuantErr: 12.50063 batch_time=0.97937
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 6.52468 (QuantReg: 12.41227) QuantErr: 12.41227 batch_time=0.91936
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 4.96878 (QuantReg: 12.76955) QuantErr: 12.76955 batch_time=0.91435
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 4.76446 (QuantReg: 12.41378) QuantErr: 12.41378 batch_time=0.90815
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 4.57992 (QuantReg: 12.58714) QuantErr: 12.58714 batch_time=0.98339
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 4.75187 (QuantReg: 12.57065) QuantErr: 12.57065 batch_time=1.04424
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 3.91299 (QuantReg: 12.79752) QuantErr: 12.79752 batch_time=0.92661
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 4.89452 (QuantReg: 12.72094) QuantErr: 12.72094 batch_time=0.97674
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 4.29031 (QuantReg: 12.54585) QuantErr: 12.54585 batch_time=0.94662
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 4.65297 (QuantReg: 12.56357) QuantErr: 12.56357 batch_time=0.95912
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 3.57231 (QuantReg: 12.99846) QuantErr: 12.99846 batch_time=0.92874
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 4.02393 (QuantReg: 12.79629) QuantErr: 12.79629 batch_time=3.80325
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 5.10728 (QuantReg: 12.60134) QuantErr: 12.60134 batch_time=0.97828
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 4.89863 (QuantReg: 13.10923) QuantErr: 13.10923 batch_time=1.04271
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 4.21154 (QuantReg: 12.68439) QuantErr: 12.68439 batch_time=1.01696
Train Epoch: 17 codebook_update_time=8.09980
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch17.pth ...
Done in 5.701s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch17.pth ...
Done in 10.697s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 4.693273175239563
quant_reg : 12.6377329788208
quant_err : 12.6377329788208
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.312
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.37824374803496
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.7905
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.48088792338978
mnt_best : 43.37824374803496
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 4.20805 (QuantReg: 12.62279) QuantErr: 12.62279 batch_time=29.81159
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 4.87069 (QuantReg: 12.47697) QuantErr: 12.47697 batch_time=0.93092
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 4.31129 (QuantReg: 12.46980) QuantErr: 12.46980 batch_time=1.00086
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 5.78816 (QuantReg: 12.65169) QuantErr: 12.65169 batch_time=0.92018
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 4.38357 (QuantReg: 12.32058) QuantErr: 12.32058 batch_time=0.88568
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 4.38094 (QuantReg: 13.08155) QuantErr: 13.08155 batch_time=0.97281
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 5.92697 (QuantReg: 12.65125) QuantErr: 12.65125 batch_time=0.90377
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 5.22621 (QuantReg: 12.53559) QuantErr: 12.53559 batch_time=0.92143
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 5.27836 (QuantReg: 12.59892) QuantErr: 12.59892 batch_time=0.97769
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 5.04099 (QuantReg: 12.71879) QuantErr: 12.71879 batch_time=0.94296
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 4.78003 (QuantReg: 12.97723) QuantErr: 12.97723 batch_time=1.11401
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 3.79664 (QuantReg: 12.90137) QuantErr: 12.90137 batch_time=0.93833
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 3.63822 (QuantReg: 12.62925) QuantErr: 12.62925 batch_time=1.48875
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 4.81321 (QuantReg: 12.90918) QuantErr: 12.90918 batch_time=0.91371
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 4.81461 (QuantReg: 12.93146) QuantErr: 12.93146 batch_time=1.01948
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 5.02519 (QuantReg: 12.52938) QuantErr: 12.52938 batch_time=0.99907
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 3.94891 (QuantReg: 12.79066) QuantErr: 12.79066 batch_time=0.99928
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 4.69619 (QuantReg: 12.58621) QuantErr: 12.58621 batch_time=0.93858
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 5.88967 (QuantReg: 12.50097) QuantErr: 12.50097 batch_time=1.09120
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 5.19370 (QuantReg: 12.46911) QuantErr: 12.46911 batch_time=0.96686
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 4.37665 (QuantReg: 13.05015) QuantErr: 13.05015 batch_time=0.97311
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 3.81732 (QuantReg: 13.05424) QuantErr: 13.05424 batch_time=0.91428
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 4.55533 (QuantReg: 12.71132) QuantErr: 12.71132 batch_time=1.02640
Train Epoch: 18 codebook_update_time=7.61721
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch18.pth ...
Done in 4.157s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 4.58987645149231
quant_reg : 12.704057807922362
quant_err : 12.704057807922362
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.474
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.32724010314757
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.0565
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.182210932783036
mnt_best : 43.37824374803496
not_improved_count: 1
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 4.66761 (QuantReg: 12.31478) QuantErr: 12.31478 batch_time=32.16276
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 4.12198 (QuantReg: 12.46523) QuantErr: 12.46523 batch_time=0.90363
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 4.53864 (QuantReg: 12.78951) QuantErr: 12.78951 batch_time=0.93182
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 4.38225 (QuantReg: 12.29114) QuantErr: 12.29114 batch_time=0.90618
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 4.47328 (QuantReg: 12.77241) QuantErr: 12.77241 batch_time=0.92991
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 5.38002 (QuantReg: 12.72467) QuantErr: 12.72467 batch_time=0.92040
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 4.60655 (QuantReg: 13.16564) QuantErr: 13.16564 batch_time=0.92575
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 4.95188 (QuantReg: 12.74011) QuantErr: 12.74011 batch_time=1.01916
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 4.28876 (QuantReg: 12.69079) QuantErr: 12.69079 batch_time=1.13678
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 4.22324 (QuantReg: 12.65416) QuantErr: 12.65416 batch_time=0.95064
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 5.32343 (QuantReg: 12.60724) QuantErr: 12.60724 batch_time=0.99594
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 5.11610 (QuantReg: 12.66482) QuantErr: 12.66482 batch_time=0.94133
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 4.93546 (QuantReg: 12.64615) QuantErr: 12.64615 batch_time=3.10821
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 4.94893 (QuantReg: 12.91899) QuantErr: 12.91899 batch_time=1.80322
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 5.14031 (QuantReg: 12.49503) QuantErr: 12.49503 batch_time=0.91605
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 4.28074 (QuantReg: 12.96413) QuantErr: 12.96413 batch_time=1.03727
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 4.68037 (QuantReg: 12.54290) QuantErr: 12.54290 batch_time=0.96215
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 3.86674 (QuantReg: 12.73893) QuantErr: 12.73893 batch_time=1.01558
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 4.10606 (QuantReg: 12.83650) QuantErr: 12.83650 batch_time=0.92697
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 4.11771 (QuantReg: 12.82211) QuantErr: 12.82211 batch_time=0.95465
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 5.18290 (QuantReg: 12.89425) QuantErr: 12.89425 batch_time=0.94951
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 4.50801 (QuantReg: 13.04972) QuantErr: 13.04972 batch_time=1.05797
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 3.91433 (QuantReg: 12.84581) QuantErr: 12.84581 batch_time=1.08162
Train Epoch: 19 codebook_update_time=7.66636
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_L31/checkpoint-epoch19.pth ...
Done in 6.172s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 4.4823583965301514
quant_reg : 12.728129306793212
quant_err : 12.728129306793212
learning_rate : 1.986071592291091e-05