-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_M64.txt
2599 lines (2599 loc) · 194 KB
/
HCQ_MSRVTT_1kA_M64.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 974.5532853603363 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 88.33325600624084 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 79.28633856773376 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch0.pth ...
Done in 1.750s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch0.pth ...
Done in 3.712s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 1.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 5.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 493.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 498.597
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.45788569702133275
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 0.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 5.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 476.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 493.279
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.45788569702133275
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.82445 (QuantReg: 21.64898) QuantErr: 21.64898 batch_time=27.88361
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.73499 (QuantReg: 21.73981) QuantErr: 21.73981 batch_time=0.61733
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.58577 (QuantReg: 21.85950) QuantErr: 21.85950 batch_time=0.60086
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.40880 (QuantReg: 21.83416) QuantErr: 21.83416 batch_time=0.64955
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.51826 (QuantReg: 21.87535) QuantErr: 21.87535 batch_time=0.60721
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.82863 (QuantReg: 21.88825) QuantErr: 21.88825 batch_time=0.60250
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.91430 (QuantReg: 21.90160) QuantErr: 21.90160 batch_time=0.61504
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.01418 (QuantReg: 21.88745) QuantErr: 21.88745 batch_time=0.61651
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.47606 (QuantReg: 21.84380) QuantErr: 21.84380 batch_time=0.63656
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.37981 (QuantReg: 21.85745) QuantErr: 21.85745 batch_time=0.60504
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 4.67008 (QuantReg: 21.90475) QuantErr: 21.90475 batch_time=0.67150
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.72617 (QuantReg: 21.89772) QuantErr: 21.89772 batch_time=0.68888
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.26558 (QuantReg: 21.77127) QuantErr: 21.77127 batch_time=0.61498
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.43846 (QuantReg: 21.80652) QuantErr: 21.80652 batch_time=2.00345
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.79476 (QuantReg: 21.84111) QuantErr: 21.84111 batch_time=1.05293
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.14413 (QuantReg: 21.80865) QuantErr: 21.80865 batch_time=0.60352
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.85463 (QuantReg: 21.82779) QuantErr: 21.82779 batch_time=0.58263
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.47437 (QuantReg: 21.80555) QuantErr: 21.80555 batch_time=0.60993
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.14839 (QuantReg: 21.80949) QuantErr: 21.80949 batch_time=0.60028
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.41726 (QuantReg: 21.84346) QuantErr: 21.84346 batch_time=1.09855
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 3.64855 (QuantReg: 21.86996) QuantErr: 21.86996 batch_time=1.07590
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.25660 (QuantReg: 21.88182) QuantErr: 21.88182 batch_time=0.58747
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.74928 (QuantReg: 21.83294) QuantErr: 21.83294 batch_time=0.58191
Train Epoch: 1 codebook_update_time=3.42150
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch1.pth ...
Done in 4.112s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch1.pth ...
Done in 8.400s
epoch : 1
loss : 5.408132533073426
quant_reg : 21.83753554534912
quant_err : 21.83753554534912
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 33.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 46.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 79.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 12.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 39.372
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.53679000668778
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 35.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 49.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 82.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 11.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 37.758
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.080500490027376
mnt_best : 26.53679000668778
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.19389 (QuantReg: 14.04108) QuantErr: 14.04108 batch_time=31.91215
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.96487 (QuantReg: 13.89687) QuantErr: 13.89687 batch_time=0.61720
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.06553 (QuantReg: 14.39344) QuantErr: 14.39344 batch_time=2.36723
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.84967 (QuantReg: 14.39817) QuantErr: 14.39817 batch_time=0.60935
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.85131 (QuantReg: 14.96866) QuantErr: 14.96866 batch_time=0.61330
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.92857 (QuantReg: 14.70997) QuantErr: 14.70997 batch_time=0.60533
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.76895 (QuantReg: 14.74725) QuantErr: 14.74725 batch_time=0.60392
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.85045 (QuantReg: 15.09814) QuantErr: 15.09814 batch_time=1.42679
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.77418 (QuantReg: 15.44170) QuantErr: 15.44170 batch_time=0.60873
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.33232 (QuantReg: 15.14600) QuantErr: 15.14600 batch_time=0.60062
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.75710 (QuantReg: 15.22931) QuantErr: 15.22931 batch_time=0.60379
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.20170 (QuantReg: 15.52842) QuantErr: 15.52842 batch_time=0.61904
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.60356 (QuantReg: 15.61037) QuantErr: 15.61037 batch_time=0.60705
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.13505 (QuantReg: 15.96534) QuantErr: 15.96534 batch_time=0.62081
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 4.02937 (QuantReg: 15.89168) QuantErr: 15.89168 batch_time=0.66997
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.48138 (QuantReg: 16.20295) QuantErr: 16.20295 batch_time=0.71248
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.84094 (QuantReg: 16.13042) QuantErr: 16.13042 batch_time=0.64703
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.69646 (QuantReg: 16.41395) QuantErr: 16.41395 batch_time=0.60030
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.35945 (QuantReg: 16.57430) QuantErr: 16.57430 batch_time=0.60913
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.34319 (QuantReg: 16.63765) QuantErr: 16.63765 batch_time=3.15782
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.45003 (QuantReg: 16.45615) QuantErr: 16.45615 batch_time=0.59465
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.62920 (QuantReg: 16.36795) QuantErr: 16.36795 batch_time=0.59960
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.74751 (QuantReg: 16.56334) QuantErr: 16.56334 batch_time=0.60453
Train Epoch: 2 codebook_update_time=3.31298
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch2.pth ...
Done in 4.337s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch2.pth ...
Done in 8.579s
removing stale ckpt [epoch 1] [took 0.02s]
removing stale ckpt [epoch 0] [took 0.03s]
epoch : 2
loss : 3.7249118251800537
quant_reg : 15.51775721359253
quant_err : 15.51775721359253
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.696
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.1612048052762
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.022
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.352413018640625
mnt_best : 33.1612048052762
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.34821 (QuantReg: 14.11727) QuantErr: 14.11727 batch_time=32.23399
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.25659 (QuantReg: 14.46440) QuantErr: 14.46440 batch_time=0.64250
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.32901 (QuantReg: 14.30644) QuantErr: 14.30644 batch_time=0.59185
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.93846 (QuantReg: 14.40044) QuantErr: 14.40044 batch_time=0.61586
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 2.94546 (QuantReg: 14.58461) QuantErr: 14.58461 batch_time=0.62127
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.45045 (QuantReg: 14.61174) QuantErr: 14.61174 batch_time=0.66134
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.72779 (QuantReg: 14.70342) QuantErr: 14.70342 batch_time=0.58679
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.52914 (QuantReg: 14.59975) QuantErr: 14.59975 batch_time=0.62761
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.56808 (QuantReg: 14.44946) QuantErr: 14.44946 batch_time=0.62487
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.72489 (QuantReg: 14.69634) QuantErr: 14.69634 batch_time=0.59812
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.42709 (QuantReg: 14.66296) QuantErr: 14.66296 batch_time=0.60761
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.15588 (QuantReg: 14.57549) QuantErr: 14.57549 batch_time=0.68879
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 2.84157 (QuantReg: 14.66420) QuantErr: 14.66420 batch_time=0.60537
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.37999 (QuantReg: 14.67638) QuantErr: 14.67638 batch_time=0.60502
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.55886 (QuantReg: 14.76317) QuantErr: 14.76317 batch_time=0.61651
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.83218 (QuantReg: 14.95629) QuantErr: 14.95629 batch_time=0.62161
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.32042 (QuantReg: 15.18442) QuantErr: 15.18442 batch_time=0.60218
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 2.92460 (QuantReg: 15.01709) QuantErr: 15.01709 batch_time=0.71012
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.00820 (QuantReg: 14.89201) QuantErr: 14.89201 batch_time=8.59089
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.10827 (QuantReg: 15.16862) QuantErr: 15.16862 batch_time=0.60800
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.03981 (QuantReg: 15.01888) QuantErr: 15.01888 batch_time=0.65042
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.09037 (QuantReg: 15.08283) QuantErr: 15.08283 batch_time=0.61286
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.15079 (QuantReg: 15.05362) QuantErr: 15.05362 batch_time=0.62349
Train Epoch: 3 codebook_update_time=3.34344
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch3.pth ...
Done in 4.712s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch3.pth ...
Done in 8.915s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.159526993751526
quant_reg : 14.71673893737793
quant_err : 14.71673893737793
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.252
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.61411038840476
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.562
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.79523749522489
mnt_best : 36.61411038840476
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.99149 (QuantReg: 14.07736) QuantErr: 14.07736 batch_time=34.35682
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.75303 (QuantReg: 14.29006) QuantErr: 14.29006 batch_time=0.59801
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.87283 (QuantReg: 14.43808) QuantErr: 14.43808 batch_time=0.98779
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.75047 (QuantReg: 14.45324) QuantErr: 14.45324 batch_time=0.62975
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.66824 (QuantReg: 14.59450) QuantErr: 14.59450 batch_time=0.61297
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.00280 (QuantReg: 14.48631) QuantErr: 14.48631 batch_time=0.59284
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.62099 (QuantReg: 14.36110) QuantErr: 14.36110 batch_time=0.61900
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.81902 (QuantReg: 14.39472) QuantErr: 14.39472 batch_time=0.60366
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.15880 (QuantReg: 14.37053) QuantErr: 14.37053 batch_time=0.69360
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.46644 (QuantReg: 14.55950) QuantErr: 14.55950 batch_time=0.60284
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.81542 (QuantReg: 14.91240) QuantErr: 14.91240 batch_time=0.60890
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.51509 (QuantReg: 14.89544) QuantErr: 14.89544 batch_time=0.60316
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.02347 (QuantReg: 14.82021) QuantErr: 14.82021 batch_time=0.64801
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.60700 (QuantReg: 14.92116) QuantErr: 14.92116 batch_time=5.48378
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.68745 (QuantReg: 14.77010) QuantErr: 14.77010 batch_time=0.60950
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.85848 (QuantReg: 14.75225) QuantErr: 14.75225 batch_time=0.61029
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.82673 (QuantReg: 14.71163) QuantErr: 14.71163 batch_time=0.59615
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.84038 (QuantReg: 14.86796) QuantErr: 14.86796 batch_time=0.61015
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.50392 (QuantReg: 14.94823) QuantErr: 14.94823 batch_time=0.66877
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.48984 (QuantReg: 14.87749) QuantErr: 14.87749 batch_time=0.64961
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.33660 (QuantReg: 15.20101) QuantErr: 15.20101 batch_time=0.62836
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.23203 (QuantReg: 15.36684) QuantErr: 15.36684 batch_time=0.60681
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.81107 (QuantReg: 14.96209) QuantErr: 14.96209 batch_time=0.60551
Train Epoch: 4 codebook_update_time=3.68161
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch4.pth ...
Done in 4.091s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch4.pth ...
Done in 8.101s
removing stale ckpt [epoch 3] [took 0.03s]
epoch : 4
loss : 2.8616314640045166
quant_reg : 14.635052577972413
quant_err : 14.635052577972413
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.383
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.80598012942461
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.797
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.83468485946031
mnt_best : 37.80598012942461
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.88474 (QuantReg: 14.08419) QuantErr: 14.08419 batch_time=34.30315
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.01903 (QuantReg: 14.33878) QuantErr: 14.33878 batch_time=0.58973
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.21732 (QuantReg: 14.28676) QuantErr: 14.28676 batch_time=0.59488
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.55094 (QuantReg: 14.56453) QuantErr: 14.56453 batch_time=0.65530
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.72130 (QuantReg: 14.56371) QuantErr: 14.56371 batch_time=0.60725
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.33858 (QuantReg: 14.54438) QuantErr: 14.54438 batch_time=0.57522
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.67237 (QuantReg: 14.45178) QuantErr: 14.45178 batch_time=0.61523
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.06175 (QuantReg: 14.12617) QuantErr: 14.12617 batch_time=0.62498
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.45760 (QuantReg: 14.66991) QuantErr: 14.66991 batch_time=0.61879
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.49127 (QuantReg: 14.30973) QuantErr: 14.30973 batch_time=0.63723
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.60855 (QuantReg: 14.57025) QuantErr: 14.57025 batch_time=0.68941
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.62112 (QuantReg: 14.58165) QuantErr: 14.58165 batch_time=0.62750
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.53438 (QuantReg: 14.78951) QuantErr: 14.78951 batch_time=0.62096
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.15517 (QuantReg: 14.46430) QuantErr: 14.46430 batch_time=0.63355
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.98412 (QuantReg: 14.56281) QuantErr: 14.56281 batch_time=0.60699
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.25666 (QuantReg: 14.94477) QuantErr: 14.94477 batch_time=0.59206
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.74472 (QuantReg: 14.69531) QuantErr: 14.69531 batch_time=0.61242
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.68304 (QuantReg: 14.69547) QuantErr: 14.69547 batch_time=0.61739
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.54066 (QuantReg: 14.94404) QuantErr: 14.94404 batch_time=0.60681
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.55795 (QuantReg: 14.80426) QuantErr: 14.80426 batch_time=0.67440
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.87859 (QuantReg: 14.83046) QuantErr: 14.83046 batch_time=0.63586
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.41901 (QuantReg: 14.87829) QuantErr: 14.87829 batch_time=0.60407
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.80605 (QuantReg: 15.04694) QuantErr: 15.04694 batch_time=0.62644
Train Epoch: 5 codebook_update_time=3.29440
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch5.pth ...
Done in 4.137s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch5.pth ...
Done in 8.471s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 2.581015509605408
quant_reg : 14.650204216003418
quant_err : 14.650204216003418
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.751
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.45410031454433
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.921
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.634500472230215
mnt_best : 40.45410031454433
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.65549 (QuantReg: 14.29245) QuantErr: 14.29245 batch_time=40.14424
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.63537 (QuantReg: 14.03003) QuantErr: 14.03003 batch_time=0.60568
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.48193 (QuantReg: 14.27854) QuantErr: 14.27854 batch_time=0.61926
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.35485 (QuantReg: 14.32023) QuantErr: 14.32023 batch_time=0.60485
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.29260 (QuantReg: 14.23530) QuantErr: 14.23530 batch_time=0.59541
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.25818 (QuantReg: 14.44039) QuantErr: 14.44039 batch_time=0.61780
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.72352 (QuantReg: 14.55580) QuantErr: 14.55580 batch_time=0.61244
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.36694 (QuantReg: 14.66571) QuantErr: 14.66571 batch_time=0.61034
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.36830 (QuantReg: 14.60436) QuantErr: 14.60436 batch_time=0.61048
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.16169 (QuantReg: 14.63546) QuantErr: 14.63546 batch_time=0.62239
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.42133 (QuantReg: 14.81202) QuantErr: 14.81202 batch_time=0.64312
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.30812 (QuantReg: 14.55057) QuantErr: 14.55057 batch_time=0.61383
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.24613 (QuantReg: 14.72735) QuantErr: 14.72735 batch_time=0.67627
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.21875 (QuantReg: 14.45447) QuantErr: 14.45447 batch_time=0.60885
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.65644 (QuantReg: 14.85128) QuantErr: 14.85128 batch_time=0.63774
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.39836 (QuantReg: 14.59793) QuantErr: 14.59793 batch_time=0.62101
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.15442 (QuantReg: 14.90708) QuantErr: 14.90708 batch_time=0.62957
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.50631 (QuantReg: 14.85711) QuantErr: 14.85711 batch_time=0.66508
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.59490 (QuantReg: 14.89635) QuantErr: 14.89635 batch_time=0.89635
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.43130 (QuantReg: 14.75171) QuantErr: 14.75171 batch_time=0.61913
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.18652 (QuantReg: 14.91192) QuantErr: 14.91192 batch_time=0.61054
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.13890 (QuantReg: 14.65321) QuantErr: 14.65321 batch_time=0.60285
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.00428 (QuantReg: 14.76037) QuantErr: 14.76037 batch_time=0.59508
Train Epoch: 6 codebook_update_time=3.26370
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch6.pth ...
Done in 4.170s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch6.pth ...
Done in 8.378s
removing stale ckpt [epoch 5] [took 0.02s]
epoch : 6
loss : 2.3865116906166075
quant_reg : 14.644343116760254
quant_err : 14.644343116760254
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.046
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.51271662853379
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.7205
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.749589344210406
mnt_best : 40.51271662853379
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.43230 (QuantReg: 14.20357) QuantErr: 14.20357 batch_time=40.49150
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.39051 (QuantReg: 14.37747) QuantErr: 14.37747 batch_time=0.66809
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.23465 (QuantReg: 14.16215) QuantErr: 14.16215 batch_time=0.60349
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 1.94319 (QuantReg: 14.53615) QuantErr: 14.53615 batch_time=0.61939
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.01053 (QuantReg: 14.49739) QuantErr: 14.49739 batch_time=0.61090
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.07819 (QuantReg: 14.82673) QuantErr: 14.82673 batch_time=0.64455
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.32132 (QuantReg: 14.73764) QuantErr: 14.73764 batch_time=2.39759
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.20694 (QuantReg: 14.57400) QuantErr: 14.57400 batch_time=0.69473
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.22824 (QuantReg: 14.76048) QuantErr: 14.76048 batch_time=0.61220
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.87346 (QuantReg: 14.65612) QuantErr: 14.65612 batch_time=0.95732
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.63877 (QuantReg: 14.55897) QuantErr: 14.55897 batch_time=0.60285
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.20095 (QuantReg: 14.80412) QuantErr: 14.80412 batch_time=0.60448
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.08564 (QuantReg: 14.76882) QuantErr: 14.76882 batch_time=0.59970
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.29902 (QuantReg: 14.68952) QuantErr: 14.68952 batch_time=0.59688
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.70739 (QuantReg: 14.61236) QuantErr: 14.61236 batch_time=1.20885
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.27093 (QuantReg: 14.61468) QuantErr: 14.61468 batch_time=0.60210
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.78012 (QuantReg: 14.99100) QuantErr: 14.99100 batch_time=0.59712
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.23894 (QuantReg: 14.66355) QuantErr: 14.66355 batch_time=0.62065
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.64540 (QuantReg: 14.87119) QuantErr: 14.87119 batch_time=0.62002
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.45612 (QuantReg: 14.87081) QuantErr: 14.87081 batch_time=0.60765
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.01993 (QuantReg: 14.90782) QuantErr: 14.90782 batch_time=0.63023
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.15949 (QuantReg: 15.03802) QuantErr: 15.03802 batch_time=0.60400
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.24867 (QuantReg: 14.96056) QuantErr: 14.96056 batch_time=0.60173
Train Epoch: 7 codebook_update_time=3.29634
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch7.pth ...
Done in 4.335s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch7.pth ...
Done in 8.588s
removing stale ckpt [epoch 6] [took 0.02s]
epoch : 7
loss : 2.2140056791305542
quant_reg : 14.68624785232544
quant_err : 14.68624785232544
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.214
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.64030584679487
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.2375
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.50501440258069
mnt_best : 42.64030584679487
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.36149 (QuantReg: 14.52645) QuantErr: 14.52645 batch_time=34.24505
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.43216 (QuantReg: 14.38848) QuantErr: 14.38848 batch_time=0.61245
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.71553 (QuantReg: 14.58437) QuantErr: 14.58437 batch_time=0.60498
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.13791 (QuantReg: 14.42494) QuantErr: 14.42494 batch_time=0.61771
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.23177 (QuantReg: 14.69539) QuantErr: 14.69539 batch_time=0.60641
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.94472 (QuantReg: 14.59832) QuantErr: 14.59832 batch_time=0.60686
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.18636 (QuantReg: 14.83589) QuantErr: 14.83589 batch_time=1.75934
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.33293 (QuantReg: 14.86763) QuantErr: 14.86763 batch_time=0.59623
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.28258 (QuantReg: 14.73390) QuantErr: 14.73390 batch_time=0.60680
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.58266 (QuantReg: 14.70570) QuantErr: 14.70570 batch_time=0.60782
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.36678 (QuantReg: 14.77991) QuantErr: 14.77991 batch_time=0.60838
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.11312 (QuantReg: 14.68789) QuantErr: 14.68789 batch_time=0.62280
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.27714 (QuantReg: 14.83455) QuantErr: 14.83455 batch_time=2.51003
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.28491 (QuantReg: 14.99976) QuantErr: 14.99976 batch_time=0.62398
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.07015 (QuantReg: 14.85258) QuantErr: 14.85258 batch_time=0.57955
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.85894 (QuantReg: 14.50820) QuantErr: 14.50820 batch_time=0.59748
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 1.93781 (QuantReg: 14.70941) QuantErr: 14.70941 batch_time=0.59308
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.65645 (QuantReg: 14.67501) QuantErr: 14.67501 batch_time=0.60211
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.27057 (QuantReg: 14.76346) QuantErr: 14.76346 batch_time=0.60011
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.26737 (QuantReg: 14.53228) QuantErr: 14.53228 batch_time=0.84600
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.21692 (QuantReg: 14.78335) QuantErr: 14.78335 batch_time=0.90389
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.66035 (QuantReg: 14.84488) QuantErr: 14.84488 batch_time=0.62972
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.40088 (QuantReg: 14.93856) QuantErr: 14.93856 batch_time=0.61321
Train Epoch: 8 codebook_update_time=3.32071
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch8.pth ...
Done in 4.469s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 2.123444999217987
quant_reg : 14.715266193389892
quant_err : 14.715266193389892
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.939
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.78871592420212
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.291
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.19974440435195
mnt_best : 42.64030584679487
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.80781 (QuantReg: 14.78163) QuantErr: 14.78163 batch_time=38.47194
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.23823 (QuantReg: 14.67068) QuantErr: 14.67068 batch_time=0.59234
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.78353 (QuantReg: 14.55507) QuantErr: 14.55507 batch_time=0.61119
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.05356 (QuantReg: 14.62508) QuantErr: 14.62508 batch_time=0.65959
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.95416 (QuantReg: 14.82016) QuantErr: 14.82016 batch_time=0.66349
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.67899 (QuantReg: 14.92032) QuantErr: 14.92032 batch_time=0.60483
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.81855 (QuantReg: 14.96400) QuantErr: 14.96400 batch_time=1.16184
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.97578 (QuantReg: 14.86876) QuantErr: 14.86876 batch_time=0.65365
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.12617 (QuantReg: 14.54727) QuantErr: 14.54727 batch_time=0.65080
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.82401 (QuantReg: 14.87644) QuantErr: 14.87644 batch_time=0.60343
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.98263 (QuantReg: 14.78202) QuantErr: 14.78202 batch_time=0.59563
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.60305 (QuantReg: 14.84366) QuantErr: 14.84366 batch_time=0.60269
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.05855 (QuantReg: 14.75452) QuantErr: 14.75452 batch_time=5.12591
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.90671 (QuantReg: 14.84293) QuantErr: 14.84293 batch_time=0.66256
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.10054 (QuantReg: 14.64787) QuantErr: 14.64787 batch_time=0.61171
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.73361 (QuantReg: 15.20177) QuantErr: 15.20177 batch_time=0.61776
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.07922 (QuantReg: 15.08860) QuantErr: 15.08860 batch_time=0.61814
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.46509 (QuantReg: 14.87655) QuantErr: 14.87655 batch_time=0.61368
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.97687 (QuantReg: 14.91355) QuantErr: 14.91355 batch_time=0.59880
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.95347 (QuantReg: 14.89790) QuantErr: 14.89790 batch_time=0.67711
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.73550 (QuantReg: 15.16414) QuantErr: 15.16414 batch_time=0.77128
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.76595 (QuantReg: 14.80116) QuantErr: 14.80116 batch_time=0.64777
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.85367 (QuantReg: 14.71801) QuantErr: 14.71801 batch_time=0.60731
Train Epoch: 9 codebook_update_time=3.57070
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch9.pth ...
Done in 7.807s
removing stale ckpt [epoch 8] [took 0.03s]
epoch : 9
loss : 1.981418164730072
quant_reg : 14.779094074249267
quant_err : 14.779094074249267
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.885
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.31469960644929
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.128
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.504303445300046
mnt_best : 42.64030584679487
not_improved_count: 2
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.81556 (QuantReg: 14.72425) QuantErr: 14.72425 batch_time=34.97089
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.96933 (QuantReg: 14.53595) QuantErr: 14.53595 batch_time=0.61949
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.19353 (QuantReg: 14.75422) QuantErr: 14.75422 batch_time=0.60799
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.07512 (QuantReg: 14.37493) QuantErr: 14.37493 batch_time=0.59692
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.69404 (QuantReg: 14.57351) QuantErr: 14.57351 batch_time=0.61716
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.88063 (QuantReg: 14.86949) QuantErr: 14.86949 batch_time=0.66423
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.28909 (QuantReg: 14.51576) QuantErr: 14.51576 batch_time=0.62239
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.96601 (QuantReg: 14.75716) QuantErr: 14.75716 batch_time=0.59553
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.89865 (QuantReg: 14.62230) QuantErr: 14.62230 batch_time=0.75308
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.89272 (QuantReg: 14.65977) QuantErr: 14.65977 batch_time=0.59232
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.69603 (QuantReg: 14.72613) QuantErr: 14.72613 batch_time=0.88196
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.07594 (QuantReg: 15.00581) QuantErr: 15.00581 batch_time=0.60362
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.92860 (QuantReg: 14.63013) QuantErr: 14.63013 batch_time=0.59428
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.57416 (QuantReg: 15.15729) QuantErr: 15.15729 batch_time=0.58818
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.86133 (QuantReg: 14.61514) QuantErr: 14.61514 batch_time=0.63125
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.08301 (QuantReg: 14.97668) QuantErr: 14.97668 batch_time=1.84051
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.03484 (QuantReg: 14.85327) QuantErr: 14.85327 batch_time=0.63286
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.83675 (QuantReg: 15.02417) QuantErr: 15.02417 batch_time=0.61381
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.51636 (QuantReg: 14.91625) QuantErr: 14.91625 batch_time=0.63138
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.49632 (QuantReg: 15.16117) QuantErr: 15.16117 batch_time=0.63534
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.93895 (QuantReg: 14.90326) QuantErr: 14.90326 batch_time=0.60918
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.13740 (QuantReg: 14.83822) QuantErr: 14.83822 batch_time=0.64289
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.98690 (QuantReg: 15.06882) QuantErr: 15.06882 batch_time=0.62097
Train Epoch: 10 codebook_update_time=3.31286
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch10.pth ...
Done in 6.590s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch10.pth ...
Done in 12.082s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 1.8789598083496093
quant_reg : 14.812705432891846
quant_err : 14.812705432891846
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.115
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.855188363777046
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.303
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.57499872342879
mnt_best : 42.855188363777046
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.81319 (QuantReg: 14.64631) QuantErr: 14.64631 batch_time=33.21017
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.09804 (QuantReg: 14.54606) QuantErr: 14.54606 batch_time=0.92365
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.09411 (QuantReg: 14.68424) QuantErr: 14.68424 batch_time=1.76438
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.98845 (QuantReg: 14.89883) QuantErr: 14.89883 batch_time=0.59506
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.04620 (QuantReg: 14.61086) QuantErr: 14.61086 batch_time=0.65404
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.35821 (QuantReg: 15.20214) QuantErr: 15.20214 batch_time=0.61355
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.49052 (QuantReg: 14.86658) QuantErr: 14.86658 batch_time=2.14756
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.92851 (QuantReg: 14.53019) QuantErr: 14.53019 batch_time=0.58947
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.80769 (QuantReg: 14.74792) QuantErr: 14.74792 batch_time=0.60967
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.67163 (QuantReg: 14.80378) QuantErr: 14.80378 batch_time=0.61042
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.55061 (QuantReg: 14.89712) QuantErr: 14.89712 batch_time=0.64147
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.57586 (QuantReg: 14.78083) QuantErr: 14.78083 batch_time=0.65299
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.93371 (QuantReg: 14.71402) QuantErr: 14.71402 batch_time=0.61659
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.69173 (QuantReg: 14.79917) QuantErr: 14.79917 batch_time=0.60536
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.44961 (QuantReg: 14.84106) QuantErr: 14.84106 batch_time=0.99403
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.21433 (QuantReg: 14.55106) QuantErr: 14.55106 batch_time=0.59782
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.71526 (QuantReg: 14.82519) QuantErr: 14.82519 batch_time=0.61847
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.72015 (QuantReg: 15.04974) QuantErr: 15.04974 batch_time=0.65607
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.82609 (QuantReg: 14.81660) QuantErr: 14.81660 batch_time=0.60059
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.74302 (QuantReg: 15.00398) QuantErr: 15.00398 batch_time=0.60150
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.68716 (QuantReg: 14.76096) QuantErr: 14.76096 batch_time=0.60324
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.81739 (QuantReg: 14.90516) QuantErr: 14.90516 batch_time=0.67907
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.73177 (QuantReg: 15.05188) QuantErr: 15.05188 batch_time=0.64400
Train Epoch: 11 codebook_update_time=3.36671
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch11.pth ...
Done in 6.798s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch11.pth ...
Done in 15.463s
removing stale ckpt [epoch 10] [took 0.41s]
epoch : 11
loss : 1.7937710738182069
quant_reg : 14.825726219177247
quant_err : 14.825726219177247
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 53.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 68.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.908
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 44.13288459980294
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 69.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.109
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.904925195871705
mnt_best : 44.13288459980294
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.55544 (QuantReg: 14.62621) QuantErr: 14.62621 batch_time=36.60927
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.67741 (QuantReg: 14.78655) QuantErr: 14.78655 batch_time=0.68473
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.83994 (QuantReg: 14.79354) QuantErr: 14.79354 batch_time=0.63724
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.03610 (QuantReg: 14.91940) QuantErr: 14.91940 batch_time=0.61198
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.69203 (QuantReg: 14.74714) QuantErr: 14.74714 batch_time=0.61466
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.73210 (QuantReg: 14.83875) QuantErr: 14.83875 batch_time=2.22701
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.93045 (QuantReg: 15.06925) QuantErr: 15.06925 batch_time=0.65359
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.89288 (QuantReg: 14.73239) QuantErr: 14.73239 batch_time=0.61823
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.69572 (QuantReg: 14.87798) QuantErr: 14.87798 batch_time=0.60307
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.71959 (QuantReg: 14.85508) QuantErr: 14.85508 batch_time=0.68118
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.54439 (QuantReg: 14.86788) QuantErr: 14.86788 batch_time=0.72749
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.52803 (QuantReg: 14.95100) QuantErr: 14.95100 batch_time=0.60366
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.57730 (QuantReg: 14.71227) QuantErr: 14.71227 batch_time=0.61238
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.72277 (QuantReg: 14.73905) QuantErr: 14.73905 batch_time=1.62803
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.92381 (QuantReg: 15.17354) QuantErr: 15.17354 batch_time=0.98790
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.64909 (QuantReg: 14.99394) QuantErr: 14.99394 batch_time=0.59369
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.93943 (QuantReg: 15.01794) QuantErr: 15.01794 batch_time=0.60041
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.48104 (QuantReg: 15.02192) QuantErr: 15.02192 batch_time=0.61315
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.84878 (QuantReg: 14.88380) QuantErr: 14.88380 batch_time=0.62053
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.50868 (QuantReg: 15.05641) QuantErr: 15.05641 batch_time=0.61733
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.34715 (QuantReg: 15.16534) QuantErr: 15.16534 batch_time=0.66892
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.79115 (QuantReg: 15.16504) QuantErr: 15.16504 batch_time=0.61679
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.62323 (QuantReg: 14.93808) QuantErr: 14.93808 batch_time=0.61576
Train Epoch: 12 codebook_update_time=3.36654
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch12.pth ...
Done in 5.413s
removing stale ckpt [epoch 11] [took 0.22s]
epoch : 12
loss : 1.7255308842658996
quant_reg : 14.866978885650635
quant_err : 14.866978885650635
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 54.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 66.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.418
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 43.34873784997596
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 54.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.1195
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 44.52140702981216
mnt_best : 44.13288459980294
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.76173 (QuantReg: 14.50530) QuantErr: 14.50530 batch_time=38.74399
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.36109 (QuantReg: 15.07061) QuantErr: 15.07061 batch_time=0.61393
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.73149 (QuantReg: 14.71612) QuantErr: 14.71612 batch_time=0.62576
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.35924 (QuantReg: 15.00793) QuantErr: 15.00793 batch_time=0.59904
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.73484 (QuantReg: 15.03187) QuantErr: 15.03187 batch_time=0.59745
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.91834 (QuantReg: 14.65396) QuantErr: 14.65396 batch_time=0.62219
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.65842 (QuantReg: 15.03797) QuantErr: 15.03797 batch_time=0.60151
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.73040 (QuantReg: 15.00919) QuantErr: 15.00919 batch_time=0.59929
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.67670 (QuantReg: 15.06229) QuantErr: 15.06229 batch_time=0.61131
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.49981 (QuantReg: 14.93061) QuantErr: 14.93061 batch_time=0.60454
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.56921 (QuantReg: 15.04136) QuantErr: 15.04136 batch_time=0.62175
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.53307 (QuantReg: 15.19539) QuantErr: 15.19539 batch_time=0.60372
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.66922 (QuantReg: 14.92934) QuantErr: 14.92934 batch_time=0.66230
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.69310 (QuantReg: 15.02953) QuantErr: 15.02953 batch_time=0.60332
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.46975 (QuantReg: 15.01277) QuantErr: 15.01277 batch_time=0.61144
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.91430 (QuantReg: 14.92122) QuantErr: 14.92122 batch_time=0.59771
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.62471 (QuantReg: 15.13332) QuantErr: 15.13332 batch_time=0.60780
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.89878 (QuantReg: 14.94307) QuantErr: 14.94307 batch_time=0.60063
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.43787 (QuantReg: 15.05833) QuantErr: 15.05833 batch_time=0.60540
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.78911 (QuantReg: 14.80357) QuantErr: 14.80357 batch_time=0.58900
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.27846 (QuantReg: 15.05841) QuantErr: 15.05841 batch_time=0.61144
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.59560 (QuantReg: 14.94751) QuantErr: 14.94751 batch_time=0.63663
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.86626 (QuantReg: 14.93256) QuantErr: 14.93256 batch_time=0.61709
Train Epoch: 13 codebook_update_time=3.69503
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch13.pth ...
Done in 5.622s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch13.pth ...
Done in 11.374s
removing stale ckpt [epoch 12] [took 0.92s]
epoch : 13
loss : 1.6671384949684143
quant_reg : 14.916769687652588
quant_err : 14.916769687652588
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 24.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 54.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 68.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.387
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 44.62831625662184
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 70.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.5115
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 45.64082601036819
mnt_best : 44.62831625662184
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.58940 (QuantReg: 14.86577) QuantErr: 14.86577 batch_time=41.70780
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.47993 (QuantReg: 14.67618) QuantErr: 14.67618 batch_time=0.64507
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.58852 (QuantReg: 14.83022) QuantErr: 14.83022 batch_time=0.60062
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.51426 (QuantReg: 14.96500) QuantErr: 14.96500 batch_time=0.58841
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.47090 (QuantReg: 14.86492) QuantErr: 14.86492 batch_time=0.59309
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.95562 (QuantReg: 14.57959) QuantErr: 14.57959 batch_time=0.59632
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.95093 (QuantReg: 14.90255) QuantErr: 14.90255 batch_time=0.58106
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.49978 (QuantReg: 14.75384) QuantErr: 14.75384 batch_time=0.60964
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.47659 (QuantReg: 14.54350) QuantErr: 14.54350 batch_time=0.59238
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.31027 (QuantReg: 14.95935) QuantErr: 14.95935 batch_time=0.63369
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.98109 (QuantReg: 14.75143) QuantErr: 14.75143 batch_time=0.68369
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.42434 (QuantReg: 14.89341) QuantErr: 14.89341 batch_time=0.59181
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.48631 (QuantReg: 14.92842) QuantErr: 14.92842 batch_time=0.66826
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.76829 (QuantReg: 14.82491) QuantErr: 14.82491 batch_time=0.69758
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.50591 (QuantReg: 14.99356) QuantErr: 14.99356 batch_time=0.71238
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.82228 (QuantReg: 14.89112) QuantErr: 14.89112 batch_time=0.60830
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.59797 (QuantReg: 15.01182) QuantErr: 15.01182 batch_time=0.59651
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.73854 (QuantReg: 14.90222) QuantErr: 14.90222 batch_time=0.73253
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.58649 (QuantReg: 15.16385) QuantErr: 15.16385 batch_time=0.64783
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.65845 (QuantReg: 14.88251) QuantErr: 14.88251 batch_time=0.62945
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.68918 (QuantReg: 14.78080) QuantErr: 14.78080 batch_time=0.66671
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.43771 (QuantReg: 15.35351) QuantErr: 15.35351 batch_time=0.88239
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.24989 (QuantReg: 15.16531) QuantErr: 15.16531 batch_time=0.94350
Train Epoch: 14 codebook_update_time=3.26992
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch14.pth ...
Done in 5.941s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch14.pth ...
Done in 11.299s
removing stale ckpt [epoch 13] [took 0.22s]
epoch : 14
loss : 1.5994117650985717
quant_reg : 14.886657775878906
quant_err : 14.886657775878906
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 24.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 55.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 67.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.182
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 44.925958240546684
MSRVTT_jsfusion_test/v2t_metrics/R1: 25.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 69.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.295
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 45.939397194387105
mnt_best : 44.925958240546684
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.43423 (QuantReg: 14.76756) QuantErr: 14.76756 batch_time=40.56995
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.09867 (QuantReg: 14.89635) QuantErr: 14.89635 batch_time=0.63186
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.56343 (QuantReg: 14.88346) QuantErr: 14.88346 batch_time=0.62855
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.36052 (QuantReg: 14.65124) QuantErr: 14.65124 batch_time=2.00957
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.78785 (QuantReg: 14.88951) QuantErr: 14.88951 batch_time=0.62962
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.64767 (QuantReg: 14.92753) QuantErr: 14.92753 batch_time=0.60125
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.78391 (QuantReg: 15.06192) QuantErr: 15.06192 batch_time=3.84613
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.29914 (QuantReg: 14.82573) QuantErr: 14.82573 batch_time=0.61415
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.90588 (QuantReg: 14.94208) QuantErr: 14.94208 batch_time=0.61753
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.36015 (QuantReg: 14.94222) QuantErr: 14.94222 batch_time=0.64387
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.79191 (QuantReg: 14.93970) QuantErr: 14.93970 batch_time=0.60575
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.25204 (QuantReg: 15.08831) QuantErr: 15.08831 batch_time=0.62805
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.58174 (QuantReg: 15.08735) QuantErr: 15.08735 batch_time=0.63557
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.47214 (QuantReg: 14.96355) QuantErr: 14.96355 batch_time=0.62485
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.56162 (QuantReg: 14.96526) QuantErr: 14.96526 batch_time=0.72610
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.40285 (QuantReg: 14.89412) QuantErr: 14.89412 batch_time=0.65631
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.38284 (QuantReg: 14.97071) QuantErr: 14.97071 batch_time=0.65616
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.57367 (QuantReg: 14.94211) QuantErr: 14.94211 batch_time=0.61505
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.89767 (QuantReg: 14.87427) QuantErr: 14.87427 batch_time=0.64840
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.32148 (QuantReg: 15.10362) QuantErr: 15.10362 batch_time=0.82199
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.28209 (QuantReg: 14.99782) QuantErr: 14.99782 batch_time=0.64933
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.67338 (QuantReg: 15.10816) QuantErr: 15.10816 batch_time=0.65640
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.47116 (QuantReg: 14.97627) QuantErr: 14.97627 batch_time=0.62685
Train Epoch: 15 codebook_update_time=3.47591
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch15.pth ...
Done in 4.783s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch15.pth ...
Done in 9.736s
removing stale ckpt [epoch 14] [took 0.02s]
epoch : 15
loss : 1.528383713722229
quant_reg : 14.923548221588135
quant_err : 14.923548221588135
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 25.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 55.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 68.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.008
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 45.89815907040152
MSRVTT_jsfusion_test/v2t_metrics/R1: 24.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 68.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.991
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 45.351785968829596
mnt_best : 45.89815907040152
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.67544 (QuantReg: 14.75023) QuantErr: 14.75023 batch_time=35.15494
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.56576 (QuantReg: 14.65257) QuantErr: 14.65257 batch_time=2.63380
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.48553 (QuantReg: 14.87454) QuantErr: 14.87454 batch_time=0.65028
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.03680 (QuantReg: 14.51218) QuantErr: 14.51218 batch_time=0.60841
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.57476 (QuantReg: 15.00349) QuantErr: 15.00349 batch_time=0.61697
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.62868 (QuantReg: 14.95755) QuantErr: 14.95755 batch_time=1.05757
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.24919 (QuantReg: 15.12326) QuantErr: 15.12326 batch_time=0.70051
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.49348 (QuantReg: 14.79630) QuantErr: 14.79630 batch_time=0.59609
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.46759 (QuantReg: 14.88569) QuantErr: 14.88569 batch_time=0.60722
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.40801 (QuantReg: 15.00046) QuantErr: 15.00046 batch_time=0.62661
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.33399 (QuantReg: 15.09231) QuantErr: 15.09231 batch_time=0.68024
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.43472 (QuantReg: 15.03469) QuantErr: 15.03469 batch_time=0.61043
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.46943 (QuantReg: 14.92658) QuantErr: 14.92658 batch_time=0.64230
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.78652 (QuantReg: 14.92955) QuantErr: 14.92955 batch_time=0.60774
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.79606 (QuantReg: 15.10196) QuantErr: 15.10196 batch_time=0.67030
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.67519 (QuantReg: 15.14654) QuantErr: 15.14654 batch_time=0.59600
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.24728 (QuantReg: 15.29938) QuantErr: 15.29938 batch_time=0.60971
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.70178 (QuantReg: 15.00850) QuantErr: 15.00850 batch_time=0.60484
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.48326 (QuantReg: 15.19775) QuantErr: 15.19775 batch_time=0.61718
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.30005 (QuantReg: 15.32317) QuantErr: 15.32317 batch_time=0.87271
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.56826 (QuantReg: 14.93237) QuantErr: 14.93237 batch_time=0.63654
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.74822 (QuantReg: 14.91516) QuantErr: 14.91516 batch_time=0.65652
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.49283 (QuantReg: 14.89188) QuantErr: 14.89188 batch_time=0.70760
Train Epoch: 16 codebook_update_time=3.64849
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch16.pth ...
Done in 6.917s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch16.pth ...
Done in 12.293s
removing stale ckpt [epoch 15] [took 0.61s]
epoch : 16
loss : 1.517564145565033
quant_reg : 14.956533226013184
quant_err : 14.956533226013184
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 25.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 55.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 68.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.823
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 46.072784899161675
MSRVTT_jsfusion_test/v2t_metrics/R1: 25.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 56.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 70.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.612
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 46.718532323456415
mnt_best : 46.072784899161675
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.26240 (QuantReg: 15.01076) QuantErr: 15.01076 batch_time=38.20995
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.58175 (QuantReg: 14.74286) QuantErr: 14.74286 batch_time=0.61359
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.33616 (QuantReg: 14.74590) QuantErr: 14.74590 batch_time=0.72779
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.96860 (QuantReg: 15.03395) QuantErr: 15.03395 batch_time=0.65243
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.49592 (QuantReg: 14.69210) QuantErr: 14.69210 batch_time=0.61555
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.44568 (QuantReg: 14.95867) QuantErr: 14.95867 batch_time=0.64403
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.73567 (QuantReg: 14.93917) QuantErr: 14.93917 batch_time=5.56494
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.20875 (QuantReg: 14.90778) QuantErr: 14.90778 batch_time=0.62779
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.41481 (QuantReg: 14.78251) QuantErr: 14.78251 batch_time=0.67490
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.54901 (QuantReg: 14.88266) QuantErr: 14.88266 batch_time=0.65516
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.56143 (QuantReg: 15.01962) QuantErr: 15.01962 batch_time=0.61440
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.49722 (QuantReg: 14.87582) QuantErr: 14.87582 batch_time=0.62026
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.64691 (QuantReg: 14.86518) QuantErr: 14.86518 batch_time=0.60844
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.44334 (QuantReg: 15.12176) QuantErr: 15.12176 batch_time=0.62197
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.82742 (QuantReg: 14.69598) QuantErr: 14.69598 batch_time=0.65020
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.41719 (QuantReg: 14.85143) QuantErr: 14.85143 batch_time=0.64533
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.56533 (QuantReg: 15.11885) QuantErr: 15.11885 batch_time=0.82108
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.28714 (QuantReg: 15.09997) QuantErr: 15.09997 batch_time=0.61968
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.47244 (QuantReg: 14.70850) QuantErr: 14.70850 batch_time=1.18664
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.36058 (QuantReg: 15.04751) QuantErr: 15.04751 batch_time=0.62333
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.40709 (QuantReg: 14.71476) QuantErr: 14.71476 batch_time=0.65208
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.46624 (QuantReg: 15.18106) QuantErr: 15.18106 batch_time=0.58703
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.24706 (QuantReg: 14.89689) QuantErr: 14.89689 batch_time=0.62758
Train Epoch: 17 codebook_update_time=3.77710
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch17.pth ...
Done in 4.819s
removing stale ckpt [epoch 16] [took 0.15s]
epoch : 17
loss : 1.4592305035591127
quant_reg : 14.93496936416626
quant_err : 14.93496936416626
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 24.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 55.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 69.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.388
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 45.69050584278528
MSRVTT_jsfusion_test/v2t_metrics/R1: 25.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 56.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 70.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 91.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.217
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 46.59584286320711
mnt_best : 46.072784899161675
not_improved_count: 1
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.51595 (QuantReg: 14.73872) QuantErr: 14.73872 batch_time=35.41813
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.39849 (QuantReg: 14.84584) QuantErr: 14.84584 batch_time=0.64262
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.23512 (QuantReg: 14.86761) QuantErr: 14.86761 batch_time=0.65756
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.21318 (QuantReg: 14.95577) QuantErr: 14.95577 batch_time=0.60150
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.40431 (QuantReg: 14.73107) QuantErr: 14.73107 batch_time=0.60170
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.26040 (QuantReg: 15.10220) QuantErr: 15.10220 batch_time=0.63349
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.05216 (QuantReg: 15.22773) QuantErr: 15.22773 batch_time=0.68668
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.18787 (QuantReg: 15.03806) QuantErr: 15.03806 batch_time=0.61327
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.66602 (QuantReg: 15.06734) QuantErr: 15.06734 batch_time=0.64860
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.37147 (QuantReg: 14.84215) QuantErr: 14.84215 batch_time=0.63531
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.75211 (QuantReg: 14.90539) QuantErr: 14.90539 batch_time=0.61842
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.57868 (QuantReg: 14.93986) QuantErr: 14.93986 batch_time=0.71353
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.52817 (QuantReg: 14.95520) QuantErr: 14.95520 batch_time=0.71479
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.69454 (QuantReg: 14.75505) QuantErr: 14.75505 batch_time=0.59515
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.55624 (QuantReg: 14.82053) QuantErr: 14.82053 batch_time=0.66864
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.49256 (QuantReg: 15.23168) QuantErr: 15.23168 batch_time=0.60138
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.29587 (QuantReg: 14.95549) QuantErr: 14.95549 batch_time=0.62453
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.20835 (QuantReg: 15.08308) QuantErr: 15.08308 batch_time=0.66211
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.32279 (QuantReg: 15.15848) QuantErr: 15.15848 batch_time=2.74320
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.08193 (QuantReg: 15.20523) QuantErr: 15.20523 batch_time=0.60798
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.78651 (QuantReg: 14.91039) QuantErr: 14.91039 batch_time=0.60659
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.25615 (QuantReg: 15.01982) QuantErr: 15.01982 batch_time=0.61777
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.43191 (QuantReg: 15.08445) QuantErr: 15.08445 batch_time=0.60406
Train Epoch: 18 codebook_update_time=3.69019
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch18.pth ...
Done in 4.698s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch18.pth ...
Done in 9.339s
removing stale ckpt [epoch 17] [took 0.02s]
epoch : 18
loss : 1.399221589565277
quant_reg : 14.98605540084839
quant_err : 14.98605540084839
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 25.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 55.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 68.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 4.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.57
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 46.11979518878253
MSRVTT_jsfusion_test/v2t_metrics/R1: 25.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 57.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 70.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 4.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.1385
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 46.88887887696562
mnt_best : 46.11979518878253
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.16126 (QuantReg: 14.98719) QuantErr: 14.98719 batch_time=35.02848
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.29037 (QuantReg: 14.96809) QuantErr: 14.96809 batch_time=0.59612
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.54693 (QuantReg: 14.66541) QuantErr: 14.66541 batch_time=0.59542
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.61541 (QuantReg: 14.78312) QuantErr: 14.78312 batch_time=0.61427
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.30158 (QuantReg: 14.89591) QuantErr: 14.89591 batch_time=0.60476
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 0.98105 (QuantReg: 15.01017) QuantErr: 15.01017 batch_time=0.59942
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.03383 (QuantReg: 15.04159) QuantErr: 15.04159 batch_time=0.62430
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.21866 (QuantReg: 14.92532) QuantErr: 14.92532 batch_time=0.65540
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.17863 (QuantReg: 15.10601) QuantErr: 15.10601 batch_time=0.59782
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.59910 (QuantReg: 14.78387) QuantErr: 14.78387 batch_time=0.64596
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.38254 (QuantReg: 14.79678) QuantErr: 14.79678 batch_time=0.63034
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.12855 (QuantReg: 15.16109) QuantErr: 15.16109 batch_time=0.60955
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.41905 (QuantReg: 15.21872) QuantErr: 15.21872 batch_time=0.88983
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.48164 (QuantReg: 14.96736) QuantErr: 14.96736 batch_time=0.63071
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.68958 (QuantReg: 14.88919) QuantErr: 14.88919 batch_time=0.66730
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.00345 (QuantReg: 14.97363) QuantErr: 14.97363 batch_time=1.74434
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.27518 (QuantReg: 14.92496) QuantErr: 14.92496 batch_time=0.77968
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.42129 (QuantReg: 15.06335) QuantErr: 15.06335 batch_time=1.05835
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.41507 (QuantReg: 14.87075) QuantErr: 14.87075 batch_time=0.60399
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.36769 (QuantReg: 15.00472) QuantErr: 15.00472 batch_time=0.98677
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.43731 (QuantReg: 15.06370) QuantErr: 15.06370 batch_time=0.64440
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.79252 (QuantReg: 15.06413) QuantErr: 15.06413 batch_time=0.59777
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.44869 (QuantReg: 15.25034) QuantErr: 15.25034 batch_time=0.60238
Train Epoch: 19 codebook_update_time=3.58008
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch19.pth ...
Done in 4.807s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M64/checkpoint-epoch19.pth ...
Done in 9.791s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 1.3725321772098542
quant_reg : 15.000696701049804
quant_err : 15.000696701049804
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 25.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 56.4