-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_M8.txt
2613 lines (2613 loc) · 193 KB
/
HCQ_MSRVTT_1kA_M8.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 1029.215271949768 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 104.19681787490845 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 71.15362191200256 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch0.pth ...
Done in 1.570s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch0.pth ...
Done in 3.127s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 5.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 496.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 498.07
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 501.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.01
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.81714 (QuantReg: 10.41575) QuantErr: 10.41575 batch_time=25.68302
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.63555 (QuantReg: 10.46349) QuantErr: 10.46349 batch_time=0.42985
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.45069 (QuantReg: 10.45829) QuantErr: 10.45829 batch_time=0.75728
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.21720 (QuantReg: 10.46290) QuantErr: 10.46290 batch_time=0.48475
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.36301 (QuantReg: 10.47482) QuantErr: 10.47482 batch_time=0.42808
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.65230 (QuantReg: 10.47504) QuantErr: 10.47504 batch_time=0.43303
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.81838 (QuantReg: 10.47536) QuantErr: 10.47536 batch_time=0.43789
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 4.89487 (QuantReg: 10.47477) QuantErr: 10.47477 batch_time=3.56599
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.43334 (QuantReg: 10.47029) QuantErr: 10.47029 batch_time=0.43601
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.24274 (QuantReg: 10.48944) QuantErr: 10.48944 batch_time=0.44501
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 4.65109 (QuantReg: 10.46901) QuantErr: 10.46901 batch_time=0.44117
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.66491 (QuantReg: 10.47411) QuantErr: 10.47411 batch_time=0.46135
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.10681 (QuantReg: 10.47465) QuantErr: 10.47465 batch_time=0.44544
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.35886 (QuantReg: 10.47472) QuantErr: 10.47472 batch_time=0.42953
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.69609 (QuantReg: 10.48206) QuantErr: 10.48206 batch_time=0.42349
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.07510 (QuantReg: 10.46540) QuantErr: 10.46540 batch_time=0.81223
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.88526 (QuantReg: 10.47409) QuantErr: 10.47409 batch_time=0.42569
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.46275 (QuantReg: 10.45769) QuantErr: 10.45769 batch_time=0.46675
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.06487 (QuantReg: 10.46845) QuantErr: 10.46845 batch_time=0.42269
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.39544 (QuantReg: 10.47771) QuantErr: 10.47771 batch_time=0.42692
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 3.55029 (QuantReg: 10.48557) QuantErr: 10.48557 batch_time=0.45231
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.16693 (QuantReg: 10.46877) QuantErr: 10.46877 batch_time=0.45272
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.66705 (QuantReg: 10.46107) QuantErr: 10.46107 batch_time=0.43500
Train Epoch: 1 codebook_update_time=0.63722
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch1.pth ...
Done in 4.367s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch1.pth ...
Done in 8.511s
epoch : 1
loss : 5.323340451240539
quant_reg : 10.47128187942505
quant_err : 10.47128187942505
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 7.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 27.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 41.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 76.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 15.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 49.465
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.677484912999752
MSRVTT_jsfusion_test/v2t_metrics/R1: 7.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 28.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 43.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 76.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 14.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 47.931
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.2720389572676
mnt_best : 20.677484912999752
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.20408 (QuantReg: 3.78055) QuantErr: 3.78055 batch_time=31.85338
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.00739 (QuantReg: 3.88626) QuantErr: 3.88626 batch_time=0.42994
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.04994 (QuantReg: 4.07516) QuantErr: 4.07516 batch_time=0.41743
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.80016 (QuantReg: 4.01517) QuantErr: 4.01517 batch_time=0.44785
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.82682 (QuantReg: 4.23654) QuantErr: 4.23654 batch_time=0.42786
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.91793 (QuantReg: 4.20813) QuantErr: 4.20813 batch_time=0.45266
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.72754 (QuantReg: 4.06402) QuantErr: 4.06402 batch_time=0.42717
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.73951 (QuantReg: 4.25174) QuantErr: 4.25174 batch_time=0.41893
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.65777 (QuantReg: 4.31652) QuantErr: 4.31652 batch_time=0.43309
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.30787 (QuantReg: 4.24693) QuantErr: 4.24693 batch_time=0.62653
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.70389 (QuantReg: 4.12388) QuantErr: 4.12388 batch_time=0.44127
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.20868 (QuantReg: 4.39634) QuantErr: 4.39634 batch_time=0.46362
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.59427 (QuantReg: 4.47988) QuantErr: 4.47988 batch_time=0.49955
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.94063 (QuantReg: 4.42050) QuantErr: 4.42050 batch_time=0.86911
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.99430 (QuantReg: 4.51127) QuantErr: 4.51127 batch_time=0.44236
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.41369 (QuantReg: 4.59930) QuantErr: 4.59930 batch_time=0.45722
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.79777 (QuantReg: 4.59091) QuantErr: 4.59091 batch_time=0.44315
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.57500 (QuantReg: 4.71082) QuantErr: 4.71082 batch_time=0.42307
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.30766 (QuantReg: 4.77548) QuantErr: 4.77548 batch_time=0.42914
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.27894 (QuantReg: 4.78223) QuantErr: 4.78223 batch_time=1.66287
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.37445 (QuantReg: 4.72642) QuantErr: 4.72642 batch_time=0.44249
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.49620 (QuantReg: 4.73311) QuantErr: 4.73311 batch_time=0.42425
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.77889 (QuantReg: 4.78578) QuantErr: 4.78578 batch_time=0.45750
Train Epoch: 2 codebook_update_time=0.53969
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch2.pth ...
Done in 8.351s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch2.pth ...
Done in 12.676s
removing stale ckpt [epoch 1] [took 0.03s]
removing stale ckpt [epoch 0] [took 0.07s]
epoch : 2
loss : 3.677021758079529
quant_reg : 4.38396501159668
quant_err : 4.38396501159668
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 10.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 34.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 49.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 80.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 11.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 41.022
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.63255339494623
MSRVTT_jsfusion_test/v2t_metrics/R1: 10.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 33.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 48.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 79.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 11.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 40.787
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.442242210208008
mnt_best : 25.63255339494623
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.25458 (QuantReg: 3.98738) QuantErr: 3.98738 batch_time=37.08669
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.12901 (QuantReg: 4.05113) QuantErr: 4.05113 batch_time=0.41297
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.33295 (QuantReg: 4.11723) QuantErr: 4.11723 batch_time=0.41421
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.87559 (QuantReg: 3.97030) QuantErr: 3.97030 batch_time=0.43785
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 2.92167 (QuantReg: 4.13957) QuantErr: 4.13957 batch_time=0.44106
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.39274 (QuantReg: 3.99087) QuantErr: 3.99087 batch_time=0.45090
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.62245 (QuantReg: 4.25633) QuantErr: 4.25633 batch_time=0.44109
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.45756 (QuantReg: 4.23337) QuantErr: 4.23337 batch_time=0.42039
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.43879 (QuantReg: 4.11016) QuantErr: 4.11016 batch_time=0.42350
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.62021 (QuantReg: 4.26996) QuantErr: 4.26996 batch_time=0.43290
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.44027 (QuantReg: 4.18842) QuantErr: 4.18842 batch_time=0.42649
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.13382 (QuantReg: 4.14035) QuantErr: 4.14035 batch_time=0.43161
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 2.87178 (QuantReg: 4.11556) QuantErr: 4.11556 batch_time=0.45280
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.29067 (QuantReg: 4.11220) QuantErr: 4.11220 batch_time=0.43547
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.55469 (QuantReg: 4.00525) QuantErr: 4.00525 batch_time=0.41851
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.79319 (QuantReg: 4.19193) QuantErr: 4.19193 batch_time=0.44238
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.22578 (QuantReg: 4.36475) QuantErr: 4.36475 batch_time=0.45679
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 2.82123 (QuantReg: 4.35595) QuantErr: 4.35595 batch_time=0.44266
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.92147 (QuantReg: 4.29392) QuantErr: 4.29392 batch_time=0.46999
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.07134 (QuantReg: 4.27824) QuantErr: 4.27824 batch_time=0.42938
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.05941 (QuantReg: 4.34350) QuantErr: 4.34350 batch_time=0.42050
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.00479 (QuantReg: 4.45200) QuantErr: 4.45200 batch_time=0.42594
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.14612 (QuantReg: 4.41630) QuantErr: 4.41630 batch_time=0.43191
Train Epoch: 3 codebook_update_time=0.51334
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch3.pth ...
Done in 4.467s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch3.pth ...
Done in 8.539s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.1272058172225954
quant_reg : 4.193632473945618
quant_err : 4.193632473945618
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 11.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 37.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 52.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 81.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 37.995
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.127814473901246
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 35.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 51.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 82.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 10.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 38.829
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.631495848260258
mnt_best : 28.127814473901246
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.03706 (QuantReg: 4.12323) QuantErr: 4.12323 batch_time=31.45380
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.74036 (QuantReg: 4.10504) QuantErr: 4.10504 batch_time=1.95180
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.85550 (QuantReg: 4.16017) QuantErr: 4.16017 batch_time=0.41441
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.69983 (QuantReg: 4.21261) QuantErr: 4.21261 batch_time=0.41866
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.68618 (QuantReg: 4.24835) QuantErr: 4.24835 batch_time=0.42183
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.92697 (QuantReg: 4.11620) QuantErr: 4.11620 batch_time=0.42807
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.56018 (QuantReg: 4.05223) QuantErr: 4.05223 batch_time=0.41652
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.76533 (QuantReg: 4.20638) QuantErr: 4.20638 batch_time=0.46398
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.04807 (QuantReg: 4.10890) QuantErr: 4.10890 batch_time=0.42302
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.46525 (QuantReg: 4.19600) QuantErr: 4.19600 batch_time=0.42634
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.70557 (QuantReg: 4.41842) QuantErr: 4.41842 batch_time=0.42973
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.51647 (QuantReg: 4.48676) QuantErr: 4.48676 batch_time=0.41922
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.03206 (QuantReg: 4.29096) QuantErr: 4.29096 batch_time=0.43535
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.63379 (QuantReg: 4.27547) QuantErr: 4.27547 batch_time=0.43349
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.63029 (QuantReg: 4.28665) QuantErr: 4.28665 batch_time=0.41694
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.81361 (QuantReg: 4.30069) QuantErr: 4.30069 batch_time=0.41742
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.75277 (QuantReg: 4.22343) QuantErr: 4.22343 batch_time=0.43077
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.72937 (QuantReg: 4.32893) QuantErr: 4.32893 batch_time=0.42986
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.57003 (QuantReg: 4.54216) QuantErr: 4.54216 batch_time=0.42732
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.43248 (QuantReg: 4.25951) QuantErr: 4.25951 batch_time=0.42663
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.36119 (QuantReg: 4.51371) QuantErr: 4.51371 batch_time=0.42863
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.15737 (QuantReg: 4.54112) QuantErr: 4.54112 batch_time=0.42749
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.83050 (QuantReg: 4.33989) QuantErr: 4.33989 batch_time=0.43189
Train Epoch: 4 codebook_update_time=0.51100
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch4.pth ...
Done in 4.045s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch4.pth ...
Done in 8.312s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 2.8315762634277344
quant_reg : 4.2362189989089964
quant_err : 4.2362189989089964
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 11.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 37.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 52.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 83.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.931
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.75183685806466
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 37.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 83.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 36.0275
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.32486753609177
mnt_best : 28.75183685806466
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.81306 (QuantReg: 4.07731) QuantErr: 4.07731 batch_time=30.93927
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.05709 (QuantReg: 4.26653) QuantErr: 4.26653 batch_time=0.42069
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.21986 (QuantReg: 4.07045) QuantErr: 4.07045 batch_time=0.50407
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.51411 (QuantReg: 4.30948) QuantErr: 4.30948 batch_time=0.43694
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.63631 (QuantReg: 4.20420) QuantErr: 4.20420 batch_time=0.45423
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.38739 (QuantReg: 4.33811) QuantErr: 4.33811 batch_time=0.41798
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.69273 (QuantReg: 4.16937) QuantErr: 4.16937 batch_time=0.96745
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.04549 (QuantReg: 4.10889) QuantErr: 4.10889 batch_time=0.42744
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.50400 (QuantReg: 4.28712) QuantErr: 4.28712 batch_time=0.42372
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.58690 (QuantReg: 4.23296) QuantErr: 4.23296 batch_time=0.43067
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.59914 (QuantReg: 4.36205) QuantErr: 4.36205 batch_time=0.43819
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.64789 (QuantReg: 4.18950) QuantErr: 4.18950 batch_time=0.46129
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.41477 (QuantReg: 4.38683) QuantErr: 4.38683 batch_time=0.43541
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.24471 (QuantReg: 4.21892) QuantErr: 4.21892 batch_time=4.09278
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.89147 (QuantReg: 4.39663) QuantErr: 4.39663 batch_time=0.95355
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.33863 (QuantReg: 4.41165) QuantErr: 4.41165 batch_time=0.49220
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.71466 (QuantReg: 4.35873) QuantErr: 4.35873 batch_time=0.42034
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.58500 (QuantReg: 4.30252) QuantErr: 4.30252 batch_time=0.42893
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.65550 (QuantReg: 4.43433) QuantErr: 4.43433 batch_time=0.44978
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.57520 (QuantReg: 4.27708) QuantErr: 4.27708 batch_time=0.47320
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.87497 (QuantReg: 4.38400) QuantErr: 4.38400 batch_time=0.43844
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.40138 (QuantReg: 4.28918) QuantErr: 4.28918 batch_time=0.44080
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.81225 (QuantReg: 4.43883) QuantErr: 4.43883 batch_time=0.42850
Train Epoch: 5 codebook_update_time=0.50804
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch5.pth ...
Done in 4.031s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch5.pth ...
Done in 8.269s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.565650972366333
quant_reg : 4.3043563823699955
quant_err : 4.3043563823699955
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 40.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 35.003
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.69742272564833
MSRVTT_jsfusion_test/v2t_metrics/R1: 12.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 38.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 54.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.953
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.759089094898094
mnt_best : 31.69742272564833
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.67673 (QuantReg: 4.22764) QuantErr: 4.22764 batch_time=33.45656
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.66905 (QuantReg: 3.99351) QuantErr: 3.99351 batch_time=0.47999
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.37804 (QuantReg: 4.18652) QuantErr: 4.18652 batch_time=0.42467
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.40280 (QuantReg: 4.23655) QuantErr: 4.23655 batch_time=0.51551
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.30083 (QuantReg: 4.14825) QuantErr: 4.14825 batch_time=0.42906
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.25636 (QuantReg: 4.18206) QuantErr: 4.18206 batch_time=0.42301
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.65638 (QuantReg: 4.33109) QuantErr: 4.33109 batch_time=0.44545
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.33542 (QuantReg: 4.37256) QuantErr: 4.37256 batch_time=0.44514
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.44635 (QuantReg: 4.36897) QuantErr: 4.36897 batch_time=0.42069
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.15808 (QuantReg: 4.22172) QuantErr: 4.22172 batch_time=0.44352
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.34820 (QuantReg: 4.52061) QuantErr: 4.52061 batch_time=0.43637
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.31512 (QuantReg: 4.23975) QuantErr: 4.23975 batch_time=0.43501
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.22766 (QuantReg: 4.39387) QuantErr: 4.39387 batch_time=0.42593
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.20640 (QuantReg: 4.27002) QuantErr: 4.27002 batch_time=0.46390
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.65987 (QuantReg: 4.52806) QuantErr: 4.52806 batch_time=0.43198
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.41792 (QuantReg: 4.39428) QuantErr: 4.39428 batch_time=0.43603
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.09638 (QuantReg: 4.42944) QuantErr: 4.42944 batch_time=0.45619
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.49141 (QuantReg: 4.35450) QuantErr: 4.35450 batch_time=0.41577
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.65523 (QuantReg: 4.43915) QuantErr: 4.43915 batch_time=0.45805
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.40034 (QuantReg: 4.41037) QuantErr: 4.41037 batch_time=0.45775
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.22633 (QuantReg: 4.37762) QuantErr: 4.37762 batch_time=0.42767
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.18130 (QuantReg: 4.34457) QuantErr: 4.34457 batch_time=0.44678
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.04837 (QuantReg: 4.43356) QuantErr: 4.43356 batch_time=0.41674
Train Epoch: 6 codebook_update_time=0.61591
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch6.pth ...
Done in 4.003s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.382354570388794
quant_reg : 4.351939671516418
quant_err : 4.351939671516418
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.483
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.853488323810026
MSRVTT_jsfusion_test/v2t_metrics/R1: 12.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 40.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 56.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 35.07
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.58140040464788
mnt_best : 31.69742272564833
not_improved_count: 1
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.45521 (QuantReg: 4.11049) QuantErr: 4.11049 batch_time=34.24508
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.44304 (QuantReg: 4.24773) QuantErr: 4.24773 batch_time=0.43732
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.22351 (QuantReg: 4.11393) QuantErr: 4.11393 batch_time=0.43264
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 1.98397 (QuantReg: 4.30040) QuantErr: 4.30040 batch_time=0.43289
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.00983 (QuantReg: 4.28128) QuantErr: 4.28128 batch_time=0.43397
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.08476 (QuantReg: 4.46147) QuantErr: 4.46147 batch_time=0.42814
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.23561 (QuantReg: 4.47216) QuantErr: 4.47216 batch_time=0.66662
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.13808 (QuantReg: 4.42233) QuantErr: 4.42233 batch_time=0.43206
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.31024 (QuantReg: 4.46263) QuantErr: 4.46263 batch_time=0.45398
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.81866 (QuantReg: 4.37519) QuantErr: 4.37519 batch_time=0.44117
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.68507 (QuantReg: 4.43513) QuantErr: 4.43513 batch_time=0.42992
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.17435 (QuantReg: 4.42093) QuantErr: 4.42093 batch_time=0.46857
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.13211 (QuantReg: 4.39186) QuantErr: 4.39186 batch_time=0.43080
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.31577 (QuantReg: 4.29121) QuantErr: 4.29121 batch_time=0.43703
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.68430 (QuantReg: 4.29528) QuantErr: 4.29528 batch_time=0.78421
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.26347 (QuantReg: 4.39804) QuantErr: 4.39804 batch_time=0.48080
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.69720 (QuantReg: 4.51082) QuantErr: 4.51082 batch_time=0.44413
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.22634 (QuantReg: 4.37752) QuantErr: 4.37752 batch_time=0.42311
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.66512 (QuantReg: 4.49450) QuantErr: 4.49450 batch_time=0.97560
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.43780 (QuantReg: 4.56110) QuantErr: 4.56110 batch_time=0.44521
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.09847 (QuantReg: 4.58274) QuantErr: 4.58274 batch_time=0.44058
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.16952 (QuantReg: 4.73121) QuantErr: 4.73121 batch_time=0.45120
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.17757 (QuantReg: 4.53468) QuantErr: 4.53468 batch_time=0.43689
Train Epoch: 7 codebook_update_time=0.50308
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch7.pth ...
Done in 4.114s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.2116307668685913
quant_reg : 4.407430799484253
quant_err : 4.407430799484253
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.807500000000005
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.128301419133415
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 56.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.1385
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.303096636850974
mnt_best : 31.69742272564833
not_improved_count: 2
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.40216 (QuantReg: 4.37177) QuantErr: 4.37177 batch_time=34.08363
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.42417 (QuantReg: 4.35494) QuantErr: 4.35494 batch_time=0.43057
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.64326 (QuantReg: 4.43619) QuantErr: 4.43619 batch_time=0.41791
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.16495 (QuantReg: 4.32475) QuantErr: 4.32475 batch_time=0.44379
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.27988 (QuantReg: 4.51790) QuantErr: 4.51790 batch_time=0.44038
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.00268 (QuantReg: 4.32550) QuantErr: 4.32550 batch_time=0.41804
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.15444 (QuantReg: 4.52898) QuantErr: 4.52898 batch_time=0.41600
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.41975 (QuantReg: 4.51302) QuantErr: 4.51302 batch_time=0.46806
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.35166 (QuantReg: 4.35183) QuantErr: 4.35183 batch_time=0.44030
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.51607 (QuantReg: 4.47580) QuantErr: 4.47580 batch_time=0.42419
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.39465 (QuantReg: 4.55211) QuantErr: 4.55211 batch_time=0.43262
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.12558 (QuantReg: 4.40952) QuantErr: 4.40952 batch_time=0.46430
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.25565 (QuantReg: 4.39526) QuantErr: 4.39526 batch_time=0.49733
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.28419 (QuantReg: 4.62447) QuantErr: 4.62447 batch_time=0.57376
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.06056 (QuantReg: 4.64081) QuantErr: 4.64081 batch_time=0.49409
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.89799 (QuantReg: 4.40616) QuantErr: 4.40616 batch_time=0.42970
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 1.97020 (QuantReg: 4.39045) QuantErr: 4.39045 batch_time=0.43158
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.64487 (QuantReg: 4.50610) QuantErr: 4.50610 batch_time=0.47869
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.23278 (QuantReg: 4.53730) QuantErr: 4.53730 batch_time=0.44579
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.27006 (QuantReg: 4.37529) QuantErr: 4.37529 batch_time=0.43446
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.18241 (QuantReg: 4.45873) QuantErr: 4.45873 batch_time=0.43052
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.72900 (QuantReg: 4.53488) QuantErr: 4.53488 batch_time=0.42343
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.29384 (QuantReg: 4.52980) QuantErr: 4.52980 batch_time=0.43924
Train Epoch: 8 codebook_update_time=0.50500
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch8.pth ...
Done in 4.112s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch8.pth ...
Done in 8.168s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 2.116631356239319
quant_reg : 4.460276000976562
quant_err : 4.460276000976562
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.7235
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.136544328188045
MSRVTT_jsfusion_test/v2t_metrics/R1: 13.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.629999999999995
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.656137358035814
mnt_best : 34.136544328188045
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.68712 (QuantReg: 4.57143) QuantErr: 4.57143 batch_time=36.92114
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.25663 (QuantReg: 4.49159) QuantErr: 4.49159 batch_time=0.43506
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.79488 (QuantReg: 4.31648) QuantErr: 4.31648 batch_time=0.42077
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.10623 (QuantReg: 4.42879) QuantErr: 4.42879 batch_time=0.43558
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.98625 (QuantReg: 4.54938) QuantErr: 4.54938 batch_time=0.42532
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.68690 (QuantReg: 4.54347) QuantErr: 4.54347 batch_time=0.41439
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.75481 (QuantReg: 4.58411) QuantErr: 4.58411 batch_time=0.46510
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.92078 (QuantReg: 4.61163) QuantErr: 4.61163 batch_time=0.42318
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.17264 (QuantReg: 4.37312) QuantErr: 4.37312 batch_time=1.25842
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.88688 (QuantReg: 4.48931) QuantErr: 4.48931 batch_time=0.46326
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.04213 (QuantReg: 4.48318) QuantErr: 4.48318 batch_time=0.46750
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.67503 (QuantReg: 4.48287) QuantErr: 4.48287 batch_time=0.45359
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.94116 (QuantReg: 4.50013) QuantErr: 4.50013 batch_time=1.69489
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.90230 (QuantReg: 4.41132) QuantErr: 4.41132 batch_time=0.42174
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.10750 (QuantReg: 4.30819) QuantErr: 4.30819 batch_time=0.53104
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.71852 (QuantReg: 4.62891) QuantErr: 4.62891 batch_time=0.45280
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.11200 (QuantReg: 4.68300) QuantErr: 4.68300 batch_time=0.43832
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.50564 (QuantReg: 4.60197) QuantErr: 4.60197 batch_time=0.91729
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.01481 (QuantReg: 4.60834) QuantErr: 4.60834 batch_time=0.43293
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.94325 (QuantReg: 4.63041) QuantErr: 4.63041 batch_time=0.42439
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.78912 (QuantReg: 4.69146) QuantErr: 4.69146 batch_time=0.43144
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.87661 (QuantReg: 4.32716) QuantErr: 4.32716 batch_time=0.43014
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.84796 (QuantReg: 4.40370) QuantErr: 4.40370 batch_time=0.43209
Train Epoch: 9 codebook_update_time=0.55311
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch9.pth ...
Done in 5.966s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 1.9878368973731995
quant_reg : 4.489647365570068
quant_err : 4.489647365570068
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.622
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.20875924105679
MSRVTT_jsfusion_test/v2t_metrics/R1: 13.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.6395
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.2720779055769
mnt_best : 34.136544328188045
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.77814 (QuantReg: 4.52641) QuantErr: 4.52641 batch_time=31.33950
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.95117 (QuantReg: 4.45411) QuantErr: 4.45411 batch_time=0.41597
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.23086 (QuantReg: 4.62201) QuantErr: 4.62201 batch_time=0.42814
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.99991 (QuantReg: 4.40729) QuantErr: 4.40729 batch_time=0.42337
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.63546 (QuantReg: 4.42319) QuantErr: 4.42319 batch_time=0.47466
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.98870 (QuantReg: 4.51683) QuantErr: 4.51683 batch_time=0.43624
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.33632 (QuantReg: 4.42464) QuantErr: 4.42464 batch_time=0.80067
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.96418 (QuantReg: 4.53138) QuantErr: 4.53138 batch_time=0.42178
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.00982 (QuantReg: 4.50192) QuantErr: 4.50192 batch_time=0.44766
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.90181 (QuantReg: 4.54740) QuantErr: 4.54740 batch_time=0.42634
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.64117 (QuantReg: 4.47359) QuantErr: 4.47359 batch_time=0.49075
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.06293 (QuantReg: 4.58496) QuantErr: 4.58496 batch_time=0.42431
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.93802 (QuantReg: 4.46929) QuantErr: 4.46929 batch_time=0.78519
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.56990 (QuantReg: 4.67464) QuantErr: 4.67464 batch_time=0.43844
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.89528 (QuantReg: 4.45857) QuantErr: 4.45857 batch_time=0.42950
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.12554 (QuantReg: 4.65447) QuantErr: 4.65447 batch_time=0.43482
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.07583 (QuantReg: 4.56919) QuantErr: 4.56919 batch_time=0.45004
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.82505 (QuantReg: 4.59490) QuantErr: 4.59490 batch_time=0.41673
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.55570 (QuantReg: 4.64487) QuantErr: 4.64487 batch_time=0.44955
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.54576 (QuantReg: 4.74885) QuantErr: 4.74885 batch_time=2.15329
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.93548 (QuantReg: 4.57703) QuantErr: 4.57703 batch_time=0.41559
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.13643 (QuantReg: 4.53394) QuantErr: 4.53394 batch_time=0.41598
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.99054 (QuantReg: 4.67118) QuantErr: 4.67118 batch_time=0.61838
Train Epoch: 10 codebook_update_time=0.61492
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch10.pth ...
Done in 7.694s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch10.pth ...
Done in 12.158s
removing stale ckpt [epoch 9] [took 0.03s]
epoch : 10
loss : 1.895401439189911
quant_reg : 4.5561936016082765
quant_err : 4.5561936016082765
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.25
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.3205
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.19587344623181
MSRVTT_jsfusion_test/v2t_metrics/R1: 13.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.2895
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.05300572390721
mnt_best : 34.19587344623181
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.75898 (QuantReg: 4.43900) QuantErr: 4.43900 batch_time=31.37517
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.12009 (QuantReg: 4.35903) QuantErr: 4.35903 batch_time=0.42889
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.15444 (QuantReg: 4.52069) QuantErr: 4.52069 batch_time=0.44517
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.05084 (QuantReg: 4.57647) QuantErr: 4.57647 batch_time=0.49036
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.04288 (QuantReg: 4.50063) QuantErr: 4.50063 batch_time=0.43515
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.30075 (QuantReg: 4.75735) QuantErr: 4.75735 batch_time=0.44873
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.50312 (QuantReg: 4.53983) QuantErr: 4.53983 batch_time=0.42327
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.05819 (QuantReg: 4.43211) QuantErr: 4.43211 batch_time=0.72747
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.75812 (QuantReg: 4.51319) QuantErr: 4.51319 batch_time=0.53484
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.71455 (QuantReg: 4.63668) QuantErr: 4.63668 batch_time=0.41887
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.50277 (QuantReg: 4.62632) QuantErr: 4.62632 batch_time=0.42230
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.61286 (QuantReg: 4.62639) QuantErr: 4.62639 batch_time=0.44103
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.92394 (QuantReg: 4.49351) QuantErr: 4.49351 batch_time=0.43238
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.68223 (QuantReg: 4.57273) QuantErr: 4.57273 batch_time=0.43233
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.46627 (QuantReg: 4.65454) QuantErr: 4.65454 batch_time=0.57492
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.24310 (QuantReg: 4.43441) QuantErr: 4.43441 batch_time=0.46986
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.68098 (QuantReg: 4.61408) QuantErr: 4.61408 batch_time=1.10827
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.72730 (QuantReg: 4.73023) QuantErr: 4.73023 batch_time=0.43930
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.85444 (QuantReg: 4.55242) QuantErr: 4.55242 batch_time=0.42032
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.73188 (QuantReg: 4.62719) QuantErr: 4.62719 batch_time=0.42504
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.75346 (QuantReg: 4.51076) QuantErr: 4.51076 batch_time=0.42774
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.85451 (QuantReg: 4.65339) QuantErr: 4.65339 batch_time=0.43653
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.71733 (QuantReg: 4.69881) QuantErr: 4.69881 batch_time=0.42856
Train Epoch: 11 codebook_update_time=0.52789
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch11.pth ...
Done in 5.064s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch11.pth ...
Done in 9.923s
removing stale ckpt [epoch 10] [took 0.01s]
epoch : 11
loss : 1.8166617856025695
quant_reg : 4.574329250335693
quant_err : 4.574329250335693
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.9115
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.90717306987326
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.75
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.623
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.25255078947561
mnt_best : 34.90717306987326
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.60801 (QuantReg: 4.49422) QuantErr: 4.49422 batch_time=31.92369
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.73556 (QuantReg: 4.53496) QuantErr: 4.53496 batch_time=1.75541
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.93173 (QuantReg: 4.60743) QuantErr: 4.60743 batch_time=0.42209
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.01673 (QuantReg: 4.58279) QuantErr: 4.58279 batch_time=0.41617
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.71804 (QuantReg: 4.53097) QuantErr: 4.53097 batch_time=0.42217
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.74843 (QuantReg: 4.62578) QuantErr: 4.62578 batch_time=0.48349
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.96487 (QuantReg: 4.71756) QuantErr: 4.71756 batch_time=1.36138
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.91537 (QuantReg: 4.40408) QuantErr: 4.40408 batch_time=0.44292
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.75710 (QuantReg: 4.54265) QuantErr: 4.54265 batch_time=0.42182
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.73611 (QuantReg: 4.57163) QuantErr: 4.57163 batch_time=0.46301
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.54889 (QuantReg: 4.62331) QuantErr: 4.62331 batch_time=0.42794
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.48054 (QuantReg: 4.69106) QuantErr: 4.69106 batch_time=0.42197
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.67773 (QuantReg: 4.53077) QuantErr: 4.53077 batch_time=0.43918
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.72781 (QuantReg: 4.47988) QuantErr: 4.47988 batch_time=3.10391
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.02960 (QuantReg: 4.76575) QuantErr: 4.76575 batch_time=0.42143
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.72728 (QuantReg: 4.67333) QuantErr: 4.67333 batch_time=0.42188
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.94390 (QuantReg: 4.73532) QuantErr: 4.73532 batch_time=0.46707
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.47016 (QuantReg: 4.68551) QuantErr: 4.68551 batch_time=0.42050
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.96752 (QuantReg: 4.59640) QuantErr: 4.59640 batch_time=0.43665
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.49646 (QuantReg: 4.68385) QuantErr: 4.68385 batch_time=0.43537
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.38243 (QuantReg: 4.74725) QuantErr: 4.74725 batch_time=0.42891
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.85698 (QuantReg: 4.75828) QuantErr: 4.75828 batch_time=0.43194
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.66047 (QuantReg: 4.65743) QuantErr: 4.65743 batch_time=0.43571
Train Epoch: 12 codebook_update_time=0.51338
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch12.pth ...
Done in 5.044s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch12.pth ...
Done in 9.725s
removing stale ckpt [epoch 11] [took 0.02s]
epoch : 12
loss : 1.7606372685432434
quant_reg : 4.602362184524536
quant_err : 4.602362184524536
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.542500000000004
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.07361702234213
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.25
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.367
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.282064135218356
mnt_best : 35.07361702234213
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.81710 (QuantReg: 4.53192) QuantErr: 4.53192 batch_time=31.09079
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.32100 (QuantReg: 4.82170) QuantErr: 4.82170 batch_time=0.46233
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.71821 (QuantReg: 4.56163) QuantErr: 4.56163 batch_time=2.52591
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.42871 (QuantReg: 4.68053) QuantErr: 4.68053 batch_time=0.42583
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.86067 (QuantReg: 4.66603) QuantErr: 4.66603 batch_time=0.55494
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.91195 (QuantReg: 4.45333) QuantErr: 4.45333 batch_time=0.42185
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.69474 (QuantReg: 4.64033) QuantErr: 4.64033 batch_time=0.42121
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.77353 (QuantReg: 4.74018) QuantErr: 4.74018 batch_time=0.42022
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.69678 (QuantReg: 4.70016) QuantErr: 4.70016 batch_time=0.43923
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.54608 (QuantReg: 4.62368) QuantErr: 4.62368 batch_time=0.43483
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.55699 (QuantReg: 4.73814) QuantErr: 4.73814 batch_time=0.42032
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.52835 (QuantReg: 4.81528) QuantErr: 4.81528 batch_time=0.43831
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.70521 (QuantReg: 4.62778) QuantErr: 4.62778 batch_time=0.42922
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.75567 (QuantReg: 4.73065) QuantErr: 4.73065 batch_time=1.17427
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.51546 (QuantReg: 4.71054) QuantErr: 4.71054 batch_time=0.44963
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.99301 (QuantReg: 4.68107) QuantErr: 4.68107 batch_time=0.42543
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.58374 (QuantReg: 4.79235) QuantErr: 4.79235 batch_time=0.42447
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.89345 (QuantReg: 4.61489) QuantErr: 4.61489 batch_time=0.43088
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.55116 (QuantReg: 4.74854) QuantErr: 4.74854 batch_time=0.80122
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.90011 (QuantReg: 4.63871) QuantErr: 4.63871 batch_time=0.76918
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.35233 (QuantReg: 4.72380) QuantErr: 4.72380 batch_time=0.44454
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.77625 (QuantReg: 4.72367) QuantErr: 4.72367 batch_time=0.43892
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.93594 (QuantReg: 4.66088) QuantErr: 4.66088 batch_time=0.43134
Train Epoch: 13 codebook_update_time=0.51265
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch13.pth ...
Done in 6.482s
removing stale ckpt [epoch 12] [took 0.05s]
epoch : 13
loss : 1.7121970357894898
quant_reg : 4.659769542694092
quant_err : 4.659769542694092
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.138
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.511149515466926
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.089
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.26949939704845
mnt_best : 35.07361702234213
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.59114 (QuantReg: 4.68842) QuantErr: 4.68842 batch_time=32.16497
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.55583 (QuantReg: 4.57969) QuantErr: 4.57969 batch_time=0.41641
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.61142 (QuantReg: 4.60650) QuantErr: 4.60650 batch_time=0.44719
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.53453 (QuantReg: 4.77231) QuantErr: 4.77231 batch_time=0.43348
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.57159 (QuantReg: 4.67708) QuantErr: 4.67708 batch_time=0.42700
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.94236 (QuantReg: 4.49731) QuantErr: 4.49731 batch_time=0.45804
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.84848 (QuantReg: 4.62496) QuantErr: 4.62496 batch_time=1.23434
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.51125 (QuantReg: 4.63985) QuantErr: 4.63985 batch_time=0.45071
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.53288 (QuantReg: 4.47867) QuantErr: 4.47867 batch_time=0.43204
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.38626 (QuantReg: 4.65849) QuantErr: 4.65849 batch_time=0.43490
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.15586 (QuantReg: 4.58507) QuantErr: 4.58507 batch_time=0.42072
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.42685 (QuantReg: 4.58280) QuantErr: 4.58280 batch_time=0.48324
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.47922 (QuantReg: 4.78154) QuantErr: 4.78154 batch_time=0.43645
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.83307 (QuantReg: 4.55822) QuantErr: 4.55822 batch_time=0.41747
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.57867 (QuantReg: 4.63870) QuantErr: 4.63870 batch_time=0.41771
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.87902 (QuantReg: 4.65532) QuantErr: 4.65532 batch_time=0.43586
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.59199 (QuantReg: 4.80633) QuantErr: 4.80633 batch_time=0.42865
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.79322 (QuantReg: 4.58659) QuantErr: 4.58659 batch_time=0.42936
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.61415 (QuantReg: 4.78603) QuantErr: 4.78603 batch_time=0.42635
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.61700 (QuantReg: 4.63821) QuantErr: 4.63821 batch_time=1.29655
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.76326 (QuantReg: 4.65613) QuantErr: 4.65613 batch_time=0.41343
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.40711 (QuantReg: 4.90088) QuantErr: 4.90088 batch_time=0.41692
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.26277 (QuantReg: 4.80662) QuantErr: 4.80662 batch_time=0.43225
Train Epoch: 14 codebook_update_time=0.51420
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch14.pth ...
Done in 5.307s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch14.pth ...
Done in 10.343s
removing stale ckpt [epoch 13] [took 0.08s]
epoch : 14
loss : 1.6488186140060426
quant_reg : 4.669720338821411
quant_err : 4.669720338821411
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.682
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.162180670582565
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.334
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.104511116393304
mnt_best : 35.162180670582565
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.47767 (QuantReg: 4.66772) QuantErr: 4.66772 batch_time=29.95639
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.19760 (QuantReg: 4.68047) QuantErr: 4.68047 batch_time=0.42120
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.65097 (QuantReg: 4.69513) QuantErr: 4.69513 batch_time=0.45824
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.42368 (QuantReg: 4.52878) QuantErr: 4.52878 batch_time=0.43383
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.87864 (QuantReg: 4.66011) QuantErr: 4.66011 batch_time=0.42419
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.71111 (QuantReg: 4.64821) QuantErr: 4.64821 batch_time=0.43193
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.76093 (QuantReg: 4.83640) QuantErr: 4.83640 batch_time=0.43416
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.35733 (QuantReg: 4.61753) QuantErr: 4.61753 batch_time=0.55576
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.88045 (QuantReg: 4.73602) QuantErr: 4.73602 batch_time=0.56323
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.55640 (QuantReg: 4.68362) QuantErr: 4.68362 batch_time=0.65654
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.88301 (QuantReg: 4.74980) QuantErr: 4.74980 batch_time=0.43270
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.30608 (QuantReg: 4.84889) QuantErr: 4.84889 batch_time=0.42757
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.58550 (QuantReg: 4.69159) QuantErr: 4.69159 batch_time=0.42237
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.48680 (QuantReg: 4.69079) QuantErr: 4.69079 batch_time=1.14617
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.68187 (QuantReg: 4.77973) QuantErr: 4.77973 batch_time=0.73914
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.49555 (QuantReg: 4.60841) QuantErr: 4.60841 batch_time=0.43394
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.46800 (QuantReg: 4.74877) QuantErr: 4.74877 batch_time=0.48375
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.60881 (QuantReg: 4.67930) QuantErr: 4.67930 batch_time=0.42329
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.90228 (QuantReg: 4.59608) QuantErr: 4.59608 batch_time=0.42235
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.39668 (QuantReg: 4.81719) QuantErr: 4.81719 batch_time=3.34487
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.40053 (QuantReg: 4.74831) QuantErr: 4.74831 batch_time=0.42153
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.61603 (QuantReg: 4.79325) QuantErr: 4.79325 batch_time=0.43325
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.49126 (QuantReg: 4.73450) QuantErr: 4.73450 batch_time=0.43824
Train Epoch: 15 codebook_update_time=0.51403
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch15.pth ...
Done in 14.011s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch15.pth ...
Done in 19.518s
removing stale ckpt [epoch 14] [took 0.03s]
epoch : 15
loss : 1.594384445667267
quant_reg : 4.694704198837281
quant_err : 4.694704198837281
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.25
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.814
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.49783098564247
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.78
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.64820594189756
mnt_best : 35.49783098564247
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.88853 (QuantReg: 4.61212) QuantErr: 4.61212 batch_time=30.86603
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.60553 (QuantReg: 4.54963) QuantErr: 4.54963 batch_time=0.45541
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.46615 (QuantReg: 4.64508) QuantErr: 4.64508 batch_time=0.41664
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.16871 (QuantReg: 4.52382) QuantErr: 4.52382 batch_time=0.44851
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.63075 (QuantReg: 4.75191) QuantErr: 4.75191 batch_time=0.47577
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.66334 (QuantReg: 4.88506) QuantErr: 4.88506 batch_time=0.42150
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.27354 (QuantReg: 4.76218) QuantErr: 4.76218 batch_time=0.42334
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.54250 (QuantReg: 4.65882) QuantErr: 4.65882 batch_time=0.42058
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.52757 (QuantReg: 4.69814) QuantErr: 4.69814 batch_time=0.88898
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.45969 (QuantReg: 4.79347) QuantErr: 4.79347 batch_time=0.43647
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.36298 (QuantReg: 4.80168) QuantErr: 4.80168 batch_time=0.47663
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.57852 (QuantReg: 4.73018) QuantErr: 4.73018 batch_time=0.41926
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.47447 (QuantReg: 4.74348) QuantErr: 4.74348 batch_time=0.41570
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.96116 (QuantReg: 4.77824) QuantErr: 4.77824 batch_time=1.64617
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.93854 (QuantReg: 4.81729) QuantErr: 4.81729 batch_time=0.41890
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.69003 (QuantReg: 4.77350) QuantErr: 4.77350 batch_time=0.42066
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.35182 (QuantReg: 4.87574) QuantErr: 4.87574 batch_time=0.42016
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.70407 (QuantReg: 4.72849) QuantErr: 4.72849 batch_time=0.43983
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.62204 (QuantReg: 4.72471) QuantErr: 4.72471 batch_time=0.43937
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.38025 (QuantReg: 4.86823) QuantErr: 4.86823 batch_time=0.41769
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.60946 (QuantReg: 4.74785) QuantErr: 4.74785 batch_time=0.41816
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.69043 (QuantReg: 4.68036) QuantErr: 4.68036 batch_time=0.77291
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.67126 (QuantReg: 4.73895) QuantErr: 4.73895 batch_time=0.86171
Train Epoch: 16 codebook_update_time=0.57032
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch16.pth ...
Done in 7.285s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch16.pth ...
Done in 12.817s
removing stale ckpt [epoch 15] [took 0.13s]
epoch : 16
loss : 1.5935309000015259
quant_reg : 4.72599268913269
quant_err : 4.72599268913269
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.460499999999996
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.181069105222406
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 30.06
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.155867384763724
mnt_best : 37.181069105222406
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.31569 (QuantReg: 4.83223) QuantErr: 4.83223 batch_time=31.83981
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.75473 (QuantReg: 4.58479) QuantErr: 4.58479 batch_time=0.47469
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.44026 (QuantReg: 4.60810) QuantErr: 4.60810 batch_time=0.81267
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.05133 (QuantReg: 4.77296) QuantErr: 4.77296 batch_time=0.43934
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.62931 (QuantReg: 4.55923) QuantErr: 4.55923 batch_time=0.43559
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.55239 (QuantReg: 4.79427) QuantErr: 4.79427 batch_time=0.44363
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.76321 (QuantReg: 4.77867) QuantErr: 4.77867 batch_time=0.65385
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.30293 (QuantReg: 4.72446) QuantErr: 4.72446 batch_time=0.41921
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.49011 (QuantReg: 4.64604) QuantErr: 4.64604 batch_time=0.41289
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.64357 (QuantReg: 4.76962) QuantErr: 4.76962 batch_time=0.42261
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.64265 (QuantReg: 4.76499) QuantErr: 4.76499 batch_time=0.47315
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.58722 (QuantReg: 4.70749) QuantErr: 4.70749 batch_time=0.42295
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.65934 (QuantReg: 4.72427) QuantErr: 4.72427 batch_time=4.30401
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.49239 (QuantReg: 4.78024) QuantErr: 4.78024 batch_time=0.41289
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.97665 (QuantReg: 4.59019) QuantErr: 4.59019 batch_time=0.42115
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.37882 (QuantReg: 4.66987) QuantErr: 4.66987 batch_time=0.43562
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.66548 (QuantReg: 4.85457) QuantErr: 4.85457 batch_time=0.44429
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.38515 (QuantReg: 4.88353) QuantErr: 4.88353 batch_time=0.41353
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.48868 (QuantReg: 4.59515) QuantErr: 4.59515 batch_time=0.45294
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.42635 (QuantReg: 4.80227) QuantErr: 4.80227 batch_time=0.41362
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.45237 (QuantReg: 4.64737) QuantErr: 4.64737 batch_time=0.44243
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.57959 (QuantReg: 4.93098) QuantErr: 4.93098 batch_time=0.46391
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.31169 (QuantReg: 4.69793) QuantErr: 4.69793 batch_time=0.43499
Train Epoch: 17 codebook_update_time=0.52232
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch17.pth ...
Done in 5.230s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch17.pth ...
Done in 11.229s
removing stale ckpt [epoch 16] [took 0.02s]
epoch : 17
loss : 1.5435816583633424
quant_reg : 4.752083864212036
quant_err : 4.752083864212036
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.9605
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.59624226819623
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.5865
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.75414919138945
mnt_best : 37.59624226819623
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.62681 (QuantReg: 4.67814) QuantErr: 4.67814 batch_time=36.30877
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.49538 (QuantReg: 4.67146) QuantErr: 4.67146 batch_time=0.42230
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.44905 (QuantReg: 4.74285) QuantErr: 4.74285 batch_time=0.42093
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.30647 (QuantReg: 4.81461) QuantErr: 4.81461 batch_time=0.43355
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.48657 (QuantReg: 4.69298) QuantErr: 4.69298 batch_time=0.50303
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.29903 (QuantReg: 4.85656) QuantErr: 4.85656 batch_time=0.42895
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.19529 (QuantReg: 4.88559) QuantErr: 4.88559 batch_time=0.42671
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.27120 (QuantReg: 4.86612) QuantErr: 4.86612 batch_time=0.42690
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.78084 (QuantReg: 4.87778) QuantErr: 4.87778 batch_time=0.42696
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.51152 (QuantReg: 4.68442) QuantErr: 4.68442 batch_time=0.42896
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.76534 (QuantReg: 4.73143) QuantErr: 4.73143 batch_time=0.42861
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.61509 (QuantReg: 4.83605) QuantErr: 4.83605 batch_time=0.44046
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.57277 (QuantReg: 4.73040) QuantErr: 4.73040 batch_time=0.42498
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.79549 (QuantReg: 4.78082) QuantErr: 4.78082 batch_time=0.44876
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.69504 (QuantReg: 4.63931) QuantErr: 4.63931 batch_time=0.43404
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.50046 (QuantReg: 4.92140) QuantErr: 4.92140 batch_time=0.44269
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.41932 (QuantReg: 4.69001) QuantErr: 4.69001 batch_time=0.43201
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.29650 (QuantReg: 4.77952) QuantErr: 4.77952 batch_time=0.42974
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.44914 (QuantReg: 4.84642) QuantErr: 4.84642 batch_time=0.42086
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.13488 (QuantReg: 4.87997) QuantErr: 4.87997 batch_time=0.47434
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.90843 (QuantReg: 4.61848) QuantErr: 4.61848 batch_time=0.43807
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.38385 (QuantReg: 4.85128) QuantErr: 4.85128 batch_time=0.43581
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.52452 (QuantReg: 4.77099) QuantErr: 4.77099 batch_time=0.41795
Train Epoch: 18 codebook_update_time=0.59412
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch18.pth ...
Done in 5.868s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 1.5007789897918702
quant_reg : 4.78233055305481
quant_err : 4.78233055305481
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.228
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.75428910095348
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.6545
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.09523070123369
mnt_best : 37.59624226819623
not_improved_count: 1
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.31236 (QuantReg: 4.80263) QuantErr: 4.80263 batch_time=33.45242
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.40781 (QuantReg: 4.77640) QuantErr: 4.77640 batch_time=0.44313
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.68039 (QuantReg: 4.67928) QuantErr: 4.67928 batch_time=0.90836
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.78781 (QuantReg: 4.72981) QuantErr: 4.72981 batch_time=0.45105
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.36713 (QuantReg: 4.76335) QuantErr: 4.76335 batch_time=0.45178
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.09675 (QuantReg: 4.86183) QuantErr: 4.86183 batch_time=0.43731
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.15226 (QuantReg: 4.85119) QuantErr: 4.85119 batch_time=0.41775
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.30971 (QuantReg: 4.75294) QuantErr: 4.75294 batch_time=2.60824
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.28231 (QuantReg: 4.74890) QuantErr: 4.74890 batch_time=0.42235
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.72007 (QuantReg: 4.60023) QuantErr: 4.60023 batch_time=0.43523
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.40599 (QuantReg: 4.69069) QuantErr: 4.69069 batch_time=0.46423
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.29678 (QuantReg: 4.89177) QuantErr: 4.89177 batch_time=0.43016
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.54068 (QuantReg: 4.94885) QuantErr: 4.94885 batch_time=0.59583
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.47857 (QuantReg: 4.85200) QuantErr: 4.85200 batch_time=2.15583
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.70898 (QuantReg: 4.70622) QuantErr: 4.70622 batch_time=0.44503
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.07717 (QuantReg: 4.80182) QuantErr: 4.80182 batch_time=0.41517
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.36990 (QuantReg: 4.84440) QuantErr: 4.84440 batch_time=0.44605
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.45462 (QuantReg: 4.73556) QuantErr: 4.73556 batch_time=0.47466
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.61482 (QuantReg: 4.67420) QuantErr: 4.67420 batch_time=0.54166
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.46850 (QuantReg: 4.81153) QuantErr: 4.81153 batch_time=0.45120
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.56166 (QuantReg: 4.96159) QuantErr: 4.96159 batch_time=0.42049
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.83857 (QuantReg: 4.78799) QuantErr: 4.78799 batch_time=1.80037
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.56444 (QuantReg: 4.97631) QuantErr: 4.97631 batch_time=0.43097
Train Epoch: 19 codebook_update_time=0.52822
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch19.pth ...
Done in 6.631s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_M8/checkpoint-epoch19.pth ...
Done in 12.168s
removing stale ckpt [epoch 18] [took 0.19s]
epoch : 19
loss : 1.4831805200576782
quant_reg : 4.80110830116272
quant_err : 4.80110830116272
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.8