-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_t0.07.txt
2593 lines (2593 loc) · 194 KB
/
HCQ_MSRVTT_1kA_t0.07.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 939.168178319931 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 83.42376852035522 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 64.12942886352539 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch0.pth ...
Done in 8.534s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch0.pth ...
Done in 10.402s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 486.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 496.278
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 6.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 509.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.537
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.5192494101851104
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.75770 (QuantReg: 22.49995) QuantErr: 22.49995 batch_time=25.04878
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.63326 (QuantReg: 22.52905) QuantErr: 22.52905 batch_time=1.38398
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.45945 (QuantReg: 22.59101) QuantErr: 22.59101 batch_time=0.52107
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.35111 (QuantReg: 22.63788) QuantErr: 22.63788 batch_time=0.50170
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.50255 (QuantReg: 22.66911) QuantErr: 22.66911 batch_time=0.50006
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 5.76515 (QuantReg: 22.63972) QuantErr: 22.63972 batch_time=0.79503
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.89738 (QuantReg: 22.58620) QuantErr: 22.58620 batch_time=0.49385
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.14738 (QuantReg: 22.62769) QuantErr: 22.62769 batch_time=0.49388
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.58241 (QuantReg: 22.62558) QuantErr: 22.62558 batch_time=0.49662
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.48778 (QuantReg: 22.65053) QuantErr: 22.65053 batch_time=0.51717
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 4.79116 (QuantReg: 22.62049) QuantErr: 22.62049 batch_time=0.52141
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.87858 (QuantReg: 22.63858) QuantErr: 22.63858 batch_time=0.48929
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.24421 (QuantReg: 22.61843) QuantErr: 22.61843 batch_time=0.49802
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.51504 (QuantReg: 22.63256) QuantErr: 22.63256 batch_time=0.51403
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.84998 (QuantReg: 22.61064) QuantErr: 22.61064 batch_time=0.50833
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.26080 (QuantReg: 22.62599) QuantErr: 22.62599 batch_time=0.48806
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.98276 (QuantReg: 22.64291) QuantErr: 22.64291 batch_time=0.50051
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.58471 (QuantReg: 22.63447) QuantErr: 22.63447 batch_time=0.48326
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.18234 (QuantReg: 22.62408) QuantErr: 22.62408 batch_time=0.49758
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.54063 (QuantReg: 22.62067) QuantErr: 22.62067 batch_time=0.54103
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 3.75853 (QuantReg: 22.67110) QuantErr: 22.67110 batch_time=0.50574
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.33773 (QuantReg: 22.65599) QuantErr: 22.65599 batch_time=0.49277
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.78661 (QuantReg: 22.64566) QuantErr: 22.64566 batch_time=0.48772
Train Epoch: 1 codebook_update_time=2.11587
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch1.pth ...
Done in 3.980s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch1.pth ...
Done in 7.787s
epoch : 1
loss : 5.457815495491028
quant_reg : 22.622351364135742
quant_err : 22.622351364135742
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 11.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 31.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 45.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 77.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 14.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 43.171
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.776306794171642
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 32.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 45.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 79.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 13.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 41.329
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.871390520807314
mnt_best : 25.776306794171642
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.29878 (QuantReg: 11.91046) QuantErr: 11.91046 batch_time=25.64396
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.14213 (QuantReg: 12.02885) QuantErr: 12.02885 batch_time=0.69414
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.10042 (QuantReg: 12.44516) QuantErr: 12.44516 batch_time=0.47757
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.99988 (QuantReg: 12.31501) QuantErr: 12.31501 batch_time=0.49222
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.96264 (QuantReg: 12.92193) QuantErr: 12.92193 batch_time=0.49250
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.09270 (QuantReg: 12.81605) QuantErr: 12.81605 batch_time=0.49656
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.83910 (QuantReg: 12.67014) QuantErr: 12.67014 batch_time=0.48843
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.93331 (QuantReg: 12.91660) QuantErr: 12.91660 batch_time=0.49958
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.74319 (QuantReg: 13.22709) QuantErr: 13.22709 batch_time=0.50732
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.34224 (QuantReg: 13.18706) QuantErr: 13.18706 batch_time=0.49175
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.84694 (QuantReg: 13.04625) QuantErr: 13.04625 batch_time=0.48892
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.45452 (QuantReg: 13.50145) QuantErr: 13.50145 batch_time=0.49064
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.75394 (QuantReg: 13.16448) QuantErr: 13.16448 batch_time=1.19959
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.21062 (QuantReg: 13.35965) QuantErr: 13.35965 batch_time=0.50489
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 4.13746 (QuantReg: 13.52014) QuantErr: 13.52014 batch_time=0.48944
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.65509 (QuantReg: 13.89092) QuantErr: 13.89092 batch_time=0.51127
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.99241 (QuantReg: 13.81053) QuantErr: 13.81053 batch_time=0.50023
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.82146 (QuantReg: 14.01621) QuantErr: 14.01621 batch_time=1.08761
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.52297 (QuantReg: 14.06157) QuantErr: 14.06157 batch_time=0.50067
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.43768 (QuantReg: 13.98330) QuantErr: 13.98330 batch_time=3.49718
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.58319 (QuantReg: 14.11912) QuantErr: 14.11912 batch_time=0.49945
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.76404 (QuantReg: 14.00258) QuantErr: 14.00258 batch_time=0.52813
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.90958 (QuantReg: 14.26298) QuantErr: 14.26298 batch_time=1.18623
Train Epoch: 2 codebook_update_time=1.62094
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch2.pth ...
Done in 18.755s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch2.pth ...
Done in 22.469s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.8444110975265504
quant_reg : 13.284837577819824
quant_err : 13.284837577819824
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 13.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 38.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 53.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 83.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 35.615
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.250699161947434
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 38.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 54.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 83.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 33.7325
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.75114129858992
mnt_best : 30.250699161947434
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.49838 (QuantReg: 11.77237) QuantErr: 11.77237 batch_time=31.63481
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.40566 (QuantReg: 11.84024) QuantErr: 11.84024 batch_time=0.48899
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.37447 (QuantReg: 11.93193) QuantErr: 11.93193 batch_time=0.49575
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.08928 (QuantReg: 11.78542) QuantErr: 11.78542 batch_time=0.49330
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.09311 (QuantReg: 11.99265) QuantErr: 11.99265 batch_time=0.49792
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.56256 (QuantReg: 11.66378) QuantErr: 11.66378 batch_time=0.48786
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.85358 (QuantReg: 12.29888) QuantErr: 12.29888 batch_time=3.68100
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.58512 (QuantReg: 12.09770) QuantErr: 12.09770 batch_time=0.51724
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.59439 (QuantReg: 12.01664) QuantErr: 12.01664 batch_time=0.48730
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.87183 (QuantReg: 12.26513) QuantErr: 12.26513 batch_time=0.49273
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.51806 (QuantReg: 12.03567) QuantErr: 12.03567 batch_time=0.48363
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.32854 (QuantReg: 11.91719) QuantErr: 11.91719 batch_time=0.48666
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.02997 (QuantReg: 11.83606) QuantErr: 11.83606 batch_time=0.49868
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.40278 (QuantReg: 11.85056) QuantErr: 11.85056 batch_time=0.49881
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.74530 (QuantReg: 11.61157) QuantErr: 11.61157 batch_time=0.49723
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.99375 (QuantReg: 11.93274) QuantErr: 11.93274 batch_time=0.49210
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.52014 (QuantReg: 12.49755) QuantErr: 12.49755 batch_time=0.49121
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.12796 (QuantReg: 12.34988) QuantErr: 12.34988 batch_time=0.51373
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.05535 (QuantReg: 12.33231) QuantErr: 12.33231 batch_time=0.84313
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.16954 (QuantReg: 12.22341) QuantErr: 12.22341 batch_time=1.06110
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.21924 (QuantReg: 12.22651) QuantErr: 12.22651 batch_time=0.49513
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.24256 (QuantReg: 12.64134) QuantErr: 12.64134 batch_time=0.49294
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.22784 (QuantReg: 12.62834) QuantErr: 12.62834 batch_time=0.54557
Train Epoch: 3 codebook_update_time=1.84865
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch3.pth ...
Done in 3.754s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch3.pth ...
Done in 7.616s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.29838449382782
quant_reg : 12.085818374633789
quant_err : 12.085818374633789
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.227
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.03843455736734
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.278
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.862680994800606
mnt_best : 33.03843455736734
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.20176 (QuantReg: 11.61950) QuantErr: 11.61950 batch_time=34.06675
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.96222 (QuantReg: 11.47842) QuantErr: 11.47842 batch_time=2.61055
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 3.04241 (QuantReg: 11.60315) QuantErr: 11.60315 batch_time=0.49106
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.92793 (QuantReg: 11.74773) QuantErr: 11.74773 batch_time=0.50343
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.92060 (QuantReg: 11.75323) QuantErr: 11.75323 batch_time=0.59612
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.05256 (QuantReg: 11.58287) QuantErr: 11.58287 batch_time=0.50992
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.78976 (QuantReg: 11.54184) QuantErr: 11.54184 batch_time=1.29641
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.96202 (QuantReg: 11.78972) QuantErr: 11.78972 batch_time=0.53429
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.21651 (QuantReg: 11.49547) QuantErr: 11.49547 batch_time=0.50951
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.60078 (QuantReg: 11.71122) QuantErr: 11.71122 batch_time=0.48930
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.95602 (QuantReg: 12.23099) QuantErr: 12.23099 batch_time=0.52143
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.68420 (QuantReg: 12.29442) QuantErr: 12.29442 batch_time=0.56704
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.17559 (QuantReg: 11.85896) QuantErr: 11.85896 batch_time=0.54760
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.75863 (QuantReg: 11.99495) QuantErr: 11.99495 batch_time=1.05845
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.86032 (QuantReg: 11.92704) QuantErr: 11.92704 batch_time=0.50818
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.97042 (QuantReg: 11.84617) QuantErr: 11.84617 batch_time=0.90173
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.94605 (QuantReg: 11.81625) QuantErr: 11.81625 batch_time=0.49195
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.97900 (QuantReg: 11.96197) QuantErr: 11.96197 batch_time=0.49500
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.81569 (QuantReg: 12.36781) QuantErr: 12.36781 batch_time=0.48769
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.67165 (QuantReg: 11.88810) QuantErr: 11.88810 batch_time=0.49471
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.44359 (QuantReg: 12.37128) QuantErr: 12.37128 batch_time=0.53351
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.36971 (QuantReg: 12.41091) QuantErr: 12.41091 batch_time=0.49899
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 3.01883 (QuantReg: 11.97498) QuantErr: 11.97498 batch_time=0.48488
Train Epoch: 4 codebook_update_time=1.67599
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch4.pth ...
Done in 3.710s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch4.pth ...
Done in 7.330s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 3.0121521730422973
quant_reg : 11.805778053283692
quant_err : 11.805778053283692
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.212
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.02898605593065
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 44.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.058
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.97650215558529
mnt_best : 35.02898605593065
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.85170 (QuantReg: 11.26032) QuantErr: 11.26032 batch_time=32.53799
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.22008 (QuantReg: 11.70573) QuantErr: 11.70573 batch_time=0.49692
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.47684 (QuantReg: 11.23778) QuantErr: 11.23778 batch_time=0.50953
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.70010 (QuantReg: 11.75146) QuantErr: 11.75146 batch_time=0.50682
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.89483 (QuantReg: 11.44058) QuantErr: 11.44058 batch_time=0.49945
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.59401 (QuantReg: 11.78171) QuantErr: 11.78171 batch_time=0.49814
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.79932 (QuantReg: 11.41721) QuantErr: 11.41721 batch_time=0.52249
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.24907 (QuantReg: 11.24996) QuantErr: 11.24996 batch_time=0.49926
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.81203 (QuantReg: 11.77327) QuantErr: 11.77327 batch_time=0.49349
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.75213 (QuantReg: 11.58409) QuantErr: 11.58409 batch_time=0.49240
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.75248 (QuantReg: 11.67718) QuantErr: 11.67718 batch_time=0.49365
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.77529 (QuantReg: 11.49786) QuantErr: 11.49786 batch_time=0.49332
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.61554 (QuantReg: 12.00043) QuantErr: 12.00043 batch_time=0.70558
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.31038 (QuantReg: 11.48089) QuantErr: 11.48089 batch_time=3.83716
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 3.14970 (QuantReg: 11.79284) QuantErr: 11.79284 batch_time=0.49726
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.44291 (QuantReg: 11.93482) QuantErr: 11.93482 batch_time=1.13817
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.88130 (QuantReg: 11.66491) QuantErr: 11.66491 batch_time=0.53219
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.77518 (QuantReg: 11.65548) QuantErr: 11.65548 batch_time=0.50329
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.75810 (QuantReg: 11.97480) QuantErr: 11.97480 batch_time=0.62531
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.69871 (QuantReg: 11.56259) QuantErr: 11.56259 batch_time=0.48454
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.98074 (QuantReg: 11.91811) QuantErr: 11.91811 batch_time=0.49422
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.61246 (QuantReg: 11.72700) QuantErr: 11.72700 batch_time=0.48911
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.99053 (QuantReg: 11.95722) QuantErr: 11.95722 batch_time=0.52087
Train Epoch: 5 codebook_update_time=1.97262
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch5.pth ...
Done in 3.812s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch5.pth ...
Done in 7.514s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 2.7489051914215086
quant_reg : 11.695408233642578
quant_err : 11.695408233642578
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.634
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.330189006724666
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.003
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.57878311073783
mnt_best : 37.330189006724666
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.82594 (QuantReg: 11.50925) QuantErr: 11.50925 batch_time=29.40365
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.85786 (QuantReg: 11.07087) QuantErr: 11.07087 batch_time=0.50436
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.68063 (QuantReg: 11.41031) QuantErr: 11.41031 batch_time=0.49095
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.59787 (QuantReg: 11.42636) QuantErr: 11.42636 batch_time=0.49689
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.55309 (QuantReg: 11.39882) QuantErr: 11.39882 batch_time=1.16529
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.39271 (QuantReg: 11.41479) QuantErr: 11.41479 batch_time=0.49919
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.77088 (QuantReg: 11.77224) QuantErr: 11.77224 batch_time=0.99384
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.48460 (QuantReg: 11.82412) QuantErr: 11.82412 batch_time=0.50011
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.54821 (QuantReg: 11.60222) QuantErr: 11.60222 batch_time=0.59454
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.45743 (QuantReg: 11.49053) QuantErr: 11.49053 batch_time=0.66231
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.50052 (QuantReg: 12.07500) QuantErr: 12.07500 batch_time=0.50456
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.56832 (QuantReg: 11.58521) QuantErr: 11.58521 batch_time=0.50393
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.39333 (QuantReg: 11.66343) QuantErr: 11.66343 batch_time=0.62146
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.47969 (QuantReg: 11.58190) QuantErr: 11.58190 batch_time=0.97425
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.92701 (QuantReg: 11.93435) QuantErr: 11.93435 batch_time=0.49813
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.62671 (QuantReg: 11.74826) QuantErr: 11.74826 batch_time=0.50480
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.28159 (QuantReg: 11.86662) QuantErr: 11.86662 batch_time=0.49655
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.72787 (QuantReg: 11.72702) QuantErr: 11.72702 batch_time=0.49577
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.82499 (QuantReg: 11.78284) QuantErr: 11.78284 batch_time=0.49011
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.52356 (QuantReg: 11.75659) QuantErr: 11.75659 batch_time=0.51117
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.34914 (QuantReg: 11.77438) QuantErr: 11.77438 batch_time=0.50125
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.29148 (QuantReg: 11.55948) QuantErr: 11.55948 batch_time=0.49525
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.20663 (QuantReg: 11.74120) QuantErr: 11.74120 batch_time=0.49167
Train Epoch: 6 codebook_update_time=1.75234
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch6.pth ...
Done in 3.484s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.5711864070892334
quant_reg : 11.699482761383056
quant_err : 11.699482761383056
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.884
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.95111635500847
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.6855
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.36594266791382
mnt_best : 37.330189006724666
not_improved_count: 1
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.65123 (QuantReg: 11.08297) QuantErr: 11.08297 batch_time=29.30091
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.55499 (QuantReg: 11.43448) QuantErr: 11.43448 batch_time=0.50064
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.43593 (QuantReg: 11.09379) QuantErr: 11.09379 batch_time=0.50217
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.18169 (QuantReg: 11.36485) QuantErr: 11.36485 batch_time=0.50914
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.13786 (QuantReg: 11.39536) QuantErr: 11.39536 batch_time=0.50281
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.14560 (QuantReg: 11.73713) QuantErr: 11.73713 batch_time=0.53980
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.41448 (QuantReg: 11.78739) QuantErr: 11.78739 batch_time=3.12620
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.44966 (QuantReg: 11.62578) QuantErr: 11.62578 batch_time=0.49458
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.45002 (QuantReg: 11.81228) QuantErr: 11.81228 batch_time=0.50050
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.07741 (QuantReg: 11.73675) QuantErr: 11.73675 batch_time=0.51637
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.85914 (QuantReg: 11.80843) QuantErr: 11.80843 batch_time=0.51342
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.47368 (QuantReg: 11.67689) QuantErr: 11.67689 batch_time=0.50172
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.32180 (QuantReg: 11.65050) QuantErr: 11.65050 batch_time=0.49553
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.41734 (QuantReg: 11.37372) QuantErr: 11.37372 batch_time=0.49983
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.96370 (QuantReg: 11.50105) QuantErr: 11.50105 batch_time=0.50474
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.46306 (QuantReg: 11.62813) QuantErr: 11.62813 batch_time=0.51319
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.98109 (QuantReg: 11.86618) QuantErr: 11.86618 batch_time=0.49724
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.39546 (QuantReg: 11.53845) QuantErr: 11.53845 batch_time=0.71463
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.87813 (QuantReg: 11.79422) QuantErr: 11.79422 batch_time=0.50050
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.63079 (QuantReg: 11.91217) QuantErr: 11.91217 batch_time=0.48696
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.34527 (QuantReg: 11.87488) QuantErr: 11.87488 batch_time=0.50974
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.28279 (QuantReg: 12.28812) QuantErr: 12.28812 batch_time=0.50170
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.40432 (QuantReg: 11.81554) QuantErr: 11.81554 batch_time=0.49496
Train Epoch: 7 codebook_update_time=1.61524
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch7.pth ...
Done in 4.761s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch7.pth ...
Done in 8.409s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 2.4022769312858583
quant_reg : 11.646035930633545
quant_err : 11.646035930633545
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.052
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.16715456361193
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.07
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.11561219026715
mnt_best : 39.16715456361193
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.61802 (QuantReg: 11.41349) QuantErr: 11.41349 batch_time=26.55590
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.60292 (QuantReg: 11.40925) QuantErr: 11.40925 batch_time=0.49693
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.85554 (QuantReg: 11.64435) QuantErr: 11.64435 batch_time=0.49051
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.40559 (QuantReg: 11.32045) QuantErr: 11.32045 batch_time=0.50361
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.47580 (QuantReg: 11.80800) QuantErr: 11.80800 batch_time=0.48281
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.14088 (QuantReg: 11.32491) QuantErr: 11.32491 batch_time=0.49298
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.36516 (QuantReg: 11.80098) QuantErr: 11.80098 batch_time=0.50164
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.54526 (QuantReg: 11.82983) QuantErr: 11.82983 batch_time=0.49383
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.51269 (QuantReg: 11.57488) QuantErr: 11.57488 batch_time=0.66536
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.70406 (QuantReg: 11.73942) QuantErr: 11.73942 batch_time=0.48773
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.58066 (QuantReg: 11.86380) QuantErr: 11.86380 batch_time=0.49920
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.28713 (QuantReg: 11.57133) QuantErr: 11.57133 batch_time=0.49906
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.34316 (QuantReg: 11.56165) QuantErr: 11.56165 batch_time=0.49248
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.51853 (QuantReg: 11.97151) QuantErr: 11.97151 batch_time=0.49927
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.25684 (QuantReg: 11.94013) QuantErr: 11.94013 batch_time=0.48135
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.13222 (QuantReg: 11.51758) QuantErr: 11.51758 batch_time=0.49017
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.16479 (QuantReg: 11.56124) QuantErr: 11.56124 batch_time=0.50503
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.79494 (QuantReg: 11.71108) QuantErr: 11.71108 batch_time=0.52254
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.50270 (QuantReg: 11.77541) QuantErr: 11.77541 batch_time=0.51299
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.46199 (QuantReg: 11.39030) QuantErr: 11.39030 batch_time=0.48991
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.39003 (QuantReg: 11.71115) QuantErr: 11.71115 batch_time=0.48449
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.89245 (QuantReg: 11.78155) QuantErr: 11.78155 batch_time=0.49326
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.56009 (QuantReg: 11.83689) QuantErr: 11.83689 batch_time=0.49369
Train Epoch: 8 codebook_update_time=1.62622
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch8.pth ...
Done in 3.570s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 2.3137659759521485
quant_reg : 11.654494705200195
quant_err : 11.654494705200195
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.036
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.874071137809516
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.7895
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.5309969479668
mnt_best : 39.16715456361193
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.94050 (QuantReg: 11.87071) QuantErr: 11.87071 batch_time=27.21038
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.42344 (QuantReg: 11.72441) QuantErr: 11.72441 batch_time=0.50455
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.91283 (QuantReg: 11.33228) QuantErr: 11.33228 batch_time=0.49323
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.33893 (QuantReg: 11.55259) QuantErr: 11.55259 batch_time=0.50696
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.15280 (QuantReg: 11.78662) QuantErr: 11.78662 batch_time=0.49927
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.90756 (QuantReg: 11.83543) QuantErr: 11.83543 batch_time=0.52305
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.99873 (QuantReg: 11.81831) QuantErr: 11.81831 batch_time=0.57427
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 2.15277 (QuantReg: 11.94272) QuantErr: 11.94272 batch_time=0.50166
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.37408 (QuantReg: 11.44879) QuantErr: 11.44879 batch_time=0.49864
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.02424 (QuantReg: 11.70028) QuantErr: 11.70028 batch_time=0.85116
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.19069 (QuantReg: 11.80773) QuantErr: 11.80773 batch_time=0.56036
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.80488 (QuantReg: 11.64323) QuantErr: 11.64323 batch_time=0.49341
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.21231 (QuantReg: 11.62040) QuantErr: 11.62040 batch_time=0.61146
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.05354 (QuantReg: 11.44669) QuantErr: 11.44669 batch_time=0.49080
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.25757 (QuantReg: 11.26276) QuantErr: 11.26276 batch_time=0.51501
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.90232 (QuantReg: 12.02817) QuantErr: 12.02817 batch_time=0.49471
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.45933 (QuantReg: 11.96879) QuantErr: 11.96879 batch_time=0.50189
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.71620 (QuantReg: 11.83312) QuantErr: 11.83312 batch_time=0.53044
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.25036 (QuantReg: 11.82197) QuantErr: 11.82197 batch_time=0.50049
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.10058 (QuantReg: 11.88523) QuantErr: 11.88523 batch_time=1.72874
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.98090 (QuantReg: 12.07209) QuantErr: 12.07209 batch_time=0.49911
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.00942 (QuantReg: 11.40877) QuantErr: 11.40877 batch_time=0.49290
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 2.03539 (QuantReg: 11.53467) QuantErr: 11.53467 batch_time=0.49167
Train Epoch: 9 codebook_update_time=1.70794
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch9.pth ...
Done in 3.998s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 2.180047679424286
quant_reg : 11.657114269256592
quant_err : 11.657114269256592
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.928
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.04372304338925
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.654
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.00579082828389
mnt_best : 39.16715456361193
not_improved_count: 2
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.94587 (QuantReg: 11.76104) QuantErr: 11.76104 batch_time=31.66100
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.19523 (QuantReg: 11.59434) QuantErr: 11.59434 batch_time=0.97279
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.49358 (QuantReg: 11.77807) QuantErr: 11.77807 batch_time=0.50480
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.11477 (QuantReg: 11.35999) QuantErr: 11.35999 batch_time=0.49695
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.83542 (QuantReg: 11.51327) QuantErr: 11.51327 batch_time=0.49211
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.08383 (QuantReg: 11.70291) QuantErr: 11.70291 batch_time=0.49830
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.50388 (QuantReg: 11.46190) QuantErr: 11.46190 batch_time=4.17313
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 2.13147 (QuantReg: 11.78397) QuantErr: 11.78397 batch_time=0.49048
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.20843 (QuantReg: 11.59874) QuantErr: 11.59874 batch_time=0.50385
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.07857 (QuantReg: 11.63511) QuantErr: 11.63511 batch_time=0.49841
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.92008 (QuantReg: 11.55277) QuantErr: 11.55277 batch_time=0.50089
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.23384 (QuantReg: 11.75201) QuantErr: 11.75201 batch_time=0.50553
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.18511 (QuantReg: 11.40049) QuantErr: 11.40049 batch_time=0.50515
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 1.83090 (QuantReg: 12.03710) QuantErr: 12.03710 batch_time=0.50541
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 2.13084 (QuantReg: 11.42074) QuantErr: 11.42074 batch_time=0.48770
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.30539 (QuantReg: 11.92447) QuantErr: 11.92447 batch_time=0.49440
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.25984 (QuantReg: 11.64336) QuantErr: 11.64336 batch_time=0.50753
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.02128 (QuantReg: 11.81517) QuantErr: 11.81517 batch_time=0.49503
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.83579 (QuantReg: 11.86671) QuantErr: 11.86671 batch_time=0.50268
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.69741 (QuantReg: 12.06993) QuantErr: 12.06993 batch_time=0.54627
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.16377 (QuantReg: 11.69491) QuantErr: 11.69491 batch_time=0.50562
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.43905 (QuantReg: 11.64402) QuantErr: 11.64402 batch_time=0.49280
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.21333 (QuantReg: 11.92410) QuantErr: 11.92410 batch_time=0.49649
Train Epoch: 10 codebook_update_time=1.65784
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch10.pth ...
Done in 4.432s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch10.pth ...
Done in 8.439s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 2.0824551396369935
quant_reg : 11.69760541152954
quant_err : 11.69760541152954
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.655
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.878212907258906
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 28.101
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.24889473965625
mnt_best : 39.878212907258906
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.00793 (QuantReg: 11.45037) QuantErr: 11.45037 batch_time=31.89953
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.32123 (QuantReg: 11.29046) QuantErr: 11.29046 batch_time=0.49966
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.19971 (QuantReg: 11.58358) QuantErr: 11.58358 batch_time=0.49548
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.16745 (QuantReg: 11.69909) QuantErr: 11.69909 batch_time=0.50223
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.15132 (QuantReg: 11.58395) QuantErr: 11.58395 batch_time=0.53914
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.50542 (QuantReg: 12.18027) QuantErr: 12.18027 batch_time=0.48437
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.70875 (QuantReg: 11.64710) QuantErr: 11.64710 batch_time=2.23607
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.17798 (QuantReg: 11.34832) QuantErr: 11.34832 batch_time=0.55917
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.04088 (QuantReg: 11.56152) QuantErr: 11.56152 batch_time=0.49414
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.96639 (QuantReg: 11.78713) QuantErr: 11.78713 batch_time=2.48706
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.74843 (QuantReg: 11.82249) QuantErr: 11.82249 batch_time=0.75205
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.78629 (QuantReg: 11.78811) QuantErr: 11.78811 batch_time=0.55121
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.06794 (QuantReg: 11.50090) QuantErr: 11.50090 batch_time=0.52474
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.87887 (QuantReg: 11.59877) QuantErr: 11.59877 batch_time=0.49177
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.71052 (QuantReg: 11.76459) QuantErr: 11.76459 batch_time=0.49392
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.38305 (QuantReg: 11.25255) QuantErr: 11.25255 batch_time=0.53548
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.88121 (QuantReg: 11.59166) QuantErr: 11.59166 batch_time=0.48883
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.96387 (QuantReg: 11.92093) QuantErr: 11.92093 batch_time=0.84885
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 2.00256 (QuantReg: 11.57096) QuantErr: 11.57096 batch_time=0.49871
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.81869 (QuantReg: 11.77925) QuantErr: 11.77925 batch_time=0.49572
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.93363 (QuantReg: 11.64817) QuantErr: 11.64817 batch_time=0.49634
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.04789 (QuantReg: 11.84145) QuantErr: 11.84145 batch_time=0.61182
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.92673 (QuantReg: 11.94153) QuantErr: 11.94153 batch_time=0.51744
Train Epoch: 11 codebook_update_time=1.59967
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch11.pth ...
Done in 4.765s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch11.pth ...
Done in 9.676s
removing stale ckpt [epoch 10] [took 0.10s]
epoch : 11
loss : 1.9959793186187744
quant_reg : 11.677843952178955
quant_err : 11.677843952178955
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.901
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.21118793155395
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.1065
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.28218151988247
mnt_best : 41.21118793155395
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.76152 (QuantReg: 11.38160) QuantErr: 11.38160 batch_time=26.74566
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.92255 (QuantReg: 11.58846) QuantErr: 11.58846 batch_time=0.49877
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 2.03885 (QuantReg: 11.66586) QuantErr: 11.66586 batch_time=0.49830
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.17615 (QuantReg: 11.64092) QuantErr: 11.64092 batch_time=0.48878
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.85206 (QuantReg: 11.57335) QuantErr: 11.57335 batch_time=0.48954
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.92388 (QuantReg: 11.67140) QuantErr: 11.67140 batch_time=0.49603
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 2.23595 (QuantReg: 11.99403) QuantErr: 11.99403 batch_time=0.51643
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 2.10303 (QuantReg: 11.28702) QuantErr: 11.28702 batch_time=1.53949
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.88974 (QuantReg: 11.67731) QuantErr: 11.67731 batch_time=0.49386
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.80708 (QuantReg: 11.74828) QuantErr: 11.74828 batch_time=0.49679
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.78699 (QuantReg: 11.73780) QuantErr: 11.73780 batch_time=0.49872
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.68654 (QuantReg: 11.84158) QuantErr: 11.84158 batch_time=0.49271
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.82397 (QuantReg: 11.49906) QuantErr: 11.49906 batch_time=0.50684
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.85774 (QuantReg: 11.41680) QuantErr: 11.41680 batch_time=0.49773
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.09738 (QuantReg: 12.06208) QuantErr: 12.06208 batch_time=0.50518
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.98188 (QuantReg: 11.79749) QuantErr: 11.79749 batch_time=0.49770
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.06638 (QuantReg: 11.91449) QuantErr: 11.91449 batch_time=0.50820
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.69662 (QuantReg: 11.82273) QuantErr: 11.82273 batch_time=0.49514
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.09918 (QuantReg: 11.63617) QuantErr: 11.63617 batch_time=3.05242
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.76744 (QuantReg: 11.83157) QuantErr: 11.83157 batch_time=0.50877
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.63352 (QuantReg: 11.96317) QuantErr: 11.96317 batch_time=0.49309
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.01678 (QuantReg: 11.93656) QuantErr: 11.93656 batch_time=0.58063
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.83019 (QuantReg: 11.73783) QuantErr: 11.73783 batch_time=0.50955
Train Epoch: 12 codebook_update_time=1.60144
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch12.pth ...
Done in 4.340s
removing stale ckpt [epoch 11] [took 0.02s]
epoch : 12
loss : 1.937915223121643
quant_reg : 11.682352642059326
quant_err : 11.682352642059326
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.12
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.34861945160592
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.394
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.308170052229144
mnt_best : 41.21118793155395
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.04340 (QuantReg: 11.42254) QuantErr: 11.42254 batch_time=39.28424
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.49075 (QuantReg: 12.00422) QuantErr: 12.00422 batch_time=0.50050
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.96609 (QuantReg: 11.44065) QuantErr: 11.44065 batch_time=0.48981
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.59893 (QuantReg: 11.86154) QuantErr: 11.86154 batch_time=0.48091
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.99761 (QuantReg: 11.99395) QuantErr: 11.99395 batch_time=0.58774
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.09933 (QuantReg: 11.35783) QuantErr: 11.35783 batch_time=0.48898
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.79815 (QuantReg: 11.85272) QuantErr: 11.85272 batch_time=0.49666
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.94640 (QuantReg: 11.75588) QuantErr: 11.75588 batch_time=0.51195
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.78186 (QuantReg: 11.81899) QuantErr: 11.81899 batch_time=0.50069
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.68966 (QuantReg: 11.70083) QuantErr: 11.70083 batch_time=0.49159
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.75626 (QuantReg: 11.85921) QuantErr: 11.85921 batch_time=0.49172
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.78625 (QuantReg: 12.06444) QuantErr: 12.06444 batch_time=0.50110
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.86503 (QuantReg: 11.64062) QuantErr: 11.64062 batch_time=0.51464
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.86273 (QuantReg: 11.79840) QuantErr: 11.79840 batch_time=0.49096
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.67372 (QuantReg: 11.80117) QuantErr: 11.80117 batch_time=0.49715
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 2.16297 (QuantReg: 11.79405) QuantErr: 11.79405 batch_time=0.51071
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.86099 (QuantReg: 11.91264) QuantErr: 11.91264 batch_time=0.49288
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 2.11972 (QuantReg: 11.63208) QuantErr: 11.63208 batch_time=0.48736
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.69646 (QuantReg: 11.83399) QuantErr: 11.83399 batch_time=0.48763
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.00456 (QuantReg: 11.70647) QuantErr: 11.70647 batch_time=0.50255
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.48150 (QuantReg: 11.87008) QuantErr: 11.87008 batch_time=0.48704
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.85294 (QuantReg: 11.75589) QuantErr: 11.75589 batch_time=0.48322
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.10735 (QuantReg: 11.69657) QuantErr: 11.69657 batch_time=0.49557
Train Epoch: 13 codebook_update_time=2.00635
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch13.pth ...
Done in 4.881s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 1.8828507232666016
quant_reg : 11.722905517578125
quant_err : 11.722905517578125
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.359
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.824552695720186
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.0915
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.3456972782244
mnt_best : 41.21118793155395
not_improved_count: 2
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.78506 (QuantReg: 11.79044) QuantErr: 11.79044 batch_time=31.20355
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.64814 (QuantReg: 11.55676) QuantErr: 11.55676 batch_time=1.56989
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.83067 (QuantReg: 11.58197) QuantErr: 11.58197 batch_time=0.49026
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.71309 (QuantReg: 11.94009) QuantErr: 11.94009 batch_time=0.76201
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.59144 (QuantReg: 11.69077) QuantErr: 11.69077 batch_time=0.49277
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.15454 (QuantReg: 11.36077) QuantErr: 11.36077 batch_time=0.49217
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.01182 (QuantReg: 11.49815) QuantErr: 11.49815 batch_time=1.20719
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.90526 (QuantReg: 11.67334) QuantErr: 11.67334 batch_time=0.48836
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.62049 (QuantReg: 11.31393) QuantErr: 11.31393 batch_time=0.50966
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.56030 (QuantReg: 11.69759) QuantErr: 11.69759 batch_time=0.48711
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.15105 (QuantReg: 11.60551) QuantErr: 11.60551 batch_time=0.50508
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.62674 (QuantReg: 11.56166) QuantErr: 11.56166 batch_time=0.49299
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.68124 (QuantReg: 11.87864) QuantErr: 11.87864 batch_time=0.48331
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 2.08098 (QuantReg: 11.45795) QuantErr: 11.45795 batch_time=0.49146
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.78483 (QuantReg: 11.64850) QuantErr: 11.64850 batch_time=0.49017
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 2.07524 (QuantReg: 11.59998) QuantErr: 11.59998 batch_time=0.50310
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.72136 (QuantReg: 11.90548) QuantErr: 11.90548 batch_time=0.50096
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.98404 (QuantReg: 11.58949) QuantErr: 11.58949 batch_time=0.49757
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.83003 (QuantReg: 11.91714) QuantErr: 11.91714 batch_time=0.49200
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.85419 (QuantReg: 11.65454) QuantErr: 11.65454 batch_time=2.53652
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.00444 (QuantReg: 11.71759) QuantErr: 11.71759 batch_time=0.49646
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.57686 (QuantReg: 12.15079) QuantErr: 12.15079 batch_time=0.53620
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.46791 (QuantReg: 11.85559) QuantErr: 11.85559 batch_time=0.48858
Train Epoch: 14 codebook_update_time=1.66118
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch14.pth ...
Done in 4.418s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch14.pth ...
Done in 8.389s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 1.8135969672203065
quant_reg : 11.683833583831786
quant_err : 11.683833583831786
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.712
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.86992067299294
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.3655
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.716821012163045
mnt_best : 41.86992067299294
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.69412 (QuantReg: 11.64164) QuantErr: 11.64164 batch_time=34.62941
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.27031 (QuantReg: 11.74502) QuantErr: 11.74502 batch_time=0.53259
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.81389 (QuantReg: 11.69112) QuantErr: 11.69112 batch_time=0.50406
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.52661 (QuantReg: 11.37015) QuantErr: 11.37015 batch_time=0.50646
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.91361 (QuantReg: 11.78189) QuantErr: 11.78189 batch_time=0.54342
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.85509 (QuantReg: 11.60126) QuantErr: 11.60126 batch_time=0.73656
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.86413 (QuantReg: 11.95014) QuantErr: 11.95014 batch_time=0.48232
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.47150 (QuantReg: 11.56355) QuantErr: 11.56355 batch_time=0.48961
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.03930 (QuantReg: 11.78081) QuantErr: 11.78081 batch_time=0.54349
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.68243 (QuantReg: 11.58844) QuantErr: 11.58844 batch_time=0.49276
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.11301 (QuantReg: 11.86027) QuantErr: 11.86027 batch_time=0.49208
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.54695 (QuantReg: 11.94890) QuantErr: 11.94890 batch_time=0.49984
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.76693 (QuantReg: 11.76665) QuantErr: 11.76665 batch_time=0.51592
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.61275 (QuantReg: 11.69785) QuantErr: 11.69785 batch_time=0.51422
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.76296 (QuantReg: 11.81032) QuantErr: 11.81032 batch_time=0.50439
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.58707 (QuantReg: 11.67765) QuantErr: 11.67765 batch_time=0.52765
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.54886 (QuantReg: 11.83716) QuantErr: 11.83716 batch_time=0.55437
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.68506 (QuantReg: 11.70525) QuantErr: 11.70525 batch_time=0.51240
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 2.16815 (QuantReg: 11.56808) QuantErr: 11.56808 batch_time=0.50930
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.60370 (QuantReg: 11.93199) QuantErr: 11.93199 batch_time=1.54800
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.50265 (QuantReg: 11.73559) QuantErr: 11.73559 batch_time=0.53886
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.82043 (QuantReg: 11.91375) QuantErr: 11.91375 batch_time=0.55262
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.64147 (QuantReg: 11.71342) QuantErr: 11.71342 batch_time=0.54522
Train Epoch: 15 codebook_update_time=1.81522
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch15.pth ...
Done in 5.940s
removing stale ckpt [epoch 14] [took 0.15s]
epoch : 15
loss : 1.7508210430145263
quant_reg : 11.703841304779052
quant_err : 11.703841304779052
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.02
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.76398205380343
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.2715
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.13902907185565
mnt_best : 41.86992067299294
not_improved_count: 1
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.97400 (QuantReg: 11.54064) QuantErr: 11.54064 batch_time=37.91599
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.81116 (QuantReg: 11.34474) QuantErr: 11.34474 batch_time=0.59718
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.66478 (QuantReg: 11.66927) QuantErr: 11.66927 batch_time=1.24857
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.29158 (QuantReg: 11.27719) QuantErr: 11.27719 batch_time=0.54851
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.69985 (QuantReg: 11.80675) QuantErr: 11.80675 batch_time=0.50748
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.90726 (QuantReg: 11.90016) QuantErr: 11.90016 batch_time=0.52507
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.46225 (QuantReg: 11.83074) QuantErr: 11.83074 batch_time=0.51616
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.76546 (QuantReg: 11.52280) QuantErr: 11.52280 batch_time=0.53039
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.72867 (QuantReg: 11.62815) QuantErr: 11.62815 batch_time=0.51736
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.61989 (QuantReg: 11.92871) QuantErr: 11.92871 batch_time=0.74630
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.48518 (QuantReg: 11.85624) QuantErr: 11.85624 batch_time=0.50953
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.65768 (QuantReg: 11.71790) QuantErr: 11.71790 batch_time=0.49617
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.59057 (QuantReg: 11.73006) QuantErr: 11.73006 batch_time=0.63125
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 2.13672 (QuantReg: 11.79545) QuantErr: 11.79545 batch_time=0.52557
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.07591 (QuantReg: 11.83274) QuantErr: 11.83274 batch_time=0.49988
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.84424 (QuantReg: 11.75590) QuantErr: 11.75590 batch_time=0.54208
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.52345 (QuantReg: 12.04464) QuantErr: 12.04464 batch_time=0.51113
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.91111 (QuantReg: 11.64525) QuantErr: 11.64525 batch_time=0.50450
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.70616 (QuantReg: 11.73325) QuantErr: 11.73325 batch_time=0.54118
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.58321 (QuantReg: 11.97224) QuantErr: 11.97224 batch_time=1.00213
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.71904 (QuantReg: 11.72902) QuantErr: 11.72902 batch_time=0.52023
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.86193 (QuantReg: 11.64134) QuantErr: 11.64134 batch_time=0.66016
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.81043 (QuantReg: 11.72597) QuantErr: 11.72597 batch_time=0.53064
Train Epoch: 16 codebook_update_time=1.77493
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch16.pth ...
Done in 4.949s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 1.7409626169204713
quant_reg : 11.697281410217284
quant_err : 11.697281410217284
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.838
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.3331179893972
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.269
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.301151822164265
mnt_best : 41.86992067299294
not_improved_count: 2
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.53200 (QuantReg: 11.84306) QuantErr: 11.84306 batch_time=34.57727
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.88573 (QuantReg: 11.40873) QuantErr: 11.40873 batch_time=0.52187
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.57836 (QuantReg: 11.44728) QuantErr: 11.44728 batch_time=0.53361
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.13589 (QuantReg: 11.77336) QuantErr: 11.77336 batch_time=0.49573
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.80306 (QuantReg: 11.41647) QuantErr: 11.41647 batch_time=0.51971
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.74899 (QuantReg: 11.80018) QuantErr: 11.80018 batch_time=0.51340
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.90876 (QuantReg: 11.72919) QuantErr: 11.72919 batch_time=0.53991
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.50672 (QuantReg: 11.63526) QuantErr: 11.63526 batch_time=0.54983
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.68113 (QuantReg: 11.44149) QuantErr: 11.44149 batch_time=0.49846
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.81090 (QuantReg: 11.82267) QuantErr: 11.82267 batch_time=0.79341
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.77494 (QuantReg: 11.76961) QuantErr: 11.76961 batch_time=0.50661
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.65930 (QuantReg: 11.67058) QuantErr: 11.67058 batch_time=0.49777
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.90802 (QuantReg: 11.64628) QuantErr: 11.64628 batch_time=0.55663
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.72185 (QuantReg: 11.85050) QuantErr: 11.85050 batch_time=0.49910
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 2.01838 (QuantReg: 11.33745) QuantErr: 11.33745 batch_time=0.49609
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.58767 (QuantReg: 11.65280) QuantErr: 11.65280 batch_time=0.51710
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.89934 (QuantReg: 11.89505) QuantErr: 11.89505 batch_time=0.49763
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.51312 (QuantReg: 12.00755) QuantErr: 12.00755 batch_time=0.51382
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.71601 (QuantReg: 11.41910) QuantErr: 11.41910 batch_time=1.00729
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.49702 (QuantReg: 11.86026) QuantErr: 11.86026 batch_time=1.02720
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.68436 (QuantReg: 11.56851) QuantErr: 11.56851 batch_time=0.56065
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.73957 (QuantReg: 12.08476) QuantErr: 12.08476 batch_time=0.52185
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.40265 (QuantReg: 11.59479) QuantErr: 11.59479 batch_time=0.55565
Train Epoch: 17 codebook_update_time=2.03429
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch17.pth ...
Done in 4.848s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch17.pth ...
Done in 9.705s
removing stale ckpt [epoch 16] [took 0.02s]
epoch : 17
loss : 1.6905624680519105
quant_reg : 11.72966760635376
quant_err : 11.72966760635376
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.635
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.45094150111946
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.5
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.6675
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.625794953110955
mnt_best : 42.45094150111946
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.77496 (QuantReg: 11.51497) QuantErr: 11.51497 batch_time=34.48347
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.63087 (QuantReg: 11.56630) QuantErr: 11.56630 batch_time=0.50368
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.52165 (QuantReg: 11.75935) QuantErr: 11.75935 batch_time=0.50794
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.39520 (QuantReg: 11.84839) QuantErr: 11.84839 batch_time=0.51702
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.63295 (QuantReg: 11.46411) QuantErr: 11.46411 batch_time=0.50839
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.47503 (QuantReg: 11.96200) QuantErr: 11.96200 batch_time=0.50457
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.24922 (QuantReg: 11.98670) QuantErr: 11.98670 batch_time=0.87732
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.47194 (QuantReg: 11.79028) QuantErr: 11.79028 batch_time=0.50235
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.94376 (QuantReg: 11.87077) QuantErr: 11.87077 batch_time=0.51311
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.56071 (QuantReg: 11.52288) QuantErr: 11.52288 batch_time=0.50031
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.91214 (QuantReg: 11.64868) QuantErr: 11.64868 batch_time=0.49378
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.79679 (QuantReg: 11.73249) QuantErr: 11.73249 batch_time=0.50821
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.64721 (QuantReg: 11.60610) QuantErr: 11.60610 batch_time=0.49204
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.90295 (QuantReg: 11.49692) QuantErr: 11.49692 batch_time=1.96175
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.68284 (QuantReg: 11.41279) QuantErr: 11.41279 batch_time=0.51045
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.67711 (QuantReg: 12.08247) QuantErr: 12.08247 batch_time=0.52460
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.54927 (QuantReg: 11.52655) QuantErr: 11.52655 batch_time=0.51174
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.50606 (QuantReg: 11.71715) QuantErr: 11.71715 batch_time=0.51000
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.60605 (QuantReg: 11.81597) QuantErr: 11.81597 batch_time=0.49998
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.32903 (QuantReg: 11.80433) QuantErr: 11.80433 batch_time=0.49556
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.95629 (QuantReg: 11.49994) QuantErr: 11.49994 batch_time=0.50361
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.56490 (QuantReg: 11.79450) QuantErr: 11.79450 batch_time=0.54442
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.61987 (QuantReg: 11.80861) QuantErr: 11.80861 batch_time=0.51115
Train Epoch: 18 codebook_update_time=2.00652
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch18.pth ...
Done in 4.437s
removing stale ckpt [epoch 17] [took 0.02s]
epoch : 18
loss : 1.6324078965187072
quant_reg : 11.732792835235596
quant_err : 11.732792835235596
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.125
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.353907428935855
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 53.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 66.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 90.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.7925
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 43.653195878473994
mnt_best : 42.45094150111946
not_improved_count: 1
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.46927 (QuantReg: 11.83071) QuantErr: 11.83071 batch_time=43.83839
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.50168 (QuantReg: 11.71607) QuantErr: 11.71607 batch_time=0.50823
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.80537 (QuantReg: 11.51280) QuantErr: 11.51280 batch_time=0.51306
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.88427 (QuantReg: 11.73114) QuantErr: 11.73114 batch_time=0.51750
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.47559 (QuantReg: 11.63965) QuantErr: 11.63965 batch_time=0.51882
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.30644 (QuantReg: 11.86008) QuantErr: 11.86008 batch_time=0.54318
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.26613 (QuantReg: 11.73852) QuantErr: 11.73852 batch_time=0.51485
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.51236 (QuantReg: 11.54187) QuantErr: 11.54187 batch_time=0.51699
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.43897 (QuantReg: 11.70592) QuantErr: 11.70592 batch_time=0.49230
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.80737 (QuantReg: 11.37287) QuantErr: 11.37287 batch_time=0.52215
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.64161 (QuantReg: 11.56489) QuantErr: 11.56489 batch_time=0.53691
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.52192 (QuantReg: 11.95199) QuantErr: 11.95199 batch_time=0.51106
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.71065 (QuantReg: 11.98440) QuantErr: 11.98440 batch_time=0.51794
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.58737 (QuantReg: 11.79988) QuantErr: 11.79988 batch_time=0.55974
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.97720 (QuantReg: 11.56878) QuantErr: 11.56878 batch_time=0.51221
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.17365 (QuantReg: 11.77995) QuantErr: 11.77995 batch_time=0.55726
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.48428 (QuantReg: 11.73509) QuantErr: 11.73509 batch_time=0.51370
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.61324 (QuantReg: 11.68927) QuantErr: 11.68927 batch_time=0.65196
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.61763 (QuantReg: 11.48047) QuantErr: 11.48047 batch_time=0.56359
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.51941 (QuantReg: 11.71028) QuantErr: 11.71028 batch_time=0.50914
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.68866 (QuantReg: 11.92160) QuantErr: 11.92160 batch_time=2.11869
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.90394 (QuantReg: 11.56173) QuantErr: 11.56173 batch_time=0.49656
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.63679 (QuantReg: 12.03522) QuantErr: 12.03522 batch_time=0.53105
Train Epoch: 19 codebook_update_time=2.03955
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch19.pth ...
Done in 4.189s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.07/checkpoint-epoch19.pth ...
Done in 8.951s
removing stale ckpt [epoch 18] [took 0.00s]
epoch : 19
loss : 1.606006980895996
quant_reg : 11.750016399383545
quant_err : 11.750016399383545
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 52.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.215
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.592766226742036
MSRVTT_jsfusion_test/v2t_metrics/R1: 23.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 55.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 67.5