-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_t0.15.txt
2603 lines (2603 loc) · 192 KB
/
HCQ_MSRVTT_1kA_t0.15.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 1061.3292739391327 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 105.40812158584595 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 51.872172117233276 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch0.pth ...
Done in 1.577s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch0.pth ...
Done in 3.182s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 486.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 496.278
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 6.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 509.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.537
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.5192494101851104
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.71507 (QuantReg: 22.49939) QuantErr: 22.49939 batch_time=28.48106
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.85176 (QuantReg: 22.56552) QuantErr: 22.56552 batch_time=2.24332
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.75011 (QuantReg: 22.65800) QuantErr: 22.65800 batch_time=0.59002
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.84697 (QuantReg: 22.59103) QuantErr: 22.59103 batch_time=0.55223
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.85477 (QuantReg: 22.62226) QuantErr: 22.62226 batch_time=0.51272
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.29332 (QuantReg: 22.65170) QuantErr: 22.65170 batch_time=0.52012
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.27988 (QuantReg: 22.56371) QuantErr: 22.56371 batch_time=0.54950
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.75210 (QuantReg: 22.67065) QuantErr: 22.67065 batch_time=3.49432
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 6.13905 (QuantReg: 22.62144) QuantErr: 22.62144 batch_time=1.35790
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.12419 (QuantReg: 22.65138) QuantErr: 22.65138 batch_time=0.53434
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.35198 (QuantReg: 22.62474) QuantErr: 22.62474 batch_time=0.52625
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.62460 (QuantReg: 22.62566) QuantErr: 22.62566 batch_time=0.55886
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.69483 (QuantReg: 22.61768) QuantErr: 22.61768 batch_time=0.55872
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 5.20831 (QuantReg: 22.64135) QuantErr: 22.64135 batch_time=0.52341
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 5.53101 (QuantReg: 22.61454) QuantErr: 22.61454 batch_time=0.50006
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 5.13933 (QuantReg: 22.56819) QuantErr: 22.56819 batch_time=0.49983
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 5.58292 (QuantReg: 22.62955) QuantErr: 22.62955 batch_time=0.55275
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 5.38089 (QuantReg: 22.63545) QuantErr: 22.63545 batch_time=0.51261
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 5.06450 (QuantReg: 22.62148) QuantErr: 22.62148 batch_time=0.51185
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 5.40943 (QuantReg: 22.61671) QuantErr: 22.61671 batch_time=2.46414
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.76435 (QuantReg: 22.64377) QuantErr: 22.64377 batch_time=0.51474
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 5.24297 (QuantReg: 22.60773) QuantErr: 22.60773 batch_time=1.75231
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.70873 (QuantReg: 22.61424) QuantErr: 22.61424 batch_time=0.50207
Train Epoch: 1 codebook_update_time=1.90364
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch1.pth ...
Done in 4.210s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch1.pth ...
Done in 8.264s
epoch : 1
loss : 6.03826442527771
quant_reg : 22.61514631652832
quant_err : 22.61514631652832
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 8.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 26.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 37.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 74.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 18.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 50.945
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.560586183179254
MSRVTT_jsfusion_test/v2t_metrics/R1: 7.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 28.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 40.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 75.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 16.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 49.68
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.212233220009455
mnt_best : 20.560586183179254
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 5.06309 (QuantReg: 8.66524) QuantErr: 8.66524 batch_time=32.67218
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.79457 (QuantReg: 8.56861) QuantErr: 8.56861 batch_time=0.51715
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.99548 (QuantReg: 9.16455) QuantErr: 9.16455 batch_time=0.51250
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.81524 (QuantReg: 9.01130) QuantErr: 9.01130 batch_time=0.49618
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.89360 (QuantReg: 9.63269) QuantErr: 9.63269 batch_time=0.51970
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.88446 (QuantReg: 9.62346) QuantErr: 9.62346 batch_time=0.51584
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.66308 (QuantReg: 9.56269) QuantErr: 9.56269 batch_time=0.51863
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.95229 (QuantReg: 9.68114) QuantErr: 9.68114 batch_time=0.51159
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.50444 (QuantReg: 9.94881) QuantErr: 9.94881 batch_time=0.51301
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 5.11892 (QuantReg: 9.78800) QuantErr: 9.78800 batch_time=0.52773
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 4.63643 (QuantReg: 9.77652) QuantErr: 9.77652 batch_time=0.50790
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.50555 (QuantReg: 10.15730) QuantErr: 10.15730 batch_time=0.53657
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 4.63996 (QuantReg: 10.03140) QuantErr: 10.03140 batch_time=0.54106
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.96574 (QuantReg: 10.04242) QuantErr: 10.04242 batch_time=4.55594
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 4.84687 (QuantReg: 9.91009) QuantErr: 9.91009 batch_time=0.55265
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 4.57052 (QuantReg: 10.51773) QuantErr: 10.51773 batch_time=0.51187
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 4.75338 (QuantReg: 10.54191) QuantErr: 10.54191 batch_time=0.51131
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 4.61558 (QuantReg: 10.34643) QuantErr: 10.34643 batch_time=0.51655
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 4.42830 (QuantReg: 10.73341) QuantErr: 10.73341 batch_time=0.52143
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 4.41489 (QuantReg: 10.40721) QuantErr: 10.40721 batch_time=0.51072
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 4.43219 (QuantReg: 10.62118) QuantErr: 10.62118 batch_time=0.51228
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 4.59575 (QuantReg: 10.59253) QuantErr: 10.59253 batch_time=0.52139
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 4.59020 (QuantReg: 10.86197) QuantErr: 10.86197 batch_time=0.49799
Train Epoch: 2 codebook_update_time=1.78157
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch2.pth ...
Done in 4.354s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch2.pth ...
Done in 8.465s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.02s]
epoch : 2
loss : 4.7098689422607425
quant_reg : 9.970110786437989
quant_err : 9.970110786437989
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 10.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 32.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 46.4
MSRVTT_jsfusion_test/t2v_metrics/R50: 79.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 12.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 42.209
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.9760795197247
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 35.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 49.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 79.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 11.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 41.6245
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.797683025356136
mnt_best : 24.9760795197247
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 4.49590 (QuantReg: 7.97601) QuantErr: 7.97601 batch_time=29.51515
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 4.38398 (QuantReg: 8.09098) QuantErr: 8.09098 batch_time=0.51243
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 4.39496 (QuantReg: 8.25556) QuantErr: 8.25556 batch_time=0.53417
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 4.32706 (QuantReg: 8.03897) QuantErr: 8.03897 batch_time=0.52716
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 4.17080 (QuantReg: 8.22381) QuantErr: 8.22381 batch_time=0.56657
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.77491 (QuantReg: 7.77047) QuantErr: 7.77047 batch_time=0.52220
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 4.72790 (QuantReg: 8.41699) QuantErr: 8.41699 batch_time=0.58270
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 4.57654 (QuantReg: 8.33924) QuantErr: 8.33924 batch_time=0.49660
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 4.50961 (QuantReg: 8.17006) QuantErr: 8.17006 batch_time=0.52330
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 4.01199 (QuantReg: 8.43796) QuantErr: 8.43796 batch_time=0.51418
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 4.25547 (QuantReg: 8.13152) QuantErr: 8.13152 batch_time=0.53115
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 4.26216 (QuantReg: 8.28319) QuantErr: 8.28319 batch_time=0.52226
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 4.05685 (QuantReg: 7.91246) QuantErr: 7.91246 batch_time=0.53207
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 4.26420 (QuantReg: 8.20767) QuantErr: 8.20767 batch_time=0.51404
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.45824 (QuantReg: 7.86393) QuantErr: 7.86393 batch_time=0.56656
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 4.06341 (QuantReg: 8.24768) QuantErr: 8.24768 batch_time=0.58705
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 4.31144 (QuantReg: 8.57831) QuantErr: 8.57831 batch_time=0.53451
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 4.29223 (QuantReg: 8.56605) QuantErr: 8.56605 batch_time=0.58653
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 4.02453 (QuantReg: 8.49747) QuantErr: 8.49747 batch_time=0.51930
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 4.08301 (QuantReg: 8.54397) QuantErr: 8.54397 batch_time=0.51129
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 4.15385 (QuantReg: 8.32943) QuantErr: 8.32943 batch_time=0.51170
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 4.26902 (QuantReg: 8.87136) QuantErr: 8.87136 batch_time=0.50018
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 4.26446 (QuantReg: 8.89293) QuantErr: 8.89293 batch_time=0.52240
Train Epoch: 3 codebook_update_time=1.72864
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch3.pth ...
Done in 4.171s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch3.pth ...
Done in 8.310s
removing stale ckpt [epoch 2] [took 0.04s]
epoch : 3
loss : 4.273318289756775
quant_reg : 8.268850830078126
quant_err : 8.268850830078126
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 35.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 49.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 81.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 11.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 39.914
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.284980813840267
MSRVTT_jsfusion_test/v2t_metrics/R1: 12.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 36.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 52.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 82.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 10.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 38.881
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.732911937422493
mnt_best : 28.284980813840267
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 4.13281 (QuantReg: 7.67988) QuantErr: 7.67988 batch_time=35.91327
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 4.07429 (QuantReg: 7.61803) QuantErr: 7.61803 batch_time=0.51657
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 4.14953 (QuantReg: 7.49242) QuantErr: 7.49242 batch_time=0.52307
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 4.02136 (QuantReg: 7.72889) QuantErr: 7.72889 batch_time=0.50979
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.98344 (QuantReg: 7.71693) QuantErr: 7.71693 batch_time=0.51496
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 4.11124 (QuantReg: 7.52727) QuantErr: 7.52727 batch_time=0.52904
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.86746 (QuantReg: 7.32280) QuantErr: 7.32280 batch_time=0.53121
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 4.06157 (QuantReg: 7.85078) QuantErr: 7.85078 batch_time=0.50358
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 4.16517 (QuantReg: 7.54091) QuantErr: 7.54091 batch_time=0.59390
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.76125 (QuantReg: 7.63397) QuantErr: 7.63397 batch_time=0.51445
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 4.04575 (QuantReg: 7.99628) QuantErr: 7.99628 batch_time=0.50667
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.84107 (QuantReg: 8.12198) QuantErr: 8.12198 batch_time=0.52981
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 4.17054 (QuantReg: 7.53461) QuantErr: 7.53461 batch_time=0.51795
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 4.01932 (QuantReg: 7.80580) QuantErr: 7.80580 batch_time=0.51938
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.91901 (QuantReg: 7.66427) QuantErr: 7.66427 batch_time=0.52032
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 4.04186 (QuantReg: 7.57764) QuantErr: 7.57764 batch_time=0.52469
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.91900 (QuantReg: 7.61988) QuantErr: 7.61988 batch_time=0.51222
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 3.96446 (QuantReg: 7.68022) QuantErr: 7.68022 batch_time=0.54362
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 3.82236 (QuantReg: 8.19657) QuantErr: 8.19657 batch_time=0.50667
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 3.68894 (QuantReg: 7.55926) QuantErr: 7.55926 batch_time=0.50689
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 3.52991 (QuantReg: 7.97103) QuantErr: 7.97103 batch_time=0.52609
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 3.56731 (QuantReg: 7.91212) QuantErr: 7.91212 batch_time=0.55230
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 4.08286 (QuantReg: 7.68884) QuantErr: 7.68884 batch_time=0.51766
Train Epoch: 4 codebook_update_time=1.77274
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch4.pth ...
Done in 4.441s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch4.pth ...
Done in 27.270s
removing stale ckpt [epoch 3] [took 0.03s]
epoch : 4
loss : 4.04977656841278
quant_reg : 7.673007284164429
quant_err : 7.673007284164429
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 35.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 52.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 83.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 10.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 38.454
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.51926306217149
MSRVTT_jsfusion_test/v2t_metrics/R1: 13.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 38.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 83.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 37.303
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.149603827482718
mnt_best : 28.51926306217149
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.85327 (QuantReg: 7.08745) QuantErr: 7.08745 batch_time=34.08916
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 4.18763 (QuantReg: 7.56980) QuantErr: 7.56980 batch_time=0.53919
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 3.55704 (QuantReg: 6.83779) QuantErr: 6.83779 batch_time=0.50079
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 4.00665 (QuantReg: 7.58658) QuantErr: 7.58658 batch_time=0.55190
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 3.92468 (QuantReg: 7.24151) QuantErr: 7.24151 batch_time=0.50102
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 3.70855 (QuantReg: 7.71350) QuantErr: 7.71350 batch_time=0.49035
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 3.82062 (QuantReg: 7.03762) QuantErr: 7.03762 batch_time=1.61406
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 4.25748 (QuantReg: 7.12534) QuantErr: 7.12534 batch_time=1.84314
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 3.87764 (QuantReg: 7.66295) QuantErr: 7.66295 batch_time=0.52212
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 3.80814 (QuantReg: 7.44403) QuantErr: 7.44403 batch_time=0.50858
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 3.84239 (QuantReg: 7.51245) QuantErr: 7.51245 batch_time=0.51687
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 3.81031 (QuantReg: 7.34043) QuantErr: 7.34043 batch_time=0.53224
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 3.72192 (QuantReg: 7.80602) QuantErr: 7.80602 batch_time=0.52891
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 3.59789 (QuantReg: 7.26705) QuantErr: 7.26705 batch_time=0.51625
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 4.19476 (QuantReg: 7.76666) QuantErr: 7.76666 batch_time=0.56869
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 3.67321 (QuantReg: 7.56805) QuantErr: 7.56805 batch_time=0.56934
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 4.01544 (QuantReg: 7.51322) QuantErr: 7.51322 batch_time=0.52179
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 3.90477 (QuantReg: 7.60936) QuantErr: 7.60936 batch_time=0.51857
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 3.85764 (QuantReg: 7.63617) QuantErr: 7.63617 batch_time=0.51685
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 3.74896 (QuantReg: 7.35072) QuantErr: 7.35072 batch_time=0.50285
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 3.97909 (QuantReg: 7.54272) QuantErr: 7.54272 batch_time=1.01350
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 3.71212 (QuantReg: 7.29323) QuantErr: 7.29323 batch_time=0.50387
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.95998 (QuantReg: 8.11188) QuantErr: 8.11188 batch_time=0.55190
Train Epoch: 5 codebook_update_time=1.88929
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch5.pth ...
Done in 19.773s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch5.pth ...
Done in 23.811s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 3.8486186122894286
quant_reg : 7.475792219161987
quant_err : 7.475792219161987
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 13.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 37.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 54.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 83.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 36.164
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.754192192229116
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 39.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 84.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 35.6435
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.31721298695106
mnt_best : 29.754192192229116
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 4.02394 (QuantReg: 7.28264) QuantErr: 7.28264 batch_time=31.08013
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 3.83569 (QuantReg: 6.93141) QuantErr: 6.93141 batch_time=0.49888
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 3.78752 (QuantReg: 6.92115) QuantErr: 6.92115 batch_time=0.49674
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 3.74653 (QuantReg: 7.22028) QuantErr: 7.22028 batch_time=0.53949
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 3.79417 (QuantReg: 7.11781) QuantErr: 7.11781 batch_time=0.52429
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 3.61123 (QuantReg: 7.02565) QuantErr: 7.02565 batch_time=0.51923
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 3.88072 (QuantReg: 7.35820) QuantErr: 7.35820 batch_time=1.24431
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 3.68741 (QuantReg: 7.23973) QuantErr: 7.23973 batch_time=0.56938
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 3.68009 (QuantReg: 7.26092) QuantErr: 7.26092 batch_time=0.55558
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 3.58052 (QuantReg: 6.97990) QuantErr: 6.97990 batch_time=0.51739
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 3.60481 (QuantReg: 7.86545) QuantErr: 7.86545 batch_time=0.49721
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 3.70723 (QuantReg: 7.30094) QuantErr: 7.30094 batch_time=0.50453
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 3.65898 (QuantReg: 7.23934) QuantErr: 7.23934 batch_time=0.52891
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 3.50983 (QuantReg: 7.35311) QuantErr: 7.35311 batch_time=0.50305
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 4.08382 (QuantReg: 7.49063) QuantErr: 7.49063 batch_time=0.49486
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 3.84546 (QuantReg: 7.59605) QuantErr: 7.59605 batch_time=0.50205
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 3.51516 (QuantReg: 7.55375) QuantErr: 7.55375 batch_time=0.64886
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 3.81879 (QuantReg: 7.03323) QuantErr: 7.03323 batch_time=0.54901
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 3.85540 (QuantReg: 7.59175) QuantErr: 7.59175 batch_time=0.50895
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 3.72830 (QuantReg: 7.43437) QuantErr: 7.43437 batch_time=0.50625
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 3.55501 (QuantReg: 7.26368) QuantErr: 7.26368 batch_time=0.54653
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 3.41963 (QuantReg: 7.08561) QuantErr: 7.08561 batch_time=0.51501
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 3.45550 (QuantReg: 7.37444) QuantErr: 7.37444 batch_time=0.53666
Train Epoch: 6 codebook_update_time=1.94946
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch6.pth ...
Done in 20.882s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch6.pth ...
Done in 24.948s
removing stale ckpt [epoch 5] [took 0.02s]
epoch : 6
loss : 3.716225005149841
quant_reg : 7.322885129928589
quant_err : 7.322885129928589
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 13.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 38.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 55.3
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 36.796
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.60884660273868
MSRVTT_jsfusion_test/v2t_metrics/R1: 13.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 54.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 84.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 36.943
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.243733868956568
mnt_best : 30.60884660273868
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 3.57141 (QuantReg: 6.71905) QuantErr: 6.71905 batch_time=32.84574
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 3.78450 (QuantReg: 6.95573) QuantErr: 6.95573 batch_time=0.51277
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 3.66731 (QuantReg: 6.66461) QuantErr: 6.66461 batch_time=0.56989
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 3.50497 (QuantReg: 7.21003) QuantErr: 7.21003 batch_time=0.50626
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 3.37387 (QuantReg: 7.12838) QuantErr: 7.12838 batch_time=0.50860
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 3.54470 (QuantReg: 7.16479) QuantErr: 7.16479 batch_time=0.53297
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 3.83147 (QuantReg: 7.34689) QuantErr: 7.34689 batch_time=0.50683
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 3.68385 (QuantReg: 7.31663) QuantErr: 7.31663 batch_time=0.51346
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 3.66877 (QuantReg: 7.41896) QuantErr: 7.41896 batch_time=0.55152
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 3.40443 (QuantReg: 7.15665) QuantErr: 7.15665 batch_time=0.51590
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 3.78429 (QuantReg: 7.35336) QuantErr: 7.35336 batch_time=0.51489
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 3.63312 (QuantReg: 7.22058) QuantErr: 7.22058 batch_time=0.51691
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 3.44121 (QuantReg: 7.23562) QuantErr: 7.23562 batch_time=0.53157
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 3.57691 (QuantReg: 6.98342) QuantErr: 6.98342 batch_time=0.51753
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 3.18668 (QuantReg: 7.14998) QuantErr: 7.14998 batch_time=0.51576
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 3.70757 (QuantReg: 7.19606) QuantErr: 7.19606 batch_time=1.62608
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 3.26135 (QuantReg: 7.37905) QuantErr: 7.37905 batch_time=0.51047
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 3.51647 (QuantReg: 6.87287) QuantErr: 6.87287 batch_time=0.54241
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 3.21276 (QuantReg: 7.41629) QuantErr: 7.41629 batch_time=0.50918
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 3.67671 (QuantReg: 7.41125) QuantErr: 7.41125 batch_time=0.51672
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 3.58751 (QuantReg: 7.34896) QuantErr: 7.34896 batch_time=0.50396
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 3.63554 (QuantReg: 8.10933) QuantErr: 8.10933 batch_time=0.51202
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 3.48628 (QuantReg: 7.34700) QuantErr: 7.34700 batch_time=0.55428
Train Epoch: 7 codebook_update_time=1.87850
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch7.pth ...
Done in 16.502s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch7.pth ...
Done in 20.942s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 3.57391588973999
quant_reg : 7.223409271240234
quant_err : 7.223409271240234
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 54.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 35.415
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.84483477081878
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 54.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.985
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.17676139210542
mnt_best : 31.84483477081878
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 3.48720 (QuantReg: 6.85562) QuantErr: 6.85562 batch_time=32.39706
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.66974 (QuantReg: 6.90665) QuantErr: 6.90665 batch_time=0.53214
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 4.07353 (QuantReg: 7.21252) QuantErr: 7.21252 batch_time=0.49839
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 3.51128 (QuantReg: 6.89243) QuantErr: 6.89243 batch_time=0.53573
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 3.68991 (QuantReg: 7.40212) QuantErr: 7.40212 batch_time=0.50568
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 3.35106 (QuantReg: 6.49298) QuantErr: 6.49298 batch_time=0.55469
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 3.64329 (QuantReg: 7.14043) QuantErr: 7.14043 batch_time=0.49299
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 3.77253 (QuantReg: 7.21690) QuantErr: 7.21690 batch_time=0.53728
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 3.57891 (QuantReg: 7.05550) QuantErr: 7.05550 batch_time=0.50654
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 3.73580 (QuantReg: 7.13678) QuantErr: 7.13678 batch_time=0.51774
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 3.70367 (QuantReg: 7.27505) QuantErr: 7.27505 batch_time=0.52028
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 3.41677 (QuantReg: 6.93877) QuantErr: 6.93877 batch_time=0.51495
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 3.41062 (QuantReg: 7.09456) QuantErr: 7.09456 batch_time=0.57023
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 3.63194 (QuantReg: 7.57217) QuantErr: 7.57217 batch_time=0.51465
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 3.53120 (QuantReg: 7.53636) QuantErr: 7.53636 batch_time=0.51323
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 3.42935 (QuantReg: 7.15578) QuantErr: 7.15578 batch_time=0.49950
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 3.32648 (QuantReg: 6.82556) QuantErr: 6.82556 batch_time=0.53560
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 3.72825 (QuantReg: 7.23181) QuantErr: 7.23181 batch_time=0.52367
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 3.66552 (QuantReg: 7.07004) QuantErr: 7.07004 batch_time=0.50114
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 3.73672 (QuantReg: 7.01182) QuantErr: 7.01182 batch_time=0.50060
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 3.59768 (QuantReg: 7.17315) QuantErr: 7.17315 batch_time=0.56490
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 3.15524 (QuantReg: 7.06060) QuantErr: 7.06060 batch_time=0.56241
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 3.78524 (QuantReg: 7.19863) QuantErr: 7.19863 batch_time=0.51087
Train Epoch: 8 codebook_update_time=1.85591
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch8.pth ...
Done in 4.362s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch8.pth ...
Done in 8.440s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 3.5046350536346433
quant_reg : 7.119911645889283
quant_err : 7.119911645889283
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 36.1
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.78520922459328
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 56.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 35.815
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.822694610057006
mnt_best : 32.78520922459328
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 3.17733 (QuantReg: 7.30120) QuantErr: 7.30120 batch_time=31.77847
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 3.57027 (QuantReg: 7.15511) QuantErr: 7.15511 batch_time=0.51388
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 3.11727 (QuantReg: 6.62091) QuantErr: 6.62091 batch_time=0.54103
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 3.55238 (QuantReg: 7.27541) QuantErr: 7.27541 batch_time=0.53678
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 3.53798 (QuantReg: 7.06443) QuantErr: 7.06443 batch_time=0.57141
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 3.19428 (QuantReg: 7.26830) QuantErr: 7.26830 batch_time=0.53006
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 3.26070 (QuantReg: 7.18226) QuantErr: 7.18226 batch_time=0.52982
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 3.42971 (QuantReg: 7.29643) QuantErr: 7.29643 batch_time=0.54553
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 3.62878 (QuantReg: 6.85673) QuantErr: 6.85673 batch_time=0.50907
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 3.19140 (QuantReg: 7.15928) QuantErr: 7.15928 batch_time=0.50708
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 3.45521 (QuantReg: 7.22764) QuantErr: 7.22764 batch_time=0.55978
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 3.21472 (QuantReg: 6.96545) QuantErr: 6.96545 batch_time=0.53573
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 3.43587 (QuantReg: 7.12041) QuantErr: 7.12041 batch_time=0.52207
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 3.31913 (QuantReg: 6.78116) QuantErr: 6.78116 batch_time=0.51350
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 3.36338 (QuantReg: 6.62581) QuantErr: 6.62581 batch_time=0.52679
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 3.24097 (QuantReg: 7.23701) QuantErr: 7.23701 batch_time=0.52926
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 3.57738 (QuantReg: 7.26919) QuantErr: 7.26919 batch_time=0.50429
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 3.17492 (QuantReg: 7.35696) QuantErr: 7.35696 batch_time=0.58133
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 3.52726 (QuantReg: 6.97239) QuantErr: 6.97239 batch_time=0.52874
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 3.26134 (QuantReg: 7.29229) QuantErr: 7.29229 batch_time=1.71229
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 3.36996 (QuantReg: 7.34140) QuantErr: 7.34140 batch_time=1.53365
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 3.08996 (QuantReg: 6.63038) QuantErr: 6.63038 batch_time=0.56241
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 3.34577 (QuantReg: 6.82710) QuantErr: 6.82710 batch_time=0.51788
Train Epoch: 9 codebook_update_time=1.77508
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch9.pth ...
Done in 6.016s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch9.pth ...
Done in 11.315s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 3.397524713516235
quant_reg : 7.043995719909668
quant_err : 7.043995719909668
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.934
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.28020238027201
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 33.7895
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.10351334697459
mnt_best : 33.28020238027201
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 3.21166 (QuantReg: 7.03927) QuantErr: 7.03927 batch_time=32.91061
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 3.42553 (QuantReg: 6.80705) QuantErr: 6.80705 batch_time=1.64332
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 3.68113 (QuantReg: 7.24539) QuantErr: 7.24539 batch_time=0.50960
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 3.33232 (QuantReg: 6.69922) QuantErr: 6.69922 batch_time=0.68183
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 3.13117 (QuantReg: 6.95322) QuantErr: 6.95322 batch_time=0.50416
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 3.28497 (QuantReg: 6.85774) QuantErr: 6.85774 batch_time=0.52481
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 3.50016 (QuantReg: 6.86462) QuantErr: 6.86462 batch_time=1.18906
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 3.49407 (QuantReg: 7.04715) QuantErr: 7.04715 batch_time=0.54480
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 3.41706 (QuantReg: 7.04074) QuantErr: 7.04074 batch_time=0.51176
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 3.42021 (QuantReg: 7.10258) QuantErr: 7.10258 batch_time=0.52215
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 3.33643 (QuantReg: 6.92522) QuantErr: 6.92522 batch_time=0.50610
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 3.40667 (QuantReg: 7.03640) QuantErr: 7.03640 batch_time=0.58223
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 3.34615 (QuantReg: 6.74104) QuantErr: 6.74104 batch_time=0.51505
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 3.06764 (QuantReg: 7.06162) QuantErr: 7.06162 batch_time=0.50919
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 3.33225 (QuantReg: 6.72858) QuantErr: 6.72858 batch_time=0.50922
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 3.59440 (QuantReg: 7.18615) QuantErr: 7.18615 batch_time=0.64916
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.37914 (QuantReg: 6.90296) QuantErr: 6.90296 batch_time=0.54234
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 3.21163 (QuantReg: 7.00194) QuantErr: 7.00194 batch_time=0.56203
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 3.11430 (QuantReg: 7.18399) QuantErr: 7.18399 batch_time=0.52775
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 3.01692 (QuantReg: 7.52719) QuantErr: 7.52719 batch_time=0.53105
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 3.40255 (QuantReg: 7.01763) QuantErr: 7.01763 batch_time=0.51519
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 3.56079 (QuantReg: 7.10333) QuantErr: 7.10333 batch_time=0.60062
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 3.37247 (QuantReg: 7.28011) QuantErr: 7.28011 batch_time=0.51841
Train Epoch: 10 codebook_update_time=1.81227
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch10.pth ...
Done in 7.062s
removing stale ckpt [epoch 9] [took 0.02s]
epoch : 10
loss : 3.3129789476394653
quant_reg : 7.0308165340423585
quant_err : 7.0308165340423585
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 56.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 36.517
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.1696140182992
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 55.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 37.381
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.21282104563246
mnt_best : 33.28020238027201
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 3.20914 (QuantReg: 6.95418) QuantErr: 6.95418 batch_time=33.48198
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 3.55665 (QuantReg: 6.39429) QuantErr: 6.39429 batch_time=0.51135
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 3.41120 (QuantReg: 6.85001) QuantErr: 6.85001 batch_time=0.50877
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 3.42650 (QuantReg: 7.05592) QuantErr: 7.05592 batch_time=0.53990
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 3.28962 (QuantReg: 7.03101) QuantErr: 7.03101 batch_time=0.52462
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 3.62868 (QuantReg: 7.50763) QuantErr: 7.50763 batch_time=0.50910
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 3.13994 (QuantReg: 6.85125) QuantErr: 6.85125 batch_time=4.45121
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 3.29249 (QuantReg: 6.81768) QuantErr: 6.81768 batch_time=0.50694
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 3.19206 (QuantReg: 6.87482) QuantErr: 6.87482 batch_time=0.52164
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 3.43935 (QuantReg: 7.10592) QuantErr: 7.10592 batch_time=0.51185
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.98908 (QuantReg: 7.08011) QuantErr: 7.08011 batch_time=0.50032
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 3.07649 (QuantReg: 7.10711) QuantErr: 7.10711 batch_time=0.51255
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 3.21479 (QuantReg: 6.74666) QuantErr: 6.74666 batch_time=0.49638
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 3.08054 (QuantReg: 6.93082) QuantErr: 6.93082 batch_time=0.51478
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 3.16781 (QuantReg: 7.19581) QuantErr: 7.19581 batch_time=0.50871
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 3.38699 (QuantReg: 6.70138) QuantErr: 6.70138 batch_time=0.51626
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 3.15345 (QuantReg: 6.78534) QuantErr: 6.78534 batch_time=0.50673
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 3.24272 (QuantReg: 7.25403) QuantErr: 7.25403 batch_time=0.50944
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 3.25328 (QuantReg: 6.74006) QuantErr: 6.74006 batch_time=0.50513
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 3.10921 (QuantReg: 6.94169) QuantErr: 6.94169 batch_time=0.52400
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 3.15133 (QuantReg: 6.98879) QuantErr: 6.98879 batch_time=0.52565
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 3.10652 (QuantReg: 7.16563) QuantErr: 7.16563 batch_time=0.50828
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 3.24513 (QuantReg: 7.32199) QuantErr: 7.32199 batch_time=0.80168
Train Epoch: 11 codebook_update_time=1.78177
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch11.pth ...
Done in 5.357s
removing stale ckpt [epoch 10] [took 0.01s]
epoch : 11
loss : 3.239279425621033
quant_reg : 6.971584667205811
quant_err : 6.971584667205811
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 14.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.3
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.681
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.97232144998274
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 42.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.9235
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.87505615704576
mnt_best : 33.28020238027201
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 3.01485 (QuantReg: 6.63919) QuantErr: 6.63919 batch_time=33.55152
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 3.22794 (QuantReg: 6.70701) QuantErr: 6.70701 batch_time=0.51642
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 3.36400 (QuantReg: 7.07095) QuantErr: 7.07095 batch_time=1.23370
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 3.34435 (QuantReg: 6.96092) QuantErr: 6.96092 batch_time=0.50312
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 3.05006 (QuantReg: 6.76103) QuantErr: 6.76103 batch_time=0.50997
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 3.25594 (QuantReg: 6.98496) QuantErr: 6.98496 batch_time=0.52781
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 3.52095 (QuantReg: 7.13793) QuantErr: 7.13793 batch_time=0.74705
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 3.26830 (QuantReg: 6.35515) QuantErr: 6.35515 batch_time=0.53188
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 3.23822 (QuantReg: 7.00738) QuantErr: 7.00738 batch_time=0.51098
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 3.00116 (QuantReg: 6.91140) QuantErr: 6.91140 batch_time=0.52421
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 3.06002 (QuantReg: 7.14084) QuantErr: 7.14084 batch_time=0.52896
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 3.11324 (QuantReg: 7.15399) QuantErr: 7.15399 batch_time=0.52890
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 3.07877 (QuantReg: 6.77955) QuantErr: 6.77955 batch_time=0.51262
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 3.05713 (QuantReg: 6.58191) QuantErr: 6.58191 batch_time=0.51779
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 3.31143 (QuantReg: 7.37680) QuantErr: 7.37680 batch_time=0.53374
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 3.21629 (QuantReg: 7.03291) QuantErr: 7.03291 batch_time=0.56134
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 3.35466 (QuantReg: 7.05344) QuantErr: 7.05344 batch_time=0.50375
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 3.03964 (QuantReg: 6.96469) QuantErr: 6.96469 batch_time=0.49864
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 3.21304 (QuantReg: 6.74234) QuantErr: 6.74234 batch_time=0.51372
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 3.01539 (QuantReg: 6.96645) QuantErr: 6.96645 batch_time=0.51246
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 3.25743 (QuantReg: 7.27577) QuantErr: 7.27577 batch_time=0.51037
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 3.26618 (QuantReg: 7.17205) QuantErr: 7.17205 batch_time=0.51107
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 3.06202 (QuantReg: 7.11320) QuantErr: 7.11320 batch_time=0.53158
Train Epoch: 12 codebook_update_time=1.83295
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch12.pth ...
Done in 25.663s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch12.pth ...
Done in 30.166s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 3.2003123331069947
quant_reg : 6.936793483734131
quant_err : 6.936793483734131
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.557
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.64483809759762
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 35.2435
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.347080551862305
mnt_best : 33.64483809759762
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 3.37718 (QuantReg: 6.79311) QuantErr: 6.79311 batch_time=32.62399
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 2.85897 (QuantReg: 7.23212) QuantErr: 7.23212 batch_time=0.50529
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 3.22837 (QuantReg: 6.55874) QuantErr: 6.55874 batch_time=0.51283
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 3.05378 (QuantReg: 6.92041) QuantErr: 6.92041 batch_time=0.49723
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 3.18072 (QuantReg: 6.96427) QuantErr: 6.96427 batch_time=0.49129
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 3.21817 (QuantReg: 6.64375) QuantErr: 6.64375 batch_time=0.51550
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 3.12469 (QuantReg: 6.98047) QuantErr: 6.98047 batch_time=0.55638
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 3.19676 (QuantReg: 7.06510) QuantErr: 7.06510 batch_time=0.53174
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 3.14908 (QuantReg: 7.26294) QuantErr: 7.26294 batch_time=0.50609
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 3.02152 (QuantReg: 6.88249) QuantErr: 6.88249 batch_time=0.52081
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 3.05353 (QuantReg: 7.19138) QuantErr: 7.19138 batch_time=0.66494
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 3.08051 (QuantReg: 7.38111) QuantErr: 7.38111 batch_time=0.54106
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 3.21022 (QuantReg: 6.92375) QuantErr: 6.92375 batch_time=0.54501
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 3.20004 (QuantReg: 7.01305) QuantErr: 7.01305 batch_time=5.27854
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.00726 (QuantReg: 6.88146) QuantErr: 6.88146 batch_time=0.51260
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 3.31194 (QuantReg: 6.94437) QuantErr: 6.94437 batch_time=0.50321
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 3.12383 (QuantReg: 7.05071) QuantErr: 7.05071 batch_time=0.52215
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 3.15613 (QuantReg: 6.76084) QuantErr: 6.76084 batch_time=0.51112
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 2.95129 (QuantReg: 7.13727) QuantErr: 7.13727 batch_time=0.51621
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 3.33615 (QuantReg: 7.09679) QuantErr: 7.09679 batch_time=0.51222
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 2.75537 (QuantReg: 6.90849) QuantErr: 6.90849 batch_time=0.55192
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 3.14764 (QuantReg: 6.80653) QuantErr: 6.80653 batch_time=0.49538
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 3.22779 (QuantReg: 6.82058) QuantErr: 6.82058 batch_time=0.52059
Train Epoch: 13 codebook_update_time=1.82313
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch13.pth ...
Done in 4.727s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch13.pth ...
Done in 9.653s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 3.143215871810913
quant_reg : 6.952013240814209
quant_err : 6.952013240814209
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.098
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.69284835048767
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.411
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.653048288492
mnt_best : 33.69284835048767
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 3.10034 (QuantReg: 7.12100) QuantErr: 7.12100 batch_time=33.40425
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.85527 (QuantReg: 6.78726) QuantErr: 6.78726 batch_time=1.19486
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 3.19949 (QuantReg: 6.79755) QuantErr: 6.79755 batch_time=0.51525
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 3.04353 (QuantReg: 7.28233) QuantErr: 7.28233 batch_time=0.51151
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.88449 (QuantReg: 6.96367) QuantErr: 6.96367 batch_time=0.52998
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 3.35520 (QuantReg: 6.64213) QuantErr: 6.64213 batch_time=0.55038
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 3.27812 (QuantReg: 6.63042) QuantErr: 6.63042 batch_time=0.53127
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 3.09518 (QuantReg: 6.99877) QuantErr: 6.99877 batch_time=0.73764
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.89850 (QuantReg: 6.46109) QuantErr: 6.46109 batch_time=0.51983
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 2.91496 (QuantReg: 6.86912) QuantErr: 6.86912 batch_time=0.51300
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 3.29122 (QuantReg: 6.76457) QuantErr: 6.76457 batch_time=0.55413
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 2.80191 (QuantReg: 6.74225) QuantErr: 6.74225 batch_time=0.51844
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 3.00227 (QuantReg: 7.24475) QuantErr: 7.24475 batch_time=0.49729
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 3.05879 (QuantReg: 6.67765) QuantErr: 6.67765 batch_time=0.51133
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 3.00146 (QuantReg: 6.94500) QuantErr: 6.94500 batch_time=0.53378
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 3.33329 (QuantReg: 6.77485) QuantErr: 6.77485 batch_time=0.52196
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 3.07533 (QuantReg: 7.02362) QuantErr: 7.02362 batch_time=0.51385
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 3.28498 (QuantReg: 6.75783) QuantErr: 6.75783 batch_time=0.52583
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 3.18260 (QuantReg: 6.94094) QuantErr: 6.94094 batch_time=0.55315
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 3.16065 (QuantReg: 6.92604) QuantErr: 6.92604 batch_time=0.50195
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 3.23535 (QuantReg: 6.93281) QuantErr: 6.93281 batch_time=0.52313
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 3.04397 (QuantReg: 7.24716) QuantErr: 7.24716 batch_time=0.54442
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.81312 (QuantReg: 6.97534) QuantErr: 6.97534 batch_time=0.52667
Train Epoch: 14 codebook_update_time=1.79512
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch14.pth ...
Done in 6.907s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch14.pth ...
Done in 12.327s
removing stale ckpt [epoch 13] [took 0.04s]
epoch : 14
loss : 3.0883258876800537
quant_reg : 6.897167345046997
quant_err : 6.897167345046997
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.541
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.84795574812987
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 44.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.6065
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.186819173793275
mnt_best : 33.84795574812987
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.91195 (QuantReg: 6.90467) QuantErr: 6.90467 batch_time=29.81757
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.72922 (QuantReg: 6.98019) QuantErr: 6.98019 batch_time=0.88966
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 3.11417 (QuantReg: 6.92674) QuantErr: 6.92674 batch_time=0.51571
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.81557 (QuantReg: 6.61687) QuantErr: 6.61687 batch_time=0.53787
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 3.15738 (QuantReg: 6.74887) QuantErr: 6.74887 batch_time=0.52237
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 2.96615 (QuantReg: 6.98841) QuantErr: 6.98841 batch_time=0.51580
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 3.04589 (QuantReg: 7.38915) QuantErr: 7.38915 batch_time=0.49513
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.89299 (QuantReg: 6.61277) QuantErr: 6.61277 batch_time=0.53790
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 3.12048 (QuantReg: 6.92479) QuantErr: 6.92479 batch_time=0.52368
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 3.02828 (QuantReg: 6.55526) QuantErr: 6.55526 batch_time=0.52799
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 3.35949 (QuantReg: 7.17267) QuantErr: 7.17267 batch_time=0.51679
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.94360 (QuantReg: 7.14673) QuantErr: 7.14673 batch_time=0.49155
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 3.04906 (QuantReg: 6.86336) QuantErr: 6.86336 batch_time=0.51146
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 2.86442 (QuantReg: 6.85388) QuantErr: 6.85388 batch_time=0.52820
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.96902 (QuantReg: 6.99212) QuantErr: 6.99212 batch_time=0.50411
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.93123 (QuantReg: 6.80412) QuantErr: 6.80412 batch_time=0.50458
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 2.87849 (QuantReg: 6.91731) QuantErr: 6.91731 batch_time=0.51253
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 2.93384 (QuantReg: 6.79874) QuantErr: 6.79874 batch_time=0.51797
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 3.36147 (QuantReg: 6.87949) QuantErr: 6.87949 batch_time=0.51353
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 3.03620 (QuantReg: 7.08871) QuantErr: 7.08871 batch_time=0.50577
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.90520 (QuantReg: 6.90737) QuantErr: 6.90737 batch_time=0.52117
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.92617 (QuantReg: 7.18755) QuantErr: 7.18755 batch_time=0.55266
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.83905 (QuantReg: 6.93127) QuantErr: 6.93127 batch_time=0.51462
Train Epoch: 15 codebook_update_time=1.85220
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch15.pth ...
Done in 5.178s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch15.pth ...
Done in 9.907s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 3.0278290615081787
quant_reg : 6.883938499450684
quant_err : 6.883938499450684
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.942
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.87007856863598
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 44.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 33.1225
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.50538150316447
mnt_best : 33.87007856863598
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 3.22680 (QuantReg: 6.76144) QuantErr: 6.76144 batch_time=31.22638
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 3.10617 (QuantReg: 6.62492) QuantErr: 6.62492 batch_time=0.51324
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 3.01008 (QuantReg: 6.92096) QuantErr: 6.92096 batch_time=1.28436
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 3.41099 (QuantReg: 6.67121) QuantErr: 6.67121 batch_time=0.51344
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 2.97284 (QuantReg: 7.05748) QuantErr: 7.05748 batch_time=0.55769
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 3.22566 (QuantReg: 7.26915) QuantErr: 7.26915 batch_time=0.52728
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.76348 (QuantReg: 6.83340) QuantErr: 6.83340 batch_time=2.34949
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 3.06597 (QuantReg: 6.57380) QuantErr: 6.57380 batch_time=0.51144
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 3.02015 (QuantReg: 7.07009) QuantErr: 7.07009 batch_time=0.50278
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 3.10977 (QuantReg: 7.06473) QuantErr: 7.06473 batch_time=0.51873
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 2.82934 (QuantReg: 6.79749) QuantErr: 6.79749 batch_time=0.51071
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 2.78289 (QuantReg: 6.73649) QuantErr: 6.73649 batch_time=0.51696
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.93314 (QuantReg: 6.84845) QuantErr: 6.84845 batch_time=0.50806
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 3.40845 (QuantReg: 7.04376) QuantErr: 7.04376 batch_time=1.99245
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 3.25622 (QuantReg: 6.76800) QuantErr: 6.76800 batch_time=0.51188
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 2.95782 (QuantReg: 6.87981) QuantErr: 6.87981 batch_time=0.50529
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 2.83050 (QuantReg: 7.06299) QuantErr: 7.06299 batch_time=0.51329
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 3.16526 (QuantReg: 6.72507) QuantErr: 6.72507 batch_time=0.58187
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.92092 (QuantReg: 6.64168) QuantErr: 6.64168 batch_time=0.52169
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 2.82068 (QuantReg: 7.19313) QuantErr: 7.19313 batch_time=0.51535
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.95458 (QuantReg: 6.94986) QuantErr: 6.94986 batch_time=0.50284
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 3.14744 (QuantReg: 6.65771) QuantErr: 6.65771 batch_time=0.57986
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 3.16273 (QuantReg: 7.05456) QuantErr: 7.05456 batch_time=0.52657
Train Epoch: 16 codebook_update_time=1.72582
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch16.pth ...
Done in 4.702s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch16.pth ...
Done in 9.541s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 3.006816719055176
quant_reg : 6.871023414611816
quant_err : 6.871023414611816
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.9295
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.114020888069206
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.258
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.339392108450326
mnt_best : 34.114020888069206
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 2.81625 (QuantReg: 7.01766) QuantErr: 7.01766 batch_time=30.84253
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 3.02491 (QuantReg: 6.68846) QuantErr: 6.68846 batch_time=5.05287
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 2.73664 (QuantReg: 6.65548) QuantErr: 6.65548 batch_time=0.51794
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 3.19341 (QuantReg: 6.92401) QuantErr: 6.92401 batch_time=0.58757
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 2.98095 (QuantReg: 6.49658) QuantErr: 6.49658 batch_time=0.50797
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 3.19430 (QuantReg: 7.01073) QuantErr: 7.01073 batch_time=0.50837
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 3.13584 (QuantReg: 7.01363) QuantErr: 7.01363 batch_time=2.27778
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 2.70476 (QuantReg: 6.82582) QuantErr: 6.82582 batch_time=0.50248
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 2.97159 (QuantReg: 6.60636) QuantErr: 6.60636 batch_time=0.54857
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 3.21025 (QuantReg: 6.98959) QuantErr: 6.98959 batch_time=0.49570
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 3.15038 (QuantReg: 7.00864) QuantErr: 7.00864 batch_time=0.51486
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 2.91345 (QuantReg: 6.84889) QuantErr: 6.84889 batch_time=0.51209
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 3.13889 (QuantReg: 6.63216) QuantErr: 6.63216 batch_time=0.50634
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 3.14282 (QuantReg: 7.09276) QuantErr: 7.09276 batch_time=0.59450
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 3.17620 (QuantReg: 6.51013) QuantErr: 6.51013 batch_time=0.52045
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.88196 (QuantReg: 6.93620) QuantErr: 6.93620 batch_time=0.51044
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 3.29300 (QuantReg: 7.12457) QuantErr: 7.12457 batch_time=0.51267
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 2.80780 (QuantReg: 7.21634) QuantErr: 7.21634 batch_time=0.56060
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.94278 (QuantReg: 6.57120) QuantErr: 6.57120 batch_time=0.54498
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 2.76909 (QuantReg: 7.08401) QuantErr: 7.08401 batch_time=0.54921
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 2.88635 (QuantReg: 6.65954) QuantErr: 6.65954 batch_time=0.74749
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 3.01896 (QuantReg: 7.23598) QuantErr: 7.23598 batch_time=0.52800
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 2.78035 (QuantReg: 6.60602) QuantErr: 6.60602 batch_time=0.51231
Train Epoch: 17 codebook_update_time=1.76523
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch17.pth ...
Done in 5.454s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch17.pth ...
Done in 10.805s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 2.9643670711517336
quant_reg : 6.903344284057617
quant_err : 6.903344284057617
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 43.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.122
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.65520412247051
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 44.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 33.51
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.93819432441713
mnt_best : 34.65520412247051
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 3.04325 (QuantReg: 6.75817) QuantErr: 6.75817 batch_time=33.55771
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 2.94647 (QuantReg: 6.61390) QuantErr: 6.61390 batch_time=0.51386
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 2.82956 (QuantReg: 6.69372) QuantErr: 6.69372 batch_time=0.53252
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 2.75711 (QuantReg: 6.83082) QuantErr: 6.83082 batch_time=0.55652
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 2.92192 (QuantReg: 6.71916) QuantErr: 6.71916 batch_time=0.50486
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 2.85192 (QuantReg: 7.13906) QuantErr: 7.13906 batch_time=0.53162
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 2.71564 (QuantReg: 7.12041) QuantErr: 7.12041 batch_time=0.51213
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 2.82030 (QuantReg: 6.98926) QuantErr: 6.98926 batch_time=0.53154
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 3.11645 (QuantReg: 7.09558) QuantErr: 7.09558 batch_time=0.51133
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 2.83613 (QuantReg: 6.66077) QuantErr: 6.66077 batch_time=0.53744
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 3.08718 (QuantReg: 6.67527) QuantErr: 6.67527 batch_time=0.55645
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 3.07752 (QuantReg: 6.93102) QuantErr: 6.93102 batch_time=0.51167
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.86399 (QuantReg: 6.83555) QuantErr: 6.83555 batch_time=0.50668
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 3.07167 (QuantReg: 7.00388) QuantErr: 7.00388 batch_time=2.67081
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 2.92670 (QuantReg: 6.52277) QuantErr: 6.52277 batch_time=0.51981
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 3.00026 (QuantReg: 7.05078) QuantErr: 7.05078 batch_time=0.51142
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 2.83574 (QuantReg: 6.51515) QuantErr: 6.51515 batch_time=0.56962
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 2.86510 (QuantReg: 6.88496) QuantErr: 6.88496 batch_time=0.51027
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.94692 (QuantReg: 7.00771) QuantErr: 7.00771 batch_time=0.56909
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 2.79350 (QuantReg: 6.87029) QuantErr: 6.87029 batch_time=2.08659
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 3.10792 (QuantReg: 6.56371) QuantErr: 6.56371 batch_time=0.58966
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.95883 (QuantReg: 6.96671) QuantErr: 6.96671 batch_time=0.51747
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 2.92395 (QuantReg: 6.87437) QuantErr: 6.87437 batch_time=0.51831
Train Epoch: 18 codebook_update_time=2.11923
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch18.pth ...
Done in 5.363s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch18.pth ...
Done in 10.322s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 2.926634873390198
quant_reg : 6.891225193023682
quant_err : 6.891225193023682
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.8
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.067
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.85369012899472
MSRVTT_jsfusion_test/v2t_metrics/R1: 15.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 59.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 33.3925
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.20423771724695
mnt_best : 35.85369012899472
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 2.93109 (QuantReg: 7.09022) QuantErr: 7.09022 batch_time=31.15749
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 2.75334 (QuantReg: 6.88160) QuantErr: 6.88160 batch_time=0.50868
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 3.06831 (QuantReg: 6.66698) QuantErr: 6.66698 batch_time=0.51234
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 3.11880 (QuantReg: 7.04509) QuantErr: 7.04509 batch_time=0.52142
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 2.89868 (QuantReg: 6.87422) QuantErr: 6.87422 batch_time=0.51299
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 2.79093 (QuantReg: 7.20234) QuantErr: 7.20234 batch_time=0.52266
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 2.61689 (QuantReg: 6.97545) QuantErr: 6.97545 batch_time=0.51399
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 2.84560 (QuantReg: 6.72297) QuantErr: 6.72297 batch_time=0.95428
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 2.69878 (QuantReg: 6.63540) QuantErr: 6.63540 batch_time=0.54153
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 2.87663 (QuantReg: 6.39129) QuantErr: 6.39129 batch_time=0.51488
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 2.81783 (QuantReg: 6.63494) QuantErr: 6.63494 batch_time=0.52366
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 2.83920 (QuantReg: 7.06076) QuantErr: 7.06076 batch_time=0.52160
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 3.01198 (QuantReg: 7.30746) QuantErr: 7.30746 batch_time=0.51883
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 2.97408 (QuantReg: 7.03889) QuantErr: 7.03889 batch_time=0.56537
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 3.15652 (QuantReg: 6.88818) QuantErr: 6.88818 batch_time=0.51521
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 2.45122 (QuantReg: 6.75204) QuantErr: 6.75204 batch_time=0.54288
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 2.89873 (QuantReg: 6.91830) QuantErr: 6.91830 batch_time=0.50834
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 2.85003 (QuantReg: 6.58664) QuantErr: 6.58664 batch_time=0.50512
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 2.81593 (QuantReg: 6.58558) QuantErr: 6.58558 batch_time=0.49480
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 2.80126 (QuantReg: 6.79658) QuantErr: 6.79658 batch_time=0.53523
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 3.14131 (QuantReg: 7.20835) QuantErr: 7.20835 batch_time=0.49842
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 2.97297 (QuantReg: 6.95053) QuantErr: 6.95053 batch_time=0.51221
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 2.85695 (QuantReg: 7.04283) QuantErr: 7.04283 batch_time=0.51742
Train Epoch: 19 codebook_update_time=1.67104
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_t0.15/checkpoint-epoch19.pth ...
Done in 14.205s
removing stale ckpt [epoch 18] [took 0.02s]
epoch : 19
loss : 2.895226366043091
quant_reg : 6.868426816940308
quant_err : 6.868426816940308
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750