-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_bs64.txt
4601 lines (4601 loc) · 423 KB
/
HCQ_MSRVTT_1kA_bs64.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 1274.2785212993622 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 73.84164261817932 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 48.81112051010132 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch0.pth ...
Done in 1.477s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch0.pth ...
Done in 2.943s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 4.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 486.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 496.278
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 1.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 6.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 509.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 503.5355
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.5192494101851104
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/500 64/32000 (0%)] Loss: 8.42179 (QuantReg: 22.48055) QuantErr: 22.48055 batch_time=17.35523
Train Epoch: 1 [9/500 576/32000 (2%)] Loss: 8.07024 (QuantReg: 22.58955) QuantErr: 22.58955 batch_time=0.39848
Train Epoch: 1 [17/500 1088/32000 (3%)] Loss: 7.08605 (QuantReg: 22.54650) QuantErr: 22.54650 batch_time=0.40290
Train Epoch: 1 [25/500 1600/32000 (5%)] Loss: 6.10758 (QuantReg: 22.59287) QuantErr: 22.59287 batch_time=0.39126
Train Epoch: 1 [33/500 2112/32000 (7%)] Loss: 6.06135 (QuantReg: 22.61736) QuantErr: 22.61736 batch_time=0.39824
Train Epoch: 1 [41/500 2624/32000 (8%)] Loss: 5.71115 (QuantReg: 22.59416) QuantErr: 22.59416 batch_time=0.38560
Train Epoch: 1 [49/500 3136/32000 (10%)] Loss: 4.87021 (QuantReg: 22.58594) QuantErr: 22.58594 batch_time=0.41387
Train Epoch: 1 [57/500 3648/32000 (11%)] Loss: 5.34070 (QuantReg: 22.60461) QuantErr: 22.60461 batch_time=0.40046
Train Epoch: 1 [65/500 4160/32000 (13%)] Loss: 4.90885 (QuantReg: 22.61914) QuantErr: 22.61914 batch_time=0.46010
Train Epoch: 1 [73/500 4672/32000 (15%)] Loss: 4.90732 (QuantReg: 22.62774) QuantErr: 22.62774 batch_time=0.43538
Train Epoch: 1 [81/500 5184/32000 (16%)] Loss: 4.50384 (QuantReg: 22.64861) QuantErr: 22.64861 batch_time=0.43536
Train Epoch: 1 [89/500 5696/32000 (18%)] Loss: 4.65365 (QuantReg: 22.66018) QuantErr: 22.66018 batch_time=0.38996
Train Epoch: 1 [97/500 6208/32000 (19%)] Loss: 4.83792 (QuantReg: 22.62110) QuantErr: 22.62110 batch_time=0.38874
Train Epoch: 1 [105/500 6720/32000 (21%)] Loss: 4.73095 (QuantReg: 22.66097) QuantErr: 22.66097 batch_time=0.44417
Train Epoch: 1 [113/500 7232/32000 (23%)] Loss: 4.24784 (QuantReg: 22.59879) QuantErr: 22.59879 batch_time=0.39493
Train Epoch: 1 [121/500 7744/32000 (24%)] Loss: 4.60036 (QuantReg: 22.64107) QuantErr: 22.64107 batch_time=0.40610
Train Epoch: 1 [129/500 8256/32000 (26%)] Loss: 3.78088 (QuantReg: 22.66199) QuantErr: 22.66199 batch_time=0.40453
Train Epoch: 1 [137/500 8768/32000 (27%)] Loss: 4.35878 (QuantReg: 22.61040) QuantErr: 22.61040 batch_time=0.43583
Train Epoch: 1 [145/500 9280/32000 (29%)] Loss: 3.27047 (QuantReg: 22.64496) QuantErr: 22.64496 batch_time=0.38996
Train Epoch: 1 [153/500 9792/32000 (31%)] Loss: 3.92429 (QuantReg: 22.66969) QuantErr: 22.66969 batch_time=0.38104
Train Epoch: 1 [161/500 10304/32000 (32%)] Loss: 3.87782 (QuantReg: 22.66512) QuantErr: 22.66512 batch_time=0.38540
Train Epoch: 1 [169/500 10816/32000 (34%)] Loss: 3.47285 (QuantReg: 22.65261) QuantErr: 22.65261 batch_time=0.38708
Train Epoch: 1 [177/500 11328/32000 (35%)] Loss: 3.92579 (QuantReg: 22.64712) QuantErr: 22.64712 batch_time=0.47100
Train Epoch: 1 [185/500 11840/32000 (37%)] Loss: 3.26597 (QuantReg: 22.61395) QuantErr: 22.61395 batch_time=0.39641
Train Epoch: 1 [193/500 12352/32000 (39%)] Loss: 3.95681 (QuantReg: 22.66005) QuantErr: 22.66005 batch_time=0.39853
Train Epoch: 1 [201/500 12864/32000 (40%)] Loss: 3.98504 (QuantReg: 22.65438) QuantErr: 22.65438 batch_time=0.47840
Train Epoch: 1 [209/500 13376/32000 (42%)] Loss: 4.30273 (QuantReg: 22.66579) QuantErr: 22.66579 batch_time=0.41473
Train Epoch: 1 [217/500 13888/32000 (43%)] Loss: 3.88799 (QuantReg: 22.64647) QuantErr: 22.64647 batch_time=0.44175
Train Epoch: 1 [225/500 14400/32000 (45%)] Loss: 3.63270 (QuantReg: 22.62845) QuantErr: 22.62845 batch_time=0.39661
Train Epoch: 1 [233/500 14912/32000 (47%)] Loss: 3.46709 (QuantReg: 22.68782) QuantErr: 22.68782 batch_time=0.41262
Train Epoch: 1 [241/500 15424/32000 (48%)] Loss: 3.39004 (QuantReg: 22.66500) QuantErr: 22.66500 batch_time=0.40127
Train Epoch: 1 [249/500 15936/32000 (50%)] Loss: 3.98725 (QuantReg: 22.65261) QuantErr: 22.65261 batch_time=0.39512
Train Epoch: 1 [257/500 16448/32000 (51%)] Loss: 3.60587 (QuantReg: 22.64394) QuantErr: 22.64394 batch_time=0.39538
Train Epoch: 1 [265/500 16960/32000 (53%)] Loss: 4.18740 (QuantReg: 22.62760) QuantErr: 22.62760 batch_time=0.42709
Train Epoch: 1 [273/500 17472/32000 (55%)] Loss: 3.73767 (QuantReg: 22.67062) QuantErr: 22.67062 batch_time=0.43085
Train Epoch: 1 [281/500 17984/32000 (56%)] Loss: 3.30122 (QuantReg: 22.69070) QuantErr: 22.69070 batch_time=0.38970
Train Epoch: 1 [289/500 18496/32000 (58%)] Loss: 3.65412 (QuantReg: 22.69288) QuantErr: 22.69288 batch_time=0.39365
Train Epoch: 1 [297/500 19008/32000 (59%)] Loss: 3.83355 (QuantReg: 22.70606) QuantErr: 22.70606 batch_time=0.38270
Train Epoch: 1 [305/500 19520/32000 (61%)] Loss: 3.85061 (QuantReg: 22.68577) QuantErr: 22.68577 batch_time=0.38995
Train Epoch: 1 [313/500 20032/32000 (63%)] Loss: 3.33344 (QuantReg: 22.68880) QuantErr: 22.68880 batch_time=0.47572
Train Epoch: 1 [321/500 20544/32000 (64%)] Loss: 3.78285 (QuantReg: 22.75968) QuantErr: 22.75968 batch_time=0.39571
Train Epoch: 1 [329/500 21056/32000 (66%)] Loss: 3.27215 (QuantReg: 22.71988) QuantErr: 22.71988 batch_time=0.42765
Train Epoch: 1 [337/500 21568/32000 (67%)] Loss: 3.46358 (QuantReg: 22.69718) QuantErr: 22.69718 batch_time=0.39322
Train Epoch: 1 [345/500 22080/32000 (69%)] Loss: 3.32685 (QuantReg: 22.70747) QuantErr: 22.70747 batch_time=0.41136
Train Epoch: 1 [353/500 22592/32000 (71%)] Loss: 3.20432 (QuantReg: 22.66489) QuantErr: 22.66489 batch_time=0.40600
Train Epoch: 1 [361/500 23104/32000 (72%)] Loss: 2.55573 (QuantReg: 22.72517) QuantErr: 22.72517 batch_time=0.41406
Train Epoch: 1 [369/500 23616/32000 (74%)] Loss: 3.13936 (QuantReg: 22.65989) QuantErr: 22.65989 batch_time=0.38943
Train Epoch: 1 [377/500 24128/32000 (75%)] Loss: 4.06648 (QuantReg: 22.65463) QuantErr: 22.65463 batch_time=0.39219
Train Epoch: 1 [385/500 24640/32000 (77%)] Loss: 4.11589 (QuantReg: 22.69662) QuantErr: 22.69662 batch_time=0.38961
Train Epoch: 1 [393/500 25152/32000 (79%)] Loss: 3.52591 (QuantReg: 22.73016) QuantErr: 22.73016 batch_time=0.39039
Train Epoch: 1 [401/500 25664/32000 (80%)] Loss: 2.77390 (QuantReg: 22.70708) QuantErr: 22.70708 batch_time=0.39126
Train Epoch: 1 [409/500 26176/32000 (82%)] Loss: 3.55209 (QuantReg: 22.70922) QuantErr: 22.70922 batch_time=0.38350
Train Epoch: 1 [417/500 26688/32000 (83%)] Loss: 3.07362 (QuantReg: 22.71288) QuantErr: 22.71288 batch_time=0.39478
Train Epoch: 1 [425/500 27200/32000 (85%)] Loss: 3.29908 (QuantReg: 22.67189) QuantErr: 22.67189 batch_time=0.40070
Train Epoch: 1 [433/500 27712/32000 (87%)] Loss: 2.75428 (QuantReg: 22.67542) QuantErr: 22.67542 batch_time=0.40802
Train Epoch: 1 [441/500 28224/32000 (88%)] Loss: 3.16129 (QuantReg: 22.72270) QuantErr: 22.72270 batch_time=0.40225
Train Epoch: 1 [449/500 28736/32000 (90%)] Loss: 2.50772 (QuantReg: 22.71757) QuantErr: 22.71757 batch_time=0.39223
Train Epoch: 1 [457/500 29248/32000 (91%)] Loss: 2.77431 (QuantReg: 22.67332) QuantErr: 22.67332 batch_time=0.38887
Train Epoch: 1 [465/500 29760/32000 (93%)] Loss: 2.97457 (QuantReg: 22.67760) QuantErr: 22.67760 batch_time=0.39077
Train Epoch: 1 [473/500 30272/32000 (95%)] Loss: 2.62806 (QuantReg: 22.64584) QuantErr: 22.64584 batch_time=0.38785
Train Epoch: 1 [481/500 30784/32000 (96%)] Loss: 3.26924 (QuantReg: 22.69742) QuantErr: 22.69742 batch_time=0.39302
Train Epoch: 1 [489/500 31296/32000 (98%)] Loss: 2.72277 (QuantReg: 22.71226) QuantErr: 22.71226 batch_time=0.40016
Train Epoch: 1 [497/500 31808/32000 (99%)] Loss: 2.98632 (QuantReg: 22.65650) QuantErr: 22.65650 batch_time=0.48003
Train Epoch: 1 codebook_update_time=1.86126
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch1.pth ...
Done in 5.164s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch1.pth ...
Done in 9.733s
epoch : 1
loss : 3.9536350874900816
quant_reg : 22.660221336364746
quant_err : 22.660221336364746
learning_rate : 5e-05
n_samples : 32000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 11.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 33.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 46.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 79.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 12.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 41.498
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.67396617781575
MSRVTT_jsfusion_test/v2t_metrics/R1: 11.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 35.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 47.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 79.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 12.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 39.491
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.1930129112048
mnt_best : 25.67396617781575
not_improved_count: 0
Train Epoch: 2 [1/500 64/32000 (0%)] Loss: 2.62006 (QuantReg: 11.22508) QuantErr: 11.22508 batch_time=18.06759
Train Epoch: 2 [9/500 576/32000 (2%)] Loss: 2.59872 (QuantReg: 11.13827) QuantErr: 11.13827 batch_time=0.39103
Train Epoch: 2 [17/500 1088/32000 (3%)] Loss: 2.30567 (QuantReg: 11.42111) QuantErr: 11.42111 batch_time=0.38219
Train Epoch: 2 [25/500 1600/32000 (5%)] Loss: 3.90870 (QuantReg: 11.59308) QuantErr: 11.59308 batch_time=0.38355
Train Epoch: 2 [33/500 2112/32000 (7%)] Loss: 2.56186 (QuantReg: 11.86573) QuantErr: 11.86573 batch_time=0.38926
Train Epoch: 2 [41/500 2624/32000 (8%)] Loss: 3.27057 (QuantReg: 11.29539) QuantErr: 11.29539 batch_time=0.39693
Train Epoch: 2 [49/500 3136/32000 (10%)] Loss: 3.03469 (QuantReg: 11.90690) QuantErr: 11.90690 batch_time=0.39778
Train Epoch: 2 [57/500 3648/32000 (11%)] Loss: 3.05091 (QuantReg: 12.26237) QuantErr: 12.26237 batch_time=0.40057
Train Epoch: 2 [65/500 4160/32000 (13%)] Loss: 3.28774 (QuantReg: 12.35887) QuantErr: 12.35887 batch_time=1.55989
Train Epoch: 2 [73/500 4672/32000 (15%)] Loss: 2.65737 (QuantReg: 12.36531) QuantErr: 12.36531 batch_time=0.39643
Train Epoch: 2 [81/500 5184/32000 (16%)] Loss: 2.03313 (QuantReg: 11.96666) QuantErr: 11.96666 batch_time=0.39405
Train Epoch: 2 [89/500 5696/32000 (18%)] Loss: 2.85253 (QuantReg: 12.42680) QuantErr: 12.42680 batch_time=0.40021
Train Epoch: 2 [97/500 6208/32000 (19%)] Loss: 3.61672 (QuantReg: 12.56135) QuantErr: 12.56135 batch_time=0.40230
Train Epoch: 2 [105/500 6720/32000 (21%)] Loss: 2.68787 (QuantReg: 12.12899) QuantErr: 12.12899 batch_time=0.45061
Train Epoch: 2 [113/500 7232/32000 (23%)] Loss: 2.39090 (QuantReg: 12.50906) QuantErr: 12.50906 batch_time=0.37588
Train Epoch: 2 [121/500 7744/32000 (24%)] Loss: 2.62116 (QuantReg: 12.84318) QuantErr: 12.84318 batch_time=0.38495
Train Epoch: 2 [129/500 8256/32000 (26%)] Loss: 2.63476 (QuantReg: 12.66758) QuantErr: 12.66758 batch_time=1.66029
Train Epoch: 2 [137/500 8768/32000 (27%)] Loss: 2.67035 (QuantReg: 12.51559) QuantErr: 12.51559 batch_time=0.37810
Train Epoch: 2 [145/500 9280/32000 (29%)] Loss: 2.30875 (QuantReg: 12.34239) QuantErr: 12.34239 batch_time=0.39307
Train Epoch: 2 [153/500 9792/32000 (31%)] Loss: 2.90315 (QuantReg: 12.88103) QuantErr: 12.88103 batch_time=0.39579
Train Epoch: 2 [161/500 10304/32000 (32%)] Loss: 2.34022 (QuantReg: 12.78667) QuantErr: 12.78667 batch_time=0.40168
Train Epoch: 2 [169/500 10816/32000 (34%)] Loss: 2.01825 (QuantReg: 12.78219) QuantErr: 12.78219 batch_time=0.40620
Train Epoch: 2 [177/500 11328/32000 (35%)] Loss: 2.16241 (QuantReg: 13.12553) QuantErr: 13.12553 batch_time=0.39852
Train Epoch: 2 [185/500 11840/32000 (37%)] Loss: 3.06039 (QuantReg: 12.70366) QuantErr: 12.70366 batch_time=0.39286
Train Epoch: 2 [193/500 12352/32000 (39%)] Loss: 2.32288 (QuantReg: 12.42189) QuantErr: 12.42189 batch_time=1.77457
Train Epoch: 2 [201/500 12864/32000 (40%)] Loss: 3.11832 (QuantReg: 12.59051) QuantErr: 12.59051 batch_time=0.44406
Train Epoch: 2 [209/500 13376/32000 (42%)] Loss: 2.98633 (QuantReg: 12.90093) QuantErr: 12.90093 batch_time=0.41088
Train Epoch: 2 [217/500 13888/32000 (43%)] Loss: 2.69851 (QuantReg: 12.79753) QuantErr: 12.79753 batch_time=0.39185
Train Epoch: 2 [225/500 14400/32000 (45%)] Loss: 3.67118 (QuantReg: 13.10180) QuantErr: 13.10180 batch_time=0.39434
Train Epoch: 2 [233/500 14912/32000 (47%)] Loss: 2.85293 (QuantReg: 12.79118) QuantErr: 12.79118 batch_time=0.39657
Train Epoch: 2 [241/500 15424/32000 (48%)] Loss: 2.30044 (QuantReg: 12.77420) QuantErr: 12.77420 batch_time=0.42304
Train Epoch: 2 [249/500 15936/32000 (50%)] Loss: 2.99484 (QuantReg: 12.93721) QuantErr: 12.93721 batch_time=0.39523
Train Epoch: 2 [257/500 16448/32000 (51%)] Loss: 2.51131 (QuantReg: 13.11358) QuantErr: 13.11358 batch_time=1.57547
Train Epoch: 2 [265/500 16960/32000 (53%)] Loss: 2.30855 (QuantReg: 13.02362) QuantErr: 13.02362 batch_time=0.39858
Train Epoch: 2 [273/500 17472/32000 (55%)] Loss: 2.88098 (QuantReg: 13.17628) QuantErr: 13.17628 batch_time=0.40179
Train Epoch: 2 [281/500 17984/32000 (56%)] Loss: 2.66678 (QuantReg: 13.67676) QuantErr: 13.67676 batch_time=0.39681
Train Epoch: 2 [289/500 18496/32000 (58%)] Loss: 2.14355 (QuantReg: 13.52485) QuantErr: 13.52485 batch_time=0.40145
Train Epoch: 2 [297/500 19008/32000 (59%)] Loss: 2.73268 (QuantReg: 13.33243) QuantErr: 13.33243 batch_time=0.45286
Train Epoch: 2 [305/500 19520/32000 (61%)] Loss: 3.37707 (QuantReg: 13.19781) QuantErr: 13.19781 batch_time=0.40463
Train Epoch: 2 [313/500 20032/32000 (63%)] Loss: 2.95430 (QuantReg: 13.27989) QuantErr: 13.27989 batch_time=0.39609
Train Epoch: 2 [321/500 20544/32000 (64%)] Loss: 3.12812 (QuantReg: 13.50870) QuantErr: 13.50870 batch_time=1.54456
Train Epoch: 2 [329/500 21056/32000 (66%)] Loss: 1.99385 (QuantReg: 13.45107) QuantErr: 13.45107 batch_time=0.42275
Train Epoch: 2 [337/500 21568/32000 (67%)] Loss: 2.57126 (QuantReg: 13.62480) QuantErr: 13.62480 batch_time=0.39648
Train Epoch: 2 [345/500 22080/32000 (69%)] Loss: 3.02131 (QuantReg: 14.00862) QuantErr: 14.00862 batch_time=0.44398
Train Epoch: 2 [353/500 22592/32000 (71%)] Loss: 2.23672 (QuantReg: 13.47744) QuantErr: 13.47744 batch_time=0.40163
Train Epoch: 2 [361/500 23104/32000 (72%)] Loss: 2.25001 (QuantReg: 14.08245) QuantErr: 14.08245 batch_time=0.40568
Train Epoch: 2 [369/500 23616/32000 (74%)] Loss: 2.01165 (QuantReg: 13.92245) QuantErr: 13.92245 batch_time=0.40579
Train Epoch: 2 [377/500 24128/32000 (75%)] Loss: 2.99366 (QuantReg: 13.58927) QuantErr: 13.58927 batch_time=0.40826
Train Epoch: 2 [385/500 24640/32000 (77%)] Loss: 2.64802 (QuantReg: 13.78279) QuantErr: 13.78279 batch_time=1.57718
Train Epoch: 2 [393/500 25152/32000 (79%)] Loss: 2.19498 (QuantReg: 13.83376) QuantErr: 13.83376 batch_time=0.43771
Train Epoch: 2 [401/500 25664/32000 (80%)] Loss: 2.56817 (QuantReg: 13.83684) QuantErr: 13.83684 batch_time=0.41372
Train Epoch: 2 [409/500 26176/32000 (82%)] Loss: 2.43078 (QuantReg: 13.51015) QuantErr: 13.51015 batch_time=0.40523
Train Epoch: 2 [417/500 26688/32000 (83%)] Loss: 2.41594 (QuantReg: 13.55619) QuantErr: 13.55619 batch_time=0.41230
Train Epoch: 2 [425/500 27200/32000 (85%)] Loss: 2.39532 (QuantReg: 13.74899) QuantErr: 13.74899 batch_time=0.39662
Train Epoch: 2 [433/500 27712/32000 (87%)] Loss: 2.23112 (QuantReg: 13.88592) QuantErr: 13.88592 batch_time=0.38973
Train Epoch: 2 [441/500 28224/32000 (88%)] Loss: 2.53970 (QuantReg: 13.92415) QuantErr: 13.92415 batch_time=0.39600
Train Epoch: 2 [449/500 28736/32000 (90%)] Loss: 2.38390 (QuantReg: 14.21874) QuantErr: 14.21874 batch_time=1.82011
Train Epoch: 2 [457/500 29248/32000 (91%)] Loss: 1.78954 (QuantReg: 14.30077) QuantErr: 14.30077 batch_time=0.39540
Train Epoch: 2 [465/500 29760/32000 (93%)] Loss: 2.48937 (QuantReg: 14.47411) QuantErr: 14.47411 batch_time=0.39023
Train Epoch: 2 [473/500 30272/32000 (95%)] Loss: 2.86483 (QuantReg: 13.79661) QuantErr: 13.79661 batch_time=0.40259
Train Epoch: 2 [481/500 30784/32000 (96%)] Loss: 2.14529 (QuantReg: 14.56754) QuantErr: 14.56754 batch_time=0.41599
Train Epoch: 2 [489/500 31296/32000 (98%)] Loss: 2.77052 (QuantReg: 14.07403) QuantErr: 14.07403 batch_time=0.40093
Train Epoch: 2 [497/500 31808/32000 (99%)] Loss: 2.30852 (QuantReg: 14.81379) QuantErr: 14.81379 batch_time=0.39565
Train Epoch: 2 codebook_update_time=1.72339
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch2.pth ...
Done in 3.889s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch2.pth ...
Done in 7.716s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.03s]
epoch : 2
loss : 2.6417448403835295
quant_reg : 13.049937316894532
quant_err : 13.049937316894532
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 13.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 39.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 53.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 33.843
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.677772180283238
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 40.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 55.4
MSRVTT_jsfusion_test/v2t_metrics/R50: 84.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 9.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 31.636
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.03602453372839
mnt_best : 30.677772180283238
not_improved_count: 0
Train Epoch: 3 [1/500 64/32000 (0%)] Loss: 2.26150 (QuantReg: 12.07461) QuantErr: 12.07461 batch_time=16.06029
Train Epoch: 3 [9/500 576/32000 (2%)] Loss: 2.65554 (QuantReg: 11.09809) QuantErr: 11.09809 batch_time=0.38707
Train Epoch: 3 [17/500 1088/32000 (3%)] Loss: 2.63913 (QuantReg: 11.19456) QuantErr: 11.19456 batch_time=0.39386
Train Epoch: 3 [25/500 1600/32000 (5%)] Loss: 2.17775 (QuantReg: 11.29367) QuantErr: 11.29367 batch_time=0.38637
Train Epoch: 3 [33/500 2112/32000 (7%)] Loss: 2.93336 (QuantReg: 11.18336) QuantErr: 11.18336 batch_time=0.41953
Train Epoch: 3 [41/500 2624/32000 (8%)] Loss: 2.63240 (QuantReg: 11.54626) QuantErr: 11.54626 batch_time=0.39270
Train Epoch: 3 [49/500 3136/32000 (10%)] Loss: 2.56952 (QuantReg: 11.43377) QuantErr: 11.43377 batch_time=0.39229
Train Epoch: 3 [57/500 3648/32000 (11%)] Loss: 2.47020 (QuantReg: 11.44704) QuantErr: 11.44704 batch_time=0.40134
Train Epoch: 3 [65/500 4160/32000 (13%)] Loss: 2.96780 (QuantReg: 11.89394) QuantErr: 11.89394 batch_time=0.39490
Train Epoch: 3 [73/500 4672/32000 (15%)] Loss: 2.18685 (QuantReg: 11.45894) QuantErr: 11.45894 batch_time=0.39741
Train Epoch: 3 [81/500 5184/32000 (16%)] Loss: 2.64047 (QuantReg: 11.64688) QuantErr: 11.64688 batch_time=0.40119
Train Epoch: 3 [89/500 5696/32000 (18%)] Loss: 2.18978 (QuantReg: 12.06723) QuantErr: 12.06723 batch_time=0.39402
Train Epoch: 3 [97/500 6208/32000 (19%)] Loss: 2.29623 (QuantReg: 11.67583) QuantErr: 11.67583 batch_time=0.38777
Train Epoch: 3 [105/500 6720/32000 (21%)] Loss: 2.48309 (QuantReg: 12.28813) QuantErr: 12.28813 batch_time=0.38311
Train Epoch: 3 [113/500 7232/32000 (23%)] Loss: 1.89286 (QuantReg: 11.92930) QuantErr: 11.92930 batch_time=0.41406
Train Epoch: 3 [121/500 7744/32000 (24%)] Loss: 2.27024 (QuantReg: 11.66505) QuantErr: 11.66505 batch_time=0.38816
Train Epoch: 3 [129/500 8256/32000 (26%)] Loss: 2.14224 (QuantReg: 11.98564) QuantErr: 11.98564 batch_time=0.39619
Train Epoch: 3 [137/500 8768/32000 (27%)] Loss: 2.18910 (QuantReg: 11.80920) QuantErr: 11.80920 batch_time=0.39919
Train Epoch: 3 [145/500 9280/32000 (29%)] Loss: 2.78102 (QuantReg: 11.94994) QuantErr: 11.94994 batch_time=0.40003
Train Epoch: 3 [153/500 9792/32000 (31%)] Loss: 2.04750 (QuantReg: 12.11440) QuantErr: 12.11440 batch_time=0.40199
Train Epoch: 3 [161/500 10304/32000 (32%)] Loss: 2.23332 (QuantReg: 12.06767) QuantErr: 12.06767 batch_time=0.39648
Train Epoch: 3 [169/500 10816/32000 (34%)] Loss: 2.23798 (QuantReg: 12.16653) QuantErr: 12.16653 batch_time=0.39084
Train Epoch: 3 [177/500 11328/32000 (35%)] Loss: 1.91046 (QuantReg: 12.14290) QuantErr: 12.14290 batch_time=0.40418
Train Epoch: 3 [185/500 11840/32000 (37%)] Loss: 2.46068 (QuantReg: 12.12275) QuantErr: 12.12275 batch_time=0.38547
Train Epoch: 3 [193/500 12352/32000 (39%)] Loss: 2.19139 (QuantReg: 12.85945) QuantErr: 12.85945 batch_time=0.38965
Train Epoch: 3 [201/500 12864/32000 (40%)] Loss: 2.67873 (QuantReg: 12.10293) QuantErr: 12.10293 batch_time=0.38739
Train Epoch: 3 [209/500 13376/32000 (42%)] Loss: 1.91280 (QuantReg: 12.68606) QuantErr: 12.68606 batch_time=0.39681
Train Epoch: 3 [217/500 13888/32000 (43%)] Loss: 2.47704 (QuantReg: 12.53888) QuantErr: 12.53888 batch_time=0.39162
Train Epoch: 3 [225/500 14400/32000 (45%)] Loss: 2.22522 (QuantReg: 12.03703) QuantErr: 12.03703 batch_time=0.38743
Train Epoch: 3 [233/500 14912/32000 (47%)] Loss: 1.93388 (QuantReg: 12.49670) QuantErr: 12.49670 batch_time=0.38776
Train Epoch: 3 [241/500 15424/32000 (48%)] Loss: 2.53157 (QuantReg: 12.48556) QuantErr: 12.48556 batch_time=0.38344
Train Epoch: 3 [249/500 15936/32000 (50%)] Loss: 1.94907 (QuantReg: 12.45824) QuantErr: 12.45824 batch_time=0.38768
Train Epoch: 3 [257/500 16448/32000 (51%)] Loss: 2.33750 (QuantReg: 12.39463) QuantErr: 12.39463 batch_time=0.37682
Train Epoch: 3 [265/500 16960/32000 (53%)] Loss: 2.46366 (QuantReg: 12.62819) QuantErr: 12.62819 batch_time=0.41602
Train Epoch: 3 [273/500 17472/32000 (55%)] Loss: 2.50957 (QuantReg: 12.25123) QuantErr: 12.25123 batch_time=0.39458
Train Epoch: 3 [281/500 17984/32000 (56%)] Loss: 2.25424 (QuantReg: 11.99255) QuantErr: 11.99255 batch_time=0.39301
Train Epoch: 3 [289/500 18496/32000 (58%)] Loss: 1.86128 (QuantReg: 12.63934) QuantErr: 12.63934 batch_time=0.39182
Train Epoch: 3 [297/500 19008/32000 (59%)] Loss: 2.32604 (QuantReg: 12.77562) QuantErr: 12.77562 batch_time=0.42636
Train Epoch: 3 [305/500 19520/32000 (61%)] Loss: 2.21253 (QuantReg: 12.29757) QuantErr: 12.29757 batch_time=0.39287
Train Epoch: 3 [313/500 20032/32000 (63%)] Loss: 1.97456 (QuantReg: 12.35938) QuantErr: 12.35938 batch_time=0.39223
Train Epoch: 3 [321/500 20544/32000 (64%)] Loss: 2.00990 (QuantReg: 12.60898) QuantErr: 12.60898 batch_time=0.39224
Train Epoch: 3 [329/500 21056/32000 (66%)] Loss: 2.03271 (QuantReg: 12.98077) QuantErr: 12.98077 batch_time=0.38751
Train Epoch: 3 [337/500 21568/32000 (67%)] Loss: 2.33176 (QuantReg: 12.61633) QuantErr: 12.61633 batch_time=0.38893
Train Epoch: 3 [345/500 22080/32000 (69%)] Loss: 2.31252 (QuantReg: 13.10382) QuantErr: 13.10382 batch_time=0.40046
Train Epoch: 3 [353/500 22592/32000 (71%)] Loss: 1.98488 (QuantReg: 12.52270) QuantErr: 12.52270 batch_time=0.39886
Train Epoch: 3 [361/500 23104/32000 (72%)] Loss: 2.16462 (QuantReg: 12.74782) QuantErr: 12.74782 batch_time=0.46145
Train Epoch: 3 [369/500 23616/32000 (74%)] Loss: 2.70604 (QuantReg: 12.10294) QuantErr: 12.10294 batch_time=0.38783
Train Epoch: 3 [377/500 24128/32000 (75%)] Loss: 2.23372 (QuantReg: 12.45160) QuantErr: 12.45160 batch_time=0.39825
Train Epoch: 3 [385/500 24640/32000 (77%)] Loss: 1.85210 (QuantReg: 12.60809) QuantErr: 12.60809 batch_time=0.40209
Train Epoch: 3 [393/500 25152/32000 (79%)] Loss: 2.17962 (QuantReg: 13.38482) QuantErr: 13.38482 batch_time=0.38811
Train Epoch: 3 [401/500 25664/32000 (80%)] Loss: 2.02209 (QuantReg: 12.79121) QuantErr: 12.79121 batch_time=0.38756
Train Epoch: 3 [409/500 26176/32000 (82%)] Loss: 1.82478 (QuantReg: 13.14603) QuantErr: 13.14603 batch_time=0.38887
Train Epoch: 3 [417/500 26688/32000 (83%)] Loss: 2.15508 (QuantReg: 12.86783) QuantErr: 12.86783 batch_time=0.44051
Train Epoch: 3 [425/500 27200/32000 (85%)] Loss: 2.18833 (QuantReg: 13.08014) QuantErr: 13.08014 batch_time=0.39262
Train Epoch: 3 [433/500 27712/32000 (87%)] Loss: 2.80134 (QuantReg: 12.37905) QuantErr: 12.37905 batch_time=0.39449
Train Epoch: 3 [441/500 28224/32000 (88%)] Loss: 2.38591 (QuantReg: 12.65982) QuantErr: 12.65982 batch_time=0.43099
Train Epoch: 3 [449/500 28736/32000 (90%)] Loss: 1.85486 (QuantReg: 12.88773) QuantErr: 12.88773 batch_time=0.40004
Train Epoch: 3 [457/500 29248/32000 (91%)] Loss: 1.87030 (QuantReg: 13.22868) QuantErr: 13.22868 batch_time=0.40655
Train Epoch: 3 [465/500 29760/32000 (93%)] Loss: 2.06836 (QuantReg: 13.11696) QuantErr: 13.11696 batch_time=0.39698
Train Epoch: 3 [473/500 30272/32000 (95%)] Loss: 1.82063 (QuantReg: 12.76066) QuantErr: 12.76066 batch_time=0.38670
Train Epoch: 3 [481/500 30784/32000 (96%)] Loss: 1.60773 (QuantReg: 12.78868) QuantErr: 12.78868 batch_time=0.53095
Train Epoch: 3 [489/500 31296/32000 (98%)] Loss: 3.03546 (QuantReg: 13.08733) QuantErr: 13.08733 batch_time=0.39424
Train Epoch: 3 [497/500 31808/32000 (99%)] Loss: 1.98842 (QuantReg: 12.86212) QuantErr: 12.86212 batch_time=0.38116
Train Epoch: 3 codebook_update_time=1.79340
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch3.pth ...
Done in 3.936s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch3.pth ...
Done in 7.646s
removing stale ckpt [epoch 2] [took 0.04s]
epoch : 3
loss : 2.218185573339462
quant_reg : 12.347946405410767
quant_err : 12.347946405410767
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 32.469
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.336879844876535
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.1
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 57.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.5125
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.17044001421866
mnt_best : 33.336879844876535
not_improved_count: 0
Train Epoch: 4 [1/500 64/32000 (0%)] Loss: 2.65958 (QuantReg: 11.66044) QuantErr: 11.66044 batch_time=17.15469
Train Epoch: 4 [9/500 576/32000 (2%)] Loss: 1.27732 (QuantReg: 11.60704) QuantErr: 11.60704 batch_time=0.42286
Train Epoch: 4 [17/500 1088/32000 (3%)] Loss: 1.94137 (QuantReg: 11.72709) QuantErr: 11.72709 batch_time=0.38883
Train Epoch: 4 [25/500 1600/32000 (5%)] Loss: 2.26076 (QuantReg: 11.80330) QuantErr: 11.80330 batch_time=0.42081
Train Epoch: 4 [33/500 2112/32000 (7%)] Loss: 2.28611 (QuantReg: 11.65627) QuantErr: 11.65627 batch_time=0.38897
Train Epoch: 4 [41/500 2624/32000 (8%)] Loss: 2.18525 (QuantReg: 12.12088) QuantErr: 12.12088 batch_time=0.41717
Train Epoch: 4 [49/500 3136/32000 (10%)] Loss: 1.97816 (QuantReg: 11.70557) QuantErr: 11.70557 batch_time=0.39011
Train Epoch: 4 [57/500 3648/32000 (11%)] Loss: 2.46010 (QuantReg: 11.90413) QuantErr: 11.90413 batch_time=0.45708
Train Epoch: 4 [65/500 4160/32000 (13%)] Loss: 1.69951 (QuantReg: 12.15346) QuantErr: 12.15346 batch_time=0.39222
Train Epoch: 4 [73/500 4672/32000 (15%)] Loss: 1.26014 (QuantReg: 12.56578) QuantErr: 12.56578 batch_time=0.39166
Train Epoch: 4 [81/500 5184/32000 (16%)] Loss: 1.94529 (QuantReg: 12.27109) QuantErr: 12.27109 batch_time=0.43972
Train Epoch: 4 [89/500 5696/32000 (18%)] Loss: 1.99649 (QuantReg: 12.28611) QuantErr: 12.28611 batch_time=0.41639
Train Epoch: 4 [97/500 6208/32000 (19%)] Loss: 1.86212 (QuantReg: 12.07768) QuantErr: 12.07768 batch_time=0.39699
Train Epoch: 4 [105/500 6720/32000 (21%)] Loss: 1.40626 (QuantReg: 11.83904) QuantErr: 11.83904 batch_time=0.39454
Train Epoch: 4 [113/500 7232/32000 (23%)] Loss: 2.18242 (QuantReg: 12.12382) QuantErr: 12.12382 batch_time=0.39799
Train Epoch: 4 [121/500 7744/32000 (24%)] Loss: 2.38294 (QuantReg: 12.46361) QuantErr: 12.46361 batch_time=0.39808
Train Epoch: 4 [129/500 8256/32000 (26%)] Loss: 2.07724 (QuantReg: 12.18068) QuantErr: 12.18068 batch_time=0.39609
Train Epoch: 4 [137/500 8768/32000 (27%)] Loss: 1.73774 (QuantReg: 12.27234) QuantErr: 12.27234 batch_time=0.39152
Train Epoch: 4 [145/500 9280/32000 (29%)] Loss: 2.20963 (QuantReg: 12.45326) QuantErr: 12.45326 batch_time=0.39131
Train Epoch: 4 [153/500 9792/32000 (31%)] Loss: 2.04822 (QuantReg: 11.67189) QuantErr: 11.67189 batch_time=0.38791
Train Epoch: 4 [161/500 10304/32000 (32%)] Loss: 2.08082 (QuantReg: 11.67893) QuantErr: 11.67893 batch_time=0.39178
Train Epoch: 4 [169/500 10816/32000 (34%)] Loss: 2.15586 (QuantReg: 12.13877) QuantErr: 12.13877 batch_time=0.40900
Train Epoch: 4 [177/500 11328/32000 (35%)] Loss: 2.21005 (QuantReg: 11.92683) QuantErr: 11.92683 batch_time=0.39697
Train Epoch: 4 [185/500 11840/32000 (37%)] Loss: 2.99855 (QuantReg: 11.90897) QuantErr: 11.90897 batch_time=0.38763
Train Epoch: 4 [193/500 12352/32000 (39%)] Loss: 2.20298 (QuantReg: 12.05097) QuantErr: 12.05097 batch_time=0.40964
Train Epoch: 4 [201/500 12864/32000 (40%)] Loss: 2.31996 (QuantReg: 12.07165) QuantErr: 12.07165 batch_time=0.38951
Train Epoch: 4 [209/500 13376/32000 (42%)] Loss: 2.34675 (QuantReg: 11.92373) QuantErr: 11.92373 batch_time=0.39299
Train Epoch: 4 [217/500 13888/32000 (43%)] Loss: 1.50377 (QuantReg: 11.84177) QuantErr: 11.84177 batch_time=0.39573
Train Epoch: 4 [225/500 14400/32000 (45%)] Loss: 2.26900 (QuantReg: 12.08553) QuantErr: 12.08553 batch_time=0.38783
Train Epoch: 4 [233/500 14912/32000 (47%)] Loss: 1.83625 (QuantReg: 11.75072) QuantErr: 11.75072 batch_time=0.38815
Train Epoch: 4 [241/500 15424/32000 (48%)] Loss: 2.15387 (QuantReg: 12.13605) QuantErr: 12.13605 batch_time=0.41074
Train Epoch: 4 [249/500 15936/32000 (50%)] Loss: 2.24192 (QuantReg: 12.41631) QuantErr: 12.41631 batch_time=0.40655
Train Epoch: 4 [257/500 16448/32000 (51%)] Loss: 1.76788 (QuantReg: 12.20451) QuantErr: 12.20451 batch_time=0.42076
Train Epoch: 4 [265/500 16960/32000 (53%)] Loss: 1.87224 (QuantReg: 11.97675) QuantErr: 11.97675 batch_time=0.39341
Train Epoch: 4 [273/500 17472/32000 (55%)] Loss: 1.93500 (QuantReg: 12.43371) QuantErr: 12.43371 batch_time=0.38172
Train Epoch: 4 [281/500 17984/32000 (56%)] Loss: 1.68349 (QuantReg: 12.41338) QuantErr: 12.41338 batch_time=0.38094
Train Epoch: 4 [289/500 18496/32000 (58%)] Loss: 1.67667 (QuantReg: 12.30222) QuantErr: 12.30222 batch_time=0.38691
Train Epoch: 4 [297/500 19008/32000 (59%)] Loss: 1.82507 (QuantReg: 12.28812) QuantErr: 12.28812 batch_time=0.39087
Train Epoch: 4 [305/500 19520/32000 (61%)] Loss: 2.59729 (QuantReg: 12.11448) QuantErr: 12.11448 batch_time=0.41520
Train Epoch: 4 [313/500 20032/32000 (63%)] Loss: 1.54347 (QuantReg: 12.31537) QuantErr: 12.31537 batch_time=0.43743
Train Epoch: 4 [321/500 20544/32000 (64%)] Loss: 1.71076 (QuantReg: 12.11332) QuantErr: 12.11332 batch_time=0.39475
Train Epoch: 4 [329/500 21056/32000 (66%)] Loss: 1.41449 (QuantReg: 12.57907) QuantErr: 12.57907 batch_time=0.38947
Train Epoch: 4 [337/500 21568/32000 (67%)] Loss: 2.22226 (QuantReg: 12.58026) QuantErr: 12.58026 batch_time=0.40268
Train Epoch: 4 [345/500 22080/32000 (69%)] Loss: 1.61972 (QuantReg: 12.87900) QuantErr: 12.87900 batch_time=0.40123
Train Epoch: 4 [353/500 22592/32000 (71%)] Loss: 1.92744 (QuantReg: 12.28973) QuantErr: 12.28973 batch_time=0.39495
Train Epoch: 4 [361/500 23104/32000 (72%)] Loss: 1.89362 (QuantReg: 11.97363) QuantErr: 11.97363 batch_time=0.39469
Train Epoch: 4 [369/500 23616/32000 (74%)] Loss: 1.80776 (QuantReg: 12.63929) QuantErr: 12.63929 batch_time=0.40677
Train Epoch: 4 [377/500 24128/32000 (75%)] Loss: 1.98841 (QuantReg: 12.62455) QuantErr: 12.62455 batch_time=0.40115
Train Epoch: 4 [385/500 24640/32000 (77%)] Loss: 2.18255 (QuantReg: 12.13752) QuantErr: 12.13752 batch_time=0.39469
Train Epoch: 4 [393/500 25152/32000 (79%)] Loss: 2.06225 (QuantReg: 12.68102) QuantErr: 12.68102 batch_time=0.38649
Train Epoch: 4 [401/500 25664/32000 (80%)] Loss: 2.24087 (QuantReg: 12.28492) QuantErr: 12.28492 batch_time=0.38634
Train Epoch: 4 [409/500 26176/32000 (82%)] Loss: 1.66890 (QuantReg: 12.30773) QuantErr: 12.30773 batch_time=0.39236
Train Epoch: 4 [417/500 26688/32000 (83%)] Loss: 1.67920 (QuantReg: 12.77246) QuantErr: 12.77246 batch_time=0.39706
Train Epoch: 4 [425/500 27200/32000 (85%)] Loss: 1.85364 (QuantReg: 12.73693) QuantErr: 12.73693 batch_time=0.38876
Train Epoch: 4 [433/500 27712/32000 (87%)] Loss: 2.09155 (QuantReg: 12.68241) QuantErr: 12.68241 batch_time=0.71234
Train Epoch: 4 [441/500 28224/32000 (88%)] Loss: 2.44958 (QuantReg: 12.66527) QuantErr: 12.66527 batch_time=0.38763
Train Epoch: 4 [449/500 28736/32000 (90%)] Loss: 1.64286 (QuantReg: 12.69690) QuantErr: 12.69690 batch_time=0.40984
Train Epoch: 4 [457/500 29248/32000 (91%)] Loss: 1.91294 (QuantReg: 12.36266) QuantErr: 12.36266 batch_time=0.39466
Train Epoch: 4 [465/500 29760/32000 (93%)] Loss: 2.34686 (QuantReg: 12.85605) QuantErr: 12.85605 batch_time=0.39897
Train Epoch: 4 [473/500 30272/32000 (95%)] Loss: 1.58185 (QuantReg: 12.92751) QuantErr: 12.92751 batch_time=0.39618
Train Epoch: 4 [481/500 30784/32000 (96%)] Loss: 2.09248 (QuantReg: 12.86954) QuantErr: 12.86954 batch_time=0.41703
Train Epoch: 4 [489/500 31296/32000 (98%)] Loss: 1.42793 (QuantReg: 12.64288) QuantErr: 12.64288 batch_time=0.39496
Train Epoch: 4 [497/500 31808/32000 (99%)] Loss: 1.71238 (QuantReg: 12.75778) QuantErr: 12.75778 batch_time=0.39838
Train Epoch: 4 codebook_update_time=1.77082
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch4.pth ...
Done in 20.369s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch4.pth ...
Done in 24.049s
removing stale ckpt [epoch 3] [took 0.02s]
epoch : 4
loss : 1.9638446147441864
quant_reg : 12.280205457687378
quant_err : 12.280205457687378
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 57.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.283
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.12588025655842
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.803
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.262127711713795
mnt_best : 34.12588025655842
not_improved_count: 0
Train Epoch: 5 [1/500 64/32000 (0%)] Loss: 2.09283 (QuantReg: 11.43460) QuantErr: 11.43460 batch_time=17.49794
Train Epoch: 5 [9/500 576/32000 (2%)] Loss: 2.68232 (QuantReg: 11.55963) QuantErr: 11.55963 batch_time=0.38511
Train Epoch: 5 [17/500 1088/32000 (3%)] Loss: 2.16102 (QuantReg: 11.41272) QuantErr: 11.41272 batch_time=0.40568
Train Epoch: 5 [25/500 1600/32000 (5%)] Loss: 1.46024 (QuantReg: 11.67547) QuantErr: 11.67547 batch_time=0.38334
Train Epoch: 5 [33/500 2112/32000 (7%)] Loss: 2.02078 (QuantReg: 11.81323) QuantErr: 11.81323 batch_time=0.38503
Train Epoch: 5 [41/500 2624/32000 (8%)] Loss: 2.45792 (QuantReg: 12.27644) QuantErr: 12.27644 batch_time=0.57072
Train Epoch: 5 [49/500 3136/32000 (10%)] Loss: 1.92939 (QuantReg: 12.12812) QuantErr: 12.12812 batch_time=0.40416
Train Epoch: 5 [57/500 3648/32000 (11%)] Loss: 1.99413 (QuantReg: 11.89506) QuantErr: 11.89506 batch_time=0.38875
Train Epoch: 5 [65/500 4160/32000 (13%)] Loss: 2.24660 (QuantReg: 11.89293) QuantErr: 11.89293 batch_time=0.39394
Train Epoch: 5 [73/500 4672/32000 (15%)] Loss: 1.90551 (QuantReg: 12.18754) QuantErr: 12.18754 batch_time=0.39695
Train Epoch: 5 [81/500 5184/32000 (16%)] Loss: 2.02547 (QuantReg: 11.75890) QuantErr: 11.75890 batch_time=0.39441
Train Epoch: 5 [89/500 5696/32000 (18%)] Loss: 2.19014 (QuantReg: 11.54982) QuantErr: 11.54982 batch_time=0.43509
Train Epoch: 5 [97/500 6208/32000 (19%)] Loss: 2.07452 (QuantReg: 11.80665) QuantErr: 11.80665 batch_time=0.43439
Train Epoch: 5 [105/500 6720/32000 (21%)] Loss: 2.21090 (QuantReg: 12.01538) QuantErr: 12.01538 batch_time=0.50993
Train Epoch: 5 [113/500 7232/32000 (23%)] Loss: 1.50722 (QuantReg: 11.97296) QuantErr: 11.97296 batch_time=0.41283
Train Epoch: 5 [121/500 7744/32000 (24%)] Loss: 1.63929 (QuantReg: 12.11858) QuantErr: 12.11858 batch_time=0.42001
Train Epoch: 5 [129/500 8256/32000 (26%)] Loss: 1.99086 (QuantReg: 12.23107) QuantErr: 12.23107 batch_time=0.45953
Train Epoch: 5 [137/500 8768/32000 (27%)] Loss: 1.76127 (QuantReg: 12.06500) QuantErr: 12.06500 batch_time=0.39824
Train Epoch: 5 [145/500 9280/32000 (29%)] Loss: 1.75368 (QuantReg: 12.11875) QuantErr: 12.11875 batch_time=0.41102
Train Epoch: 5 [153/500 9792/32000 (31%)] Loss: 1.61636 (QuantReg: 12.47781) QuantErr: 12.47781 batch_time=0.38600
Train Epoch: 5 [161/500 10304/32000 (32%)] Loss: 1.71456 (QuantReg: 12.43005) QuantErr: 12.43005 batch_time=0.43217
Train Epoch: 5 [169/500 10816/32000 (34%)] Loss: 1.80098 (QuantReg: 12.12861) QuantErr: 12.12861 batch_time=0.49853
Train Epoch: 5 [177/500 11328/32000 (35%)] Loss: 1.55229 (QuantReg: 12.56070) QuantErr: 12.56070 batch_time=0.39884
Train Epoch: 5 [185/500 11840/32000 (37%)] Loss: 2.06403 (QuantReg: 12.44549) QuantErr: 12.44549 batch_time=0.38603
Train Epoch: 5 [193/500 12352/32000 (39%)] Loss: 1.65255 (QuantReg: 12.15595) QuantErr: 12.15595 batch_time=0.39310
Train Epoch: 5 [201/500 12864/32000 (40%)] Loss: 2.05796 (QuantReg: 12.13556) QuantErr: 12.13556 batch_time=0.39499
Train Epoch: 5 [209/500 13376/32000 (42%)] Loss: 1.75415 (QuantReg: 12.35286) QuantErr: 12.35286 batch_time=0.39049
Train Epoch: 5 [217/500 13888/32000 (43%)] Loss: 1.60993 (QuantReg: 12.28843) QuantErr: 12.28843 batch_time=0.39273
Train Epoch: 5 [225/500 14400/32000 (45%)] Loss: 1.80466 (QuantReg: 12.43720) QuantErr: 12.43720 batch_time=0.45033
Train Epoch: 5 [233/500 14912/32000 (47%)] Loss: 1.74580 (QuantReg: 12.53188) QuantErr: 12.53188 batch_time=0.49024
Train Epoch: 5 [241/500 15424/32000 (48%)] Loss: 1.74060 (QuantReg: 12.30914) QuantErr: 12.30914 batch_time=0.37847
Train Epoch: 5 [249/500 15936/32000 (50%)] Loss: 1.77846 (QuantReg: 12.86011) QuantErr: 12.86011 batch_time=0.39018
Train Epoch: 5 [257/500 16448/32000 (51%)] Loss: 2.14054 (QuantReg: 12.53497) QuantErr: 12.53497 batch_time=0.39200
Train Epoch: 5 [265/500 16960/32000 (53%)] Loss: 2.09205 (QuantReg: 12.75517) QuantErr: 12.75517 batch_time=0.38794
Train Epoch: 5 [273/500 17472/32000 (55%)] Loss: 1.14447 (QuantReg: 12.22155) QuantErr: 12.22155 batch_time=0.39059
Train Epoch: 5 [281/500 17984/32000 (56%)] Loss: 2.24775 (QuantReg: 12.28502) QuantErr: 12.28502 batch_time=0.38036
Train Epoch: 5 [289/500 18496/32000 (58%)] Loss: 2.47481 (QuantReg: 12.59245) QuantErr: 12.59245 batch_time=0.51804
Train Epoch: 5 [297/500 19008/32000 (59%)] Loss: 1.23164 (QuantReg: 12.90435) QuantErr: 12.90435 batch_time=0.50087
Train Epoch: 5 [305/500 19520/32000 (61%)] Loss: 1.31532 (QuantReg: 12.64684) QuantErr: 12.64684 batch_time=0.39493
Train Epoch: 5 [313/500 20032/32000 (63%)] Loss: 1.72958 (QuantReg: 12.07564) QuantErr: 12.07564 batch_time=0.39628
Train Epoch: 5 [321/500 20544/32000 (64%)] Loss: 1.56038 (QuantReg: 12.60845) QuantErr: 12.60845 batch_time=0.39795
Train Epoch: 5 [329/500 21056/32000 (66%)] Loss: 2.09696 (QuantReg: 12.53109) QuantErr: 12.53109 batch_time=0.38453
Train Epoch: 5 [337/500 21568/32000 (67%)] Loss: 2.06931 (QuantReg: 12.32182) QuantErr: 12.32182 batch_time=0.39029
Train Epoch: 5 [345/500 22080/32000 (69%)] Loss: 2.14520 (QuantReg: 12.59765) QuantErr: 12.59765 batch_time=0.39910
Train Epoch: 5 [353/500 22592/32000 (71%)] Loss: 1.80188 (QuantReg: 12.23917) QuantErr: 12.23917 batch_time=0.39855
Train Epoch: 5 [361/500 23104/32000 (72%)] Loss: 1.70227 (QuantReg: 12.27056) QuantErr: 12.27056 batch_time=0.51502
Train Epoch: 5 [369/500 23616/32000 (74%)] Loss: 2.13886 (QuantReg: 12.47216) QuantErr: 12.47216 batch_time=0.40481
Train Epoch: 5 [377/500 24128/32000 (75%)] Loss: 1.75067 (QuantReg: 12.56833) QuantErr: 12.56833 batch_time=0.40080
Train Epoch: 5 [385/500 24640/32000 (77%)] Loss: 2.15460 (QuantReg: 12.04207) QuantErr: 12.04207 batch_time=0.39257
Train Epoch: 5 [393/500 25152/32000 (79%)] Loss: 1.60546 (QuantReg: 12.73046) QuantErr: 12.73046 batch_time=0.38631
Train Epoch: 5 [401/500 25664/32000 (80%)] Loss: 1.16978 (QuantReg: 12.73383) QuantErr: 12.73383 batch_time=0.39045
Train Epoch: 5 [409/500 26176/32000 (82%)] Loss: 1.49889 (QuantReg: 12.75822) QuantErr: 12.75822 batch_time=0.39450
Train Epoch: 5 [417/500 26688/32000 (83%)] Loss: 1.76299 (QuantReg: 12.58125) QuantErr: 12.58125 batch_time=0.38425
Train Epoch: 5 [425/500 27200/32000 (85%)] Loss: 1.62904 (QuantReg: 12.37911) QuantErr: 12.37911 batch_time=0.50784
Train Epoch: 5 [433/500 27712/32000 (87%)] Loss: 1.97494 (QuantReg: 12.44461) QuantErr: 12.44461 batch_time=0.39417
Train Epoch: 5 [441/500 28224/32000 (88%)] Loss: 2.20145 (QuantReg: 12.53308) QuantErr: 12.53308 batch_time=0.41324
Train Epoch: 5 [449/500 28736/32000 (90%)] Loss: 1.52158 (QuantReg: 12.83965) QuantErr: 12.83965 batch_time=0.39691
Train Epoch: 5 [457/500 29248/32000 (91%)] Loss: 1.60203 (QuantReg: 12.65594) QuantErr: 12.65594 batch_time=0.40077
Train Epoch: 5 [465/500 29760/32000 (93%)] Loss: 1.90956 (QuantReg: 12.78954) QuantErr: 12.78954 batch_time=0.40687
Train Epoch: 5 [473/500 30272/32000 (95%)] Loss: 1.45293 (QuantReg: 12.51598) QuantErr: 12.51598 batch_time=0.41063
Train Epoch: 5 [481/500 30784/32000 (96%)] Loss: 2.08172 (QuantReg: 12.37862) QuantErr: 12.37862 batch_time=0.42877
Train Epoch: 5 [489/500 31296/32000 (98%)] Loss: 1.39011 (QuantReg: 12.97742) QuantErr: 12.97742 batch_time=0.53310
Train Epoch: 5 [497/500 31808/32000 (99%)] Loss: 1.67952 (QuantReg: 12.64383) QuantErr: 12.64383 batch_time=0.49744
Train Epoch: 5 codebook_update_time=1.66293
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch5.pth ...
Done in 3.749s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch5.pth ...
Done in 7.487s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 1.7595801063776015
quant_reg : 12.30203327178955
quant_err : 12.30203327178955
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 17.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 44.0
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.088
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.842418224108826
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.5
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 29.246
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.5100571543746
mnt_best : 35.842418224108826
not_improved_count: 0
Train Epoch: 6 [1/500 64/32000 (0%)] Loss: 1.90391 (QuantReg: 11.94095) QuantErr: 11.94095 batch_time=17.67860
Train Epoch: 6 [9/500 576/32000 (2%)] Loss: 1.54514 (QuantReg: 12.14671) QuantErr: 12.14671 batch_time=0.39911
Train Epoch: 6 [17/500 1088/32000 (3%)] Loss: 2.28117 (QuantReg: 12.29377) QuantErr: 12.29377 batch_time=0.37879
Train Epoch: 6 [25/500 1600/32000 (5%)] Loss: 1.75065 (QuantReg: 11.65415) QuantErr: 11.65415 batch_time=0.69997
Train Epoch: 6 [33/500 2112/32000 (7%)] Loss: 1.62443 (QuantReg: 11.62109) QuantErr: 11.62109 batch_time=0.42547
Train Epoch: 6 [41/500 2624/32000 (8%)] Loss: 1.17833 (QuantReg: 11.63527) QuantErr: 11.63527 batch_time=0.40393
Train Epoch: 6 [49/500 3136/32000 (10%)] Loss: 1.81712 (QuantReg: 11.58916) QuantErr: 11.58916 batch_time=0.38385
Train Epoch: 6 [57/500 3648/32000 (11%)] Loss: 2.00544 (QuantReg: 12.13044) QuantErr: 12.13044 batch_time=0.44280
Train Epoch: 6 [65/500 4160/32000 (13%)] Loss: 1.85696 (QuantReg: 12.32132) QuantErr: 12.32132 batch_time=0.86413
Train Epoch: 6 [73/500 4672/32000 (15%)] Loss: 1.99894 (QuantReg: 12.26821) QuantErr: 12.26821 batch_time=0.37906
Train Epoch: 6 [81/500 5184/32000 (16%)] Loss: 1.44814 (QuantReg: 12.09482) QuantErr: 12.09482 batch_time=0.38856
Train Epoch: 6 [89/500 5696/32000 (18%)] Loss: 1.67972 (QuantReg: 11.79106) QuantErr: 11.79106 batch_time=0.70708
Train Epoch: 6 [97/500 6208/32000 (19%)] Loss: 1.89028 (QuantReg: 11.59347) QuantErr: 11.59347 batch_time=0.38498
Train Epoch: 6 [105/500 6720/32000 (21%)] Loss: 2.13066 (QuantReg: 12.38327) QuantErr: 12.38327 batch_time=0.40023
Train Epoch: 6 [113/500 7232/32000 (23%)] Loss: 1.80187 (QuantReg: 11.93601) QuantErr: 11.93601 batch_time=0.38611
Train Epoch: 6 [121/500 7744/32000 (24%)] Loss: 1.81788 (QuantReg: 12.27907) QuantErr: 12.27907 batch_time=0.44101
Train Epoch: 6 [129/500 8256/32000 (26%)] Loss: 1.80339 (QuantReg: 12.37981) QuantErr: 12.37981 batch_time=0.87503
Train Epoch: 6 [137/500 8768/32000 (27%)] Loss: 1.60496 (QuantReg: 12.43741) QuantErr: 12.43741 batch_time=0.39542
Train Epoch: 6 [145/500 9280/32000 (29%)] Loss: 2.27130 (QuantReg: 12.72842) QuantErr: 12.72842 batch_time=0.39034
Train Epoch: 6 [153/500 9792/32000 (31%)] Loss: 1.69220 (QuantReg: 12.52103) QuantErr: 12.52103 batch_time=0.70543
Train Epoch: 6 [161/500 10304/32000 (32%)] Loss: 1.32175 (QuantReg: 12.38202) QuantErr: 12.38202 batch_time=0.38037
Train Epoch: 6 [169/500 10816/32000 (34%)] Loss: 1.35649 (QuantReg: 12.93257) QuantErr: 12.93257 batch_time=0.39102
Train Epoch: 6 [177/500 11328/32000 (35%)] Loss: 1.64072 (QuantReg: 12.86839) QuantErr: 12.86839 batch_time=0.39667
Train Epoch: 6 [185/500 11840/32000 (37%)] Loss: 1.67441 (QuantReg: 12.92309) QuantErr: 12.92309 batch_time=0.43706
Train Epoch: 6 [193/500 12352/32000 (39%)] Loss: 1.17780 (QuantReg: 12.32174) QuantErr: 12.32174 batch_time=0.86419
Train Epoch: 6 [201/500 12864/32000 (40%)] Loss: 1.67322 (QuantReg: 12.51669) QuantErr: 12.51669 batch_time=0.38132
Train Epoch: 6 [209/500 13376/32000 (42%)] Loss: 1.99239 (QuantReg: 12.88783) QuantErr: 12.88783 batch_time=0.38156
Train Epoch: 6 [217/500 13888/32000 (43%)] Loss: 2.29074 (QuantReg: 11.81640) QuantErr: 11.81640 batch_time=0.77527
Train Epoch: 6 [225/500 14400/32000 (45%)] Loss: 1.82043 (QuantReg: 12.31451) QuantErr: 12.31451 batch_time=0.38955
Train Epoch: 6 [233/500 14912/32000 (47%)] Loss: 1.77100 (QuantReg: 12.41087) QuantErr: 12.41087 batch_time=0.39116
Train Epoch: 6 [241/500 15424/32000 (48%)] Loss: 1.18018 (QuantReg: 12.32046) QuantErr: 12.32046 batch_time=0.39447
Train Epoch: 6 [249/500 15936/32000 (50%)] Loss: 1.76579 (QuantReg: 12.42726) QuantErr: 12.42726 batch_time=0.46044
Train Epoch: 6 [257/500 16448/32000 (51%)] Loss: 1.12335 (QuantReg: 12.57835) QuantErr: 12.57835 batch_time=0.93679
Train Epoch: 6 [265/500 16960/32000 (53%)] Loss: 1.26397 (QuantReg: 12.10742) QuantErr: 12.10742 batch_time=0.39214
Train Epoch: 6 [273/500 17472/32000 (55%)] Loss: 1.77880 (QuantReg: 12.13589) QuantErr: 12.13589 batch_time=0.39568
Train Epoch: 6 [281/500 17984/32000 (56%)] Loss: 1.43823 (QuantReg: 12.78405) QuantErr: 12.78405 batch_time=0.75380
Train Epoch: 6 [289/500 18496/32000 (58%)] Loss: 1.14571 (QuantReg: 12.34823) QuantErr: 12.34823 batch_time=0.38483
Train Epoch: 6 [297/500 19008/32000 (59%)] Loss: 1.32704 (QuantReg: 12.11411) QuantErr: 12.11411 batch_time=0.38594
Train Epoch: 6 [305/500 19520/32000 (61%)] Loss: 1.87099 (QuantReg: 12.73763) QuantErr: 12.73763 batch_time=0.38555
Train Epoch: 6 [313/500 20032/32000 (63%)] Loss: 1.72255 (QuantReg: 12.33585) QuantErr: 12.33585 batch_time=0.88339
Train Epoch: 6 [321/500 20544/32000 (64%)] Loss: 1.47350 (QuantReg: 12.38504) QuantErr: 12.38504 batch_time=0.89562
Train Epoch: 6 [329/500 21056/32000 (66%)] Loss: 1.66410 (QuantReg: 12.42916) QuantErr: 12.42916 batch_time=0.37493
Train Epoch: 6 [337/500 21568/32000 (67%)] Loss: 1.27679 (QuantReg: 12.33350) QuantErr: 12.33350 batch_time=0.39481
Train Epoch: 6 [345/500 22080/32000 (69%)] Loss: 1.78962 (QuantReg: 12.58657) QuantErr: 12.58657 batch_time=0.72186
Train Epoch: 6 [353/500 22592/32000 (71%)] Loss: 1.58077 (QuantReg: 12.46031) QuantErr: 12.46031 batch_time=0.39747
Train Epoch: 6 [361/500 23104/32000 (72%)] Loss: 1.81734 (QuantReg: 12.57031) QuantErr: 12.57031 batch_time=0.38409
Train Epoch: 6 [369/500 23616/32000 (74%)] Loss: 1.38330 (QuantReg: 12.31428) QuantErr: 12.31428 batch_time=0.39617
Train Epoch: 6 [377/500 24128/32000 (75%)] Loss: 1.96060 (QuantReg: 12.41982) QuantErr: 12.41982 batch_time=0.46000
Train Epoch: 6 [385/500 24640/32000 (77%)] Loss: 1.19654 (QuantReg: 12.61383) QuantErr: 12.61383 batch_time=0.94877
Train Epoch: 6 [393/500 25152/32000 (79%)] Loss: 2.02921 (QuantReg: 12.45117) QuantErr: 12.45117 batch_time=0.41092
Train Epoch: 6 [401/500 25664/32000 (80%)] Loss: 1.19058 (QuantReg: 13.03049) QuantErr: 13.03049 batch_time=0.38433
Train Epoch: 6 [409/500 26176/32000 (82%)] Loss: 1.77809 (QuantReg: 12.34816) QuantErr: 12.34816 batch_time=0.75282
Train Epoch: 6 [417/500 26688/32000 (83%)] Loss: 1.15095 (QuantReg: 13.06307) QuantErr: 13.06307 batch_time=0.38221
Train Epoch: 6 [425/500 27200/32000 (85%)] Loss: 1.97394 (QuantReg: 12.57477) QuantErr: 12.57477 batch_time=0.39182
Train Epoch: 6 [433/500 27712/32000 (87%)] Loss: 1.36473 (QuantReg: 12.62806) QuantErr: 12.62806 batch_time=0.38378
Train Epoch: 6 [441/500 28224/32000 (88%)] Loss: 1.78527 (QuantReg: 12.58648) QuantErr: 12.58648 batch_time=0.44719
Train Epoch: 6 [449/500 28736/32000 (90%)] Loss: 1.23412 (QuantReg: 12.84067) QuantErr: 12.84067 batch_time=0.94461
Train Epoch: 6 [457/500 29248/32000 (91%)] Loss: 1.41695 (QuantReg: 12.43817) QuantErr: 12.43817 batch_time=0.39125
Train Epoch: 6 [465/500 29760/32000 (93%)] Loss: 1.30175 (QuantReg: 12.47916) QuantErr: 12.47916 batch_time=0.37778
Train Epoch: 6 [473/500 30272/32000 (95%)] Loss: 1.82651 (QuantReg: 12.54855) QuantErr: 12.54855 batch_time=0.73849
Train Epoch: 6 [481/500 30784/32000 (96%)] Loss: 1.51890 (QuantReg: 12.79037) QuantErr: 12.79037 batch_time=0.38215
Train Epoch: 6 [489/500 31296/32000 (98%)] Loss: 1.74287 (QuantReg: 12.57113) QuantErr: 12.57113 batch_time=0.41457
Train Epoch: 6 [497/500 31808/32000 (99%)] Loss: 1.48715 (QuantReg: 12.68404) QuantErr: 12.68404 batch_time=0.38468
Train Epoch: 6 codebook_update_time=2.04159
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch6.pth ...
Done in 3.853s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch6.pth ...
Done in 7.842s
removing stale ckpt [epoch 5] [took 0.03s]
epoch : 6
loss : 1.6083833993673324
quant_reg : 12.38238554573059
quant_err : 12.38238554573059
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.325
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.08405052500629
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.878
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.85469904379261
mnt_best : 36.08405052500629
not_improved_count: 0
Train Epoch: 7 [1/500 64/32000 (0%)] Loss: 1.57391 (QuantReg: 11.52560) QuantErr: 11.52560 batch_time=17.44629
Train Epoch: 7 [9/500 576/32000 (2%)] Loss: 1.43025 (QuantReg: 11.93150) QuantErr: 11.93150 batch_time=0.45227
Train Epoch: 7 [17/500 1088/32000 (3%)] Loss: 1.99759 (QuantReg: 12.15316) QuantErr: 12.15316 batch_time=1.15627
Train Epoch: 7 [25/500 1600/32000 (5%)] Loss: 1.99372 (QuantReg: 11.77746) QuantErr: 11.77746 batch_time=0.39759
Train Epoch: 7 [33/500 2112/32000 (7%)] Loss: 1.35430 (QuantReg: 12.20814) QuantErr: 12.20814 batch_time=0.39843
Train Epoch: 7 [41/500 2624/32000 (8%)] Loss: 1.40931 (QuantReg: 11.83734) QuantErr: 11.83734 batch_time=0.39612
Train Epoch: 7 [49/500 3136/32000 (10%)] Loss: 2.00513 (QuantReg: 12.05783) QuantErr: 12.05783 batch_time=0.42866
Train Epoch: 7 [57/500 3648/32000 (11%)] Loss: 1.84390 (QuantReg: 11.90660) QuantErr: 11.90660 batch_time=0.38970
Train Epoch: 7 [65/500 4160/32000 (13%)] Loss: 1.25251 (QuantReg: 12.28854) QuantErr: 12.28854 batch_time=0.38784
Train Epoch: 7 [73/500 4672/32000 (15%)] Loss: 1.45810 (QuantReg: 12.24385) QuantErr: 12.24385 batch_time=0.38605
Train Epoch: 7 [81/500 5184/32000 (16%)] Loss: 0.99568 (QuantReg: 12.21873) QuantErr: 12.21873 batch_time=1.13204
Train Epoch: 7 [89/500 5696/32000 (18%)] Loss: 1.29518 (QuantReg: 11.67810) QuantErr: 11.67810 batch_time=0.38432
Train Epoch: 7 [97/500 6208/32000 (19%)] Loss: 1.66920 (QuantReg: 12.52048) QuantErr: 12.52048 batch_time=0.38090
Train Epoch: 7 [105/500 6720/32000 (21%)] Loss: 1.40191 (QuantReg: 12.27122) QuantErr: 12.27122 batch_time=0.38068
Train Epoch: 7 [113/500 7232/32000 (23%)] Loss: 1.18725 (QuantReg: 12.41704) QuantErr: 12.41704 batch_time=0.38719
Train Epoch: 7 [121/500 7744/32000 (24%)] Loss: 1.70972 (QuantReg: 12.45685) QuantErr: 12.45685 batch_time=0.39606
Train Epoch: 7 [129/500 8256/32000 (26%)] Loss: 1.90720 (QuantReg: 12.13695) QuantErr: 12.13695 batch_time=0.40376
Train Epoch: 7 [137/500 8768/32000 (27%)] Loss: 1.29969 (QuantReg: 12.82502) QuantErr: 12.82502 batch_time=0.46553
Train Epoch: 7 [145/500 9280/32000 (29%)] Loss: 1.64236 (QuantReg: 12.16683) QuantErr: 12.16683 batch_time=1.12541
Train Epoch: 7 [153/500 9792/32000 (31%)] Loss: 1.33119 (QuantReg: 12.49818) QuantErr: 12.49818 batch_time=0.38684
Train Epoch: 7 [161/500 10304/32000 (32%)] Loss: 1.24276 (QuantReg: 12.74734) QuantErr: 12.74734 batch_time=0.38557
Train Epoch: 7 [169/500 10816/32000 (34%)] Loss: 1.60522 (QuantReg: 12.73194) QuantErr: 12.73194 batch_time=0.41547
Train Epoch: 7 [177/500 11328/32000 (35%)] Loss: 1.62509 (QuantReg: 12.11903) QuantErr: 12.11903 batch_time=0.38980
Train Epoch: 7 [185/500 11840/32000 (37%)] Loss: 1.58711 (QuantReg: 12.72521) QuantErr: 12.72521 batch_time=0.39998
Train Epoch: 7 [193/500 12352/32000 (39%)] Loss: 2.23692 (QuantReg: 12.30229) QuantErr: 12.30229 batch_time=0.44385
Train Epoch: 7 [201/500 12864/32000 (40%)] Loss: 1.81909 (QuantReg: 12.32690) QuantErr: 12.32690 batch_time=0.44195
Train Epoch: 7 [209/500 13376/32000 (42%)] Loss: 1.35153 (QuantReg: 12.06672) QuantErr: 12.06672 batch_time=1.13970
Train Epoch: 7 [217/500 13888/32000 (43%)] Loss: 1.98435 (QuantReg: 11.80646) QuantErr: 11.80646 batch_time=0.39410
Train Epoch: 7 [225/500 14400/32000 (45%)] Loss: 1.94963 (QuantReg: 12.47742) QuantErr: 12.47742 batch_time=0.39280
Train Epoch: 7 [233/500 14912/32000 (47%)] Loss: 1.50983 (QuantReg: 12.37857) QuantErr: 12.37857 batch_time=0.39627
Train Epoch: 7 [241/500 15424/32000 (48%)] Loss: 1.24962 (QuantReg: 12.80173) QuantErr: 12.80173 batch_time=0.41134
Train Epoch: 7 [249/500 15936/32000 (50%)] Loss: 2.12747 (QuantReg: 12.24427) QuantErr: 12.24427 batch_time=0.39630
Train Epoch: 7 [257/500 16448/32000 (51%)] Loss: 1.44181 (QuantReg: 12.12385) QuantErr: 12.12385 batch_time=0.38614
Train Epoch: 7 [265/500 16960/32000 (53%)] Loss: 1.22868 (QuantReg: 12.33404) QuantErr: 12.33404 batch_time=0.38678
Train Epoch: 7 [273/500 17472/32000 (55%)] Loss: 1.89606 (QuantReg: 12.52978) QuantErr: 12.52978 batch_time=1.23267
Train Epoch: 7 [281/500 17984/32000 (56%)] Loss: 1.33831 (QuantReg: 12.35811) QuantErr: 12.35811 batch_time=0.39084
Train Epoch: 7 [289/500 18496/32000 (58%)] Loss: 1.88343 (QuantReg: 12.50326) QuantErr: 12.50326 batch_time=0.38892
Train Epoch: 7 [297/500 19008/32000 (59%)] Loss: 2.26450 (QuantReg: 12.29451) QuantErr: 12.29451 batch_time=0.39043
Train Epoch: 7 [305/500 19520/32000 (61%)] Loss: 1.47651 (QuantReg: 12.73297) QuantErr: 12.73297 batch_time=0.40075
Train Epoch: 7 [313/500 20032/32000 (63%)] Loss: 1.30282 (QuantReg: 12.28851) QuantErr: 12.28851 batch_time=0.39438
Train Epoch: 7 [321/500 20544/32000 (64%)] Loss: 1.47924 (QuantReg: 12.79596) QuantErr: 12.79596 batch_time=0.37681
Train Epoch: 7 [329/500 21056/32000 (66%)] Loss: 1.43071 (QuantReg: 12.36051) QuantErr: 12.36051 batch_time=0.40144
Train Epoch: 7 [337/500 21568/32000 (67%)] Loss: 2.01688 (QuantReg: 12.58380) QuantErr: 12.58380 batch_time=1.10797
Train Epoch: 7 [345/500 22080/32000 (69%)] Loss: 1.25702 (QuantReg: 12.93409) QuantErr: 12.93409 batch_time=0.39054
Train Epoch: 7 [353/500 22592/32000 (71%)] Loss: 1.18994 (QuantReg: 12.90978) QuantErr: 12.90978 batch_time=0.38855
Train Epoch: 7 [361/500 23104/32000 (72%)] Loss: 1.62455 (QuantReg: 12.70672) QuantErr: 12.70672 batch_time=0.38119
Train Epoch: 7 [369/500 23616/32000 (74%)] Loss: 1.34308 (QuantReg: 13.21257) QuantErr: 13.21257 batch_time=0.39265
Train Epoch: 7 [377/500 24128/32000 (75%)] Loss: 1.27109 (QuantReg: 12.45664) QuantErr: 12.45664 batch_time=0.38238
Train Epoch: 7 [385/500 24640/32000 (77%)] Loss: 1.23691 (QuantReg: 12.14960) QuantErr: 12.14960 batch_time=0.38101
Train Epoch: 7 [393/500 25152/32000 (79%)] Loss: 1.34844 (QuantReg: 12.75487) QuantErr: 12.75487 batch_time=0.39746
Train Epoch: 7 [401/500 25664/32000 (80%)] Loss: 1.64176 (QuantReg: 12.77072) QuantErr: 12.77072 batch_time=1.12809
Train Epoch: 7 [409/500 26176/32000 (82%)] Loss: 1.24157 (QuantReg: 12.85606) QuantErr: 12.85606 batch_time=0.38706
Train Epoch: 7 [417/500 26688/32000 (83%)] Loss: 1.24490 (QuantReg: 12.93826) QuantErr: 12.93826 batch_time=0.38701
Train Epoch: 7 [425/500 27200/32000 (85%)] Loss: 1.16822 (QuantReg: 12.66522) QuantErr: 12.66522 batch_time=0.41242
Train Epoch: 7 [433/500 27712/32000 (87%)] Loss: 1.35567 (QuantReg: 12.96339) QuantErr: 12.96339 batch_time=0.39076
Train Epoch: 7 [441/500 28224/32000 (88%)] Loss: 1.95702 (QuantReg: 12.88913) QuantErr: 12.88913 batch_time=0.38910
Train Epoch: 7 [449/500 28736/32000 (90%)] Loss: 1.27279 (QuantReg: 12.72001) QuantErr: 12.72001 batch_time=0.39044
Train Epoch: 7 [457/500 29248/32000 (91%)] Loss: 1.67259 (QuantReg: 13.03327) QuantErr: 13.03327 batch_time=0.39031
Train Epoch: 7 [465/500 29760/32000 (93%)] Loss: 1.58671 (QuantReg: 12.31660) QuantErr: 12.31660 batch_time=1.12122
Train Epoch: 7 [473/500 30272/32000 (95%)] Loss: 1.56941 (QuantReg: 12.66557) QuantErr: 12.66557 batch_time=0.39770
Train Epoch: 7 [481/500 30784/32000 (96%)] Loss: 1.87366 (QuantReg: 13.09348) QuantErr: 13.09348 batch_time=0.38953
Train Epoch: 7 [489/500 31296/32000 (98%)] Loss: 1.35074 (QuantReg: 12.50569) QuantErr: 12.50569 batch_time=0.39339
Train Epoch: 7 [497/500 31808/32000 (99%)] Loss: 1.57302 (QuantReg: 13.04582) QuantErr: 13.04582 batch_time=0.38349
Train Epoch: 7 codebook_update_time=1.69709
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch7.pth ...
Done in 6.589s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch7.pth ...
Done in 11.610s
removing stale ckpt [epoch 6] [took 0.02s]
epoch : 7
loss : 1.5051208713054658
quant_reg : 12.501774110794067
quant_err : 12.501774110794067
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 45.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 60.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.5
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 30.118
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.41788253272113
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 46.9
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.0
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.9985
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.275386339985786
mnt_best : 37.41788253272113
not_improved_count: 0
Train Epoch: 8 [1/500 64/32000 (0%)] Loss: 1.66472 (QuantReg: 12.46550) QuantErr: 12.46550 batch_time=17.24138
Train Epoch: 8 [9/500 576/32000 (2%)] Loss: 1.55434 (QuantReg: 12.00653) QuantErr: 12.00653 batch_time=0.39094
Train Epoch: 8 [17/500 1088/32000 (3%)] Loss: 2.24864 (QuantReg: 11.77843) QuantErr: 11.77843 batch_time=0.39938
Train Epoch: 8 [25/500 1600/32000 (5%)] Loss: 1.50939 (QuantReg: 12.28538) QuantErr: 12.28538 batch_time=0.57396
Train Epoch: 8 [33/500 2112/32000 (7%)] Loss: 1.37420 (QuantReg: 12.33344) QuantErr: 12.33344 batch_time=0.39137
Train Epoch: 8 [41/500 2624/32000 (8%)] Loss: 1.06412 (QuantReg: 12.36967) QuantErr: 12.36967 batch_time=0.39687
Train Epoch: 8 [49/500 3136/32000 (10%)] Loss: 1.55419 (QuantReg: 12.67851) QuantErr: 12.67851 batch_time=0.39337
Train Epoch: 8 [57/500 3648/32000 (11%)] Loss: 1.67789 (QuantReg: 12.90914) QuantErr: 12.90914 batch_time=0.40541
Train Epoch: 8 [65/500 4160/32000 (13%)] Loss: 1.78504 (QuantReg: 12.56886) QuantErr: 12.56886 batch_time=0.45308
Train Epoch: 8 [73/500 4672/32000 (15%)] Loss: 1.48328 (QuantReg: 12.92658) QuantErr: 12.92658 batch_time=0.39137
Train Epoch: 8 [81/500 5184/32000 (16%)] Loss: 1.63748 (QuantReg: 12.45586) QuantErr: 12.45586 batch_time=0.39260
Train Epoch: 8 [89/500 5696/32000 (18%)] Loss: 1.95157 (QuantReg: 12.64638) QuantErr: 12.64638 batch_time=0.56963
Train Epoch: 8 [97/500 6208/32000 (19%)] Loss: 1.95589 (QuantReg: 12.80142) QuantErr: 12.80142 batch_time=0.39096
Train Epoch: 8 [105/500 6720/32000 (21%)] Loss: 1.54546 (QuantReg: 11.96908) QuantErr: 11.96908 batch_time=0.39107
Train Epoch: 8 [113/500 7232/32000 (23%)] Loss: 1.27474 (QuantReg: 12.33202) QuantErr: 12.33202 batch_time=0.39344
Train Epoch: 8 [121/500 7744/32000 (24%)] Loss: 1.29679 (QuantReg: 12.87939) QuantErr: 12.87939 batch_time=0.38956
Train Epoch: 8 [129/500 8256/32000 (26%)] Loss: 1.49165 (QuantReg: 12.70084) QuantErr: 12.70084 batch_time=0.43780
Train Epoch: 8 [137/500 8768/32000 (27%)] Loss: 1.75431 (QuantReg: 12.62730) QuantErr: 12.62730 batch_time=0.39927
Train Epoch: 8 [145/500 9280/32000 (29%)] Loss: 1.15790 (QuantReg: 12.20667) QuantErr: 12.20667 batch_time=0.43369
Train Epoch: 8 [153/500 9792/32000 (31%)] Loss: 1.33655 (QuantReg: 11.97945) QuantErr: 11.97945 batch_time=0.57213
Train Epoch: 8 [161/500 10304/32000 (32%)] Loss: 1.32361 (QuantReg: 12.40961) QuantErr: 12.40961 batch_time=0.40633
Train Epoch: 8 [169/500 10816/32000 (34%)] Loss: 1.39732 (QuantReg: 12.55658) QuantErr: 12.55658 batch_time=0.39418
Train Epoch: 8 [177/500 11328/32000 (35%)] Loss: 1.63709 (QuantReg: 12.19050) QuantErr: 12.19050 batch_time=0.38955
Train Epoch: 8 [185/500 11840/32000 (37%)] Loss: 1.19245 (QuantReg: 12.60962) QuantErr: 12.60962 batch_time=0.39087
Train Epoch: 8 [193/500 12352/32000 (39%)] Loss: 1.97267 (QuantReg: 12.03733) QuantErr: 12.03733 batch_time=0.46227
Train Epoch: 8 [201/500 12864/32000 (40%)] Loss: 1.30008 (QuantReg: 12.54200) QuantErr: 12.54200 batch_time=0.39500
Train Epoch: 8 [209/500 13376/32000 (42%)] Loss: 1.47875 (QuantReg: 12.30043) QuantErr: 12.30043 batch_time=0.39442
Train Epoch: 8 [217/500 13888/32000 (43%)] Loss: 1.19206 (QuantReg: 12.89801) QuantErr: 12.89801 batch_time=0.59145
Train Epoch: 8 [225/500 14400/32000 (45%)] Loss: 1.06042 (QuantReg: 12.53726) QuantErr: 12.53726 batch_time=0.40501
Train Epoch: 8 [233/500 14912/32000 (47%)] Loss: 1.30831 (QuantReg: 12.37671) QuantErr: 12.37671 batch_time=0.41221
Train Epoch: 8 [241/500 15424/32000 (48%)] Loss: 1.62073 (QuantReg: 12.80197) QuantErr: 12.80197 batch_time=0.39576
Train Epoch: 8 [249/500 15936/32000 (50%)] Loss: 1.33171 (QuantReg: 12.68009) QuantErr: 12.68009 batch_time=0.39248
Train Epoch: 8 [257/500 16448/32000 (51%)] Loss: 1.16250 (QuantReg: 13.12788) QuantErr: 13.12788 batch_time=0.45855
Train Epoch: 8 [265/500 16960/32000 (53%)] Loss: 1.05366 (QuantReg: 12.83234) QuantErr: 12.83234 batch_time=0.38885
Train Epoch: 8 [273/500 17472/32000 (55%)] Loss: 1.28529 (QuantReg: 12.39958) QuantErr: 12.39958 batch_time=0.39304
Train Epoch: 8 [281/500 17984/32000 (56%)] Loss: 1.52237 (QuantReg: 12.76219) QuantErr: 12.76219 batch_time=0.56533
Train Epoch: 8 [289/500 18496/32000 (58%)] Loss: 1.65598 (QuantReg: 12.91351) QuantErr: 12.91351 batch_time=0.38563
Train Epoch: 8 [297/500 19008/32000 (59%)] Loss: 1.13697 (QuantReg: 12.49892) QuantErr: 12.49892 batch_time=0.39549
Train Epoch: 8 [305/500 19520/32000 (61%)] Loss: 2.01352 (QuantReg: 12.30511) QuantErr: 12.30511 batch_time=0.40619
Train Epoch: 8 [313/500 20032/32000 (63%)] Loss: 1.06736 (QuantReg: 12.34055) QuantErr: 12.34055 batch_time=0.39657
Train Epoch: 8 [321/500 20544/32000 (64%)] Loss: 1.41705 (QuantReg: 12.74534) QuantErr: 12.74534 batch_time=0.44400
Train Epoch: 8 [329/500 21056/32000 (66%)] Loss: 1.06555 (QuantReg: 12.89157) QuantErr: 12.89157 batch_time=0.38462
Train Epoch: 8 [337/500 21568/32000 (67%)] Loss: 1.58392 (QuantReg: 12.77572) QuantErr: 12.77572 batch_time=0.39098
Train Epoch: 8 [345/500 22080/32000 (69%)] Loss: 1.68710 (QuantReg: 12.73551) QuantErr: 12.73551 batch_time=0.57364
Train Epoch: 8 [353/500 22592/32000 (71%)] Loss: 1.20625 (QuantReg: 12.59612) QuantErr: 12.59612 batch_time=0.42037
Train Epoch: 8 [361/500 23104/32000 (72%)] Loss: 1.32991 (QuantReg: 12.70931) QuantErr: 12.70931 batch_time=0.38988
Train Epoch: 8 [369/500 23616/32000 (74%)] Loss: 1.36337 (QuantReg: 12.59079) QuantErr: 12.59079 batch_time=0.38829
Train Epoch: 8 [377/500 24128/32000 (75%)] Loss: 1.31439 (QuantReg: 12.88050) QuantErr: 12.88050 batch_time=0.46039
Train Epoch: 8 [385/500 24640/32000 (77%)] Loss: 1.43895 (QuantReg: 12.55658) QuantErr: 12.55658 batch_time=0.44990
Train Epoch: 8 [393/500 25152/32000 (79%)] Loss: 1.15145 (QuantReg: 12.85669) QuantErr: 12.85669 batch_time=0.38840
Train Epoch: 8 [401/500 25664/32000 (80%)] Loss: 1.42291 (QuantReg: 12.25670) QuantErr: 12.25670 batch_time=0.39777
Train Epoch: 8 [409/500 26176/32000 (82%)] Loss: 1.30850 (QuantReg: 12.14458) QuantErr: 12.14458 batch_time=0.54680
Train Epoch: 8 [417/500 26688/32000 (83%)] Loss: 1.42594 (QuantReg: 12.79269) QuantErr: 12.79269 batch_time=0.40183
Train Epoch: 8 [425/500 27200/32000 (85%)] Loss: 1.29991 (QuantReg: 12.67039) QuantErr: 12.67039 batch_time=0.39117
Train Epoch: 8 [433/500 27712/32000 (87%)] Loss: 1.19524 (QuantReg: 12.78636) QuantErr: 12.78636 batch_time=0.39895
Train Epoch: 8 [441/500 28224/32000 (88%)] Loss: 1.18566 (QuantReg: 13.10266) QuantErr: 13.10266 batch_time=0.39176
Train Epoch: 8 [449/500 28736/32000 (90%)] Loss: 1.18862 (QuantReg: 13.35276) QuantErr: 13.35276 batch_time=0.70921
Train Epoch: 8 [457/500 29248/32000 (91%)] Loss: 0.99798 (QuantReg: 13.21336) QuantErr: 13.21336 batch_time=0.39483
Train Epoch: 8 [465/500 29760/32000 (93%)] Loss: 1.80158 (QuantReg: 12.81538) QuantErr: 12.81538 batch_time=0.42385
Train Epoch: 8 [473/500 30272/32000 (95%)] Loss: 1.76842 (QuantReg: 12.94466) QuantErr: 12.94466 batch_time=0.56201
Train Epoch: 8 [481/500 30784/32000 (96%)] Loss: 1.00761 (QuantReg: 12.92771) QuantErr: 12.92771 batch_time=0.39751
Train Epoch: 8 [489/500 31296/32000 (98%)] Loss: 1.29564 (QuantReg: 12.49514) QuantErr: 12.49514 batch_time=0.38777
Train Epoch: 8 [497/500 31808/32000 (99%)] Loss: 1.68951 (QuantReg: 12.01618) QuantErr: 12.01618 batch_time=0.41336
Train Epoch: 8 codebook_update_time=1.64099
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch8.pth ...
Done in 5.561s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch8.pth ...
Done in 10.667s
removing stale ckpt [epoch 7] [took 0.02s]
epoch : 8
loss : 1.3876902730464935
quant_reg : 12.58097562789917
quant_err : 12.58097562789917
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 29.676
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.27895846263565
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.074
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.781195053623875
mnt_best : 38.27895846263565
not_improved_count: 0
Train Epoch: 9 [1/500 64/32000 (0%)] Loss: 1.11694 (QuantReg: 12.40683) QuantErr: 12.40683 batch_time=18.23960
Train Epoch: 9 [9/500 576/32000 (2%)] Loss: 1.62707 (QuantReg: 12.51502) QuantErr: 12.51502 batch_time=0.38698
Train Epoch: 9 [17/500 1088/32000 (3%)] Loss: 1.00417 (QuantReg: 12.34313) QuantErr: 12.34313 batch_time=0.39466
Train Epoch: 9 [25/500 1600/32000 (5%)] Loss: 1.38781 (QuantReg: 12.64220) QuantErr: 12.64220 batch_time=0.79446
Train Epoch: 9 [33/500 2112/32000 (7%)] Loss: 1.23564 (QuantReg: 12.57654) QuantErr: 12.57654 batch_time=0.39706
Train Epoch: 9 [41/500 2624/32000 (8%)] Loss: 1.09597 (QuantReg: 12.82699) QuantErr: 12.82699 batch_time=0.39303
Train Epoch: 9 [49/500 3136/32000 (10%)] Loss: 1.70103 (QuantReg: 12.39120) QuantErr: 12.39120 batch_time=0.39768
Train Epoch: 9 [57/500 3648/32000 (11%)] Loss: 1.58566 (QuantReg: 12.40341) QuantErr: 12.40341 batch_time=0.39165
Train Epoch: 9 [65/500 4160/32000 (13%)] Loss: 1.50208 (QuantReg: 12.91872) QuantErr: 12.91872 batch_time=0.99329
Train Epoch: 9 [73/500 4672/32000 (15%)] Loss: 1.50727 (QuantReg: 12.01018) QuantErr: 12.01018 batch_time=0.40146
Train Epoch: 9 [81/500 5184/32000 (16%)] Loss: 1.75929 (QuantReg: 12.17904) QuantErr: 12.17904 batch_time=0.39694
Train Epoch: 9 [89/500 5696/32000 (18%)] Loss: 1.36100 (QuantReg: 12.95789) QuantErr: 12.95789 batch_time=0.72689
Train Epoch: 9 [97/500 6208/32000 (19%)] Loss: 1.38224 (QuantReg: 12.68473) QuantErr: 12.68473 batch_time=0.38527
Train Epoch: 9 [105/500 6720/32000 (21%)] Loss: 1.47543 (QuantReg: 12.35239) QuantErr: 12.35239 batch_time=0.38706
Train Epoch: 9 [113/500 7232/32000 (23%)] Loss: 1.18567 (QuantReg: 12.35077) QuantErr: 12.35077 batch_time=0.39293
Train Epoch: 9 [121/500 7744/32000 (24%)] Loss: 1.37984 (QuantReg: 12.91609) QuantErr: 12.91609 batch_time=0.38830
Train Epoch: 9 [129/500 8256/32000 (26%)] Loss: 1.53018 (QuantReg: 12.80286) QuantErr: 12.80286 batch_time=1.00821
Train Epoch: 9 [137/500 8768/32000 (27%)] Loss: 1.44993 (QuantReg: 12.52950) QuantErr: 12.52950 batch_time=0.40130
Train Epoch: 9 [145/500 9280/32000 (29%)] Loss: 1.09349 (QuantReg: 13.10928) QuantErr: 13.10928 batch_time=0.39497
Train Epoch: 9 [153/500 9792/32000 (31%)] Loss: 1.49269 (QuantReg: 11.85662) QuantErr: 11.85662 batch_time=0.67762
Train Epoch: 9 [161/500 10304/32000 (32%)] Loss: 1.77109 (QuantReg: 12.52746) QuantErr: 12.52746 batch_time=0.67162
Train Epoch: 9 [169/500 10816/32000 (34%)] Loss: 1.28452 (QuantReg: 12.39639) QuantErr: 12.39639 batch_time=0.38467
Train Epoch: 9 [177/500 11328/32000 (35%)] Loss: 1.48312 (QuantReg: 12.18731) QuantErr: 12.18731 batch_time=0.38842
Train Epoch: 9 [185/500 11840/32000 (37%)] Loss: 1.30183 (QuantReg: 12.90518) QuantErr: 12.90518 batch_time=0.38969
Train Epoch: 9 [193/500 12352/32000 (39%)] Loss: 1.30526 (QuantReg: 12.45727) QuantErr: 12.45727 batch_time=1.02096
Train Epoch: 9 [201/500 12864/32000 (40%)] Loss: 1.19960 (QuantReg: 12.47402) QuantErr: 12.47402 batch_time=0.42626
Train Epoch: 9 [209/500 13376/32000 (42%)] Loss: 1.17053 (QuantReg: 12.71996) QuantErr: 12.71996 batch_time=0.39674
Train Epoch: 9 [217/500 13888/32000 (43%)] Loss: 1.06693 (QuantReg: 12.83737) QuantErr: 12.83737 batch_time=0.69510
Train Epoch: 9 [225/500 14400/32000 (45%)] Loss: 1.35915 (QuantReg: 12.45746) QuantErr: 12.45746 batch_time=0.40056
Train Epoch: 9 [233/500 14912/32000 (47%)] Loss: 0.78822 (QuantReg: 12.98041) QuantErr: 12.98041 batch_time=0.39742
Train Epoch: 9 [241/500 15424/32000 (48%)] Loss: 1.62938 (QuantReg: 12.57870) QuantErr: 12.57870 batch_time=0.39614
Train Epoch: 9 [249/500 15936/32000 (50%)] Loss: 1.44815 (QuantReg: 12.37022) QuantErr: 12.37022 batch_time=0.41142
Train Epoch: 9 [257/500 16448/32000 (51%)] Loss: 1.54155 (QuantReg: 12.77389) QuantErr: 12.77389 batch_time=1.00527
Train Epoch: 9 [265/500 16960/32000 (53%)] Loss: 1.30919 (QuantReg: 12.92681) QuantErr: 12.92681 batch_time=0.39508
Train Epoch: 9 [273/500 17472/32000 (55%)] Loss: 1.01891 (QuantReg: 12.58328) QuantErr: 12.58328 batch_time=0.39504
Train Epoch: 9 [281/500 17984/32000 (56%)] Loss: 1.34103 (QuantReg: 12.60287) QuantErr: 12.60287 batch_time=0.74784
Train Epoch: 9 [289/500 18496/32000 (58%)] Loss: 1.25061 (QuantReg: 12.92915) QuantErr: 12.92915 batch_time=0.38598
Train Epoch: 9 [297/500 19008/32000 (59%)] Loss: 1.71873 (QuantReg: 13.10741) QuantErr: 13.10741 batch_time=0.39272
Train Epoch: 9 [305/500 19520/32000 (61%)] Loss: 1.02261 (QuantReg: 12.90493) QuantErr: 12.90493 batch_time=0.38865
Train Epoch: 9 [313/500 20032/32000 (63%)] Loss: 1.40102 (QuantReg: 12.43208) QuantErr: 12.43208 batch_time=0.40859
Train Epoch: 9 [321/500 20544/32000 (64%)] Loss: 1.28241 (QuantReg: 12.95629) QuantErr: 12.95629 batch_time=1.04286
Train Epoch: 9 [329/500 21056/32000 (66%)] Loss: 1.26697 (QuantReg: 12.74267) QuantErr: 12.74267 batch_time=0.39054
Train Epoch: 9 [337/500 21568/32000 (67%)] Loss: 1.44497 (QuantReg: 13.31601) QuantErr: 13.31601 batch_time=0.40115
Train Epoch: 9 [345/500 22080/32000 (69%)] Loss: 1.86199 (QuantReg: 12.78634) QuantErr: 12.78634 batch_time=0.68722
Train Epoch: 9 [353/500 22592/32000 (71%)] Loss: 1.18317 (QuantReg: 12.83574) QuantErr: 12.83574 batch_time=0.39429
Train Epoch: 9 [361/500 23104/32000 (72%)] Loss: 1.16804 (QuantReg: 12.89106) QuantErr: 12.89106 batch_time=0.39007
Train Epoch: 9 [369/500 23616/32000 (74%)] Loss: 1.18641 (QuantReg: 13.19099) QuantErr: 13.19099 batch_time=0.41005
Train Epoch: 9 [377/500 24128/32000 (75%)] Loss: 1.44932 (QuantReg: 12.91891) QuantErr: 12.91891 batch_time=0.39573
Train Epoch: 9 [385/500 24640/32000 (77%)] Loss: 0.99103 (QuantReg: 12.62557) QuantErr: 12.62557 batch_time=1.02554
Train Epoch: 9 [393/500 25152/32000 (79%)] Loss: 1.57176 (QuantReg: 12.81067) QuantErr: 12.81067 batch_time=0.41923
Train Epoch: 9 [401/500 25664/32000 (80%)] Loss: 1.36223 (QuantReg: 12.96061) QuantErr: 12.96061 batch_time=0.39733
Train Epoch: 9 [409/500 26176/32000 (82%)] Loss: 1.28075 (QuantReg: 12.33234) QuantErr: 12.33234 batch_time=0.67257
Train Epoch: 9 [417/500 26688/32000 (83%)] Loss: 1.88261 (QuantReg: 12.70833) QuantErr: 12.70833 batch_time=0.39388
Train Epoch: 9 [425/500 27200/32000 (85%)] Loss: 1.54923 (QuantReg: 12.59919) QuantErr: 12.59919 batch_time=0.39147
Train Epoch: 9 [433/500 27712/32000 (87%)] Loss: 1.54421 (QuantReg: 12.78975) QuantErr: 12.78975 batch_time=0.38985
Train Epoch: 9 [441/500 28224/32000 (88%)] Loss: 1.55012 (QuantReg: 12.93843) QuantErr: 12.93843 batch_time=0.40536
Train Epoch: 9 [449/500 28736/32000 (90%)] Loss: 1.43947 (QuantReg: 12.49295) QuantErr: 12.49295 batch_time=1.03718
Train Epoch: 9 [457/500 29248/32000 (91%)] Loss: 1.41308 (QuantReg: 13.00617) QuantErr: 13.00617 batch_time=0.39826
Train Epoch: 9 [465/500 29760/32000 (93%)] Loss: 1.24028 (QuantReg: 13.02754) QuantErr: 13.02754 batch_time=0.40599
Train Epoch: 9 [473/500 30272/32000 (95%)] Loss: 1.09062 (QuantReg: 12.65441) QuantErr: 12.65441 batch_time=0.67498
Train Epoch: 9 [481/500 30784/32000 (96%)] Loss: 1.22523 (QuantReg: 12.70884) QuantErr: 12.70884 batch_time=0.40482
Train Epoch: 9 [489/500 31296/32000 (98%)] Loss: 1.26969 (QuantReg: 12.46730) QuantErr: 12.46730 batch_time=0.40507
Train Epoch: 9 [497/500 31808/32000 (99%)] Loss: 1.83685 (QuantReg: 12.39955) QuantErr: 12.39955 batch_time=0.39112
Train Epoch: 9 codebook_update_time=1.84910
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch9.pth ...
Done in 6.688s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch9.pth ...
Done in 12.481s
removing stale ckpt [epoch 8] [took 0.03s]
epoch : 9
loss : 1.326055295586586
quant_reg : 12.619954467773438
quant_err : 12.619954467773438
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 4500
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.815
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.387260303590324
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.8
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.25
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.137
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.43717103823166
mnt_best : 40.387260303590324
not_improved_count: 0
Train Epoch: 10 [1/500 64/32000 (0%)] Loss: 1.06526 (QuantReg: 12.35770) QuantErr: 12.35770 batch_time=18.17801
Train Epoch: 10 [9/500 576/32000 (2%)] Loss: 0.96323 (QuantReg: 12.58889) QuantErr: 12.58889 batch_time=0.39246
Train Epoch: 10 [17/500 1088/32000 (3%)] Loss: 0.90981 (QuantReg: 12.49405) QuantErr: 12.49405 batch_time=0.39361
Train Epoch: 10 [25/500 1600/32000 (5%)] Loss: 0.74548 (QuantReg: 12.50273) QuantErr: 12.50273 batch_time=0.40569
Train Epoch: 10 [33/500 2112/32000 (7%)] Loss: 1.29475 (QuantReg: 12.97423) QuantErr: 12.97423 batch_time=0.39506
Train Epoch: 10 [41/500 2624/32000 (8%)] Loss: 1.14344 (QuantReg: 12.87450) QuantErr: 12.87450 batch_time=0.37959
Train Epoch: 10 [49/500 3136/32000 (10%)] Loss: 1.20606 (QuantReg: 12.30503) QuantErr: 12.30503 batch_time=0.46957
Train Epoch: 10 [57/500 3648/32000 (11%)] Loss: 1.45237 (QuantReg: 12.47041) QuantErr: 12.47041 batch_time=0.39135
Train Epoch: 10 [65/500 4160/32000 (13%)] Loss: 1.06861 (QuantReg: 12.62032) QuantErr: 12.62032 batch_time=0.39412
Train Epoch: 10 [73/500 4672/32000 (15%)] Loss: 1.35695 (QuantReg: 12.51781) QuantErr: 12.51781 batch_time=0.38349
Train Epoch: 10 [81/500 5184/32000 (16%)] Loss: 1.05913 (QuantReg: 12.88052) QuantErr: 12.88052 batch_time=0.46098
Train Epoch: 10 [89/500 5696/32000 (18%)] Loss: 1.21753 (QuantReg: 12.35158) QuantErr: 12.35158 batch_time=0.38442
Train Epoch: 10 [97/500 6208/32000 (19%)] Loss: 1.27124 (QuantReg: 12.70398) QuantErr: 12.70398 batch_time=0.39487
Train Epoch: 10 [105/500 6720/32000 (21%)] Loss: 0.78821 (QuantReg: 12.71186) QuantErr: 12.71186 batch_time=0.38430
Train Epoch: 10 [113/500 7232/32000 (23%)] Loss: 1.55907 (QuantReg: 12.69761) QuantErr: 12.69761 batch_time=0.39554
Train Epoch: 10 [121/500 7744/32000 (24%)] Loss: 0.72051 (QuantReg: 12.60051) QuantErr: 12.60051 batch_time=0.41210
Train Epoch: 10 [129/500 8256/32000 (26%)] Loss: 1.53074 (QuantReg: 12.43745) QuantErr: 12.43745 batch_time=0.48065
Train Epoch: 10 [137/500 8768/32000 (27%)] Loss: 0.76005 (QuantReg: 13.07347) QuantErr: 13.07347 batch_time=0.40921
Train Epoch: 10 [145/500 9280/32000 (29%)] Loss: 1.02089 (QuantReg: 12.93517) QuantErr: 12.93517 batch_time=0.39117
Train Epoch: 10 [153/500 9792/32000 (31%)] Loss: 1.37709 (QuantReg: 12.72596) QuantErr: 12.72596 batch_time=0.38798
Train Epoch: 10 [161/500 10304/32000 (32%)] Loss: 1.29473 (QuantReg: 12.58808) QuantErr: 12.58808 batch_time=0.39302
Train Epoch: 10 [169/500 10816/32000 (34%)] Loss: 1.25438 (QuantReg: 12.61736) QuantErr: 12.61736 batch_time=0.39050
Train Epoch: 10 [177/500 11328/32000 (35%)] Loss: 1.12527 (QuantReg: 12.54941) QuantErr: 12.54941 batch_time=0.38849
Train Epoch: 10 [185/500 11840/32000 (37%)] Loss: 1.25038 (QuantReg: 12.79461) QuantErr: 12.79461 batch_time=0.38449
Train Epoch: 10 [193/500 12352/32000 (39%)] Loss: 1.43940 (QuantReg: 12.99457) QuantErr: 12.99457 batch_time=0.38915
Train Epoch: 10 [201/500 12864/32000 (40%)] Loss: 1.59323 (QuantReg: 12.26372) QuantErr: 12.26372 batch_time=0.38246
Train Epoch: 10 [209/500 13376/32000 (42%)] Loss: 1.42047 (QuantReg: 12.73018) QuantErr: 12.73018 batch_time=0.38517
Train Epoch: 10 [217/500 13888/32000 (43%)] Loss: 1.05901 (QuantReg: 13.10758) QuantErr: 13.10758 batch_time=0.40435
Train Epoch: 10 [225/500 14400/32000 (45%)] Loss: 0.99287 (QuantReg: 13.37206) QuantErr: 13.37206 batch_time=0.38406
Train Epoch: 10 [233/500 14912/32000 (47%)] Loss: 0.95222 (QuantReg: 12.79100) QuantErr: 12.79100 batch_time=0.39081
Train Epoch: 10 [241/500 15424/32000 (48%)] Loss: 1.11959 (QuantReg: 12.48613) QuantErr: 12.48613 batch_time=0.41709
Train Epoch: 10 [249/500 15936/32000 (50%)] Loss: 1.36829 (QuantReg: 12.49568) QuantErr: 12.49568 batch_time=0.39304
Train Epoch: 10 [257/500 16448/32000 (51%)] Loss: 1.53861 (QuantReg: 12.87021) QuantErr: 12.87021 batch_time=0.39560
Train Epoch: 10 [265/500 16960/32000 (53%)] Loss: 1.06264 (QuantReg: 12.65668) QuantErr: 12.65668 batch_time=0.38312
Train Epoch: 10 [273/500 17472/32000 (55%)] Loss: 1.59705 (QuantReg: 13.23549) QuantErr: 13.23549 batch_time=0.39193
Train Epoch: 10 [281/500 17984/32000 (56%)] Loss: 1.57638 (QuantReg: 12.95210) QuantErr: 12.95210 batch_time=0.41027
Train Epoch: 10 [289/500 18496/32000 (58%)] Loss: 1.06698 (QuantReg: 12.45169) QuantErr: 12.45169 batch_time=0.39137
Train Epoch: 10 [297/500 19008/32000 (59%)] Loss: 1.37505 (QuantReg: 12.65191) QuantErr: 12.65191 batch_time=0.39583
Train Epoch: 10 [305/500 19520/32000 (61%)] Loss: 1.17172 (QuantReg: 13.37060) QuantErr: 13.37060 batch_time=0.38678
Train Epoch: 10 [313/500 20032/32000 (63%)] Loss: 1.04678 (QuantReg: 12.60240) QuantErr: 12.60240 batch_time=0.38921
Train Epoch: 10 [321/500 20544/32000 (64%)] Loss: 1.29043 (QuantReg: 12.93709) QuantErr: 12.93709 batch_time=0.39230
Train Epoch: 10 [329/500 21056/32000 (66%)] Loss: 1.03904 (QuantReg: 12.85841) QuantErr: 12.85841 batch_time=0.38769
Train Epoch: 10 [337/500 21568/32000 (67%)] Loss: 1.31832 (QuantReg: 13.01828) QuantErr: 13.01828 batch_time=0.38789
Train Epoch: 10 [345/500 22080/32000 (69%)] Loss: 0.84017 (QuantReg: 13.03595) QuantErr: 13.03595 batch_time=0.39341
Train Epoch: 10 [353/500 22592/32000 (71%)] Loss: 1.35613 (QuantReg: 12.91237) QuantErr: 12.91237 batch_time=0.39203
Train Epoch: 10 [361/500 23104/32000 (72%)] Loss: 1.13669 (QuantReg: 12.81594) QuantErr: 12.81594 batch_time=0.38807
Train Epoch: 10 [369/500 23616/32000 (74%)] Loss: 1.24750 (QuantReg: 13.36549) QuantErr: 13.36549 batch_time=0.39083
Train Epoch: 10 [377/500 24128/32000 (75%)] Loss: 1.02538 (QuantReg: 12.90402) QuantErr: 12.90402 batch_time=0.38955
Train Epoch: 10 [385/500 24640/32000 (77%)] Loss: 0.88245 (QuantReg: 13.26641) QuantErr: 13.26641 batch_time=0.39135
Train Epoch: 10 [393/500 25152/32000 (79%)] Loss: 1.10172 (QuantReg: 13.09942) QuantErr: 13.09942 batch_time=0.39655
Train Epoch: 10 [401/500 25664/32000 (80%)] Loss: 1.36370 (QuantReg: 12.81647) QuantErr: 12.81647 batch_time=0.39952
Train Epoch: 10 [409/500 26176/32000 (82%)] Loss: 1.13329 (QuantReg: 12.83367) QuantErr: 12.83367 batch_time=0.38276
Train Epoch: 10 [417/500 26688/32000 (83%)] Loss: 1.01273 (QuantReg: 13.22180) QuantErr: 13.22180 batch_time=0.41403
Train Epoch: 10 [425/500 27200/32000 (85%)] Loss: 1.03457 (QuantReg: 13.17237) QuantErr: 13.17237 batch_time=0.38823
Train Epoch: 10 [433/500 27712/32000 (87%)] Loss: 0.86887 (QuantReg: 13.36805) QuantErr: 13.36805 batch_time=0.38785
Train Epoch: 10 [441/500 28224/32000 (88%)] Loss: 0.91638 (QuantReg: 12.68407) QuantErr: 12.68407 batch_time=0.45573
Train Epoch: 10 [449/500 28736/32000 (90%)] Loss: 1.61205 (QuantReg: 12.50339) QuantErr: 12.50339 batch_time=0.38634
Train Epoch: 10 [457/500 29248/32000 (91%)] Loss: 1.62569 (QuantReg: 12.71425) QuantErr: 12.71425 batch_time=0.38376
Train Epoch: 10 [465/500 29760/32000 (93%)] Loss: 1.06988 (QuantReg: 12.76612) QuantErr: 12.76612 batch_time=0.39675
Train Epoch: 10 [473/500 30272/32000 (95%)] Loss: 1.46154 (QuantReg: 13.18792) QuantErr: 13.18792 batch_time=0.38257
Train Epoch: 10 [481/500 30784/32000 (96%)] Loss: 1.21195 (QuantReg: 12.79129) QuantErr: 12.79129 batch_time=0.58512
Train Epoch: 10 [489/500 31296/32000 (98%)] Loss: 1.16040 (QuantReg: 13.01865) QuantErr: 13.01865 batch_time=0.40267
Train Epoch: 10 [497/500 31808/32000 (99%)] Loss: 1.17887 (QuantReg: 13.37302) QuantErr: 13.37302 batch_time=0.39863
Train Epoch: 10 codebook_update_time=1.76188
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch10.pth ...
Done in 5.442s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_bs64/checkpoint-epoch10.pth ...
Done in 12.586s
removing stale ckpt [epoch 9] [took 0.10s]
epoch : 10
loss : 1.2380561178922653
quant_reg : 12.785605012893678
quant_err : 12.785605012893678
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 5000
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.25
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.65048602303446
MSRVTT_jsfusion_test/v2t_metrics/R1: 18.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.686
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.21747732562143
mnt_best : 40.65048602303446
not_improved_count: 0
Train Epoch: 11 [1/500 64/32000 (0%)] Loss: 1.18687 (QuantReg: 12.76305) QuantErr: 12.76305 batch_time=22.74253
Train Epoch: 11 [9/500 576/32000 (2%)] Loss: 1.13792 (QuantReg: 12.91145) QuantErr: 12.91145 batch_time=0.41918
Train Epoch: 11 [17/500 1088/32000 (3%)] Loss: 1.10997 (QuantReg: 12.38864) QuantErr: 12.38864 batch_time=0.38717
Train Epoch: 11 [25/500 1600/32000 (5%)] Loss: 1.08754 (QuantReg: 12.34055) QuantErr: 12.34055 batch_time=0.75683
Train Epoch: 11 [33/500 2112/32000 (7%)] Loss: 1.09268 (QuantReg: 12.51842) QuantErr: 12.51842 batch_time=0.45387
Train Epoch: 11 [41/500 2624/32000 (8%)] Loss: 0.69345 (QuantReg: 12.44044) QuantErr: 12.44044 batch_time=0.39976
Train Epoch: 11 [49/500 3136/32000 (10%)] Loss: 1.40313 (QuantReg: 12.64209) QuantErr: 12.64209 batch_time=0.40761
Train Epoch: 11 [57/500 3648/32000 (11%)] Loss: 1.21274 (QuantReg: 12.51860) QuantErr: 12.51860 batch_time=0.40080
Train Epoch: 11 [65/500 4160/32000 (13%)] Loss: 1.37001 (QuantReg: 12.38971) QuantErr: 12.38971 batch_time=0.50761
Train Epoch: 11 [73/500 4672/32000 (15%)] Loss: 1.42194 (QuantReg: 12.75328) QuantErr: 12.75328 batch_time=0.39996
Train Epoch: 11 [81/500 5184/32000 (16%)] Loss: 1.28158 (QuantReg: 12.29140) QuantErr: 12.29140 batch_time=0.39939
Train Epoch: 11 [89/500 5696/32000 (18%)] Loss: 1.38909 (QuantReg: 12.62450) QuantErr: 12.62450 batch_time=0.39886
Train Epoch: 11 [97/500 6208/32000 (19%)] Loss: 1.01093 (QuantReg: 12.85216) QuantErr: 12.85216 batch_time=0.39987
Train Epoch: 11 [105/500 6720/32000 (21%)] Loss: 1.37810 (QuantReg: 12.81371) QuantErr: 12.81371 batch_time=0.40236
Train Epoch: 11 [113/500 7232/32000 (23%)] Loss: 1.67984 (QuantReg: 12.61533) QuantErr: 12.61533 batch_time=0.39879
Train Epoch: 11 [121/500 7744/32000 (24%)] Loss: 1.46386 (QuantReg: 12.75508) QuantErr: 12.75508 batch_time=0.45218
Train Epoch: 11 [129/500 8256/32000 (26%)] Loss: 1.23128 (QuantReg: 12.55862) QuantErr: 12.55862 batch_time=0.51234
Train Epoch: 11 [137/500 8768/32000 (27%)] Loss: 1.08721 (QuantReg: 12.56723) QuantErr: 12.56723 batch_time=0.44590
Train Epoch: 11 [145/500 9280/32000 (29%)] Loss: 1.21631 (QuantReg: 12.92429) QuantErr: 12.92429 batch_time=0.38680
Train Epoch: 11 [153/500 9792/32000 (31%)] Loss: 1.59345 (QuantReg: 12.56082) QuantErr: 12.56082 batch_time=0.38849
Train Epoch: 11 [161/500 10304/32000 (32%)] Loss: 0.93211 (QuantReg: 12.93705) QuantErr: 12.93705 batch_time=0.38509
Train Epoch: 11 [169/500 10816/32000 (34%)] Loss: 1.21626 (QuantReg: 12.82586) QuantErr: 12.82586 batch_time=0.46556
Train Epoch: 11 [177/500 11328/32000 (35%)] Loss: 1.06557 (QuantReg: 12.81720) QuantErr: 12.81720 batch_time=0.38858
Train Epoch: 11 [185/500 11840/32000 (37%)] Loss: 1.33771 (QuantReg: 13.04710) QuantErr: 13.04710 batch_time=0.38688
Train Epoch: 11 [193/500 12352/32000 (39%)] Loss: 0.91181 (QuantReg: 12.57541) QuantErr: 12.57541 batch_time=0.50164
Train Epoch: 11 [201/500 12864/32000 (40%)] Loss: 1.78957 (QuantReg: 12.88727) QuantErr: 12.88727 batch_time=0.39706
Train Epoch: 11 [209/500 13376/32000 (42%)] Loss: 0.87449 (QuantReg: 12.74671) QuantErr: 12.74671 batch_time=0.38749
Train Epoch: 11 [217/500 13888/32000 (43%)] Loss: 0.90685 (QuantReg: 12.27593) QuantErr: 12.27593 batch_time=0.38796
Train Epoch: 11 [225/500 14400/32000 (45%)] Loss: 0.83693 (QuantReg: 12.42501) QuantErr: 12.42501 batch_time=0.42276
Train Epoch: 11 [233/500 14912/32000 (47%)] Loss: 1.15874 (QuantReg: 12.73951) QuantErr: 12.73951 batch_time=0.39756
Train Epoch: 11 [241/500 15424/32000 (48%)] Loss: 0.94681 (QuantReg: 13.04705) QuantErr: 13.04705 batch_time=0.39466
Train Epoch: 11 [249/500 15936/32000 (50%)] Loss: 1.26856 (QuantReg: 13.08866) QuantErr: 13.08866 batch_time=0.38794
Train Epoch: 11 [257/500 16448/32000 (51%)] Loss: 1.09535 (QuantReg: 12.95442) QuantErr: 12.95442 batch_time=0.38511
Train Epoch: 11 [265/500 16960/32000 (53%)] Loss: 1.52557 (QuantReg: 12.47016) QuantErr: 12.47016 batch_time=0.45455
Train Epoch: 11 [273/500 17472/32000 (55%)] Loss: 1.14662 (QuantReg: 12.66918) QuantErr: 12.66918 batch_time=0.39428
Train Epoch: 11 [281/500 17984/32000 (56%)] Loss: 1.41052 (QuantReg: 12.66264) QuantErr: 12.66264 batch_time=0.40619
Train Epoch: 11 [289/500 18496/32000 (58%)] Loss: 1.50514 (QuantReg: 12.55703) QuantErr: 12.55703 batch_time=0.40195
Train Epoch: 11 [297/500 19008/32000 (59%)] Loss: 1.10419 (QuantReg: 12.57442) QuantErr: 12.57442 batch_time=0.40011
Train Epoch: 11 [305/500 19520/32000 (61%)] Loss: 1.13942 (QuantReg: 13.00551) QuantErr: 13.00551 batch_time=0.39768
Train Epoch: 11 [313/500 20032/32000 (63%)] Loss: 0.82037 (QuantReg: 12.91786) QuantErr: 12.91786 batch_time=0.40003
Train Epoch: 11 [321/500 20544/32000 (64%)] Loss: 1.33849 (QuantReg: 12.57077) QuantErr: 12.57077 batch_time=0.39558
Train Epoch: 11 [329/500 21056/32000 (66%)] Loss: 0.90507 (QuantReg: 12.81976) QuantErr: 12.81976 batch_time=0.38870
Train Epoch: 11 [337/500 21568/32000 (67%)] Loss: 1.80234 (QuantReg: 12.54964) QuantErr: 12.54964 batch_time=0.38686
Train Epoch: 11 [345/500 22080/32000 (69%)] Loss: 1.05201 (QuantReg: 12.75527) QuantErr: 12.75527 batch_time=0.37652
Train Epoch: 11 [353/500 22592/32000 (71%)] Loss: 1.04371 (QuantReg: 12.95948) QuantErr: 12.95948 batch_time=0.38519
Train Epoch: 11 [361/500 23104/32000 (72%)] Loss: 1.33589 (QuantReg: 12.45922) QuantErr: 12.45922 batch_time=0.40549