-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kA_roberta-base.txt
2757 lines (2757 loc) · 206 KB
/
HCQ_MSRVTT_1kA_roberta-base.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 303.1627972126007 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 30.62748885154724 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 14.081108331680298 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch0.pth ...
Done in 8.467s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch0.pth ...
Done in 10.155s
epoch : 0
loss : 0
learning_rate : 0.0001
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 5.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 494.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 500.367
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 0.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 4.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 517.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 511.0715
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.79148 (QuantReg: 22.29398) QuantErr: 22.29398 batch_time=17.60762
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.15695 (QuantReg: 22.36269) QuantErr: 22.36269 batch_time=0.48463
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.71432 (QuantReg: 22.55618) QuantErr: 22.55618 batch_time=0.49085
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 7.11308 (QuantReg: 22.62117) QuantErr: 22.62117 batch_time=0.50021
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.44017 (QuantReg: 22.59572) QuantErr: 22.59572 batch_time=0.48779
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.58747 (QuantReg: 22.62014) QuantErr: 22.62014 batch_time=0.48433
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.96107 (QuantReg: 22.64347) QuantErr: 22.64347 batch_time=0.49536
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.63668 (QuantReg: 22.60108) QuantErr: 22.60108 batch_time=0.48929
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.72397 (QuantReg: 22.61637) QuantErr: 22.61637 batch_time=0.49659
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.23563 (QuantReg: 22.62866) QuantErr: 22.62866 batch_time=0.48468
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.45254 (QuantReg: 22.62077) QuantErr: 22.62077 batch_time=0.82563
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.08708 (QuantReg: 22.61661) QuantErr: 22.61661 batch_time=0.49912
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.94262 (QuantReg: 22.62331) QuantErr: 22.62331 batch_time=0.52268
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.81687 (QuantReg: 22.61462) QuantErr: 22.61462 batch_time=0.50104
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.77283 (QuantReg: 22.62066) QuantErr: 22.62066 batch_time=0.49504
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.68481 (QuantReg: 22.57771) QuantErr: 22.57771 batch_time=0.49749
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.67921 (QuantReg: 22.61409) QuantErr: 22.61409 batch_time=0.49539
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.72821 (QuantReg: 22.64111) QuantErr: 22.64111 batch_time=0.85462
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.48433 (QuantReg: 22.66157) QuantErr: 22.66157 batch_time=0.49805
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.36722 (QuantReg: 22.65427) QuantErr: 22.65427 batch_time=0.49232
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.99173 (QuantReg: 22.66599) QuantErr: 22.66599 batch_time=0.52043
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.20352 (QuantReg: 22.67728) QuantErr: 22.67728 batch_time=0.61435
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.95817 (QuantReg: 22.63443) QuantErr: 22.63443 batch_time=0.49045
Train Epoch: 1 codebook_update_time=2.23457
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch1.pth ...
Done in 3.955s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch1.pth ...
Done in 8.698s
epoch : 1
loss : 5.616883320808411
quant_reg : 22.600959770202635
quant_err : 22.600959770202635
learning_rate : 0.0001
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 8.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 28.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 43.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 77.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 14.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 44.858
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.016578413053768
MSRVTT_jsfusion_test/v2t_metrics/R1: 10.9
MSRVTT_jsfusion_test/v2t_metrics/R5: 32.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 45.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 79.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 13.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 43.66
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.25602614068817
mnt_best : 22.016578413053768
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.04907 (QuantReg: 10.60452) QuantErr: 10.60452 batch_time=23.30103
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.34646 (QuantReg: 10.80864) QuantErr: 10.80864 batch_time=0.49790
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.60635 (QuantReg: 11.76969) QuantErr: 11.76969 batch_time=0.49703
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.81537 (QuantReg: 11.45857) QuantErr: 11.45857 batch_time=0.49729
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.11797 (QuantReg: 12.05406) QuantErr: 12.05406 batch_time=1.79420
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.83013 (QuantReg: 12.03797) QuantErr: 12.03797 batch_time=0.50672
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.92122 (QuantReg: 12.43412) QuantErr: 12.43412 batch_time=0.49214
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.64327 (QuantReg: 12.09956) QuantErr: 12.09956 batch_time=0.50409
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.49058 (QuantReg: 12.25521) QuantErr: 12.25521 batch_time=0.72971
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.10961 (QuantReg: 12.64125) QuantErr: 12.64125 batch_time=0.48888
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 4.15006 (QuantReg: 12.49328) QuantErr: 12.49328 batch_time=0.50195
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.51093 (QuantReg: 12.68871) QuantErr: 12.68871 batch_time=0.49796
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.98043 (QuantReg: 13.36201) QuantErr: 13.36201 batch_time=0.51262
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.86710 (QuantReg: 13.21151) QuantErr: 13.21151 batch_time=0.88571
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.82423 (QuantReg: 13.16813) QuantErr: 13.16813 batch_time=0.48950
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.73893 (QuantReg: 13.30185) QuantErr: 13.30185 batch_time=0.81079
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.79339 (QuantReg: 13.35253) QuantErr: 13.35253 batch_time=0.49659
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.67709 (QuantReg: 13.89095) QuantErr: 13.89095 batch_time=0.50372
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.82995 (QuantReg: 13.43280) QuantErr: 13.43280 batch_time=0.49974
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.38052 (QuantReg: 14.48529) QuantErr: 14.48529 batch_time=0.49899
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.40227 (QuantReg: 14.10329) QuantErr: 14.10329 batch_time=0.49885
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.53674 (QuantReg: 14.06636) QuantErr: 14.06636 batch_time=0.49845
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.54245 (QuantReg: 13.86805) QuantErr: 13.86805 batch_time=0.49654
Train Epoch: 2 codebook_update_time=1.63846
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch2.pth ...
Done in 4.008s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch2.pth ...
Done in 7.965s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 3.765118082046509
quant_reg : 12.80198962020874
quant_err : 12.80198962020874
learning_rate : 9.5e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.6
MSRVTT_jsfusion_test/t2v_metrics/R5: 38.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 53.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 83.0
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 38.109
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.504149564706317
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.0
MSRVTT_jsfusion_test/v2t_metrics/R10: 54.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 82.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 36.9775
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.767428379579528
mnt_best : 29.504149564706317
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.25211 (QuantReg: 11.03418) QuantErr: 11.03418 batch_time=31.24523
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.13154 (QuantReg: 10.90020) QuantErr: 10.90020 batch_time=1.01906
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.63055 (QuantReg: 10.98746) QuantErr: 10.98746 batch_time=0.60388
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.78538 (QuantReg: 11.46998) QuantErr: 11.46998 batch_time=0.49342
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.29590 (QuantReg: 11.61458) QuantErr: 11.61458 batch_time=0.50218
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.24607 (QuantReg: 11.64419) QuantErr: 11.64419 batch_time=0.49858
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.02486 (QuantReg: 11.87529) QuantErr: 11.87529 batch_time=0.49025
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.30610 (QuantReg: 11.80598) QuantErr: 11.80598 batch_time=0.49721
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 2.99747 (QuantReg: 11.75561) QuantErr: 11.75561 batch_time=0.50923
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.42340 (QuantReg: 11.71822) QuantErr: 11.71822 batch_time=0.47473
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.41941 (QuantReg: 11.96877) QuantErr: 11.96877 batch_time=0.51663
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 2.89199 (QuantReg: 12.10991) QuantErr: 12.10991 batch_time=0.49706
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.01953 (QuantReg: 11.85675) QuantErr: 11.85675 batch_time=0.49092
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 2.88530 (QuantReg: 12.20796) QuantErr: 12.20796 batch_time=0.49438
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.54684 (QuantReg: 12.28975) QuantErr: 12.28975 batch_time=0.97184
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.25400 (QuantReg: 12.45780) QuantErr: 12.45780 batch_time=0.49434
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.31546 (QuantReg: 12.38459) QuantErr: 12.38459 batch_time=0.48706
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.49139 (QuantReg: 12.67160) QuantErr: 12.67160 batch_time=0.50433
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.25192 (QuantReg: 12.82726) QuantErr: 12.82726 batch_time=0.50374
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.56871 (QuantReg: 12.84011) QuantErr: 12.84011 batch_time=0.53537
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base
Preparing the dataloaders ...
Loading dataset MSRVTT_jsfusion_trainval in ram ...
Finish loading dataset MSRVTT_jsfusion_trainval in ram, taking 328.4512679576874 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 29.23418879508972 s.
Loading dataset MSRVTT_jsfusion_test in ram ...
Finish loading dataset MSRVTT_jsfusion_test in ram, taking 35.05871081352234 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch0.pth ...
Done in 1.778s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch0.pth ...
Done in 3.466s
epoch : 0
loss : 0
learning_rate : 0.0001
n_samples : 0
n_steps : 0
MSRVTT_jsfusion_test/t2v_metrics/R1: 0.0
MSRVTT_jsfusion_test/t2v_metrics/R5: 0.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 0.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 5.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 494.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 500.367
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R1: 0.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 0.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 0.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 4.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 517.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 511.0715
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.78876 (QuantReg: 22.29451) QuantErr: 22.29451 batch_time=18.99975
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.30404 (QuantReg: 22.39879) QuantErr: 22.39879 batch_time=0.48281
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.64905 (QuantReg: 22.55591) QuantErr: 22.55591 batch_time=1.85077
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.89103 (QuantReg: 22.61416) QuantErr: 22.61416 batch_time=0.51771
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.13362 (QuantReg: 22.61382) QuantErr: 22.61382 batch_time=0.47768
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.50737 (QuantReg: 22.62199) QuantErr: 22.62199 batch_time=0.47708
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.75723 (QuantReg: 22.60867) QuantErr: 22.60867 batch_time=3.18233
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.54827 (QuantReg: 22.61583) QuantErr: 22.61583 batch_time=0.47735
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.77014 (QuantReg: 22.61644) QuantErr: 22.61644 batch_time=0.46949
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.39464 (QuantReg: 22.65991) QuantErr: 22.65991 batch_time=0.95060
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.26279 (QuantReg: 22.58437) QuantErr: 22.58437 batch_time=0.48064
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.28473 (QuantReg: 22.60606) QuantErr: 22.60606 batch_time=0.47193
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.04621 (QuantReg: 22.61851) QuantErr: 22.61851 batch_time=0.47749
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.88421 (QuantReg: 22.65601) QuantErr: 22.65601 batch_time=1.93702
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.91598 (QuantReg: 22.61003) QuantErr: 22.61003 batch_time=0.48354
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.64491 (QuantReg: 22.57845) QuantErr: 22.57845 batch_time=0.48431
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.66547 (QuantReg: 22.62090) QuantErr: 22.62090 batch_time=0.48655
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.60874 (QuantReg: 22.63664) QuantErr: 22.63664 batch_time=0.47929
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.60505 (QuantReg: 22.66799) QuantErr: 22.66799 batch_time=0.53260
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.38503 (QuantReg: 22.64332) QuantErr: 22.64332 batch_time=0.52277
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.85373 (QuantReg: 22.64572) QuantErr: 22.64572 batch_time=0.49509
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.29711 (QuantReg: 22.63468) QuantErr: 22.63468 batch_time=0.49191
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.94186 (QuantReg: 22.62863) QuantErr: 22.62863 batch_time=0.49433
Train Epoch: 1 codebook_update_time=1.93285
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch1.pth ...
Done in 4.030s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch1.pth ...
Done in 7.815s
epoch : 1
loss : 5.564510670661926
quant_reg : 22.60241316986084
quant_err : 22.60241316986084
learning_rate : 0.0001
n_samples : 32000
n_steps : 250
MSRVTT_jsfusion_test/t2v_metrics/R1: 8.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 31.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 43.6
MSRVTT_jsfusion_test/t2v_metrics/R50: 77.8
MSRVTT_jsfusion_test/t2v_metrics/MedR: 14.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 45.846
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.948795983939718
MSRVTT_jsfusion_test/v2t_metrics/R1: 10.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 32.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 45.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 78.8
MSRVTT_jsfusion_test/v2t_metrics/MedR: 13.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 44.232
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.834063709381603
mnt_best : 22.948795983939718
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.01795 (QuantReg: 10.23188) QuantErr: 10.23188 batch_time=30.37116
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.32586 (QuantReg: 10.71046) QuantErr: 10.71046 batch_time=0.49735
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.59775 (QuantReg: 11.60484) QuantErr: 11.60484 batch_time=0.48314
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.92952 (QuantReg: 11.18868) QuantErr: 11.18868 batch_time=0.51841
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.20868 (QuantReg: 12.33174) QuantErr: 12.33174 batch_time=0.48483
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.93033 (QuantReg: 12.05905) QuantErr: 12.05905 batch_time=0.51162
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.93864 (QuantReg: 12.12665) QuantErr: 12.12665 batch_time=1.28985
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.40910 (QuantReg: 12.18132) QuantErr: 12.18132 batch_time=0.56978
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.36989 (QuantReg: 12.34477) QuantErr: 12.34477 batch_time=0.49137
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.22926 (QuantReg: 12.52924) QuantErr: 12.52924 batch_time=0.48766
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 4.18998 (QuantReg: 12.50600) QuantErr: 12.50600 batch_time=0.60203
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.45194 (QuantReg: 12.72742) QuantErr: 12.72742 batch_time=0.47668
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.94796 (QuantReg: 13.37875) QuantErr: 13.37875 batch_time=0.47666
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.75051 (QuantReg: 12.89451) QuantErr: 12.89451 batch_time=1.44926
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.77229 (QuantReg: 13.22000) QuantErr: 13.22000 batch_time=0.49122
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.86418 (QuantReg: 13.10659) QuantErr: 13.10659 batch_time=0.48962
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.75562 (QuantReg: 13.26286) QuantErr: 13.26286 batch_time=0.48977
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.66688 (QuantReg: 13.56225) QuantErr: 13.56225 batch_time=0.49005
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.76441 (QuantReg: 13.37771) QuantErr: 13.37771 batch_time=0.47702
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.21566 (QuantReg: 14.14134) QuantErr: 14.14134 batch_time=0.48356
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.39800 (QuantReg: 14.33235) QuantErr: 14.33235 batch_time=0.48234
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.57317 (QuantReg: 13.53979) QuantErr: 13.53979 batch_time=0.48461
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.50857 (QuantReg: 14.14395) QuantErr: 14.14395 batch_time=0.51660
Train Epoch: 2 codebook_update_time=1.56825
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch2.pth ...
Done in 3.901s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch2.pth ...
Done in 7.683s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
epoch : 2
loss : 3.7399247140884397
quant_reg : 12.718965816497803
quant_err : 12.718965816497803
learning_rate : 9.5e-05
n_samples : 64000
n_steps : 500
MSRVTT_jsfusion_test/t2v_metrics/R1: 12.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 38.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 52.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 82.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 9.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 38.477
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.36442857266928
MSRVTT_jsfusion_test/v2t_metrics/R1: 14.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 41.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 53.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 83.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 36.805
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.08834992262977
mnt_best : 29.36442857266928
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.15178 (QuantReg: 11.03813) QuantErr: 11.03813 batch_time=26.70025
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.06007 (QuantReg: 11.13581) QuantErr: 11.13581 batch_time=0.49942
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.58578 (QuantReg: 11.22079) QuantErr: 11.22079 batch_time=0.47661
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.71836 (QuantReg: 11.38439) QuantErr: 11.38439 batch_time=0.48815
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.29878 (QuantReg: 11.44952) QuantErr: 11.44952 batch_time=2.08895
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.19271 (QuantReg: 11.47770) QuantErr: 11.47770 batch_time=0.48750
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.96248 (QuantReg: 11.93103) QuantErr: 11.93103 batch_time=0.48299
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.37287 (QuantReg: 11.95112) QuantErr: 11.95112 batch_time=0.59302
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.06161 (QuantReg: 11.85408) QuantErr: 11.85408 batch_time=0.48702
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.31058 (QuantReg: 11.99900) QuantErr: 11.99900 batch_time=0.70888
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.29464 (QuantReg: 12.22749) QuantErr: 12.22749 batch_time=0.48338
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 2.86962 (QuantReg: 12.16106) QuantErr: 12.16106 batch_time=0.48513
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.11707 (QuantReg: 11.89912) QuantErr: 11.89912 batch_time=0.52716
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 2.84270 (QuantReg: 12.55656) QuantErr: 12.55656 batch_time=1.18250
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.33766 (QuantReg: 12.45936) QuantErr: 12.45936 batch_time=0.49124
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.36388 (QuantReg: 12.49098) QuantErr: 12.49098 batch_time=0.48499
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.31371 (QuantReg: 12.46542) QuantErr: 12.46542 batch_time=0.71517
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.52455 (QuantReg: 12.81858) QuantErr: 12.81858 batch_time=0.47951
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.35949 (QuantReg: 12.90587) QuantErr: 12.90587 batch_time=1.15812
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.76305 (QuantReg: 12.78317) QuantErr: 12.78317 batch_time=0.49799
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.44795 (QuantReg: 12.55824) QuantErr: 12.55824 batch_time=0.48738
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.73534 (QuantReg: 13.00282) QuantErr: 13.00282 batch_time=0.47939
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.87192 (QuantReg: 12.93090) QuantErr: 12.93090 batch_time=0.47764
Train Epoch: 3 codebook_update_time=1.58670
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch3.pth ...
Done in 4.054s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch3.pth ...
Done in 7.970s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 3.1283362617492676
quant_reg : 12.163154209136962
quant_err : 12.163154209136962
learning_rate : 9.025e-05
n_samples : 96000
n_steps : 750
MSRVTT_jsfusion_test/t2v_metrics/R1: 15.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 41.4
MSRVTT_jsfusion_test/t2v_metrics/R10: 56.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 84.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 8.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 34.576
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.239117172571916
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 43.7
MSRVTT_jsfusion_test/v2t_metrics/R10: 55.9
MSRVTT_jsfusion_test/v2t_metrics/R50: 83.3
MSRVTT_jsfusion_test/v2t_metrics/MedR: 8.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 34.732
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.07763286970777
mnt_best : 33.239117172571916
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.06300 (QuantReg: 11.84448) QuantErr: 11.84448 batch_time=27.18164
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.54424 (QuantReg: 10.76435) QuantErr: 10.76435 batch_time=0.49415
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 3.38015 (QuantReg: 11.40240) QuantErr: 11.40240 batch_time=0.48159
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.64742 (QuantReg: 11.34886) QuantErr: 11.34886 batch_time=0.48140
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.01379 (QuantReg: 12.18349) QuantErr: 12.18349 batch_time=0.49461
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.55050 (QuantReg: 11.48080) QuantErr: 11.48080 batch_time=0.47714
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.40061 (QuantReg: 11.57647) QuantErr: 11.57647 batch_time=0.49128
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 3.15815 (QuantReg: 11.91097) QuantErr: 11.91097 batch_time=0.48135
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.73815 (QuantReg: 12.43271) QuantErr: 12.43271 batch_time=0.51588
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.57571 (QuantReg: 11.91706) QuantErr: 11.91706 batch_time=0.49455
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.63150 (QuantReg: 12.01877) QuantErr: 12.01877 batch_time=0.49695
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.42360 (QuantReg: 12.67567) QuantErr: 12.67567 batch_time=1.24645
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.85849 (QuantReg: 12.16191) QuantErr: 12.16191 batch_time=0.50880
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.68433 (QuantReg: 12.28422) QuantErr: 12.28422 batch_time=1.52101
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.68495 (QuantReg: 12.33066) QuantErr: 12.33066 batch_time=0.49722
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.86905 (QuantReg: 12.80900) QuantErr: 12.80900 batch_time=0.49390
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.10884 (QuantReg: 12.38535) QuantErr: 12.38535 batch_time=0.48715
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.65596 (QuantReg: 12.40719) QuantErr: 12.40719 batch_time=0.48579
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.63663 (QuantReg: 12.31523) QuantErr: 12.31523 batch_time=0.80468
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.64818 (QuantReg: 12.21047) QuantErr: 12.21047 batch_time=0.49285
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.51407 (QuantReg: 12.81056) QuantErr: 12.81056 batch_time=0.49143
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 3.15287 (QuantReg: 12.52741) QuantErr: 12.52741 batch_time=0.53050
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.86564 (QuantReg: 12.79766) QuantErr: 12.79766 batch_time=0.49037
Train Epoch: 4 codebook_update_time=1.62475
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch4.pth ...
Done in 3.957s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch4.pth ...
Done in 7.853s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 2.808939471244812
quant_reg : 12.140612907409668
quant_err : 12.140612907409668
learning_rate : 8.573749999999999e-05
n_samples : 128000
n_steps : 1000
MSRVTT_jsfusion_test/t2v_metrics/R1: 16.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 42.9
MSRVTT_jsfusion_test/t2v_metrics/R10: 58.2
MSRVTT_jsfusion_test/t2v_metrics/R50: 85.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 31.338
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.46736169725739
MSRVTT_jsfusion_test/v2t_metrics/R1: 16.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 45.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 58.5
MSRVTT_jsfusion_test/v2t_metrics/R50: 85.7
MSRVTT_jsfusion_test/v2t_metrics/MedR: 7.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 32.195499999999996
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.16436433145058
mnt_best : 34.46736169725739
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.89774 (QuantReg: 11.58838) QuantErr: 11.58838 batch_time=30.39497
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.70143 (QuantReg: 11.85547) QuantErr: 11.85547 batch_time=0.47623
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.77560 (QuantReg: 12.05634) QuantErr: 12.05634 batch_time=0.49081
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.44685 (QuantReg: 11.72468) QuantErr: 11.72468 batch_time=0.48791
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.87022 (QuantReg: 12.22706) QuantErr: 12.22706 batch_time=0.49004
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.72972 (QuantReg: 11.92283) QuantErr: 11.92283 batch_time=0.49194
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 3.25264 (QuantReg: 11.76211) QuantErr: 11.76211 batch_time=0.54063
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.64480 (QuantReg: 12.14319) QuantErr: 12.14319 batch_time=0.49814
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.02798 (QuantReg: 12.34831) QuantErr: 12.34831 batch_time=0.54747
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.58624 (QuantReg: 11.93959) QuantErr: 11.93959 batch_time=0.49801
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.09420 (QuantReg: 11.93083) QuantErr: 11.93083 batch_time=0.49596
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.34979 (QuantReg: 12.13651) QuantErr: 12.13651 batch_time=0.82641
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.36777 (QuantReg: 12.43271) QuantErr: 12.43271 batch_time=0.47940
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.13230 (QuantReg: 12.36703) QuantErr: 12.36703 batch_time=0.49224
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.32208 (QuantReg: 12.62380) QuantErr: 12.62380 batch_time=0.48984
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.28415 (QuantReg: 12.31069) QuantErr: 12.31069 batch_time=0.53337
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.24140 (QuantReg: 12.27098) QuantErr: 12.27098 batch_time=0.48510
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.20690 (QuantReg: 12.69590) QuantErr: 12.69590 batch_time=0.49392
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.07039 (QuantReg: 12.45418) QuantErr: 12.45418 batch_time=0.48761
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.27886 (QuantReg: 12.48251) QuantErr: 12.48251 batch_time=0.49248
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.28371 (QuantReg: 12.45471) QuantErr: 12.45471 batch_time=0.49063
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.24813 (QuantReg: 12.66427) QuantErr: 12.66427 batch_time=0.49366
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.61138 (QuantReg: 12.59312) QuantErr: 12.59312 batch_time=1.25534
Train Epoch: 5 codebook_update_time=1.58527
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch5.pth ...
Done in 9.594s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch5.pth ...
Done in 13.734s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.4375512285232546
quant_reg : 12.257071594238282
quant_err : 12.257071594238282
learning_rate : 8.145062499999998e-05
n_samples : 160000
n_steps : 1250
MSRVTT_jsfusion_test/t2v_metrics/R1: 18.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 46.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 59.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 86.7
MSRVTT_jsfusion_test/t2v_metrics/MedR: 7.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 28.272
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.83568701588683
MSRVTT_jsfusion_test/v2t_metrics/R1: 17.2
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 60.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 86.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.862
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.63318677622725
mnt_best : 36.83568701588683
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.41294 (QuantReg: 11.91495) QuantErr: 11.91495 batch_time=29.25734
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.84078 (QuantReg: 11.70439) QuantErr: 11.70439 batch_time=0.48424
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.47814 (QuantReg: 12.13412) QuantErr: 12.13412 batch_time=0.48945
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.60994 (QuantReg: 12.03449) QuantErr: 12.03449 batch_time=0.47681
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.35396 (QuantReg: 12.17746) QuantErr: 12.17746 batch_time=0.49132
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.24814 (QuantReg: 12.35949) QuantErr: 12.35949 batch_time=0.49446
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.37996 (QuantReg: 12.07577) QuantErr: 12.07577 batch_time=0.48553
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.35173 (QuantReg: 12.36160) QuantErr: 12.36160 batch_time=0.47510
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.21855 (QuantReg: 12.10989) QuantErr: 12.10989 batch_time=0.49193
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.13394 (QuantReg: 12.20411) QuantErr: 12.20411 batch_time=0.60585
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.04014 (QuantReg: 12.29963) QuantErr: 12.29963 batch_time=0.71555
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.31610 (QuantReg: 12.24338) QuantErr: 12.24338 batch_time=0.48990
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.25748 (QuantReg: 12.47534) QuantErr: 12.47534 batch_time=1.40772
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.42721 (QuantReg: 12.47520) QuantErr: 12.47520 batch_time=0.49137
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.36910 (QuantReg: 12.50132) QuantErr: 12.50132 batch_time=0.49583
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.15343 (QuantReg: 12.51692) QuantErr: 12.51692 batch_time=0.48633
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 1.92438 (QuantReg: 12.09575) QuantErr: 12.09575 batch_time=0.48350
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.13901 (QuantReg: 12.64707) QuantErr: 12.64707 batch_time=0.49306
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.24646 (QuantReg: 12.57708) QuantErr: 12.57708 batch_time=0.48378
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.13720 (QuantReg: 12.76378) QuantErr: 12.76378 batch_time=1.34892
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.14121 (QuantReg: 12.80715) QuantErr: 12.80715 batch_time=0.47698
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.01622 (QuantReg: 12.83145) QuantErr: 12.83145 batch_time=0.48412
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.24617 (QuantReg: 12.48486) QuantErr: 12.48486 batch_time=0.48058
Train Epoch: 6 codebook_update_time=1.66467
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch6.pth ...
Done in 3.668s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch6.pth ...
Done in 7.747s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.255416241645813
quant_reg : 12.342533512115478
quant_err : 12.342533512115478
learning_rate : 7.737809374999998e-05
n_samples : 192000
n_steps : 1500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.6
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.762
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.7798244205739
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 27.5135
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.068280997847424
mnt_best : 38.7798244205739
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.05536 (QuantReg: 11.59041) QuantErr: 11.59041 batch_time=25.86617
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.14430 (QuantReg: 12.11360) QuantErr: 12.11360 batch_time=0.47282
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 1.88668 (QuantReg: 12.34693) QuantErr: 12.34693 batch_time=0.48341
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.42834 (QuantReg: 12.24511) QuantErr: 12.24511 batch_time=0.48752
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.20325 (QuantReg: 12.18710) QuantErr: 12.18710 batch_time=0.48853
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 1.81228 (QuantReg: 12.42776) QuantErr: 12.42776 batch_time=0.49744
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 1.89141 (QuantReg: 12.69475) QuantErr: 12.69475 batch_time=0.47473
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 1.72835 (QuantReg: 12.67097) QuantErr: 12.67097 batch_time=0.47686
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 1.86961 (QuantReg: 12.55931) QuantErr: 12.55931 batch_time=0.48042
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.45854 (QuantReg: 12.90419) QuantErr: 12.90419 batch_time=0.47996
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.68176 (QuantReg: 12.23573) QuantErr: 12.23573 batch_time=0.48077
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.04609 (QuantReg: 12.78741) QuantErr: 12.78741 batch_time=0.49053
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.17915 (QuantReg: 12.63390) QuantErr: 12.63390 batch_time=0.48608
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.15453 (QuantReg: 12.76260) QuantErr: 12.76260 batch_time=0.47464
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.63151 (QuantReg: 12.25673) QuantErr: 12.25673 batch_time=0.49662
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.66394 (QuantReg: 12.29717) QuantErr: 12.29717 batch_time=0.49343
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.92588 (QuantReg: 12.81239) QuantErr: 12.81239 batch_time=0.48698
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.16696 (QuantReg: 12.20279) QuantErr: 12.20279 batch_time=0.49073
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.91448 (QuantReg: 12.55758) QuantErr: 12.55758 batch_time=0.48205
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.30352 (QuantReg: 12.69962) QuantErr: 12.69962 batch_time=0.52325
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.08887 (QuantReg: 12.73307) QuantErr: 12.73307 batch_time=0.49789
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.37808 (QuantReg: 12.68667) QuantErr: 12.68667 batch_time=0.47273
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 1.97285 (QuantReg: 12.78296) QuantErr: 12.78296 batch_time=0.54664
Train Epoch: 7 codebook_update_time=1.60531
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch7.pth ...
Done in 4.333s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.0865332236289977
quant_reg : 12.498834690093995
quant_err : 12.498834690093995
learning_rate : 7.350918906249998e-05
n_samples : 224000
n_steps : 1750
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.1
MSRVTT_jsfusion_test/t2v_metrics/R5: 47.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 61.5
MSRVTT_jsfusion_test/t2v_metrics/R50: 87.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 27.79
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.10451720677924
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 47.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 61.8
MSRVTT_jsfusion_test/v2t_metrics/R50: 87.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 26.828
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.3922944608152
mnt_best : 38.7798244205739
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.98904 (QuantReg: 11.93523) QuantErr: 11.93523 batch_time=31.93371
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.16851 (QuantReg: 12.43986) QuantErr: 12.43986 batch_time=0.47591
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.23057 (QuantReg: 12.47167) QuantErr: 12.47167 batch_time=0.47465
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.33309 (QuantReg: 12.39509) QuantErr: 12.39509 batch_time=0.49286
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 1.54938 (QuantReg: 12.51083) QuantErr: 12.51083 batch_time=0.47784
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.78841 (QuantReg: 12.09640) QuantErr: 12.09640 batch_time=0.48859
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.76342 (QuantReg: 12.19335) QuantErr: 12.19335 batch_time=0.47674
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 1.83692 (QuantReg: 12.48532) QuantErr: 12.48532 batch_time=0.47886
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.42906 (QuantReg: 12.11625) QuantErr: 12.11625 batch_time=0.59653
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.16212 (QuantReg: 12.23075) QuantErr: 12.23075 batch_time=0.49976
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 1.87364 (QuantReg: 12.70621) QuantErr: 12.70621 batch_time=0.49121
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 1.94881 (QuantReg: 12.63151) QuantErr: 12.63151 batch_time=0.48696
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.02373 (QuantReg: 12.21127) QuantErr: 12.21127 batch_time=0.50310
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.09444 (QuantReg: 12.79032) QuantErr: 12.79032 batch_time=0.48789
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 1.91097 (QuantReg: 12.51278) QuantErr: 12.51278 batch_time=0.69265
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.16385 (QuantReg: 12.50268) QuantErr: 12.50268 batch_time=0.48912
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 1.47028 (QuantReg: 12.65635) QuantErr: 12.65635 batch_time=0.48614
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.29269 (QuantReg: 12.41413) QuantErr: 12.41413 batch_time=0.49007
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.90664 (QuantReg: 12.65866) QuantErr: 12.65866 batch_time=0.60781
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.84593 (QuantReg: 12.76096) QuantErr: 12.76096 batch_time=0.84388
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.79270 (QuantReg: 12.65623) QuantErr: 12.65623 batch_time=0.51617
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.70496 (QuantReg: 12.45304) QuantErr: 12.45304 batch_time=0.47928
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.61879 (QuantReg: 12.82245) QuantErr: 12.82245 batch_time=0.49176
Train Epoch: 8 codebook_update_time=1.59266
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch8.pth ...
Done in 4.144s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch8.pth ...
Done in 7.929s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 1.9299684081077575
quant_reg : 12.55632773590088
quant_err : 12.55632773590088
learning_rate : 6.983372960937497e-05
n_samples : 256000
n_steps : 2000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.7
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.056
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.982862658837
MSRVTT_jsfusion_test/v2t_metrics/R1: 19.6
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.1
MSRVTT_jsfusion_test/v2t_metrics/R10: 62.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.0
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 25.8405
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.48696723273632
mnt_best : 39.982862658837
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 1.80150 (QuantReg: 12.49873) QuantErr: 12.49873 batch_time=34.05183
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.88849 (QuantReg: 12.22593) QuantErr: 12.22593 batch_time=0.48113
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.59646 (QuantReg: 12.61866) QuantErr: 12.61866 batch_time=0.48088
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.25994 (QuantReg: 12.38442) QuantErr: 12.38442 batch_time=0.47885
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.61608 (QuantReg: 12.48756) QuantErr: 12.48756 batch_time=0.61046
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.87112 (QuantReg: 12.41489) QuantErr: 12.41489 batch_time=0.72507
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.92773 (QuantReg: 12.45654) QuantErr: 12.45654 batch_time=0.53497
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.65231 (QuantReg: 12.47909) QuantErr: 12.47909 batch_time=0.53008
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.67191 (QuantReg: 12.44047) QuantErr: 12.44047 batch_time=0.52046
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.95157 (QuantReg: 12.62650) QuantErr: 12.62650 batch_time=0.79599
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.90296 (QuantReg: 12.66943) QuantErr: 12.66943 batch_time=0.49319
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.95108 (QuantReg: 12.62761) QuantErr: 12.62761 batch_time=0.49755
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.79747 (QuantReg: 12.50569) QuantErr: 12.50569 batch_time=0.62007
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.65645 (QuantReg: 12.39413) QuantErr: 12.39413 batch_time=1.53003
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.54549 (QuantReg: 12.94353) QuantErr: 12.94353 batch_time=0.49245
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.96887 (QuantReg: 12.67731) QuantErr: 12.67731 batch_time=0.47942
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.55461 (QuantReg: 12.54632) QuantErr: 12.54632 batch_time=0.48314
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.84935 (QuantReg: 12.60620) QuantErr: 12.60620 batch_time=0.48375
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.96604 (QuantReg: 12.69258) QuantErr: 12.69258 batch_time=0.49453
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.05535 (QuantReg: 12.74539) QuantErr: 12.74539 batch_time=0.47697
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.34000 (QuantReg: 12.55416) QuantErr: 12.55416 batch_time=0.50054
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.02732 (QuantReg: 12.71974) QuantErr: 12.71974 batch_time=0.52520
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.54750 (QuantReg: 12.92064) QuantErr: 12.92064 batch_time=0.50388
Train Epoch: 9 codebook_update_time=1.60541
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch9.pth ...
Done in 4.207s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 1.7930238671302796
quant_reg : 12.590286758422852
quant_err : 12.590286758422852
learning_rate : 6.634204312890622e-05
n_samples : 288000
n_steps : 2250
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.4
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 62.7
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.1
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.134
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.87515501914658
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.0
MSRVTT_jsfusion_test/v2t_metrics/R5: 48.4
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.3
MSRVTT_jsfusion_test/v2t_metrics/R50: 88.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 6.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.8895
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.423909491713985
mnt_best : 39.982862658837
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.73845 (QuantReg: 12.38152) QuantErr: 12.38152 batch_time=28.96621
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.92685 (QuantReg: 12.32197) QuantErr: 12.32197 batch_time=0.49426
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.72292 (QuantReg: 12.61821) QuantErr: 12.61821 batch_time=0.48332
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.36728 (QuantReg: 12.62850) QuantErr: 12.62850 batch_time=0.48250
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.45317 (QuantReg: 12.76411) QuantErr: 12.76411 batch_time=0.48403
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.12765 (QuantReg: 12.60278) QuantErr: 12.60278 batch_time=0.48790
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.73666 (QuantReg: 12.60190) QuantErr: 12.60190 batch_time=0.48665
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.69921 (QuantReg: 12.57515) QuantErr: 12.57515 batch_time=0.49923
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.80952 (QuantReg: 12.15483) QuantErr: 12.15483 batch_time=0.53507
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.98236 (QuantReg: 12.51050) QuantErr: 12.51050 batch_time=0.48429
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.77192 (QuantReg: 12.67617) QuantErr: 12.67617 batch_time=0.48683
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.50730 (QuantReg: 12.52141) QuantErr: 12.52141 batch_time=0.47932
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.50456 (QuantReg: 12.97289) QuantErr: 12.97289 batch_time=0.48291
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.16583 (QuantReg: 12.59621) QuantErr: 12.59621 batch_time=0.65308
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.47780 (QuantReg: 12.90536) QuantErr: 12.90536 batch_time=0.47514
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.95613 (QuantReg: 12.71877) QuantErr: 12.71877 batch_time=0.49153
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 1.45937 (QuantReg: 12.67209) QuantErr: 12.67209 batch_time=0.47908
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.35043 (QuantReg: 12.68090) QuantErr: 12.68090 batch_time=0.47930
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.97876 (QuantReg: 12.86640) QuantErr: 12.86640 batch_time=0.51616
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.11996 (QuantReg: 12.92795) QuantErr: 12.92795 batch_time=1.00739
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.72061 (QuantReg: 12.67925) QuantErr: 12.67925 batch_time=0.48515
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.84988 (QuantReg: 12.56503) QuantErr: 12.56503 batch_time=0.55129
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.50238 (QuantReg: 12.82526) QuantErr: 12.82526 batch_time=0.60645
Train Epoch: 10 codebook_update_time=1.58768
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch10.pth ...
Done in 3.904s
removing stale ckpt [epoch 9] [took 0.03s]
epoch : 10
loss : 1.714939702987671
quant_reg : 12.675204723358155
quant_err : 12.675204723358155
learning_rate : 6.30249409724609e-05
n_samples : 320000
n_steps : 2500
MSRVTT_jsfusion_test/t2v_metrics/R1: 19.7
MSRVTT_jsfusion_test/t2v_metrics/R5: 48.5
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.126
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.37642413594324
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 63.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.2
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.5
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.319
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.82250136487754
mnt_best : 39.982862658837
not_improved_count: 2
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.51887 (QuantReg: 12.37076) QuantErr: 12.37076 batch_time=23.32701
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.61270 (QuantReg: 12.80917) QuantErr: 12.80917 batch_time=0.52350
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.74465 (QuantReg: 12.44467) QuantErr: 12.44467 batch_time=0.47342
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.77197 (QuantReg: 12.40222) QuantErr: 12.40222 batch_time=0.49851
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.65589 (QuantReg: 12.82126) QuantErr: 12.82126 batch_time=0.49599
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.55550 (QuantReg: 12.52989) QuantErr: 12.52989 batch_time=0.49188
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.74116 (QuantReg: 13.02409) QuantErr: 13.02409 batch_time=0.48006
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.50774 (QuantReg: 12.81854) QuantErr: 12.81854 batch_time=0.47540
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.22177 (QuantReg: 12.42133) QuantErr: 12.42133 batch_time=0.48838
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.53107 (QuantReg: 12.66232) QuantErr: 12.66232 batch_time=0.59382
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.32439 (QuantReg: 12.70685) QuantErr: 12.70685 batch_time=0.48678
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.77809 (QuantReg: 12.79892) QuantErr: 12.79892 batch_time=0.49113
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.85601 (QuantReg: 12.91447) QuantErr: 12.91447 batch_time=0.70629
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.54449 (QuantReg: 13.16202) QuantErr: 13.16202 batch_time=2.01950
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.86760 (QuantReg: 12.73079) QuantErr: 12.73079 batch_time=0.48729
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.82189 (QuantReg: 12.73665) QuantErr: 12.73665 batch_time=0.49188
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.62684 (QuantReg: 12.93566) QuantErr: 12.93566 batch_time=0.49608
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.65344 (QuantReg: 12.89565) QuantErr: 12.89565 batch_time=0.48521
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.54493 (QuantReg: 12.94157) QuantErr: 12.94157 batch_time=0.49839
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.60680 (QuantReg: 12.66587) QuantErr: 12.66587 batch_time=0.49567
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.73722 (QuantReg: 12.67920) QuantErr: 12.67920 batch_time=0.48154
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.43002 (QuantReg: 12.82396) QuantErr: 12.82396 batch_time=0.48741
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.44807 (QuantReg: 13.34846) QuantErr: 13.34846 batch_time=0.48009
Train Epoch: 11 codebook_update_time=1.64598
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch11.pth ...
Done in 4.182s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch11.pth ...
Done in 8.037s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 1.6231155576705933
quant_reg : 12.763282180786133
quant_err : 12.763282180786133
learning_rate : 5.987369392383786e-05
n_samples : 352000
n_steps : 2750
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.9
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.1
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.6
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 24.343
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.15617878602743
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 50.8
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.1
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 22.242
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.08662312343042
mnt_best : 40.15617878602743
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.55471 (QuantReg: 12.50087) QuantErr: 12.50087 batch_time=24.00727
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.80777 (QuantReg: 12.42166) QuantErr: 12.42166 batch_time=0.49604
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.91452 (QuantReg: 12.39489) QuantErr: 12.39489 batch_time=0.50502
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.26912 (QuantReg: 13.17751) QuantErr: 13.17751 batch_time=0.48685
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.70683 (QuantReg: 12.41596) QuantErr: 12.41596 batch_time=0.47925
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.69104 (QuantReg: 12.90051) QuantErr: 12.90051 batch_time=0.47528
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.49341 (QuantReg: 12.77176) QuantErr: 12.77176 batch_time=0.48690
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.65041 (QuantReg: 12.62924) QuantErr: 12.62924 batch_time=0.47114
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.53382 (QuantReg: 12.67207) QuantErr: 12.67207 batch_time=0.69009
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 2.18497 (QuantReg: 11.97397) QuantErr: 11.97397 batch_time=0.80179
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.25403 (QuantReg: 12.64635) QuantErr: 12.64635 batch_time=0.49931
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.60028 (QuantReg: 12.61513) QuantErr: 12.61513 batch_time=0.48735
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.76922 (QuantReg: 12.77681) QuantErr: 12.77681 batch_time=0.48190
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.70009 (QuantReg: 12.51700) QuantErr: 12.51700 batch_time=0.49164
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.68556 (QuantReg: 13.03814) QuantErr: 13.03814 batch_time=0.48211
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.29456 (QuantReg: 12.93789) QuantErr: 12.93789 batch_time=0.48580
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.63271 (QuantReg: 12.81011) QuantErr: 12.81011 batch_time=0.48535
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.47576 (QuantReg: 12.61897) QuantErr: 12.61897 batch_time=0.48128
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.51913 (QuantReg: 12.60090) QuantErr: 12.60090 batch_time=0.48011
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.25844 (QuantReg: 13.15169) QuantErr: 13.15169 batch_time=0.51891
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.32345 (QuantReg: 13.05747) QuantErr: 13.05747 batch_time=0.53454
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.66706 (QuantReg: 12.89366) QuantErr: 12.89366 batch_time=0.49283
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.50077 (QuantReg: 12.99018) QuantErr: 12.99018 batch_time=0.71309
Train Epoch: 12 codebook_update_time=1.89660
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch12.pth ...
Done in 4.032s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch12.pth ...
Done in 8.078s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 1.5383011283874513
quant_reg : 12.778900856018067
quant_err : 12.778900856018067
learning_rate : 5.688000922764596e-05
n_samples : 384000
n_steps : 3000
MSRVTT_jsfusion_test/t2v_metrics/R1: 22.5
MSRVTT_jsfusion_test/t2v_metrics/R5: 49.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.0
MSRVTT_jsfusion_test/t2v_metrics/R50: 88.9
MSRVTT_jsfusion_test/t2v_metrics/MedR: 6.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 26.239
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.3505535831905
MSRVTT_jsfusion_test/v2t_metrics/R1: 21.4
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 64.2
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 24.7165
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.27999038519095
mnt_best : 41.3505535831905
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.53680 (QuantReg: 12.75366) QuantErr: 12.75366 batch_time=25.45378
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.38285 (QuantReg: 12.58187) QuantErr: 12.58187 batch_time=0.48077
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.60631 (QuantReg: 12.24889) QuantErr: 12.24889 batch_time=1.08944
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.58489 (QuantReg: 12.76478) QuantErr: 12.76478 batch_time=0.49764
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.37290 (QuantReg: 12.69887) QuantErr: 12.69887 batch_time=0.48106
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.24582 (QuantReg: 12.66505) QuantErr: 12.66505 batch_time=0.49970
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.35255 (QuantReg: 12.82354) QuantErr: 12.82354 batch_time=0.48558
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.30705 (QuantReg: 12.56483) QuantErr: 12.56483 batch_time=0.48837
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.49875 (QuantReg: 12.81720) QuantErr: 12.81720 batch_time=0.49789
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.64584 (QuantReg: 12.65969) QuantErr: 12.65969 batch_time=0.53811
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.46295 (QuantReg: 12.72056) QuantErr: 12.72056 batch_time=0.49238
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.64694 (QuantReg: 12.69824) QuantErr: 12.69824 batch_time=0.49567
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.20366 (QuantReg: 13.04296) QuantErr: 13.04296 batch_time=0.49286
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.23679 (QuantReg: 12.88866) QuantErr: 12.88866 batch_time=3.02000
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.56723 (QuantReg: 12.92328) QuantErr: 12.92328 batch_time=0.52421
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.34051 (QuantReg: 12.84668) QuantErr: 12.84668 batch_time=0.74578
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.21430 (QuantReg: 13.09309) QuantErr: 13.09309 batch_time=0.49402
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.41553 (QuantReg: 13.05190) QuantErr: 13.05190 batch_time=0.52735
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.77948 (QuantReg: 12.94874) QuantErr: 12.94874 batch_time=0.71578
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.32693 (QuantReg: 12.90221) QuantErr: 12.90221 batch_time=0.47487
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.11811 (QuantReg: 12.99386) QuantErr: 12.99386 batch_time=0.47915
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.66525 (QuantReg: 12.83367) QuantErr: 12.83367 batch_time=0.49576
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.24226 (QuantReg: 13.13151) QuantErr: 13.13151 batch_time=0.49318
Train Epoch: 13 codebook_update_time=1.65551
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch13.pth ...
Done in 13.685s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 1.4744682831764222
quant_reg : 12.833291820526123
quant_err : 12.833291820526123
learning_rate : 5.4036008766263664e-05
n_samples : 416000
n_steps : 3250
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.3
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.3
MSRVTT_jsfusion_test/t2v_metrics/R10: 63.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 90.2
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 24.791
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.50214529102553
MSRVTT_jsfusion_test/v2t_metrics/R1: 22.3
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.2
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.1
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.6
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 22.616
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 42.045411200309466
mnt_best : 41.3505535831905
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.45016 (QuantReg: 12.87608) QuantErr: 12.87608 batch_time=25.36863
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.56168 (QuantReg: 13.07539) QuantErr: 13.07539 batch_time=0.49168
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.36316 (QuantReg: 12.67556) QuantErr: 12.67556 batch_time=0.48579
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.10013 (QuantReg: 12.72080) QuantErr: 12.72080 batch_time=0.48316
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.13192 (QuantReg: 13.02445) QuantErr: 13.02445 batch_time=0.48452
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.46095 (QuantReg: 12.80134) QuantErr: 12.80134 batch_time=0.48775
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.28055 (QuantReg: 12.88980) QuantErr: 12.88980 batch_time=1.27152
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.09143 (QuantReg: 12.88371) QuantErr: 12.88371 batch_time=0.48670
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.73628 (QuantReg: 12.92683) QuantErr: 12.92683 batch_time=0.50258
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.75223 (QuantReg: 12.68729) QuantErr: 12.68729 batch_time=0.48256
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.65696 (QuantReg: 12.66437) QuantErr: 12.66437 batch_time=0.48719
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.16409 (QuantReg: 13.20796) QuantErr: 13.20796 batch_time=0.48464
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.81849 (QuantReg: 12.89984) QuantErr: 12.89984 batch_time=0.49376
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.42182 (QuantReg: 12.99421) QuantErr: 12.99421 batch_time=0.49505
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.41244 (QuantReg: 12.77862) QuantErr: 12.77862 batch_time=0.71101
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.38422 (QuantReg: 12.66807) QuantErr: 12.66807 batch_time=0.49529
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.35911 (QuantReg: 12.98553) QuantErr: 12.98553 batch_time=1.05544
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.34188 (QuantReg: 12.73097) QuantErr: 12.73097 batch_time=0.48437
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.58856 (QuantReg: 12.83863) QuantErr: 12.83863 batch_time=0.50045
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.13729 (QuantReg: 13.36257) QuantErr: 13.36257 batch_time=0.49254
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.48462 (QuantReg: 13.36749) QuantErr: 13.36749 batch_time=0.48819
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.34147 (QuantReg: 13.12947) QuantErr: 13.12947 batch_time=0.49781
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.23828 (QuantReg: 13.15486) QuantErr: 13.15486 batch_time=0.49129
Train Epoch: 14 codebook_update_time=1.75678
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch14.pth ...
Done in 4.020s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch14.pth ...
Done in 24.385s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 1.3964751441478729
quant_reg : 12.953193359375
quant_err : 12.953193359375
learning_rate : 5.133420832795048e-05
n_samples : 448000
n_steps : 3500
MSRVTT_jsfusion_test/t2v_metrics/R1: 21.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 51.2
MSRVTT_jsfusion_test/t2v_metrics/R10: 65.9
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.4
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.681
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.51105255525371
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.5
MSRVTT_jsfusion_test/v2t_metrics/R5: 52.3
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.6
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.9
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.4765
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.27810927682134
mnt_best : 41.51105255525371
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.39598 (QuantReg: 13.07109) QuantErr: 13.07109 batch_time=26.23247
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.57174 (QuantReg: 12.90043) QuantErr: 12.90043 batch_time=0.82421
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.51437 (QuantReg: 12.81585) QuantErr: 12.81585 batch_time=1.43528
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.46503 (QuantReg: 12.66467) QuantErr: 12.66467 batch_time=0.48290
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.61961 (QuantReg: 12.59982) QuantErr: 12.59982 batch_time=0.49238
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.30888 (QuantReg: 13.08152) QuantErr: 13.08152 batch_time=0.48681
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.34714 (QuantReg: 13.20964) QuantErr: 13.20964 batch_time=0.49229
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.50099 (QuantReg: 13.22906) QuantErr: 13.22906 batch_time=0.49208
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.44051 (QuantReg: 12.78274) QuantErr: 12.78274 batch_time=0.59959
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.05045 (QuantReg: 12.86462) QuantErr: 12.86462 batch_time=0.48111
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.02284 (QuantReg: 13.10267) QuantErr: 13.10267 batch_time=0.47382
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.29459 (QuantReg: 13.07288) QuantErr: 13.07288 batch_time=0.72568
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.35649 (QuantReg: 13.26511) QuantErr: 13.26511 batch_time=0.48126
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.60983 (QuantReg: 13.04230) QuantErr: 13.04230 batch_time=0.48440
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.32180 (QuantReg: 13.07542) QuantErr: 13.07542 batch_time=0.48611
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.31836 (QuantReg: 12.88226) QuantErr: 12.88226 batch_time=0.48295
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.33392 (QuantReg: 13.01900) QuantErr: 13.01900 batch_time=0.48539
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.36416 (QuantReg: 12.84391) QuantErr: 12.84391 batch_time=0.48244
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.46462 (QuantReg: 13.50912) QuantErr: 13.50912 batch_time=0.50018
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.56603 (QuantReg: 12.99232) QuantErr: 12.99232 batch_time=0.48289
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.51479 (QuantReg: 13.34737) QuantErr: 13.34737 batch_time=0.48733
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.37176 (QuantReg: 13.39953) QuantErr: 13.39953 batch_time=0.47346
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.36372 (QuantReg: 13.00748) QuantErr: 13.00748 batch_time=0.83591
Train Epoch: 15 codebook_update_time=1.61014
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch15.pth ...
Done in 4.021s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch15.pth ...
Done in 7.993s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 1.337243925333023
quant_reg : 12.992572593688966
quant_err : 12.992572593688966
learning_rate : 4.876749791155295e-05
n_samples : 480000
n_steps : 3750
MSRVTT_jsfusion_test/t2v_metrics/R1: 23.2
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.1
MSRVTT_jsfusion_test/t2v_metrics/R10: 64.8
MSRVTT_jsfusion_test/t2v_metrics/R50: 89.5
MSRVTT_jsfusion_test/t2v_metrics/MedR: 5.0
MSRVTT_jsfusion_test/t2v_metrics/MeanR: 25.639
MSRVTT_jsfusion_test/t2v_metrics/geometric_mean_R1-R5-R10: 42.231214598490844
MSRVTT_jsfusion_test/v2t_metrics/R1: 20.7
MSRVTT_jsfusion_test/v2t_metrics/R5: 51.6
MSRVTT_jsfusion_test/v2t_metrics/R10: 65.7
MSRVTT_jsfusion_test/v2t_metrics/R50: 89.4
MSRVTT_jsfusion_test/v2t_metrics/MedR: 5.0
MSRVTT_jsfusion_test/v2t_metrics/MeanR: 23.605
MSRVTT_jsfusion_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.24726328951273
mnt_best : 42.231214598490844
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.47112 (QuantReg: 12.66747) QuantErr: 12.66747 batch_time=28.65316
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.04482 (QuantReg: 12.91136) QuantErr: 12.91136 batch_time=0.48054
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.20600 (QuantReg: 12.95227) QuantErr: 12.95227 batch_time=0.69958
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.17886 (QuantReg: 12.99290) QuantErr: 12.99290 batch_time=0.47500
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.25344 (QuantReg: 13.00930) QuantErr: 13.00930 batch_time=0.48857
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.24100 (QuantReg: 13.24392) QuantErr: 13.24392 batch_time=0.47398
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.27682 (QuantReg: 12.95119) QuantErr: 12.95119 batch_time=0.47895
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.37431 (QuantReg: 12.90666) QuantErr: 12.90666 batch_time=0.48050
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.24280 (QuantReg: 12.86123) QuantErr: 12.86123 batch_time=0.47159
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.27380 (QuantReg: 12.85417) QuantErr: 12.85417 batch_time=0.48251
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.62249 (QuantReg: 12.98039) QuantErr: 12.98039 batch_time=0.48345
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.37555 (QuantReg: 13.17805) QuantErr: 13.17805 batch_time=0.46962
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.37282 (QuantReg: 13.12674) QuantErr: 13.12674 batch_time=0.48009
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.59639 (QuantReg: 12.76551) QuantErr: 12.76551 batch_time=0.52187
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.44740 (QuantReg: 13.49054) QuantErr: 13.49054 batch_time=0.47413
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.40160 (QuantReg: 13.34806) QuantErr: 13.34806 batch_time=0.47119
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.29097 (QuantReg: 13.08693) QuantErr: 13.08693 batch_time=0.47195
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.52462 (QuantReg: 13.10785) QuantErr: 13.10785 batch_time=0.46922
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.48923 (QuantReg: 12.76772) QuantErr: 12.76772 batch_time=0.48477
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.26484 (QuantReg: 12.82559) QuantErr: 12.82559 batch_time=0.48541
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.35573 (QuantReg: 12.91451) QuantErr: 12.91451 batch_time=0.58657
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.05985 (QuantReg: 13.11012) QuantErr: 13.11012 batch_time=0.58155
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.11124 (QuantReg: 13.10870) QuantErr: 13.10870 batch_time=0.48303
Train Epoch: 16 codebook_update_time=1.73951
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kA_roberta-base/checkpoint-epoch16.pth ...
Done in 12.521s
removing stale ckpt [epoch 15] [took 0.00s]
epoch : 16
loss : 1.2834236388206481
quant_reg : 12.99853598022461
quant_err : 12.99853598022461
learning_rate : 4.6329123015975305e-05
n_samples : 512000
n_steps : 4000
MSRVTT_jsfusion_test/t2v_metrics/R1: 20.8
MSRVTT_jsfusion_test/t2v_metrics/R5: 50.6