-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kB_t0.1.txt
2593 lines (2593 loc) · 190 KB
/
HCQ_MSRVTT_1kB_t0.1.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1
Preparing the dataloaders ...
Loading dataset MSRVTT_miech_trainval in ram ...
Finish loading dataset MSRVTT_miech_trainval in ram, taking 943.3343877792358 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 121.37042355537415 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 97.25923752784729 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch0.pth ...
Done in 8.292s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch0.pth ...
Done in 10.161s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_miech_test/t2v_metrics/R1: 0.1
MSRVTT_miech_test/t2v_metrics/R5: 0.6
MSRVTT_miech_test/t2v_metrics/R10: 1.0
MSRVTT_miech_test/t2v_metrics/R50: 5.0
MSRVTT_miech_test/t2v_metrics/MedR: 503.0
MSRVTT_miech_test/t2v_metrics/MeanR: 505.193
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3914867641168864
MSRVTT_miech_test/v2t_metrics/R1: 0.1
MSRVTT_miech_test/v2t_metrics/R5: 0.4
MSRVTT_miech_test/v2t_metrics/R10: 1.0
MSRVTT_miech_test/v2t_metrics/R50: 5.4
MSRVTT_miech_test/v2t_metrics/MedR: 511.5
MSRVTT_miech_test/v2t_metrics/MeanR: 499.894
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.3419951893353394
mnt_best : 0.3914867641168864
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.72230 (QuantReg: 22.47940) QuantErr: 22.47940 batch_time=41.67038
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.64002 (QuantReg: 22.50946) QuantErr: 22.50946 batch_time=0.50814
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.47146 (QuantReg: 22.62173) QuantErr: 22.62173 batch_time=0.51138
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.64771 (QuantReg: 22.65363) QuantErr: 22.65363 batch_time=0.50414
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.67464 (QuantReg: 22.66921) QuantErr: 22.66921 batch_time=0.51926
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.34200 (QuantReg: 22.67845) QuantErr: 22.67845 batch_time=0.53705
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.26114 (QuantReg: 22.64800) QuantErr: 22.64800 batch_time=0.52590
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.80840 (QuantReg: 22.64152) QuantErr: 22.64152 batch_time=0.53129
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.47011 (QuantReg: 22.61757) QuantErr: 22.61757 batch_time=0.52324
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.40088 (QuantReg: 22.66103) QuantErr: 22.66103 batch_time=0.50726
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.34114 (QuantReg: 22.67394) QuantErr: 22.67394 batch_time=0.53271
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.51318 (QuantReg: 22.60574) QuantErr: 22.60574 batch_time=0.58605
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.98978 (QuantReg: 22.67563) QuantErr: 22.67563 batch_time=0.51087
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 5.02869 (QuantReg: 22.66515) QuantErr: 22.66515 batch_time=0.53764
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.84557 (QuantReg: 22.68286) QuantErr: 22.68286 batch_time=0.52153
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.72654 (QuantReg: 22.63576) QuantErr: 22.63576 batch_time=0.51716
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.51949 (QuantReg: 22.66547) QuantErr: 22.66547 batch_time=0.51608
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.92298 (QuantReg: 22.66599) QuantErr: 22.66599 batch_time=0.50655
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.81338 (QuantReg: 22.69030) QuantErr: 22.69030 batch_time=0.53462
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.87545 (QuantReg: 22.69123) QuantErr: 22.69123 batch_time=0.51104
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.82267 (QuantReg: 22.64202) QuantErr: 22.64202 batch_time=0.51289
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 5.07401 (QuantReg: 22.63673) QuantErr: 22.63673 batch_time=0.50804
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.41478 (QuantReg: 22.65041) QuantErr: 22.65041 batch_time=0.51008
Train Epoch: 1 codebook_update_time=1.84366
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch1.pth ...
Done in 4.284s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch1.pth ...
Done in 8.429s
epoch : 1
loss : 5.6730799369812015
quant_reg : 22.640385612487794
quant_err : 22.640385612487794
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_miech_test/t2v_metrics/R1: 8.0
MSRVTT_miech_test/t2v_metrics/R5: 28.3
MSRVTT_miech_test/t2v_metrics/R10: 41.7
MSRVTT_miech_test/t2v_metrics/R50: 77.5
MSRVTT_miech_test/t2v_metrics/MedR: 15.0
MSRVTT_miech_test/t2v_metrics/MeanR: 48.802
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.135092804534903
MSRVTT_miech_test/v2t_metrics/R1: 8.7
MSRVTT_miech_test/v2t_metrics/R5: 29.8
MSRVTT_miech_test/v2t_metrics/R10: 42.9
MSRVTT_miech_test/v2t_metrics/R50: 77.1
MSRVTT_miech_test/v2t_metrics/MedR: 14.0
MSRVTT_miech_test/v2t_metrics/MeanR: 48.3175
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.32188859419169
mnt_best : 21.135092804534903
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.35883 (QuantReg: 10.94703) QuantErr: 10.94703 batch_time=36.06653
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.27459 (QuantReg: 11.19126) QuantErr: 11.19126 batch_time=1.48216
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.57432 (QuantReg: 11.30709) QuantErr: 11.30709 batch_time=0.50953
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.36338 (QuantReg: 11.22923) QuantErr: 11.22923 batch_time=0.53402
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.02154 (QuantReg: 11.39223) QuantErr: 11.39223 batch_time=0.55477
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.99587 (QuantReg: 11.23298) QuantErr: 11.23298 batch_time=0.53049
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.14185 (QuantReg: 11.63540) QuantErr: 11.63540 batch_time=0.51139
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.87280 (QuantReg: 11.46874) QuantErr: 11.46874 batch_time=0.76299
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.87990 (QuantReg: 11.81813) QuantErr: 11.81813 batch_time=0.51276
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.23988 (QuantReg: 12.11911) QuantErr: 12.11911 batch_time=0.52158
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 4.00805 (QuantReg: 11.97370) QuantErr: 11.97370 batch_time=0.54191
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.46303 (QuantReg: 11.99310) QuantErr: 11.99310 batch_time=0.51490
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.98622 (QuantReg: 12.11781) QuantErr: 12.11781 batch_time=4.83811
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.18213 (QuantReg: 12.75301) QuantErr: 12.75301 batch_time=0.77265
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 4.04008 (QuantReg: 12.23265) QuantErr: 12.23265 batch_time=0.50972
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 4.41825 (QuantReg: 12.05157) QuantErr: 12.05157 batch_time=0.51206
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.93034 (QuantReg: 12.06025) QuantErr: 12.06025 batch_time=0.51102
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.39619 (QuantReg: 12.32296) QuantErr: 12.32296 batch_time=0.51255
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 4.28937 (QuantReg: 12.55606) QuantErr: 12.55606 batch_time=0.51655
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.99720 (QuantReg: 12.28420) QuantErr: 12.28420 batch_time=0.51633
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 4.07309 (QuantReg: 12.69162) QuantErr: 12.69162 batch_time=0.52092
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.64540 (QuantReg: 12.96565) QuantErr: 12.96565 batch_time=0.55540
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.63158 (QuantReg: 12.98433) QuantErr: 12.98433 batch_time=0.50948
Train Epoch: 2 codebook_update_time=1.95846
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch2.pth ...
Done in 3.854s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch2.pth ...
Done in 7.529s
removing stale ckpt [epoch 1] [took 0.02s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 4.075274127960205
quant_reg : 12.03143357849121
quant_err : 12.03143357849121
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_miech_test/t2v_metrics/R1: 11.8
MSRVTT_miech_test/t2v_metrics/R5: 34.9
MSRVTT_miech_test/t2v_metrics/R10: 49.7
MSRVTT_miech_test/t2v_metrics/R50: 82.4
MSRVTT_miech_test/t2v_metrics/MedR: 11.0
MSRVTT_miech_test/t2v_metrics/MeanR: 42.922
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.354027193577803
MSRVTT_miech_test/v2t_metrics/R1: 12.9
MSRVTT_miech_test/v2t_metrics/R5: 35.2
MSRVTT_miech_test/v2t_metrics/R10: 49.5
MSRVTT_miech_test/v2t_metrics/R50: 82.8
MSRVTT_miech_test/v2t_metrics/MedR: 11.0
MSRVTT_miech_test/v2t_metrics/MeanR: 42.553
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.22144136646335
mnt_best : 27.354027193577803
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 4.07650 (QuantReg: 10.13940) QuantErr: 10.13940 batch_time=33.01003
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.75603 (QuantReg: 10.26713) QuantErr: 10.26713 batch_time=0.52152
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.79755 (QuantReg: 10.09891) QuantErr: 10.09891 batch_time=0.53644
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 4.09943 (QuantReg: 9.88274) QuantErr: 9.88274 batch_time=0.54878
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.65762 (QuantReg: 10.70682) QuantErr: 10.70682 batch_time=0.52331
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.72619 (QuantReg: 10.33597) QuantErr: 10.33597 batch_time=0.52554
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.41488 (QuantReg: 10.22931) QuantErr: 10.22931 batch_time=0.51602
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.43139 (QuantReg: 10.01607) QuantErr: 10.01607 batch_time=0.55743
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.61417 (QuantReg: 10.49059) QuantErr: 10.49059 batch_time=0.53099
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.67896 (QuantReg: 10.49957) QuantErr: 10.49957 batch_time=0.52006
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.47806 (QuantReg: 10.17428) QuantErr: 10.17428 batch_time=0.52393
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.59978 (QuantReg: 10.59207) QuantErr: 10.59207 batch_time=0.51956
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.73462 (QuantReg: 10.74407) QuantErr: 10.74407 batch_time=0.52186
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.50814 (QuantReg: 10.57357) QuantErr: 10.57357 batch_time=1.08605
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.42659 (QuantReg: 10.29569) QuantErr: 10.29569 batch_time=0.53777
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.32433 (QuantReg: 10.42011) QuantErr: 10.42011 batch_time=0.52166
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.15849 (QuantReg: 10.57942) QuantErr: 10.57942 batch_time=0.54597
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.62642 (QuantReg: 10.53445) QuantErr: 10.53445 batch_time=0.52359
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.85007 (QuantReg: 11.01294) QuantErr: 11.01294 batch_time=0.71866
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.28840 (QuantReg: 10.77015) QuantErr: 10.77015 batch_time=0.54764
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.66740 (QuantReg: 10.86063) QuantErr: 10.86063 batch_time=0.51731
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.07587 (QuantReg: 10.80838) QuantErr: 10.80838 batch_time=0.51831
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.11673 (QuantReg: 10.86524) QuantErr: 10.86524 batch_time=0.50591
Train Epoch: 3 codebook_update_time=1.79792
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch3.pth ...
Done in 4.216s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch3.pth ...
Done in 8.381s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.580486979484558
quant_reg : 10.458792362213135
quant_err : 10.458792362213135
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_miech_test/t2v_metrics/R1: 13.8
MSRVTT_miech_test/t2v_metrics/R5: 37.9
MSRVTT_miech_test/t2v_metrics/R10: 50.6
MSRVTT_miech_test/t2v_metrics/R50: 84.3
MSRVTT_miech_test/t2v_metrics/MedR: 10.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.144
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.80045793029321
MSRVTT_miech_test/v2t_metrics/R1: 14.8
MSRVTT_miech_test/v2t_metrics/R5: 38.8
MSRVTT_miech_test/v2t_metrics/R10: 53.6
MSRVTT_miech_test/v2t_metrics/R50: 84.8
MSRVTT_miech_test/v2t_metrics/MedR: 9.0
MSRVTT_miech_test/v2t_metrics/MeanR: 38.892
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.339068011278247
mnt_best : 29.80045793029321
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.28303 (QuantReg: 9.71070) QuantErr: 9.71070 batch_time=32.21929
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.30833 (QuantReg: 10.14216) QuantErr: 10.14216 batch_time=0.52976
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 3.04573 (QuantReg: 9.44252) QuantErr: 9.44252 batch_time=0.53513
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.49841 (QuantReg: 10.03769) QuantErr: 10.03769 batch_time=0.53225
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.36002 (QuantReg: 10.00690) QuantErr: 10.00690 batch_time=0.58821
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.98409 (QuantReg: 10.00256) QuantErr: 10.00256 batch_time=0.61075
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.51213 (QuantReg: 9.83049) QuantErr: 9.83049 batch_time=2.11262
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.96155 (QuantReg: 9.73350) QuantErr: 9.73350 batch_time=0.54112
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.25133 (QuantReg: 9.98772) QuantErr: 9.98772 batch_time=0.52290
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.19425 (QuantReg: 10.14650) QuantErr: 10.14650 batch_time=0.54693
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.42727 (QuantReg: 10.13355) QuantErr: 10.13355 batch_time=0.52436
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.29740 (QuantReg: 9.78214) QuantErr: 9.78214 batch_time=0.55345
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.02734 (QuantReg: 10.20604) QuantErr: 10.20604 batch_time=0.51601
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 3.13042 (QuantReg: 9.85673) QuantErr: 9.85673 batch_time=0.51241
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.14934 (QuantReg: 10.11750) QuantErr: 10.11750 batch_time=0.50383
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 3.47005 (QuantReg: 9.89787) QuantErr: 9.89787 batch_time=0.52331
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.28595 (QuantReg: 10.38414) QuantErr: 10.38414 batch_time=0.60595
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 3.35976 (QuantReg: 9.93310) QuantErr: 9.93310 batch_time=0.67262
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 3.32949 (QuantReg: 10.63635) QuantErr: 10.63635 batch_time=0.57634
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 3.01591 (QuantReg: 9.91793) QuantErr: 9.91793 batch_time=0.54740
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.96696 (QuantReg: 10.49404) QuantErr: 10.49404 batch_time=0.76339
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.92909 (QuantReg: 10.26286) QuantErr: 10.26286 batch_time=0.52487
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.91172 (QuantReg: 10.22083) QuantErr: 10.22083 batch_time=0.52419
Train Epoch: 4 codebook_update_time=1.73101
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch4.pth ...
Done in 22.729s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch4.pth ...
Done in 27.131s
removing stale ckpt [epoch 3] [took 0.02s]
epoch : 4
loss : 3.2285922117233277
quant_reg : 10.043009880065918
quant_err : 10.043009880065918
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_miech_test/t2v_metrics/R1: 14.3
MSRVTT_miech_test/t2v_metrics/R5: 39.1
MSRVTT_miech_test/t2v_metrics/R10: 52.5
MSRVTT_miech_test/t2v_metrics/R50: 84.8
MSRVTT_miech_test/t2v_metrics/MedR: 9.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.599
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.84778836874538
MSRVTT_miech_test/v2t_metrics/R1: 16.3
MSRVTT_miech_test/v2t_metrics/R5: 41.3
MSRVTT_miech_test/v2t_metrics/R10: 56.3
MSRVTT_miech_test/v2t_metrics/R50: 85.5
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.706
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.590413518429244
mnt_best : 30.84778836874538
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.64044 (QuantReg: 9.59756) QuantErr: 9.59756 batch_time=32.41814
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.76622 (QuantReg: 9.80322) QuantErr: 9.80322 batch_time=8.26792
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.96392 (QuantReg: 9.71893) QuantErr: 9.71893 batch_time=0.51497
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 3.02113 (QuantReg: 9.73229) QuantErr: 9.73229 batch_time=0.73829
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 3.37525 (QuantReg: 9.83774) QuantErr: 9.83774 batch_time=0.51464
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.70062 (QuantReg: 9.82506) QuantErr: 9.82506 batch_time=0.51222
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 3.17877 (QuantReg: 9.79034) QuantErr: 9.79034 batch_time=0.55401
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.11674 (QuantReg: 9.63100) QuantErr: 9.63100 batch_time=0.52175
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.92160 (QuantReg: 9.63146) QuantErr: 9.63146 batch_time=0.51633
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.88713 (QuantReg: 10.05327) QuantErr: 10.05327 batch_time=0.54627
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 3.04236 (QuantReg: 10.05292) QuantErr: 10.05292 batch_time=0.56440
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 3.14907 (QuantReg: 9.69807) QuantErr: 9.69807 batch_time=0.52024
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 3.15750 (QuantReg: 10.10869) QuantErr: 10.10869 batch_time=0.65739
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 3.09971 (QuantReg: 9.63304) QuantErr: 9.63304 batch_time=0.52656
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.87441 (QuantReg: 9.50782) QuantErr: 9.50782 batch_time=0.52125
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.43543 (QuantReg: 10.00522) QuantErr: 10.00522 batch_time=0.55639
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 3.02525 (QuantReg: 10.02216) QuantErr: 10.02216 batch_time=0.51224
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 3.00020 (QuantReg: 9.98160) QuantErr: 9.98160 batch_time=0.57224
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.92672 (QuantReg: 10.01576) QuantErr: 10.01576 batch_time=0.80088
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 3.10399 (QuantReg: 9.99133) QuantErr: 9.99133 batch_time=0.53284
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 3.08859 (QuantReg: 9.90429) QuantErr: 9.90429 batch_time=0.53186
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.41198 (QuantReg: 9.71272) QuantErr: 9.71272 batch_time=0.52510
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.73577 (QuantReg: 10.14740) QuantErr: 10.14740 batch_time=0.52148
Train Epoch: 5 codebook_update_time=1.80370
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch5.pth ...
Done in 3.688s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch5.pth ...
Done in 7.408s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 3.0155208778381346
quant_reg : 9.820034397125244
quant_err : 9.820034397125244
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_miech_test/t2v_metrics/R1: 15.0
MSRVTT_miech_test/t2v_metrics/R5: 41.4
MSRVTT_miech_test/t2v_metrics/R10: 55.7
MSRVTT_miech_test/t2v_metrics/R50: 85.8
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.162
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.582339538082884
MSRVTT_miech_test/v2t_metrics/R1: 16.2
MSRVTT_miech_test/v2t_metrics/R5: 41.9
MSRVTT_miech_test/v2t_metrics/R10: 56.5
MSRVTT_miech_test/v2t_metrics/R50: 85.8
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.2755
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.72297095186919
mnt_best : 32.582339538082884
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.92711 (QuantReg: 9.72973) QuantErr: 9.72973 batch_time=29.69190
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.64735 (QuantReg: 9.80222) QuantErr: 9.80222 batch_time=0.53673
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.97693 (QuantReg: 9.31839) QuantErr: 9.31839 batch_time=1.33506
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 3.19522 (QuantReg: 9.58435) QuantErr: 9.58435 batch_time=0.51448
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.80124 (QuantReg: 9.31759) QuantErr: 9.31759 batch_time=0.59478
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 3.04602 (QuantReg: 9.78138) QuantErr: 9.78138 batch_time=0.55125
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.90385 (QuantReg: 9.74302) QuantErr: 9.74302 batch_time=0.51440
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.84223 (QuantReg: 9.37304) QuantErr: 9.37304 batch_time=0.50214
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.90608 (QuantReg: 9.49182) QuantErr: 9.49182 batch_time=0.52185
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.73125 (QuantReg: 9.65800) QuantErr: 9.65800 batch_time=1.40496
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.67163 (QuantReg: 9.84934) QuantErr: 9.84934 batch_time=0.52512
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 3.12696 (QuantReg: 10.03584) QuantErr: 10.03584 batch_time=0.55330
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.87378 (QuantReg: 9.62717) QuantErr: 9.62717 batch_time=0.57005
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 3.21268 (QuantReg: 10.02894) QuantErr: 10.02894 batch_time=0.52558
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.99810 (QuantReg: 9.51510) QuantErr: 9.51510 batch_time=0.51553
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.74044 (QuantReg: 9.80906) QuantErr: 9.80906 batch_time=0.54072
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.99306 (QuantReg: 9.72490) QuantErr: 9.72490 batch_time=0.51769
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 3.12882 (QuantReg: 9.85230) QuantErr: 9.85230 batch_time=0.51113
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.43263 (QuantReg: 9.74819) QuantErr: 9.74819 batch_time=0.51645
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 3.04845 (QuantReg: 9.66003) QuantErr: 9.66003 batch_time=0.99936
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.82802 (QuantReg: 9.91221) QuantErr: 9.91221 batch_time=0.51664
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.69337 (QuantReg: 10.12782) QuantErr: 10.12782 batch_time=0.52350
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.62216 (QuantReg: 9.65920) QuantErr: 9.65920 batch_time=0.57511
Train Epoch: 6 codebook_update_time=1.73085
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch6.pth ...
Done in 10.845s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch6.pth ...
Done in 28.198s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.8092761640548707
quant_reg : 9.757548286437988
quant_err : 9.757548286437988
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_miech_test/t2v_metrics/R1: 16.9
MSRVTT_miech_test/t2v_metrics/R5: 42.0
MSRVTT_miech_test/t2v_metrics/R10: 57.1
MSRVTT_miech_test/t2v_metrics/R50: 86.4
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.674
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.34978589562058
MSRVTT_miech_test/v2t_metrics/R1: 17.5
MSRVTT_miech_test/v2t_metrics/R5: 43.9
MSRVTT_miech_test/v2t_metrics/R10: 57.6
MSRVTT_miech_test/v2t_metrics/R50: 87.1
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.6055
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.370539503435126
mnt_best : 34.34978589562058
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.92468 (QuantReg: 9.69301) QuantErr: 9.69301 batch_time=24.93908
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.75190 (QuantReg: 9.45318) QuantErr: 9.45318 batch_time=0.50622
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.85400 (QuantReg: 10.05752) QuantErr: 10.05752 batch_time=0.51138
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.96827 (QuantReg: 9.57611) QuantErr: 9.57611 batch_time=0.51140
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.51866 (QuantReg: 9.74565) QuantErr: 9.74565 batch_time=0.52430
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.69599 (QuantReg: 9.55149) QuantErr: 9.55149 batch_time=0.51560
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.74184 (QuantReg: 9.86889) QuantErr: 9.86889 batch_time=0.56371
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.71126 (QuantReg: 9.55437) QuantErr: 9.55437 batch_time=0.53265
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.60686 (QuantReg: 9.85972) QuantErr: 9.85972 batch_time=0.55328
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.74004 (QuantReg: 9.93062) QuantErr: 9.93062 batch_time=0.52483
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.72927 (QuantReg: 9.53924) QuantErr: 9.53924 batch_time=0.53930
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.96167 (QuantReg: 9.81576) QuantErr: 9.81576 batch_time=0.52292
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.51415 (QuantReg: 9.54499) QuantErr: 9.54499 batch_time=4.43979
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.43558 (QuantReg: 9.70588) QuantErr: 9.70588 batch_time=1.21242
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.69725 (QuantReg: 9.94485) QuantErr: 9.94485 batch_time=0.74487
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.82530 (QuantReg: 10.00070) QuantErr: 10.00070 batch_time=0.51444
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 2.42684 (QuantReg: 9.81124) QuantErr: 9.81124 batch_time=0.55992
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.70126 (QuantReg: 9.52020) QuantErr: 9.52020 batch_time=0.50643
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.55004 (QuantReg: 9.56504) QuantErr: 9.56504 batch_time=0.50423
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.45545 (QuantReg: 9.73131) QuantErr: 9.73131 batch_time=0.52103
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.65736 (QuantReg: 9.97995) QuantErr: 9.97995 batch_time=0.50959
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.68513 (QuantReg: 10.01625) QuantErr: 10.01625 batch_time=0.84570
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.60339 (QuantReg: 9.86453) QuantErr: 9.86453 batch_time=0.50815
Train Epoch: 7 codebook_update_time=1.78416
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch7.pth ...
Done in 10.541s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch7.pth ...
Done in 25.781s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 2.668061067581177
quant_reg : 9.696174083709717
quant_err : 9.696174083709717
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_miech_test/t2v_metrics/R1: 16.5
MSRVTT_miech_test/t2v_metrics/R5: 44.3
MSRVTT_miech_test/t2v_metrics/R10: 58.1
MSRVTT_miech_test/t2v_metrics/R50: 86.4
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.031
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.88895280654772
MSRVTT_miech_test/v2t_metrics/R1: 16.9
MSRVTT_miech_test/v2t_metrics/R5: 44.4
MSRVTT_miech_test/v2t_metrics/R10: 58.7
MSRVTT_miech_test/v2t_metrics/R50: 86.9
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.165
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.31581707634575
mnt_best : 34.88895280654772
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.57715 (QuantReg: 9.69018) QuantErr: 9.69018 batch_time=30.00020
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.71379 (QuantReg: 9.78714) QuantErr: 9.78714 batch_time=0.60086
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.85334 (QuantReg: 9.63240) QuantErr: 9.63240 batch_time=0.56595
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.72238 (QuantReg: 9.49598) QuantErr: 9.49598 batch_time=0.52014
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.61763 (QuantReg: 10.04095) QuantErr: 10.04095 batch_time=0.53118
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.72207 (QuantReg: 9.52671) QuantErr: 9.52671 batch_time=0.54244
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 3.27202 (QuantReg: 9.62102) QuantErr: 9.62102 batch_time=0.53299
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.56305 (QuantReg: 9.70456) QuantErr: 9.70456 batch_time=1.16945
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.37667 (QuantReg: 9.52952) QuantErr: 9.52952 batch_time=0.51554
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.41693 (QuantReg: 9.89548) QuantErr: 9.89548 batch_time=0.51125
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.82024 (QuantReg: 9.79994) QuantErr: 9.79994 batch_time=0.52411
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.36743 (QuantReg: 9.41108) QuantErr: 9.41108 batch_time=0.51567
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.42859 (QuantReg: 9.81475) QuantErr: 9.81475 batch_time=0.51383
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.53231 (QuantReg: 9.87050) QuantErr: 9.87050 batch_time=1.56500
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.09213 (QuantReg: 9.26202) QuantErr: 9.26202 batch_time=0.52393
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.53735 (QuantReg: 9.67099) QuantErr: 9.67099 batch_time=0.55769
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.71773 (QuantReg: 9.66489) QuantErr: 9.66489 batch_time=0.51689
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.46980 (QuantReg: 9.91462) QuantErr: 9.91462 batch_time=0.56962
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.45105 (QuantReg: 9.39432) QuantErr: 9.39432 batch_time=0.54453
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.39537 (QuantReg: 9.93317) QuantErr: 9.93317 batch_time=0.53743
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.38900 (QuantReg: 9.39476) QuantErr: 9.39476 batch_time=0.57839
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 2.56780 (QuantReg: 9.88808) QuantErr: 9.88808 batch_time=0.52116
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.83071 (QuantReg: 9.76489) QuantErr: 9.76489 batch_time=0.50344
Train Epoch: 8 codebook_update_time=1.88513
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch8.pth ...
Done in 3.813s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 2.5449611053466796
quant_reg : 9.637869525909425
quant_err : 9.637869525909425
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_miech_test/t2v_metrics/R1: 15.1
MSRVTT_miech_test/t2v_metrics/R5: 44.0
MSRVTT_miech_test/t2v_metrics/R10: 58.4
MSRVTT_miech_test/t2v_metrics/R50: 87.5
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.515
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.85432483376466
MSRVTT_miech_test/v2t_metrics/R1: 16.8
MSRVTT_miech_test/v2t_metrics/R5: 44.5
MSRVTT_miech_test/v2t_metrics/R10: 60.1
MSRVTT_miech_test/v2t_metrics/R50: 87.2
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.2765
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.55068076975978
mnt_best : 34.88895280654772
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.80539 (QuantReg: 9.40013) QuantErr: 9.40013 batch_time=34.46087
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.39355 (QuantReg: 9.10765) QuantErr: 9.10765 batch_time=0.56723
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.25501 (QuantReg: 9.25133) QuantErr: 9.25133 batch_time=1.61876
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.54391 (QuantReg: 9.85733) QuantErr: 9.85733 batch_time=0.50914
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.36218 (QuantReg: 9.38241) QuantErr: 9.38241 batch_time=0.54678
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.40416 (QuantReg: 9.53420) QuantErr: 9.53420 batch_time=0.50957
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.42298 (QuantReg: 9.77448) QuantErr: 9.77448 batch_time=0.51695
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 2.35314 (QuantReg: 9.44153) QuantErr: 9.44153 batch_time=0.56373
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.29403 (QuantReg: 9.27110) QuantErr: 9.27110 batch_time=0.51368
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.30184 (QuantReg: 9.28349) QuantErr: 9.28349 batch_time=0.54274
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.41212 (QuantReg: 9.66293) QuantErr: 9.66293 batch_time=0.52841
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 2.36801 (QuantReg: 9.86501) QuantErr: 9.86501 batch_time=0.51663
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.12393 (QuantReg: 9.33018) QuantErr: 9.33018 batch_time=0.52109
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.33948 (QuantReg: 9.99006) QuantErr: 9.99006 batch_time=0.51560
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.99037 (QuantReg: 9.54710) QuantErr: 9.54710 batch_time=0.52530
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 2.30065 (QuantReg: 9.86584) QuantErr: 9.86584 batch_time=0.53133
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.35099 (QuantReg: 9.59997) QuantErr: 9.59997 batch_time=0.51823
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.95006 (QuantReg: 9.57536) QuantErr: 9.57536 batch_time=0.57547
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.18941 (QuantReg: 9.35324) QuantErr: 9.35324 batch_time=0.51231
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.63549 (QuantReg: 9.43110) QuantErr: 9.43110 batch_time=0.51464
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.64806 (QuantReg: 9.61218) QuantErr: 9.61218 batch_time=0.52984
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.36804 (QuantReg: 9.53757) QuantErr: 9.53757 batch_time=0.53244
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 2.14981 (QuantReg: 9.66498) QuantErr: 9.66498 batch_time=0.51773
Train Epoch: 9 codebook_update_time=1.67573
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch9.pth ...
Done in 4.241s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 2.435631613254547
quant_reg : 9.586739177703857
quant_err : 9.586739177703857
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_miech_test/t2v_metrics/R1: 16.0
MSRVTT_miech_test/t2v_metrics/R5: 43.6
MSRVTT_miech_test/t2v_metrics/R10: 58.1
MSRVTT_miech_test/t2v_metrics/R50: 85.9
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.165
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.35006275118189
MSRVTT_miech_test/v2t_metrics/R1: 17.3
MSRVTT_miech_test/v2t_metrics/R5: 43.4
MSRVTT_miech_test/v2t_metrics/R10: 58.9
MSRVTT_miech_test/v2t_metrics/R50: 86.4
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 36.368
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.36310380080047
mnt_best : 34.88895280654772
not_improved_count: 2
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 2.51127 (QuantReg: 9.49109) QuantErr: 9.49109 batch_time=28.69100
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.37950 (QuantReg: 9.89163) QuantErr: 9.89163 batch_time=0.51107
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.58434 (QuantReg: 9.41902) QuantErr: 9.41902 batch_time=0.55147
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.59443 (QuantReg: 9.52061) QuantErr: 9.52061 batch_time=1.44833
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 2.21198 (QuantReg: 9.64472) QuantErr: 9.64472 batch_time=0.52142
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.35730 (QuantReg: 9.80223) QuantErr: 9.80223 batch_time=0.51972
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.18006 (QuantReg: 9.78973) QuantErr: 9.78973 batch_time=0.52039
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 2.32699 (QuantReg: 9.64865) QuantErr: 9.64865 batch_time=0.57785
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.05677 (QuantReg: 9.49511) QuantErr: 9.49511 batch_time=4.12367
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.49527 (QuantReg: 9.93151) QuantErr: 9.93151 batch_time=0.59934
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.75869 (QuantReg: 9.80342) QuantErr: 9.80342 batch_time=0.56003
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.63488 (QuantReg: 9.45281) QuantErr: 9.45281 batch_time=0.52389
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.11056 (QuantReg: 9.71655) QuantErr: 9.71655 batch_time=0.51349
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.90768 (QuantReg: 9.61291) QuantErr: 9.61291 batch_time=0.55311
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 2.40261 (QuantReg: 9.68505) QuantErr: 9.68505 batch_time=0.52703
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.29928 (QuantReg: 9.46315) QuantErr: 9.46315 batch_time=0.60113
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.62610 (QuantReg: 9.26055) QuantErr: 9.26055 batch_time=0.52269
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.17416 (QuantReg: 9.81508) QuantErr: 9.81508 batch_time=0.59271
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.58516 (QuantReg: 9.73428) QuantErr: 9.73428 batch_time=0.50633
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.37237 (QuantReg: 9.95380) QuantErr: 9.95380 batch_time=0.55027
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.36325 (QuantReg: 9.74219) QuantErr: 9.74219 batch_time=0.52597
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.17239 (QuantReg: 9.25317) QuantErr: 9.25317 batch_time=0.52357
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.35686 (QuantReg: 9.34588) QuantErr: 9.34588 batch_time=0.56160
Train Epoch: 10 codebook_update_time=1.72621
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch10.pth ...
Done in 4.852s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch10.pth ...
Done in 9.835s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 2.3439470787048338
quant_reg : 9.558702236175536
quant_err : 9.558702236175536
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_miech_test/t2v_metrics/R1: 16.9
MSRVTT_miech_test/t2v_metrics/R5: 45.3
MSRVTT_miech_test/t2v_metrics/R10: 59.4
MSRVTT_miech_test/t2v_metrics/R50: 87.2
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 34.542
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.693608064008075
MSRVTT_miech_test/v2t_metrics/R1: 17.8
MSRVTT_miech_test/v2t_metrics/R5: 46.1
MSRVTT_miech_test/v2t_metrics/R10: 59.7
MSRVTT_miech_test/v2t_metrics/R50: 88.0
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.744
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.590225524451064
mnt_best : 35.693608064008075
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.41502 (QuantReg: 9.43481) QuantErr: 9.43481 batch_time=28.09313
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.03273 (QuantReg: 9.72442) QuantErr: 9.72442 batch_time=0.50746
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.46951 (QuantReg: 9.77438) QuantErr: 9.77438 batch_time=0.52407
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.52063 (QuantReg: 9.18320) QuantErr: 9.18320 batch_time=0.51584
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.32104 (QuantReg: 9.87733) QuantErr: 9.87733 batch_time=0.51099
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.41516 (QuantReg: 9.25049) QuantErr: 9.25049 batch_time=0.52582
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 2.15607 (QuantReg: 9.53666) QuantErr: 9.53666 batch_time=0.52096
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.42723 (QuantReg: 9.36946) QuantErr: 9.36946 batch_time=0.52394
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.38798 (QuantReg: 9.64403) QuantErr: 9.64403 batch_time=0.53565
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.23253 (QuantReg: 9.30608) QuantErr: 9.30608 batch_time=0.74947
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.40856 (QuantReg: 9.51913) QuantErr: 9.51913 batch_time=0.78576
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.16107 (QuantReg: 9.59194) QuantErr: 9.59194 batch_time=0.53245
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.49415 (QuantReg: 9.46592) QuantErr: 9.46592 batch_time=0.54731
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.40209 (QuantReg: 9.46210) QuantErr: 9.46210 batch_time=2.05585
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 2.30590 (QuantReg: 9.43414) QuantErr: 9.43414 batch_time=0.51465
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.42789 (QuantReg: 9.31856) QuantErr: 9.31856 batch_time=0.53108
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.44266 (QuantReg: 9.52196) QuantErr: 9.52196 batch_time=0.52351
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 2.06046 (QuantReg: 9.47345) QuantErr: 9.47345 batch_time=0.51077
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 2.44791 (QuantReg: 9.36609) QuantErr: 9.36609 batch_time=0.51000
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.94771 (QuantReg: 9.76325) QuantErr: 9.76325 batch_time=0.54056
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 2.17285 (QuantReg: 9.36752) QuantErr: 9.36752 batch_time=0.74224
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.38009 (QuantReg: 9.62002) QuantErr: 9.62002 batch_time=0.53674
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 2.17624 (QuantReg: 9.42052) QuantErr: 9.42052 batch_time=0.54149
Train Epoch: 11 codebook_update_time=1.73350
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch11.pth ...
Done in 5.030s
removing stale ckpt [epoch 10] [took 0.24s]
epoch : 11
loss : 2.286825608253479
quant_reg : 9.551147151947022
quant_err : 9.551147151947022
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_miech_test/t2v_metrics/R1: 16.6
MSRVTT_miech_test/t2v_metrics/R5: 43.9
MSRVTT_miech_test/t2v_metrics/R10: 58.9
MSRVTT_miech_test/t2v_metrics/R50: 87.2
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 34.885
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.012998165391025
MSRVTT_miech_test/v2t_metrics/R1: 17.8
MSRVTT_miech_test/v2t_metrics/R5: 45.9
MSRVTT_miech_test/v2t_metrics/R10: 60.4
MSRVTT_miech_test/v2t_metrics/R50: 87.4
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.048
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.679483269744466
mnt_best : 35.693608064008075
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 2.40761 (QuantReg: 9.11573) QuantErr: 9.11573 batch_time=30.24543
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 2.11262 (QuantReg: 9.50427) QuantErr: 9.50427 batch_time=0.53240
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 2.22304 (QuantReg: 9.52913) QuantErr: 9.52913 batch_time=0.52315
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.21133 (QuantReg: 9.64788) QuantErr: 9.64788 batch_time=0.57825
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.11835 (QuantReg: 9.45206) QuantErr: 9.45206 batch_time=0.52668
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 2.28023 (QuantReg: 9.52704) QuantErr: 9.52704 batch_time=0.52665
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 2.26630 (QuantReg: 9.36977) QuantErr: 9.36977 batch_time=0.54915
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 2.05083 (QuantReg: 9.64336) QuantErr: 9.64336 batch_time=0.52408
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 2.21718 (QuantReg: 9.01232) QuantErr: 9.01232 batch_time=0.51829
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 2.03457 (QuantReg: 9.62427) QuantErr: 9.62427 batch_time=0.57462
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 2.14803 (QuantReg: 9.79410) QuantErr: 9.79410 batch_time=0.51471
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 2.24726 (QuantReg: 9.48733) QuantErr: 9.48733 batch_time=0.52365
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 2.03736 (QuantReg: 9.16983) QuantErr: 9.16983 batch_time=0.50936
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 2.29933 (QuantReg: 9.39629) QuantErr: 9.39629 batch_time=0.52917
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.19437 (QuantReg: 9.73243) QuantErr: 9.73243 batch_time=0.99737
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 2.24470 (QuantReg: 9.12687) QuantErr: 9.12687 batch_time=0.55663
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.33121 (QuantReg: 9.54735) QuantErr: 9.54735 batch_time=0.54098
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 2.37912 (QuantReg: 9.53264) QuantErr: 9.53264 batch_time=0.52659
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.08322 (QuantReg: 9.87616) QuantErr: 9.87616 batch_time=5.44313
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 2.32214 (QuantReg: 9.59720) QuantErr: 9.59720 batch_time=0.74977
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 2.14199 (QuantReg: 9.29905) QuantErr: 9.29905 batch_time=1.03665
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.22987 (QuantReg: 9.54947) QuantErr: 9.54947 batch_time=0.53703
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 2.21083 (QuantReg: 9.77291) QuantErr: 9.77291 batch_time=0.51305
Train Epoch: 12 codebook_update_time=1.67420
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch12.pth ...
Done in 5.407s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch12.pth ...
Done in 10.880s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 2.2045254788398743
quant_reg : 9.506489719390869
quant_err : 9.506489719390869
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_miech_test/t2v_metrics/R1: 18.1
MSRVTT_miech_test/t2v_metrics/R5: 44.5
MSRVTT_miech_test/t2v_metrics/R10: 59.2
MSRVTT_miech_test/t2v_metrics/R50: 87.0
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 34.581
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.26214004801967
MSRVTT_miech_test/v2t_metrics/R1: 18.4
MSRVTT_miech_test/v2t_metrics/R5: 46.3
MSRVTT_miech_test/v2t_metrics/R10: 61.2
MSRVTT_miech_test/v2t_metrics/R50: 87.0
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.6445
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.35798230494012
mnt_best : 36.26214004801967
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.03504 (QuantReg: 9.52062) QuantErr: 9.52062 batch_time=29.44313
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 2.21869 (QuantReg: 9.31963) QuantErr: 9.31963 batch_time=0.52819
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 2.10418 (QuantReg: 9.51353) QuantErr: 9.51353 batch_time=0.57861
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 2.56500 (QuantReg: 9.63916) QuantErr: 9.63916 batch_time=0.52123
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.26653 (QuantReg: 9.57088) QuantErr: 9.57088 batch_time=0.51964
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.42896 (QuantReg: 9.35837) QuantErr: 9.35837 batch_time=0.52615
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 2.01115 (QuantReg: 9.64353) QuantErr: 9.64353 batch_time=0.50258
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 2.25688 (QuantReg: 9.66019) QuantErr: 9.66019 batch_time=0.51409
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.84345 (QuantReg: 9.47097) QuantErr: 9.47097 batch_time=0.76275
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 2.25065 (QuantReg: 9.29211) QuantErr: 9.29211 batch_time=0.52337
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 2.28965 (QuantReg: 9.51688) QuantErr: 9.51688 batch_time=0.51925
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 2.01360 (QuantReg: 9.70621) QuantErr: 9.70621 batch_time=0.59422
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 2.46409 (QuantReg: 9.16884) QuantErr: 9.16884 batch_time=1.01561
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 2.41775 (QuantReg: 9.73666) QuantErr: 9.73666 batch_time=0.50859
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 2.15629 (QuantReg: 9.71502) QuantErr: 9.71502 batch_time=0.57483
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 2.33715 (QuantReg: 9.30127) QuantErr: 9.30127 batch_time=0.53000
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 2.21946 (QuantReg: 9.86461) QuantErr: 9.86461 batch_time=0.76143
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 2.27558 (QuantReg: 9.52260) QuantErr: 9.52260 batch_time=0.53770
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.94707 (QuantReg: 9.55515) QuantErr: 9.55515 batch_time=0.51227
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.19205 (QuantReg: 9.77137) QuantErr: 9.77137 batch_time=0.57291
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.91325 (QuantReg: 9.36832) QuantErr: 9.36832 batch_time=0.53796
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.85954 (QuantReg: 9.14102) QuantErr: 9.14102 batch_time=0.69342
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.50296 (QuantReg: 9.42247) QuantErr: 9.42247 batch_time=0.51828
Train Epoch: 13 codebook_update_time=1.70503
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch13.pth ...
Done in 4.721s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 2.152361847400665
quant_reg : 9.506782958984376
quant_err : 9.506782958984376
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_miech_test/t2v_metrics/R1: 16.5
MSRVTT_miech_test/t2v_metrics/R5: 45.2
MSRVTT_miech_test/t2v_metrics/R10: 59.5
MSRVTT_miech_test/t2v_metrics/R50: 86.8
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.368
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.403520349578805
MSRVTT_miech_test/v2t_metrics/R1: 17.1
MSRVTT_miech_test/v2t_metrics/R5: 47.3
MSRVTT_miech_test/v2t_metrics/R10: 60.3
MSRVTT_miech_test/v2t_metrics/R50: 87.4
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.411
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.536324381341586
mnt_best : 36.26214004801967
not_improved_count: 1
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 2.34513 (QuantReg: 9.24169) QuantErr: 9.24169 batch_time=34.25936
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.00824 (QuantReg: 9.51633) QuantErr: 9.51633 batch_time=0.52439
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.88597 (QuantReg: 9.11186) QuantErr: 9.11186 batch_time=0.50712
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.12598 (QuantReg: 9.74841) QuantErr: 9.74841 batch_time=0.52433
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.27304 (QuantReg: 9.36503) QuantErr: 9.36503 batch_time=0.55762
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.01460 (QuantReg: 9.64944) QuantErr: 9.64944 batch_time=0.51422
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.07210 (QuantReg: 9.29822) QuantErr: 9.29822 batch_time=0.51724
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 2.53010 (QuantReg: 9.24622) QuantErr: 9.24622 batch_time=0.55758
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.31447 (QuantReg: 9.62372) QuantErr: 9.62372 batch_time=0.52627
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 2.08508 (QuantReg: 9.55162) QuantErr: 9.55162 batch_time=0.55750
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.26339 (QuantReg: 9.46567) QuantErr: 9.46567 batch_time=0.58987
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 2.02072 (QuantReg: 9.40771) QuantErr: 9.40771 batch_time=0.56456
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 2.12654 (QuantReg: 9.26213) QuantErr: 9.26213 batch_time=0.62883
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 2.00142 (QuantReg: 9.50031) QuantErr: 9.50031 batch_time=0.53058
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.87213 (QuantReg: 9.24686) QuantErr: 9.24686 batch_time=0.53566
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.69537 (QuantReg: 9.58315) QuantErr: 9.58315 batch_time=0.52325
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 2.21035 (QuantReg: 9.15745) QuantErr: 9.15745 batch_time=0.51590
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 2.03528 (QuantReg: 9.67783) QuantErr: 9.67783 batch_time=0.57815
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.75343 (QuantReg: 9.48071) QuantErr: 9.48071 batch_time=0.51705
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 2.37148 (QuantReg: 9.73028) QuantErr: 9.73028 batch_time=0.51044
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.92492 (QuantReg: 9.66792) QuantErr: 9.66792 batch_time=0.57339
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.86513 (QuantReg: 9.25332) QuantErr: 9.25332 batch_time=0.83651
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.13456 (QuantReg: 9.59812) QuantErr: 9.59812 batch_time=0.59946
Train Epoch: 14 codebook_update_time=1.71605
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch14.pth ...
Done in 6.755s
removing stale ckpt [epoch 13] [took 0.03s]
epoch : 14
loss : 2.0864813013076784
quant_reg : 9.519416397094727
quant_err : 9.519416397094727
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_miech_test/t2v_metrics/R1: 17.1
MSRVTT_miech_test/t2v_metrics/R5: 44.8
MSRVTT_miech_test/t2v_metrics/R10: 60.7
MSRVTT_miech_test/t2v_metrics/R50: 86.6
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.937
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.9601039507381
MSRVTT_miech_test/v2t_metrics/R1: 18.1
MSRVTT_miech_test/v2t_metrics/R5: 47.1
MSRVTT_miech_test/v2t_metrics/R10: 60.8
MSRVTT_miech_test/v2t_metrics/R50: 87.3
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 33.655
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.285017713846685
mnt_best : 36.26214004801967
not_improved_count: 2
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.17690 (QuantReg: 9.60639) QuantErr: 9.60639 batch_time=32.91039
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.31418 (QuantReg: 9.35723) QuantErr: 9.35723 batch_time=0.52774
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.30883 (QuantReg: 9.55046) QuantErr: 9.55046 batch_time=0.53732
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.37087 (QuantReg: 9.27040) QuantErr: 9.27040 batch_time=0.51581
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.21424 (QuantReg: 9.50558) QuantErr: 9.50558 batch_time=0.51703
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.69405 (QuantReg: 9.45388) QuantErr: 9.45388 batch_time=0.59486
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.18276 (QuantReg: 9.33859) QuantErr: 9.33859 batch_time=0.56040
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.72821 (QuantReg: 9.51850) QuantErr: 9.51850 batch_time=0.53559
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.33010 (QuantReg: 9.48387) QuantErr: 9.48387 batch_time=0.56539
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.94817 (QuantReg: 9.44333) QuantErr: 9.44333 batch_time=0.53186
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.81555 (QuantReg: 9.57943) QuantErr: 9.57943 batch_time=0.52399
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.10770 (QuantReg: 9.40578) QuantErr: 9.40578 batch_time=0.53870
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.31401 (QuantReg: 9.63765) QuantErr: 9.63765 batch_time=2.52438
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.95659 (QuantReg: 9.43073) QuantErr: 9.43073 batch_time=1.08105
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.98260 (QuantReg: 9.38778) QuantErr: 9.38778 batch_time=0.52964
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.96035 (QuantReg: 9.42863) QuantErr: 9.42863 batch_time=0.53982
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.88734 (QuantReg: 9.47156) QuantErr: 9.47156 batch_time=0.56867
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 2.30193 (QuantReg: 9.62707) QuantErr: 9.62707 batch_time=0.52552
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.61813 (QuantReg: 9.32764) QuantErr: 9.32764 batch_time=0.51758
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.69652 (QuantReg: 9.56732) QuantErr: 9.56732 batch_time=0.53963
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.83718 (QuantReg: 9.72101) QuantErr: 9.72101 batch_time=0.52690
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.09206 (QuantReg: 9.64170) QuantErr: 9.64170 batch_time=0.52259
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.18528 (QuantReg: 9.79269) QuantErr: 9.79269 batch_time=0.53124
Train Epoch: 15 codebook_update_time=1.73541
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch15.pth ...
Done in 6.619s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 2.0603098335266115
quant_reg : 9.472115520477296
quant_err : 9.472115520477296
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_miech_test/t2v_metrics/R1: 16.5
MSRVTT_miech_test/t2v_metrics/R5: 44.7
MSRVTT_miech_test/t2v_metrics/R10: 60.5
MSRVTT_miech_test/t2v_metrics/R50: 86.7
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 34.66
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.4690002472851
MSRVTT_miech_test/v2t_metrics/R1: 17.8
MSRVTT_miech_test/v2t_metrics/R5: 46.0
MSRVTT_miech_test/v2t_metrics/R10: 60.9
MSRVTT_miech_test/v2t_metrics/R50: 87.2
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 33.8
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.80710913379426
mnt_best : 36.26214004801967
not_improved_count: 3
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 2.08272 (QuantReg: 9.15069) QuantErr: 9.15069 batch_time=36.58362
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 2.20438 (QuantReg: 9.17474) QuantErr: 9.17474 batch_time=0.54712
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.70173 (QuantReg: 9.24186) QuantErr: 9.24186 batch_time=0.55895
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.33613 (QuantReg: 9.21506) QuantErr: 9.21506 batch_time=0.52021
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 2.12719 (QuantReg: 9.38570) QuantErr: 9.38570 batch_time=0.51452
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 2.36691 (QuantReg: 9.63158) QuantErr: 9.63158 batch_time=0.51809
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.32126 (QuantReg: 9.55612) QuantErr: 9.55612 batch_time=0.53688
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.97189 (QuantReg: 9.86092) QuantErr: 9.86092 batch_time=0.51259
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.92554 (QuantReg: 9.06102) QuantErr: 9.06102 batch_time=0.53375
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.81016 (QuantReg: 9.46097) QuantErr: 9.46097 batch_time=0.51074
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.87132 (QuantReg: 9.65197) QuantErr: 9.65197 batch_time=0.57116
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.85894 (QuantReg: 9.45276) QuantErr: 9.45276 batch_time=0.57496
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.07292 (QuantReg: 9.40593) QuantErr: 9.40593 batch_time=0.58931
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 2.04244 (QuantReg: 9.41936) QuantErr: 9.41936 batch_time=0.51295
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.09371 (QuantReg: 9.51205) QuantErr: 9.51205 batch_time=0.52737
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.96410 (QuantReg: 9.45669) QuantErr: 9.45669 batch_time=0.52787
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.93155 (QuantReg: 9.68791) QuantErr: 9.68791 batch_time=0.52767
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 2.02889 (QuantReg: 9.09968) QuantErr: 9.09968 batch_time=0.59444
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.04356 (QuantReg: 9.52481) QuantErr: 9.52481 batch_time=0.51667
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.96382 (QuantReg: 9.41549) QuantErr: 9.41549 batch_time=0.53067
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.05028 (QuantReg: 9.39244) QuantErr: 9.39244 batch_time=0.64369
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.83925 (QuantReg: 9.63016) QuantErr: 9.63016 batch_time=0.54502
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 2.00271 (QuantReg: 9.43880) QuantErr: 9.43880 batch_time=0.52375
Train Epoch: 16 codebook_update_time=1.68313
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch16.pth ...
Done in 4.633s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch16.pth ...
Done in 10.284s
removing stale ckpt [epoch 15] [took 0.36s]
epoch : 16
loss : 2.0005235538482666
quant_reg : 9.491786014556885
quant_err : 9.491786014556885
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_miech_test/t2v_metrics/R1: 17.6
MSRVTT_miech_test/t2v_metrics/R5: 46.6
MSRVTT_miech_test/t2v_metrics/R10: 61.5
MSRVTT_miech_test/t2v_metrics/R50: 87.6
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.299
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.948025393101666
MSRVTT_miech_test/v2t_metrics/R1: 19.1
MSRVTT_miech_test/v2t_metrics/R5: 47.5
MSRVTT_miech_test/v2t_metrics/R10: 61.7
MSRVTT_miech_test/v2t_metrics/R50: 87.9
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 33.5665
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.25345918279096
mnt_best : 36.948025393101666
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 2.12973 (QuantReg: 9.19374) QuantErr: 9.19374 batch_time=25.96902
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.82984 (QuantReg: 9.72170) QuantErr: 9.72170 batch_time=0.51594
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.92935 (QuantReg: 9.72180) QuantErr: 9.72180 batch_time=0.51169
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.15253 (QuantReg: 9.27571) QuantErr: 9.27571 batch_time=0.50578
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 2.02997 (QuantReg: 9.70041) QuantErr: 9.70041 batch_time=0.53261
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 2.05125 (QuantReg: 9.60869) QuantErr: 9.60869 batch_time=0.51787
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 2.26932 (QuantReg: 9.32138) QuantErr: 9.32138 batch_time=0.51351
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.97317 (QuantReg: 9.66931) QuantErr: 9.66931 batch_time=0.52267
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.92981 (QuantReg: 9.48551) QuantErr: 9.48551 batch_time=0.52569
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.70408 (QuantReg: 9.89924) QuantErr: 9.89924 batch_time=0.51090
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.74625 (QuantReg: 9.75370) QuantErr: 9.75370 batch_time=0.51513
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.86702 (QuantReg: 9.44675) QuantErr: 9.44675 batch_time=0.52235
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 2.16518 (QuantReg: 9.29617) QuantErr: 9.29617 batch_time=4.20420
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.13872 (QuantReg: 9.46691) QuantErr: 9.46691 batch_time=0.54560
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 2.12263 (QuantReg: 9.36013) QuantErr: 9.36013 batch_time=0.50415
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.21353 (QuantReg: 9.36466) QuantErr: 9.36466 batch_time=0.51428
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.63341 (QuantReg: 9.21791) QuantErr: 9.21791 batch_time=0.50850
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.81040 (QuantReg: 9.30311) QuantErr: 9.30311 batch_time=0.52607
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.20262 (QuantReg: 9.77513) QuantErr: 9.77513 batch_time=0.50587
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 2.04065 (QuantReg: 9.57145) QuantErr: 9.57145 batch_time=0.62757
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.77336 (QuantReg: 9.57097) QuantErr: 9.57097 batch_time=0.51431
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.98538 (QuantReg: 9.34942) QuantErr: 9.34942 batch_time=0.93182
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.95917 (QuantReg: 9.46147) QuantErr: 9.46147 batch_time=1.06464
Train Epoch: 17 codebook_update_time=1.76202
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch17.pth ...
Done in 6.809s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch17.pth ...
Done in 11.806s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 1.9651159501075746
quant_reg : 9.489287570953369
quant_err : 9.489287570953369
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_miech_test/t2v_metrics/R1: 18.2
MSRVTT_miech_test/t2v_metrics/R5: 46.0
MSRVTT_miech_test/t2v_metrics/R10: 61.7
MSRVTT_miech_test/t2v_metrics/R50: 86.2
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.145
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.2424400975769
MSRVTT_miech_test/v2t_metrics/R1: 18.2
MSRVTT_miech_test/v2t_metrics/R5: 48.1
MSRVTT_miech_test/v2t_metrics/R10: 62.7
MSRVTT_miech_test/v2t_metrics/R50: 86.1
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.4735
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.00388556759254
mnt_best : 37.2424400975769
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.03332 (QuantReg: 9.24820) QuantErr: 9.24820 batch_time=25.57219
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.97912 (QuantReg: 9.34221) QuantErr: 9.34221 batch_time=1.10865
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 2.01177 (QuantReg: 9.27680) QuantErr: 9.27680 batch_time=0.52072
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 2.23863 (QuantReg: 9.44726) QuantErr: 9.44726 batch_time=0.50816
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.90992 (QuantReg: 9.60918) QuantErr: 9.60918 batch_time=0.55359
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.98089 (QuantReg: 9.43861) QuantErr: 9.43861 batch_time=0.52443
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.70362 (QuantReg: 9.53249) QuantErr: 9.53249 batch_time=1.86838
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.97882 (QuantReg: 9.44645) QuantErr: 9.44645 batch_time=0.58597
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 2.09443 (QuantReg: 9.82525) QuantErr: 9.82525 batch_time=1.86294
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 2.19135 (QuantReg: 9.45504) QuantErr: 9.45504 batch_time=0.54234
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.71956 (QuantReg: 9.49887) QuantErr: 9.49887 batch_time=0.52131
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.76134 (QuantReg: 9.87863) QuantErr: 9.87863 batch_time=0.51404
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.03668 (QuantReg: 9.49520) QuantErr: 9.49520 batch_time=0.67736
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.82063 (QuantReg: 9.33083) QuantErr: 9.33083 batch_time=0.66185
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.80362 (QuantReg: 9.38479) QuantErr: 9.38479 batch_time=0.51465
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.97597 (QuantReg: 9.51103) QuantErr: 9.51103 batch_time=0.52104
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 2.10420 (QuantReg: 9.67009) QuantErr: 9.67009 batch_time=0.53922
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.98308 (QuantReg: 9.44415) QuantErr: 9.44415 batch_time=0.53700
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.00623 (QuantReg: 9.44383) QuantErr: 9.44383 batch_time=0.61765
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.97720 (QuantReg: 9.44487) QuantErr: 9.44487 batch_time=0.53217
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.91996 (QuantReg: 9.23867) QuantErr: 9.23867 batch_time=0.55680
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.05724 (QuantReg: 9.64557) QuantErr: 9.64557 batch_time=0.50503
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.92785 (QuantReg: 9.44831) QuantErr: 9.44831 batch_time=0.52013
Train Epoch: 18 codebook_update_time=1.84466
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch18.pth ...
Done in 5.715s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch18.pth ...
Done in 10.931s
removing stale ckpt [epoch 17] [took 0.18s]
epoch : 18
loss : 1.9385207352638245
quant_reg : 9.49444730758667
quant_err : 9.49444730758667
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_miech_test/t2v_metrics/R1: 19.9
MSRVTT_miech_test/t2v_metrics/R5: 46.8
MSRVTT_miech_test/t2v_metrics/R10: 61.0
MSRVTT_miech_test/t2v_metrics/R50: 85.6
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.0835
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.44231987942381
MSRVTT_miech_test/v2t_metrics/R1: 20.3
MSRVTT_miech_test/v2t_metrics/R5: 47.6
MSRVTT_miech_test/v2t_metrics/R10: 61.8
MSRVTT_miech_test/v2t_metrics/R50: 86.6
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.1285
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.086833477774725
mnt_best : 38.44231987942381
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 2.05643 (QuantReg: 9.38074) QuantErr: 9.38074 batch_time=29.16066
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.77272 (QuantReg: 9.65947) QuantErr: 9.65947 batch_time=0.55121
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.90343 (QuantReg: 9.05611) QuantErr: 9.05611 batch_time=0.52846
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 2.02841 (QuantReg: 9.27087) QuantErr: 9.27087 batch_time=0.53121
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.98132 (QuantReg: 9.57483) QuantErr: 9.57483 batch_time=0.58330
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 2.06179 (QuantReg: 9.69037) QuantErr: 9.69037 batch_time=0.51879
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.98718 (QuantReg: 9.41853) QuantErr: 9.41853 batch_time=0.83344
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.70786 (QuantReg: 9.59278) QuantErr: 9.59278 batch_time=1.90636
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 2.13809 (QuantReg: 9.53811) QuantErr: 9.53811 batch_time=0.52342
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.65914 (QuantReg: 9.16823) QuantErr: 9.16823 batch_time=0.79066
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.84915 (QuantReg: 9.55661) QuantErr: 9.55661 batch_time=0.51459
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.97851 (QuantReg: 9.34791) QuantErr: 9.34791 batch_time=0.51891
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.80735 (QuantReg: 9.58688) QuantErr: 9.58688 batch_time=0.52581
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.76753 (QuantReg: 9.32740) QuantErr: 9.32740 batch_time=2.95120
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 2.31303 (QuantReg: 9.44260) QuantErr: 9.44260 batch_time=0.51602
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.64949 (QuantReg: 9.65295) QuantErr: 9.65295 batch_time=0.54040
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.84291 (QuantReg: 9.59292) QuantErr: 9.59292 batch_time=0.54398
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.82055 (QuantReg: 9.43426) QuantErr: 9.43426 batch_time=0.52298
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 2.08890 (QuantReg: 9.55842) QuantErr: 9.55842 batch_time=0.52146
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.92742 (QuantReg: 9.15410) QuantErr: 9.15410 batch_time=0.51696
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.92676 (QuantReg: 9.55703) QuantErr: 9.55703 batch_time=0.55706
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.83013 (QuantReg: 9.35476) QuantErr: 9.35476 batch_time=1.08524
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 2.01229 (QuantReg: 9.17174) QuantErr: 9.17174 batch_time=0.50746
Train Epoch: 19 codebook_update_time=1.69711
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.1/checkpoint-epoch19.pth ...
Done in 5.114s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 1.9024231023788452
quant_reg : 9.49955237197876
quant_err : 9.49955237197876
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_miech_test/t2v_metrics/R1: 18.9
MSRVTT_miech_test/t2v_metrics/R5: 45.9
MSRVTT_miech_test/t2v_metrics/R10: 61.1
MSRVTT_miech_test/t2v_metrics/R50: 86.1
MSRVTT_miech_test/t2v_metrics/MedR: 6.5
MSRVTT_miech_test/t2v_metrics/MeanR: 33.89
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.564005891069236
MSRVTT_miech_test/v2t_metrics/R1: 19.6