-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kB_t0.12.txt
2591 lines (2591 loc) · 189 KB
/
HCQ_MSRVTT_1kB_t0.12.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12
Preparing the dataloaders ...
Loading dataset MSRVTT_miech_trainval in ram ...
Finish loading dataset MSRVTT_miech_trainval in ram, taking 693.0185225009918 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 109.40281200408936 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 77.56351327896118 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch0.pth ...
Done in 1.494s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch0.pth ...
Done in 2.957s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_miech_test/t2v_metrics/R1: 0.1
MSRVTT_miech_test/t2v_metrics/R5: 0.6
MSRVTT_miech_test/t2v_metrics/R10: 1.0
MSRVTT_miech_test/t2v_metrics/R50: 5.0
MSRVTT_miech_test/t2v_metrics/MedR: 503.0
MSRVTT_miech_test/t2v_metrics/MeanR: 505.193
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3914867641168864
MSRVTT_miech_test/v2t_metrics/R1: 0.1
MSRVTT_miech_test/v2t_metrics/R5: 0.4
MSRVTT_miech_test/v2t_metrics/R10: 1.0
MSRVTT_miech_test/v2t_metrics/R50: 5.4
MSRVTT_miech_test/v2t_metrics/MedR: 511.5
MSRVTT_miech_test/v2t_metrics/MeanR: 499.894
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.3419951893353394
mnt_best : 0.3914867641168864
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.71350 (QuantReg: 22.48371) QuantErr: 22.48371 batch_time=20.59480
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.69578 (QuantReg: 22.51129) QuantErr: 22.51129 batch_time=0.62377
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.48810 (QuantReg: 22.63348) QuantErr: 22.63348 batch_time=0.50646
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.78123 (QuantReg: 22.65992) QuantErr: 22.65992 batch_time=0.56163
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.79127 (QuantReg: 22.67678) QuantErr: 22.67678 batch_time=0.52848
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.39461 (QuantReg: 22.68974) QuantErr: 22.68974 batch_time=0.61686
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.31673 (QuantReg: 22.64101) QuantErr: 22.64101 batch_time=0.52172
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.95055 (QuantReg: 22.64018) QuantErr: 22.64018 batch_time=0.50454
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.57678 (QuantReg: 22.62092) QuantErr: 22.62092 batch_time=0.50168
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.56236 (QuantReg: 22.64747) QuantErr: 22.64747 batch_time=0.51570
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.54627 (QuantReg: 22.66624) QuantErr: 22.66624 batch_time=0.53561
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.65384 (QuantReg: 22.61428) QuantErr: 22.61428 batch_time=0.51629
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.17681 (QuantReg: 22.66196) QuantErr: 22.66196 batch_time=0.50099
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 5.18884 (QuantReg: 22.65871) QuantErr: 22.65871 batch_time=0.49840
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 5.01865 (QuantReg: 22.67710) QuantErr: 22.67710 batch_time=0.51329
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.88827 (QuantReg: 22.64832) QuantErr: 22.64832 batch_time=0.49954
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.75110 (QuantReg: 22.65484) QuantErr: 22.65484 batch_time=0.53218
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 5.09933 (QuantReg: 22.66242) QuantErr: 22.66242 batch_time=0.50714
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 5.01605 (QuantReg: 22.68533) QuantErr: 22.68533 batch_time=0.53459
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 5.02459 (QuantReg: 22.67793) QuantErr: 22.67793 batch_time=0.49811
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 5.00940 (QuantReg: 22.63878) QuantErr: 22.63878 batch_time=0.50311
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 5.20436 (QuantReg: 22.63622) QuantErr: 22.63622 batch_time=0.74491
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.64518 (QuantReg: 22.66110) QuantErr: 22.66110 batch_time=0.52642
Train Epoch: 1 codebook_update_time=2.48140
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch1.pth ...
Done in 4.122s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch1.pth ...
Done in 8.395s
epoch : 1
loss : 5.820175867080689
quant_reg : 22.640720153808594
quant_err : 22.640720153808594
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_miech_test/t2v_metrics/R1: 8.0
MSRVTT_miech_test/t2v_metrics/R5: 26.4
MSRVTT_miech_test/t2v_metrics/R10: 40.2
MSRVTT_miech_test/t2v_metrics/R50: 75.4
MSRVTT_miech_test/t2v_metrics/MedR: 16.0
MSRVTT_miech_test/t2v_metrics/MeanR: 50.829
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.40046135058137
MSRVTT_miech_test/v2t_metrics/R1: 9.6
MSRVTT_miech_test/v2t_metrics/R5: 29.5
MSRVTT_miech_test/v2t_metrics/R10: 42.1
MSRVTT_miech_test/v2t_metrics/R50: 76.0
MSRVTT_miech_test/v2t_metrics/MedR: 15.0
MSRVTT_miech_test/v2t_metrics/MeanR: 49.632
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.84503257289785
mnt_best : 20.40046135058137
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.58528 (QuantReg: 10.00180) QuantErr: 10.00180 batch_time=26.80723
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.52217 (QuantReg: 10.14652) QuantErr: 10.14652 batch_time=0.49911
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.79816 (QuantReg: 10.34727) QuantErr: 10.34727 batch_time=0.50092
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.59871 (QuantReg: 10.21026) QuantErr: 10.21026 batch_time=0.51007
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.24743 (QuantReg: 10.47844) QuantErr: 10.47844 batch_time=0.94690
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.24482 (QuantReg: 10.24428) QuantErr: 10.24428 batch_time=0.53978
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.38499 (QuantReg: 10.68754) QuantErr: 10.68754 batch_time=0.51263
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 5.03303 (QuantReg: 10.48849) QuantErr: 10.48849 batch_time=0.51297
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.11037 (QuantReg: 10.65143) QuantErr: 10.65143 batch_time=0.49645
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.48441 (QuantReg: 11.04049) QuantErr: 11.04049 batch_time=0.49582
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 4.20340 (QuantReg: 10.94552) QuantErr: 10.94552 batch_time=0.51465
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.62017 (QuantReg: 10.94785) QuantErr: 10.94785 batch_time=0.51572
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 4.24491 (QuantReg: 11.16712) QuantErr: 11.16712 batch_time=0.51215
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.39327 (QuantReg: 11.81011) QuantErr: 11.81011 batch_time=0.51406
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 4.30356 (QuantReg: 11.27739) QuantErr: 11.27739 batch_time=0.51050
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 4.60014 (QuantReg: 11.03268) QuantErr: 11.03268 batch_time=0.51840
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 4.15682 (QuantReg: 11.10193) QuantErr: 11.10193 batch_time=0.51955
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.66863 (QuantReg: 11.40316) QuantErr: 11.40316 batch_time=0.51554
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 4.47525 (QuantReg: 11.64583) QuantErr: 11.64583 batch_time=0.49687
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 4.24245 (QuantReg: 11.32094) QuantErr: 11.32094 batch_time=0.52231
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 4.34411 (QuantReg: 11.78198) QuantErr: 11.78198 batch_time=0.55521
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.85752 (QuantReg: 12.05566) QuantErr: 12.05566 batch_time=0.51329
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.88199 (QuantReg: 12.08155) QuantErr: 12.08155 batch_time=0.57152
Train Epoch: 2 codebook_update_time=1.95112
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch2.pth ...
Done in 22.280s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch2.pth ...
Done in 26.379s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 4.307337514877319
quant_reg : 11.070338775634765
quant_err : 11.070338775634765
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_miech_test/t2v_metrics/R1: 11.3
MSRVTT_miech_test/t2v_metrics/R5: 33.4
MSRVTT_miech_test/t2v_metrics/R10: 47.6
MSRVTT_miech_test/t2v_metrics/R50: 82.4
MSRVTT_miech_test/t2v_metrics/MedR: 11.0
MSRVTT_miech_test/t2v_metrics/MeanR: 44.146
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.19050993630703
MSRVTT_miech_test/v2t_metrics/R1: 12.3
MSRVTT_miech_test/v2t_metrics/R5: 34.3
MSRVTT_miech_test/v2t_metrics/R10: 47.9
MSRVTT_miech_test/v2t_metrics/R50: 81.8
MSRVTT_miech_test/v2t_metrics/MedR: 12.0
MSRVTT_miech_test/v2t_metrics/MeanR: 44.437
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.238190208401477
mnt_best : 26.19050993630703
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 4.28499 (QuantReg: 9.05177) QuantErr: 9.05177 batch_time=25.00676
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 4.03010 (QuantReg: 9.26416) QuantErr: 9.26416 batch_time=0.51081
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 4.11979 (QuantReg: 9.09654) QuantErr: 9.09654 batch_time=0.86316
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 4.26955 (QuantReg: 8.90187) QuantErr: 8.90187 batch_time=0.51472
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.93158 (QuantReg: 9.66069) QuantErr: 9.66069 batch_time=0.49893
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 4.03342 (QuantReg: 9.32530) QuantErr: 9.32530 batch_time=0.50324
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.68075 (QuantReg: 9.21780) QuantErr: 9.21780 batch_time=0.50284
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.76892 (QuantReg: 8.91633) QuantErr: 8.91633 batch_time=0.50006
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.87999 (QuantReg: 9.42015) QuantErr: 9.42015 batch_time=0.50777
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.97358 (QuantReg: 9.41666) QuantErr: 9.41666 batch_time=0.51109
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.72532 (QuantReg: 9.18845) QuantErr: 9.18845 batch_time=0.49889
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.85554 (QuantReg: 9.55422) QuantErr: 9.55422 batch_time=0.51552
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.99655 (QuantReg: 9.67535) QuantErr: 9.67535 batch_time=0.51083
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.77095 (QuantReg: 9.49008) QuantErr: 9.49008 batch_time=0.50168
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.69917 (QuantReg: 9.26249) QuantErr: 9.26249 batch_time=0.50006
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.65169 (QuantReg: 9.24094) QuantErr: 9.24094 batch_time=0.50379
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.49460 (QuantReg: 9.48797) QuantErr: 9.48797 batch_time=0.50482
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.86013 (QuantReg: 9.43784) QuantErr: 9.43784 batch_time=0.50532
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 4.12478 (QuantReg: 9.89019) QuantErr: 9.89019 batch_time=0.50168
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.64879 (QuantReg: 9.78254) QuantErr: 9.78254 batch_time=1.58562
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.93355 (QuantReg: 9.74990) QuantErr: 9.74990 batch_time=0.51036
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.33265 (QuantReg: 9.74513) QuantErr: 9.74513 batch_time=0.51897
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.40526 (QuantReg: 9.78330) QuantErr: 9.78330 batch_time=0.52082
Train Epoch: 3 codebook_update_time=1.82756
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch3.pth ...
Done in 3.709s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch3.pth ...
Done in 7.772s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.844153156280518
quant_reg : 9.402127494812012
quant_err : 9.402127494812012
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_miech_test/t2v_metrics/R1: 12.6
MSRVTT_miech_test/t2v_metrics/R5: 36.7
MSRVTT_miech_test/t2v_metrics/R10: 50.3
MSRVTT_miech_test/t2v_metrics/R50: 83.1
MSRVTT_miech_test/t2v_metrics/MedR: 10.0
MSRVTT_miech_test/t2v_metrics/MeanR: 40.434
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.54531664073665
MSRVTT_miech_test/v2t_metrics/R1: 14.2
MSRVTT_miech_test/v2t_metrics/R5: 37.1
MSRVTT_miech_test/v2t_metrics/R10: 51.5
MSRVTT_miech_test/v2t_metrics/R50: 83.6
MSRVTT_miech_test/v2t_metrics/MedR: 10.0
MSRVTT_miech_test/v2t_metrics/MeanR: 40.733
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.04852517164174
mnt_best : 28.54531664073665
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.59086 (QuantReg: 8.62110) QuantErr: 8.62110 batch_time=24.50133
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.57254 (QuantReg: 8.97310) QuantErr: 8.97310 batch_time=0.48702
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 3.37211 (QuantReg: 8.33414) QuantErr: 8.33414 batch_time=5.84220
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.71682 (QuantReg: 8.92876) QuantErr: 8.92876 batch_time=0.54724
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.66609 (QuantReg: 8.92282) QuantErr: 8.92282 batch_time=0.52862
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.29499 (QuantReg: 8.88344) QuantErr: 8.88344 batch_time=0.50597
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.75313 (QuantReg: 8.61064) QuantErr: 8.61064 batch_time=1.97328
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 3.33933 (QuantReg: 8.67573) QuantErr: 8.67573 batch_time=0.49279
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.52726 (QuantReg: 8.77917) QuantErr: 8.77917 batch_time=0.51032
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.48697 (QuantReg: 9.06530) QuantErr: 9.06530 batch_time=0.50587
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.71363 (QuantReg: 9.03978) QuantErr: 9.03978 batch_time=0.55081
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.49767 (QuantReg: 8.61157) QuantErr: 8.61157 batch_time=0.52409
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.31218 (QuantReg: 9.08998) QuantErr: 9.08998 batch_time=0.73345
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 3.38215 (QuantReg: 8.71513) QuantErr: 8.71513 batch_time=0.63578
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.47527 (QuantReg: 8.94675) QuantErr: 8.94675 batch_time=0.49344
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 3.72129 (QuantReg: 8.71146) QuantErr: 8.71146 batch_time=0.50038
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.58212 (QuantReg: 9.23367) QuantErr: 9.23367 batch_time=0.50653
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 3.60428 (QuantReg: 8.76464) QuantErr: 8.76464 batch_time=0.50133
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 3.59399 (QuantReg: 9.39857) QuantErr: 9.39857 batch_time=0.50116
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 3.30771 (QuantReg: 8.71208) QuantErr: 8.71208 batch_time=0.84656
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 3.33040 (QuantReg: 9.28375) QuantErr: 9.28375 batch_time=0.49790
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 3.16269 (QuantReg: 9.12388) QuantErr: 9.12388 batch_time=0.48423
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 3.14952 (QuantReg: 9.06298) QuantErr: 9.06298 batch_time=0.50286
Train Epoch: 4 codebook_update_time=1.82162
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch4.pth ...
Done in 3.954s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch4.pth ...
Done in 7.875s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 3.5177882318496705
quant_reg : 8.886661056518555
quant_err : 8.886661056518555
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_miech_test/t2v_metrics/R1: 12.7
MSRVTT_miech_test/t2v_metrics/R5: 36.5
MSRVTT_miech_test/t2v_metrics/R10: 50.4
MSRVTT_miech_test/t2v_metrics/R50: 83.5
MSRVTT_miech_test/t2v_metrics/MedR: 10.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.194
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.587469040551056
MSRVTT_miech_test/v2t_metrics/R1: 15.2
MSRVTT_miech_test/v2t_metrics/R5: 38.7
MSRVTT_miech_test/v2t_metrics/R10: 53.8
MSRVTT_miech_test/v2t_metrics/R50: 84.8
MSRVTT_miech_test/v2t_metrics/MedR: 9.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.605
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.63095295353805
mnt_best : 28.587469040551056
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.88434 (QuantReg: 8.47526) QuantErr: 8.47526 batch_time=23.82650
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.08489 (QuantReg: 8.71117) QuantErr: 8.71117 batch_time=0.52214
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 3.22259 (QuantReg: 8.51342) QuantErr: 8.51342 batch_time=0.52024
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 3.30612 (QuantReg: 8.64833) QuantErr: 8.64833 batch_time=0.50204
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 3.70489 (QuantReg: 8.72252) QuantErr: 8.72252 batch_time=0.54271
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 3.04672 (QuantReg: 8.71307) QuantErr: 8.71307 batch_time=0.70987
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 3.50752 (QuantReg: 8.66770) QuantErr: 8.66770 batch_time=0.50871
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.39836 (QuantReg: 8.50150) QuantErr: 8.50150 batch_time=0.50882
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 3.24014 (QuantReg: 8.51773) QuantErr: 8.51773 batch_time=0.51491
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 3.20151 (QuantReg: 8.84759) QuantErr: 8.84759 batch_time=0.49559
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 3.39775 (QuantReg: 8.94933) QuantErr: 8.94933 batch_time=0.49790
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 3.46970 (QuantReg: 8.45927) QuantErr: 8.45927 batch_time=0.50825
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 3.53149 (QuantReg: 8.90425) QuantErr: 8.90425 batch_time=1.42289
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 3.40465 (QuantReg: 8.45391) QuantErr: 8.45391 batch_time=3.05234
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 3.18497 (QuantReg: 8.34907) QuantErr: 8.34907 batch_time=0.51273
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.77602 (QuantReg: 8.76293) QuantErr: 8.76293 batch_time=0.49094
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 3.34817 (QuantReg: 8.82309) QuantErr: 8.82309 batch_time=0.71710
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 3.32290 (QuantReg: 8.76887) QuantErr: 8.76887 batch_time=0.55145
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 3.27160 (QuantReg: 8.86262) QuantErr: 8.86262 batch_time=0.51360
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 3.36495 (QuantReg: 8.75768) QuantErr: 8.75768 batch_time=0.50986
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 3.44650 (QuantReg: 8.68902) QuantErr: 8.68902 batch_time=0.52578
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.79729 (QuantReg: 8.53074) QuantErr: 8.53074 batch_time=0.49043
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.11475 (QuantReg: 8.98428) QuantErr: 8.98428 batch_time=0.52649
Train Epoch: 5 codebook_update_time=1.86385
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch5.pth ...
Done in 3.859s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch5.pth ...
Done in 7.715s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 3.3190775108337403
quant_reg : 8.645332328796387
quant_err : 8.645332328796387
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_miech_test/t2v_metrics/R1: 14.6
MSRVTT_miech_test/t2v_metrics/R5: 39.6
MSRVTT_miech_test/t2v_metrics/R10: 54.4
MSRVTT_miech_test/t2v_metrics/R50: 83.8
MSRVTT_miech_test/t2v_metrics/MedR: 9.0
MSRVTT_miech_test/t2v_metrics/MeanR: 39.409
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.5657161693417
MSRVTT_miech_test/v2t_metrics/R1: 15.2
MSRVTT_miech_test/v2t_metrics/R5: 40.4
MSRVTT_miech_test/v2t_metrics/R10: 55.8
MSRVTT_miech_test/v2t_metrics/R50: 84.9
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 38.9
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.48027647446801
mnt_best : 31.5657161693417
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 3.27521 (QuantReg: 8.62107) QuantErr: 8.62107 batch_time=24.05816
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.95948 (QuantReg: 8.64002) QuantErr: 8.64002 batch_time=0.50898
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 3.27275 (QuantReg: 8.12126) QuantErr: 8.12126 batch_time=0.49659
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 3.44899 (QuantReg: 8.37577) QuantErr: 8.37577 batch_time=0.48333
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 3.14343 (QuantReg: 8.09248) QuantErr: 8.09248 batch_time=0.49108
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 3.31257 (QuantReg: 8.56506) QuantErr: 8.56506 batch_time=0.48710
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 3.16486 (QuantReg: 8.53231) QuantErr: 8.53231 batch_time=0.50923
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 3.11170 (QuantReg: 8.25495) QuantErr: 8.25495 batch_time=0.50284
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 3.25427 (QuantReg: 8.35543) QuantErr: 8.35543 batch_time=0.49168
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 3.10664 (QuantReg: 8.38653) QuantErr: 8.38653 batch_time=0.50419
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.97068 (QuantReg: 8.68363) QuantErr: 8.68363 batch_time=0.50709
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 3.42079 (QuantReg: 8.84126) QuantErr: 8.84126 batch_time=0.53479
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 3.20953 (QuantReg: 8.39008) QuantErr: 8.39008 batch_time=0.48402
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 3.57816 (QuantReg: 8.81016) QuantErr: 8.81016 batch_time=0.48794
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 3.30790 (QuantReg: 8.32854) QuantErr: 8.32854 batch_time=0.48360
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 3.08887 (QuantReg: 8.56348) QuantErr: 8.56348 batch_time=0.51017
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 3.30096 (QuantReg: 8.55773) QuantErr: 8.55773 batch_time=0.48687
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 3.34917 (QuantReg: 8.68504) QuantErr: 8.68504 batch_time=0.49283
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.78852 (QuantReg: 8.55737) QuantErr: 8.55737 batch_time=0.50490
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 3.38140 (QuantReg: 8.49281) QuantErr: 8.49281 batch_time=1.28178
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 3.14517 (QuantReg: 8.71110) QuantErr: 8.71110 batch_time=1.69907
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 3.01594 (QuantReg: 9.07815) QuantErr: 9.07815 batch_time=0.50371
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.91075 (QuantReg: 8.50459) QuantErr: 8.50459 batch_time=0.53272
Train Epoch: 6 codebook_update_time=2.19239
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch6.pth ...
Done in 4.181s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch6.pth ...
Done in 8.814s
removing stale ckpt [epoch 5] [took 0.05s]
epoch : 6
loss : 3.120764910697937
quant_reg : 8.56220118713379
quant_err : 8.56220118713379
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_miech_test/t2v_metrics/R1: 15.2
MSRVTT_miech_test/t2v_metrics/R5: 41.4
MSRVTT_miech_test/t2v_metrics/R10: 55.6
MSRVTT_miech_test/t2v_metrics/R50: 85.8
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.56
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.70691434181951
MSRVTT_miech_test/v2t_metrics/R1: 15.9
MSRVTT_miech_test/v2t_metrics/R5: 42.5
MSRVTT_miech_test/v2t_metrics/R10: 56.3
MSRVTT_miech_test/v2t_metrics/R50: 85.8
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 38.3045
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.63293875193561
mnt_best : 32.70691434181951
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 3.27725 (QuantReg: 8.52699) QuantErr: 8.52699 batch_time=27.82440
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 3.14435 (QuantReg: 8.28977) QuantErr: 8.28977 batch_time=0.60956
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 3.20243 (QuantReg: 8.84775) QuantErr: 8.84775 batch_time=0.52190
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 3.26686 (QuantReg: 8.31435) QuantErr: 8.31435 batch_time=0.48978
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.83454 (QuantReg: 8.51680) QuantErr: 8.51680 batch_time=0.51616
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 3.09748 (QuantReg: 8.36318) QuantErr: 8.36318 batch_time=0.76869
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 3.06317 (QuantReg: 8.62308) QuantErr: 8.62308 batch_time=0.50926
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 3.03201 (QuantReg: 8.24662) QuantErr: 8.24662 batch_time=0.53670
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.92618 (QuantReg: 8.64543) QuantErr: 8.64543 batch_time=0.50974
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 3.04766 (QuantReg: 8.65791) QuantErr: 8.65791 batch_time=0.58691
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.97965 (QuantReg: 8.25815) QuantErr: 8.25815 batch_time=0.54017
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 3.27022 (QuantReg: 8.52891) QuantErr: 8.52891 batch_time=0.60359
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.92145 (QuantReg: 8.28786) QuantErr: 8.28786 batch_time=0.82078
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.77028 (QuantReg: 8.48781) QuantErr: 8.48781 batch_time=0.49897
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 3.09630 (QuantReg: 8.75089) QuantErr: 8.75089 batch_time=0.51299
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 3.18293 (QuantReg: 8.74188) QuantErr: 8.74188 batch_time=0.52117
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 2.82265 (QuantReg: 8.52950) QuantErr: 8.52950 batch_time=0.58719
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 3.04278 (QuantReg: 8.30809) QuantErr: 8.30809 batch_time=0.50456
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.85407 (QuantReg: 8.31621) QuantErr: 8.31621 batch_time=0.49953
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.73584 (QuantReg: 8.53808) QuantErr: 8.53808 batch_time=0.52590
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 3.00733 (QuantReg: 8.75557) QuantErr: 8.75557 batch_time=0.51073
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 3.02291 (QuantReg: 8.73390) QuantErr: 8.73390 batch_time=0.52393
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.89788 (QuantReg: 8.63274) QuantErr: 8.63274 batch_time=0.53181
Train Epoch: 7 codebook_update_time=1.76726
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch7.pth ...
Done in 4.133s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 2.99194287109375
quant_reg : 8.458193584442139
quant_err : 8.458193584442139
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_miech_test/t2v_metrics/R1: 14.9
MSRVTT_miech_test/t2v_metrics/R5: 41.2
MSRVTT_miech_test/t2v_metrics/R10: 56.5
MSRVTT_miech_test/t2v_metrics/R50: 85.5
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.098
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.61199077142814
MSRVTT_miech_test/v2t_metrics/R1: 14.7
MSRVTT_miech_test/v2t_metrics/R5: 42.5
MSRVTT_miech_test/v2t_metrics/R10: 57.2
MSRVTT_miech_test/v2t_metrics/R50: 85.4
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.7445
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.93826843201215
mnt_best : 32.70691434181951
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.88591 (QuantReg: 8.48051) QuantErr: 8.48051 batch_time=27.78690
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.07208 (QuantReg: 8.59849) QuantErr: 8.59849 batch_time=0.57432
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 3.15299 (QuantReg: 8.49028) QuantErr: 8.49028 batch_time=0.48627
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 3.04266 (QuantReg: 8.27029) QuantErr: 8.27029 batch_time=0.66000
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.92793 (QuantReg: 8.76581) QuantErr: 8.76581 batch_time=0.54274
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 3.08733 (QuantReg: 8.33280) QuantErr: 8.33280 batch_time=0.54257
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 3.53791 (QuantReg: 8.43910) QuantErr: 8.43910 batch_time=0.84451
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.88973 (QuantReg: 8.49926) QuantErr: 8.49926 batch_time=0.56576
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.74606 (QuantReg: 8.27347) QuantErr: 8.27347 batch_time=0.49808
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.77323 (QuantReg: 8.71337) QuantErr: 8.71337 batch_time=0.49512
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 3.14316 (QuantReg: 8.51680) QuantErr: 8.51680 batch_time=0.51032
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.71543 (QuantReg: 8.10996) QuantErr: 8.10996 batch_time=0.54347
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.77230 (QuantReg: 8.55248) QuantErr: 8.55248 batch_time=0.48489
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.84801 (QuantReg: 8.61900) QuantErr: 8.61900 batch_time=0.49716
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.41215 (QuantReg: 8.02598) QuantErr: 8.02598 batch_time=0.52272
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.86745 (QuantReg: 8.33927) QuantErr: 8.33927 batch_time=0.54742
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 3.03576 (QuantReg: 8.42374) QuantErr: 8.42374 batch_time=0.52149
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.75939 (QuantReg: 8.56111) QuantErr: 8.56111 batch_time=0.49152
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.82045 (QuantReg: 8.16336) QuantErr: 8.16336 batch_time=0.55327
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.75544 (QuantReg: 8.65866) QuantErr: 8.65866 batch_time=1.73525
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.72531 (QuantReg: 8.12929) QuantErr: 8.12929 batch_time=0.49099
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 2.90138 (QuantReg: 8.61460) QuantErr: 8.61460 batch_time=0.59268
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 3.14551 (QuantReg: 8.45840) QuantErr: 8.45840 batch_time=0.58091
Train Epoch: 8 codebook_update_time=1.91786
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch8.pth ...
Done in 3.800s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch8.pth ...
Done in 7.547s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 2.8725391607284547
quant_reg : 8.390648866653443
quant_err : 8.390648866653443
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_miech_test/t2v_metrics/R1: 14.9
MSRVTT_miech_test/t2v_metrics/R5: 42.2
MSRVTT_miech_test/t2v_metrics/R10: 56.1
MSRVTT_miech_test/t2v_metrics/R50: 86.3
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.568
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.79597350434354
MSRVTT_miech_test/v2t_metrics/R1: 16.1
MSRVTT_miech_test/v2t_metrics/R5: 42.6
MSRVTT_miech_test/v2t_metrics/R10: 57.1
MSRVTT_miech_test/v2t_metrics/R50: 86.2
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.78
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.95917996191334
mnt_best : 32.79597350434354
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 3.10006 (QuantReg: 8.17565) QuantErr: 8.17565 batch_time=35.75252
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.67813 (QuantReg: 7.85094) QuantErr: 7.85094 batch_time=0.51413
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.59324 (QuantReg: 7.93162) QuantErr: 7.93162 batch_time=0.68760
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.89149 (QuantReg: 8.50761) QuantErr: 8.50761 batch_time=0.55608
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.71301 (QuantReg: 8.08003) QuantErr: 8.08003 batch_time=0.58106
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.84321 (QuantReg: 8.26953) QuantErr: 8.26953 batch_time=0.50898
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.70990 (QuantReg: 8.58934) QuantErr: 8.58934 batch_time=0.49559
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 2.71771 (QuantReg: 8.14633) QuantErr: 8.14633 batch_time=0.56689
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.61083 (QuantReg: 7.97769) QuantErr: 7.97769 batch_time=0.51528
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.52864 (QuantReg: 8.01796) QuantErr: 8.01796 batch_time=0.51266
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.76057 (QuantReg: 8.37931) QuantErr: 8.37931 batch_time=0.53705
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 2.73221 (QuantReg: 8.57781) QuantErr: 8.57781 batch_time=0.55146
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.48486 (QuantReg: 8.02502) QuantErr: 8.02502 batch_time=0.51087
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.70155 (QuantReg: 8.64294) QuantErr: 8.64294 batch_time=0.55071
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.31575 (QuantReg: 8.27785) QuantErr: 8.27785 batch_time=0.51741
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 2.58839 (QuantReg: 8.58668) QuantErr: 8.58668 batch_time=0.52036
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.69529 (QuantReg: 8.30858) QuantErr: 8.30858 batch_time=0.53536
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 3.27830 (QuantReg: 8.31364) QuantErr: 8.31364 batch_time=0.52431
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.49383 (QuantReg: 8.06689) QuantErr: 8.06689 batch_time=0.50707
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.94237 (QuantReg: 8.13247) QuantErr: 8.13247 batch_time=0.51132
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 3.05194 (QuantReg: 8.26602) QuantErr: 8.26602 batch_time=0.57647
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.71221 (QuantReg: 8.26312) QuantErr: 8.26312 batch_time=0.61790
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 2.47337 (QuantReg: 8.34440) QuantErr: 8.34440 batch_time=0.50269
Train Epoch: 9 codebook_update_time=1.77118
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch9.pth ...
Done in 4.067s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 2.770172918319702
quant_reg : 8.306956399917603
quant_err : 8.306956399917603
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_miech_test/t2v_metrics/R1: 15.0
MSRVTT_miech_test/t2v_metrics/R5: 41.9
MSRVTT_miech_test/t2v_metrics/R10: 55.2
MSRVTT_miech_test/t2v_metrics/R50: 85.7
MSRVTT_miech_test/t2v_metrics/MedR: 9.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.591
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.61480502342901
MSRVTT_miech_test/v2t_metrics/R1: 14.8
MSRVTT_miech_test/v2t_metrics/R5: 41.7
MSRVTT_miech_test/v2t_metrics/R10: 57.3
MSRVTT_miech_test/v2t_metrics/R50: 85.6
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 38.257
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.823442739382884
mnt_best : 32.79597350434354
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 2.86761 (QuantReg: 8.22727) QuantErr: 8.22727 batch_time=25.35220
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.72656 (QuantReg: 8.62396) QuantErr: 8.62396 batch_time=0.50971
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.87519 (QuantReg: 8.15536) QuantErr: 8.15536 batch_time=0.52575
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.94268 (QuantReg: 8.22187) QuantErr: 8.22187 batch_time=0.51558
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 2.51880 (QuantReg: 8.37259) QuantErr: 8.37259 batch_time=0.55559
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.70864 (QuantReg: 8.52958) QuantErr: 8.52958 batch_time=0.48504
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.55018 (QuantReg: 8.57384) QuantErr: 8.57384 batch_time=0.54936
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 2.71143 (QuantReg: 8.39116) QuantErr: 8.39116 batch_time=0.52151
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.42855 (QuantReg: 8.19551) QuantErr: 8.19551 batch_time=0.49815
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.90934 (QuantReg: 8.65381) QuantErr: 8.65381 batch_time=0.48992
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 3.09441 (QuantReg: 8.49121) QuantErr: 8.49121 batch_time=0.54331
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.85441 (QuantReg: 8.24864) QuantErr: 8.24864 batch_time=0.51457
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.47195 (QuantReg: 8.41979) QuantErr: 8.41979 batch_time=0.53676
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 3.21015 (QuantReg: 8.31393) QuantErr: 8.31393 batch_time=1.06819
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 2.73244 (QuantReg: 8.37114) QuantErr: 8.37114 batch_time=0.55629
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.55754 (QuantReg: 8.25398) QuantErr: 8.25398 batch_time=0.53084
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.00699 (QuantReg: 7.96041) QuantErr: 7.96041 batch_time=0.49460
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.45199 (QuantReg: 8.52403) QuantErr: 8.52403 batch_time=0.52835
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.86595 (QuantReg: 8.35115) QuantErr: 8.35115 batch_time=1.94571
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.71558 (QuantReg: 8.69987) QuantErr: 8.69987 batch_time=0.50364
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.78600 (QuantReg: 8.49358) QuantErr: 8.49358 batch_time=0.52955
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.52158 (QuantReg: 8.02260) QuantErr: 8.02260 batch_time=0.51818
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.69432 (QuantReg: 8.09035) QuantErr: 8.09035 batch_time=0.68986
Train Epoch: 10 codebook_update_time=1.68063
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch10.pth ...
Done in 23.387s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch10.pth ...
Done in 27.694s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 2.679824920654297
quant_reg : 8.276396621704102
quant_err : 8.276396621704102
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_miech_test/t2v_metrics/R1: 15.1
MSRVTT_miech_test/t2v_metrics/R5: 42.5
MSRVTT_miech_test/t2v_metrics/R10: 55.9
MSRVTT_miech_test/t2v_metrics/R50: 85.4
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.025
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.98065134513912
MSRVTT_miech_test/v2t_metrics/R1: 15.6
MSRVTT_miech_test/v2t_metrics/R5: 44.7
MSRVTT_miech_test/v2t_metrics/R10: 58.6
MSRVTT_miech_test/v2t_metrics/R50: 87.0
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 36.356
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.44370908670192
mnt_best : 32.98065134513912
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.80416 (QuantReg: 8.22800) QuantErr: 8.22800 batch_time=26.04554
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.36981 (QuantReg: 8.40451) QuantErr: 8.40451 batch_time=0.48045
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.80473 (QuantReg: 8.50681) QuantErr: 8.50681 batch_time=0.48140
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.79525 (QuantReg: 7.90385) QuantErr: 7.90385 batch_time=0.51610
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.61811 (QuantReg: 8.57360) QuantErr: 8.57360 batch_time=0.58259
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.75976 (QuantReg: 7.95270) QuantErr: 7.95270 batch_time=0.53838
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 2.46668 (QuantReg: 8.17805) QuantErr: 8.17805 batch_time=7.15656
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.85380 (QuantReg: 8.09612) QuantErr: 8.09612 batch_time=0.53081
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.64924 (QuantReg: 8.35368) QuantErr: 8.35368 batch_time=0.48428
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.57285 (QuantReg: 7.99860) QuantErr: 7.99860 batch_time=0.49872
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.71100 (QuantReg: 8.20275) QuantErr: 8.20275 batch_time=0.49466
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.49082 (QuantReg: 8.33398) QuantErr: 8.33398 batch_time=0.52391
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.78598 (QuantReg: 8.20717) QuantErr: 8.20717 batch_time=0.51296
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.76441 (QuantReg: 8.10775) QuantErr: 8.10775 batch_time=0.54653
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 2.54916 (QuantReg: 8.08860) QuantErr: 8.08860 batch_time=0.56931
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.77399 (QuantReg: 8.00228) QuantErr: 8.00228 batch_time=0.49677
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.82659 (QuantReg: 8.20185) QuantErr: 8.20185 batch_time=0.49866
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 2.43684 (QuantReg: 8.24761) QuantErr: 8.24761 batch_time=0.58350
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 2.74996 (QuantReg: 8.02624) QuantErr: 8.02624 batch_time=0.52107
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 2.32235 (QuantReg: 8.45786) QuantErr: 8.45786 batch_time=0.53492
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 2.47699 (QuantReg: 8.03068) QuantErr: 8.03068 batch_time=0.53745
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.68462 (QuantReg: 8.33055) QuantErr: 8.33055 batch_time=0.62999
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 2.50528 (QuantReg: 8.15891) QuantErr: 8.15891 batch_time=0.54302
Train Epoch: 11 codebook_update_time=1.73864
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch11.pth ...
Done in 11.478s
removing stale ckpt [epoch 10] [took 0.02s]
epoch : 11
loss : 2.625202124595642
quant_reg : 8.248076276779175
quant_err : 8.248076276779175
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_miech_test/t2v_metrics/R1: 14.9
MSRVTT_miech_test/t2v_metrics/R5: 42.0
MSRVTT_miech_test/t2v_metrics/R10: 56.6
MSRVTT_miech_test/t2v_metrics/R50: 86.3
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.876
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.84107238243554
MSRVTT_miech_test/v2t_metrics/R1: 16.3
MSRVTT_miech_test/v2t_metrics/R5: 44.8
MSRVTT_miech_test/v2t_metrics/R10: 58.7
MSRVTT_miech_test/v2t_metrics/R50: 86.0
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.333
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.997302649271035
mnt_best : 32.98065134513912
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 2.77370 (QuantReg: 7.77587) QuantErr: 7.77587 batch_time=28.04065
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 2.45387 (QuantReg: 8.17625) QuantErr: 8.17625 batch_time=1.15082
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 2.63596 (QuantReg: 8.18223) QuantErr: 8.18223 batch_time=0.53235
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.53012 (QuantReg: 8.27887) QuantErr: 8.27887 batch_time=0.50882
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.51017 (QuantReg: 8.22072) QuantErr: 8.22072 batch_time=0.50113
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 2.62760 (QuantReg: 8.28099) QuantErr: 8.28099 batch_time=0.54524
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 2.59635 (QuantReg: 8.07933) QuantErr: 8.07933 batch_time=0.62816
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 2.41409 (QuantReg: 8.42186) QuantErr: 8.42186 batch_time=0.54808
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 2.49355 (QuantReg: 7.67369) QuantErr: 7.67369 batch_time=0.53250
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 2.38797 (QuantReg: 8.32334) QuantErr: 8.32334 batch_time=0.54575
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 2.46681 (QuantReg: 8.52682) QuantErr: 8.52682 batch_time=0.83433
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 2.58078 (QuantReg: 8.19656) QuantErr: 8.19656 batch_time=0.49097
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 2.33004 (QuantReg: 7.86305) QuantErr: 7.86305 batch_time=2.17680
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 2.61688 (QuantReg: 8.18273) QuantErr: 8.18273 batch_time=0.60683
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.53608 (QuantReg: 8.37104) QuantErr: 8.37104 batch_time=0.51656
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 2.59229 (QuantReg: 7.81661) QuantErr: 7.81661 batch_time=0.58517
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.64540 (QuantReg: 8.19917) QuantErr: 8.19917 batch_time=0.51802
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 2.70752 (QuantReg: 8.19199) QuantErr: 8.19199 batch_time=0.56203
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.42136 (QuantReg: 8.57574) QuantErr: 8.57574 batch_time=0.48823
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 2.60996 (QuantReg: 8.26402) QuantErr: 8.26402 batch_time=0.55616
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 2.44138 (QuantReg: 8.06237) QuantErr: 8.06237 batch_time=0.56058
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.52534 (QuantReg: 8.16169) QuantErr: 8.16169 batch_time=0.67840
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 2.52963 (QuantReg: 8.44377) QuantErr: 8.44377 batch_time=0.51371
Train Epoch: 12 codebook_update_time=1.95922
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch12.pth ...
Done in 5.373s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 2.5419753894805908
quant_reg : 8.20388597869873
quant_err : 8.20388597869873
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_miech_test/t2v_metrics/R1: 14.5
MSRVTT_miech_test/t2v_metrics/R5: 41.4
MSRVTT_miech_test/t2v_metrics/R10: 57.4
MSRVTT_miech_test/t2v_metrics/R50: 86.7
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.295
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.540689076620716
MSRVTT_miech_test/v2t_metrics/R1: 16.4
MSRVTT_miech_test/v2t_metrics/R5: 45.5
MSRVTT_miech_test/v2t_metrics/R10: 59.4
MSRVTT_miech_test/v2t_metrics/R50: 86.5
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.0635
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.39000004817215
mnt_best : 32.98065134513912
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.47214 (QuantReg: 8.20849) QuantErr: 8.20849 batch_time=28.20758
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 2.58397 (QuantReg: 8.01029) QuantErr: 8.01029 batch_time=0.63654
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 2.44860 (QuantReg: 8.22330) QuantErr: 8.22330 batch_time=0.50814
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 2.91522 (QuantReg: 8.22167) QuantErr: 8.22167 batch_time=0.52988
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.64407 (QuantReg: 8.27215) QuantErr: 8.27215 batch_time=0.56100
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.78825 (QuantReg: 8.17856) QuantErr: 8.17856 batch_time=0.56104
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 2.32629 (QuantReg: 8.33524) QuantErr: 8.33524 batch_time=0.50952
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 2.54998 (QuantReg: 8.35142) QuantErr: 8.35142 batch_time=0.50598
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 2.20207 (QuantReg: 8.09597) QuantErr: 8.09597 batch_time=0.53674
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 2.53892 (QuantReg: 8.00377) QuantErr: 8.00377 batch_time=0.54569
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 2.62480 (QuantReg: 8.21701) QuantErr: 8.21701 batch_time=0.53317
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 2.33276 (QuantReg: 8.40452) QuantErr: 8.40452 batch_time=1.12417
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 2.76481 (QuantReg: 7.82840) QuantErr: 7.82840 batch_time=0.58486
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 2.70583 (QuantReg: 8.38052) QuantErr: 8.38052 batch_time=0.57236
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 2.47480 (QuantReg: 8.45312) QuantErr: 8.45312 batch_time=0.55694
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 2.64227 (QuantReg: 7.97779) QuantErr: 7.97779 batch_time=0.56853
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 2.56243 (QuantReg: 8.55466) QuantErr: 8.55466 batch_time=0.54898
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 2.58621 (QuantReg: 8.17546) QuantErr: 8.17546 batch_time=0.54737
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 2.24893 (QuantReg: 8.23452) QuantErr: 8.23452 batch_time=0.51818
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.49097 (QuantReg: 8.49326) QuantErr: 8.49326 batch_time=2.63222
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 2.25734 (QuantReg: 8.03143) QuantErr: 8.03143 batch_time=0.50817
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 2.16804 (QuantReg: 7.86279) QuantErr: 7.86279 batch_time=0.54057
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.89481 (QuantReg: 8.04045) QuantErr: 8.04045 batch_time=0.53442
Train Epoch: 13 codebook_update_time=1.69235
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch13.pth ...
Done in 5.176s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch13.pth ...
Done in 10.108s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 2.4920988187789916
quant_reg : 8.187079141616822
quant_err : 8.187079141616822
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_miech_test/t2v_metrics/R1: 15.4
MSRVTT_miech_test/t2v_metrics/R5: 42.7
MSRVTT_miech_test/t2v_metrics/R10: 56.5
MSRVTT_miech_test/t2v_metrics/R50: 85.7
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.137
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.36816680841286
MSRVTT_miech_test/v2t_metrics/R1: 15.0
MSRVTT_miech_test/v2t_metrics/R5: 44.2
MSRVTT_miech_test/v2t_metrics/R10: 59.1
MSRVTT_miech_test/v2t_metrics/R50: 86.2
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 36.4095
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.96516039067291
mnt_best : 33.36816680841286
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 2.66735 (QuantReg: 7.91284) QuantErr: 7.91284 batch_time=32.01012
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.28888 (QuantReg: 8.19693) QuantErr: 8.19693 batch_time=0.51420
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 2.21685 (QuantReg: 7.87203) QuantErr: 7.87203 batch_time=0.50998
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.51207 (QuantReg: 8.47474) QuantErr: 8.47474 batch_time=0.52219
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.59853 (QuantReg: 7.99346) QuantErr: 7.99346 batch_time=0.50440
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.34428 (QuantReg: 8.30202) QuantErr: 8.30202 batch_time=0.53219
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.43463 (QuantReg: 7.97907) QuantErr: 7.97907 batch_time=0.61413
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 2.80605 (QuantReg: 7.88919) QuantErr: 7.88919 batch_time=0.60207
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.63609 (QuantReg: 8.21991) QuantErr: 8.21991 batch_time=0.78376
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 2.44719 (QuantReg: 8.26948) QuantErr: 8.26948 batch_time=0.51385
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.58015 (QuantReg: 8.10745) QuantErr: 8.10745 batch_time=0.53717
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 2.37752 (QuantReg: 8.07264) QuantErr: 8.07264 batch_time=0.52065
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 2.39569 (QuantReg: 7.90318) QuantErr: 7.90318 batch_time=0.54147
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 2.33254 (QuantReg: 8.22089) QuantErr: 8.22089 batch_time=0.51120
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 2.24998 (QuantReg: 7.88002) QuantErr: 7.88002 batch_time=0.60321
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 2.07541 (QuantReg: 8.19973) QuantErr: 8.19973 batch_time=0.51397
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 2.54011 (QuantReg: 7.88333) QuantErr: 7.88333 batch_time=0.50378
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 2.38683 (QuantReg: 8.23847) QuantErr: 8.23847 batch_time=0.51100
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 2.18305 (QuantReg: 8.09467) QuantErr: 8.09467 batch_time=0.55295
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 2.65503 (QuantReg: 8.36142) QuantErr: 8.36142 batch_time=1.76735
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.33332 (QuantReg: 8.29734) QuantErr: 8.29734 batch_time=0.55906
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.17936 (QuantReg: 7.90248) QuantErr: 7.90248 batch_time=0.63458
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.41943 (QuantReg: 8.22751) QuantErr: 8.22751 batch_time=0.53067
Train Epoch: 14 codebook_update_time=2.05163
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch14.pth ...
Done in 5.246s
removing stale ckpt [epoch 13] [took 0.03s]
epoch : 14
loss : 2.42850532579422
quant_reg : 8.174701370239259
quant_err : 8.174701370239259
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_miech_test/t2v_metrics/R1: 14.1
MSRVTT_miech_test/t2v_metrics/R5: 43.2
MSRVTT_miech_test/t2v_metrics/R10: 58.0
MSRVTT_miech_test/t2v_metrics/R50: 85.4
MSRVTT_miech_test/t2v_metrics/MedR: 7.5
MSRVTT_miech_test/t2v_metrics/MeanR: 36.813
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.812824649849084
MSRVTT_miech_test/v2t_metrics/R1: 16.8
MSRVTT_miech_test/v2t_metrics/R5: 45.3
MSRVTT_miech_test/v2t_metrics/R10: 58.0
MSRVTT_miech_test/v2t_metrics/R50: 85.4
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 36.6105
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.340972202282636
mnt_best : 33.36816680841286
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.46821 (QuantReg: 8.32763) QuantErr: 8.32763 batch_time=26.34461
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.70174 (QuantReg: 8.06931) QuantErr: 8.06931 batch_time=0.49644
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.65006 (QuantReg: 8.19021) QuantErr: 8.19021 batch_time=0.51601
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.68842 (QuantReg: 7.95053) QuantErr: 7.95053 batch_time=0.52925
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.60339 (QuantReg: 8.15555) QuantErr: 8.15555 batch_time=0.52499
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 2.10274 (QuantReg: 8.09702) QuantErr: 8.09702 batch_time=0.96591
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.51596 (QuantReg: 8.04068) QuantErr: 8.04068 batch_time=0.49243
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.10422 (QuantReg: 8.13989) QuantErr: 8.13989 batch_time=0.51346
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.64455 (QuantReg: 8.11262) QuantErr: 8.11262 batch_time=0.57035
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 2.33995 (QuantReg: 8.12512) QuantErr: 8.12512 batch_time=0.52360
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.15482 (QuantReg: 8.27905) QuantErr: 8.27905 batch_time=0.55184
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.46584 (QuantReg: 8.14099) QuantErr: 8.14099 batch_time=0.51111
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.58438 (QuantReg: 8.29018) QuantErr: 8.29018 batch_time=0.54337
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 2.27349 (QuantReg: 8.06217) QuantErr: 8.06217 batch_time=2.97811
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.31853 (QuantReg: 8.09594) QuantErr: 8.09594 batch_time=0.49555
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.25333 (QuantReg: 8.06284) QuantErr: 8.06284 batch_time=1.00244
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 2.20337 (QuantReg: 8.07242) QuantErr: 8.07242 batch_time=0.53143
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 2.61696 (QuantReg: 8.27684) QuantErr: 8.27684 batch_time=0.50154
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.95292 (QuantReg: 7.87646) QuantErr: 7.87646 batch_time=0.50739
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.04342 (QuantReg: 8.21353) QuantErr: 8.21353 batch_time=0.59697
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.19590 (QuantReg: 8.26524) QuantErr: 8.26524 batch_time=0.52531
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.45848 (QuantReg: 8.24152) QuantErr: 8.24152 batch_time=0.50419
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.56281 (QuantReg: 8.49624) QuantErr: 8.49624 batch_time=0.58920
Train Epoch: 15 codebook_update_time=1.65935
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch15.pth ...
Done in 5.266s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch15.pth ...
Done in 10.716s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 2.4011084957122804
quant_reg : 8.124391672134399
quant_err : 8.124391672134399
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_miech_test/t2v_metrics/R1: 16.1
MSRVTT_miech_test/t2v_metrics/R5: 43.2
MSRVTT_miech_test/t2v_metrics/R10: 58.2
MSRVTT_miech_test/t2v_metrics/R50: 85.8
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 36.687
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.33556533912691
MSRVTT_miech_test/v2t_metrics/R1: 16.7
MSRVTT_miech_test/v2t_metrics/R5: 45.3
MSRVTT_miech_test/v2t_metrics/R10: 59.0
MSRVTT_miech_test/v2t_metrics/R50: 86.5
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 36.0945
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.472262933574676
mnt_best : 34.33556533912691
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 2.42756 (QuantReg: 7.75138) QuantErr: 7.75138 batch_time=35.04855
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 2.49994 (QuantReg: 7.84446) QuantErr: 7.84446 batch_time=0.54676
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 2.07160 (QuantReg: 7.83734) QuantErr: 7.83734 batch_time=0.55178
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.64163 (QuantReg: 7.82976) QuantErr: 7.82976 batch_time=0.48675
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 2.46513 (QuantReg: 7.99365) QuantErr: 7.99365 batch_time=0.52599
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 2.79269 (QuantReg: 8.28435) QuantErr: 8.28435 batch_time=0.50046
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.56657 (QuantReg: 8.16350) QuantErr: 8.16350 batch_time=0.47993
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 2.34308 (QuantReg: 8.55569) QuantErr: 8.55569 batch_time=0.50463
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 2.21176 (QuantReg: 7.71669) QuantErr: 7.71669 batch_time=0.53604
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 2.15770 (QuantReg: 8.14730) QuantErr: 8.14730 batch_time=1.15730
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 2.21439 (QuantReg: 8.43914) QuantErr: 8.43914 batch_time=0.48757
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 2.23442 (QuantReg: 8.11638) QuantErr: 8.11638 batch_time=0.48887
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.43252 (QuantReg: 8.10154) QuantErr: 8.10154 batch_time=0.62248
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 2.38702 (QuantReg: 7.97778) QuantErr: 7.97778 batch_time=0.65070
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.46948 (QuantReg: 8.20708) QuantErr: 8.20708 batch_time=0.48889
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 2.35342 (QuantReg: 8.03382) QuantErr: 8.03382 batch_time=0.56935
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 2.33446 (QuantReg: 8.33916) QuantErr: 8.33916 batch_time=0.52300
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 2.34494 (QuantReg: 7.68796) QuantErr: 7.68796 batch_time=0.54559
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.40679 (QuantReg: 8.13751) QuantErr: 8.13751 batch_time=0.48070
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 2.33895 (QuantReg: 8.02692) QuantErr: 8.02692 batch_time=0.52790
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.43316 (QuantReg: 8.17694) QuantErr: 8.17694 batch_time=0.50146
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 2.23305 (QuantReg: 8.25926) QuantErr: 8.25926 batch_time=0.52696
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 2.34685 (QuantReg: 8.04234) QuantErr: 8.04234 batch_time=0.53715
Train Epoch: 16 codebook_update_time=1.88645
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch16.pth ...
Done in 5.976s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch16.pth ...
Done in 12.528s
removing stale ckpt [epoch 15] [took 0.04s]
epoch : 16
loss : 2.3424986572265625
quant_reg : 8.12967359161377
quant_err : 8.12967359161377
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_miech_test/t2v_metrics/R1: 16.1
MSRVTT_miech_test/t2v_metrics/R5: 44.9
MSRVTT_miech_test/t2v_metrics/R10: 58.3
MSRVTT_miech_test/t2v_metrics/R50: 86.2
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.584
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.80008119723864
MSRVTT_miech_test/v2t_metrics/R1: 17.9
MSRVTT_miech_test/v2t_metrics/R5: 46.2
MSRVTT_miech_test/v2t_metrics/R10: 60.2
MSRVTT_miech_test/v2t_metrics/R50: 86.9
MSRVTT_miech_test/v2t_metrics/MedR: 6.25
MSRVTT_miech_test/v2t_metrics/MeanR: 35.7595
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.78723663396232
mnt_best : 34.80008119723864
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 2.47783 (QuantReg: 7.88265) QuantErr: 7.88265 batch_time=31.87273
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 2.22396 (QuantReg: 8.42834) QuantErr: 8.42834 batch_time=0.48968
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 2.28995 (QuantReg: 8.41820) QuantErr: 8.41820 batch_time=0.59673
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.48979 (QuantReg: 7.82900) QuantErr: 7.82900 batch_time=0.53518
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 2.39307 (QuantReg: 8.33981) QuantErr: 8.33981 batch_time=0.61445
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 2.38294 (QuantReg: 8.28969) QuantErr: 8.28969 batch_time=0.59365
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 2.58959 (QuantReg: 8.01079) QuantErr: 8.01079 batch_time=6.30496
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 2.31930 (QuantReg: 8.39964) QuantErr: 8.39964 batch_time=0.57598
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 2.23189 (QuantReg: 8.12218) QuantErr: 8.12218 batch_time=0.54473
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 2.03204 (QuantReg: 8.60221) QuantErr: 8.60221 batch_time=0.48379
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 2.10727 (QuantReg: 8.36191) QuantErr: 8.36191 batch_time=0.51478
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 2.23292 (QuantReg: 8.11020) QuantErr: 8.11020 batch_time=0.52033
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 2.44616 (QuantReg: 7.87976) QuantErr: 7.87976 batch_time=0.51633
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.46540 (QuantReg: 8.14278) QuantErr: 8.14278 batch_time=0.51815
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 2.42702 (QuantReg: 8.06473) QuantErr: 8.06473 batch_time=0.53893
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.58193 (QuantReg: 7.95890) QuantErr: 7.95890 batch_time=0.52471
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.99846 (QuantReg: 7.86321) QuantErr: 7.86321 batch_time=0.79304
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 2.11041 (QuantReg: 7.89702) QuantErr: 7.89702 batch_time=0.54536
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.51067 (QuantReg: 8.41430) QuantErr: 8.41430 batch_time=0.50304
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 2.42471 (QuantReg: 8.19680) QuantErr: 8.19680 batch_time=0.58262
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 2.16256 (QuantReg: 8.19260) QuantErr: 8.19260 batch_time=0.51730
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 2.31427 (QuantReg: 7.99237) QuantErr: 7.99237 batch_time=0.59698
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 2.31977 (QuantReg: 8.06656) QuantErr: 8.06656 batch_time=0.54783
Train Epoch: 17 codebook_update_time=2.17267
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch17.pth ...
Done in 6.176s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch17.pth ...
Done in 11.714s
removing stale ckpt [epoch 16] [took 0.23s]
epoch : 17
loss : 2.3063598408699035
quant_reg : 8.133119201660156
quant_err : 8.133119201660156
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_miech_test/t2v_metrics/R1: 16.6
MSRVTT_miech_test/t2v_metrics/R5: 43.9
MSRVTT_miech_test/t2v_metrics/R10: 57.9
MSRVTT_miech_test/t2v_metrics/R50: 85.5
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.3035
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.81371668285987
MSRVTT_miech_test/v2t_metrics/R1: 18.2
MSRVTT_miech_test/v2t_metrics/R5: 46.6
MSRVTT_miech_test/v2t_metrics/R10: 60.0
MSRVTT_miech_test/v2t_metrics/R50: 85.7
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.087
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.05693693062076
mnt_best : 34.81371668285987
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.34315 (QuantReg: 7.90152) QuantErr: 7.90152 batch_time=32.05444
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 2.29718 (QuantReg: 8.03017) QuantErr: 8.03017 batch_time=0.55306
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 2.36169 (QuantReg: 7.92491) QuantErr: 7.92491 batch_time=2.06098
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 2.55083 (QuantReg: 8.14744) QuantErr: 8.14744 batch_time=0.52065
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 2.28923 (QuantReg: 8.21781) QuantErr: 8.21781 batch_time=0.50993
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 2.36225 (QuantReg: 8.11454) QuantErr: 8.11454 batch_time=0.50932
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 2.08581 (QuantReg: 8.19886) QuantErr: 8.19886 batch_time=0.60623
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 2.39714 (QuantReg: 8.09042) QuantErr: 8.09042 batch_time=0.53544
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 2.45651 (QuantReg: 8.37291) QuantErr: 8.37291 batch_time=0.56946
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 2.52771 (QuantReg: 8.12086) QuantErr: 8.12086 batch_time=0.50886
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 2.09591 (QuantReg: 8.16735) QuantErr: 8.16735 batch_time=0.60034
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 2.16445 (QuantReg: 8.55513) QuantErr: 8.55513 batch_time=0.51199
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.38562 (QuantReg: 8.15970) QuantErr: 8.15970 batch_time=0.52602
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 2.14362 (QuantReg: 7.95134) QuantErr: 7.95134 batch_time=0.52247
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 2.12958 (QuantReg: 7.95089) QuantErr: 7.95089 batch_time=0.49148
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 2.26775 (QuantReg: 8.00556) QuantErr: 8.00556 batch_time=0.88675
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 2.42027 (QuantReg: 8.30281) QuantErr: 8.30281 batch_time=0.52584
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 2.32651 (QuantReg: 8.06395) QuantErr: 8.06395 batch_time=0.51911
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.37746 (QuantReg: 8.08204) QuantErr: 8.08204 batch_time=0.48241
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 2.32731 (QuantReg: 8.13230) QuantErr: 8.13230 batch_time=0.49713
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 2.25245 (QuantReg: 7.81881) QuantErr: 7.81881 batch_time=0.57300
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.39768 (QuantReg: 8.23777) QuantErr: 8.23777 batch_time=0.49920
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 2.29823 (QuantReg: 8.01976) QuantErr: 8.01976 batch_time=0.51112
Train Epoch: 18 codebook_update_time=1.84978
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch18.pth ...
Done in 5.921s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch18.pth ...
Done in 10.816s
removing stale ckpt [epoch 17] [took 0.27s]
epoch : 18
loss : 2.28190527009964
quant_reg : 8.129754409790038
quant_err : 8.129754409790038
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_miech_test/t2v_metrics/R1: 18.2
MSRVTT_miech_test/t2v_metrics/R5: 44.0
MSRVTT_miech_test/t2v_metrics/R10: 58.9
MSRVTT_miech_test/t2v_metrics/R50: 85.5
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.7295
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.13098375115593
MSRVTT_miech_test/v2t_metrics/R1: 17.7
MSRVTT_miech_test/v2t_metrics/R5: 46.3
MSRVTT_miech_test/v2t_metrics/R10: 60.4
MSRVTT_miech_test/v2t_metrics/R50: 85.6
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.4525
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.716707689878724
mnt_best : 36.13098375115593
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 2.40054 (QuantReg: 8.04415) QuantErr: 8.04415 batch_time=27.74128
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 2.12572 (QuantReg: 8.26923) QuantErr: 8.26923 batch_time=0.51228
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 2.23522 (QuantReg: 7.67859) QuantErr: 7.67859 batch_time=0.58025
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 2.38427 (QuantReg: 7.90475) QuantErr: 7.90475 batch_time=0.50662
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 2.34416 (QuantReg: 8.23674) QuantErr: 8.23674 batch_time=0.58337
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 2.48219 (QuantReg: 8.31901) QuantErr: 8.31901 batch_time=0.76187
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 2.33520 (QuantReg: 8.07737) QuantErr: 8.07737 batch_time=0.54600
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 2.09142 (QuantReg: 8.26735) QuantErr: 8.26735 batch_time=0.52099
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 2.47472 (QuantReg: 8.09922) QuantErr: 8.09922 batch_time=0.58853
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.97702 (QuantReg: 7.81405) QuantErr: 7.81405 batch_time=0.55380
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 2.19733 (QuantReg: 8.19395) QuantErr: 8.19395 batch_time=0.51475
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 2.30931 (QuantReg: 7.99930) QuantErr: 7.99930 batch_time=0.54869
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 2.16451 (QuantReg: 8.16805) QuantErr: 8.16805 batch_time=0.60815
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 2.08301 (QuantReg: 7.91315) QuantErr: 7.91315 batch_time=0.51549
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 2.67230 (QuantReg: 8.03700) QuantErr: 8.03700 batch_time=0.49683
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 2.07248 (QuantReg: 8.39178) QuantErr: 8.39178 batch_time=0.81316
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 2.16135 (QuantReg: 8.21142) QuantErr: 8.21142 batch_time=0.51062
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 2.10974 (QuantReg: 8.08402) QuantErr: 8.08402 batch_time=0.50291
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 2.45399 (QuantReg: 8.15528) QuantErr: 8.15528 batch_time=0.54743
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 2.21671 (QuantReg: 7.81355) QuantErr: 7.81355 batch_time=0.51571
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 2.28050 (QuantReg: 8.20419) QuantErr: 8.20419 batch_time=0.54593
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 2.17876 (QuantReg: 7.99798) QuantErr: 7.99798 batch_time=0.52030
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 2.34360 (QuantReg: 7.82744) QuantErr: 7.82744 batch_time=0.50483
Train Epoch: 19 codebook_update_time=2.61752
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.12/checkpoint-epoch19.pth ...
Done in 5.663s
removing stale ckpt [epoch 18] [took 0.03s]
epoch : 19
loss : 2.247394281387329
quant_reg : 8.128447959899903
quant_err : 8.128447959899903
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_miech_test/t2v_metrics/R1: 16.6
MSRVTT_miech_test/t2v_metrics/R5: 44.2
MSRVTT_miech_test/t2v_metrics/R10: 59.0
MSRVTT_miech_test/t2v_metrics/R50: 85.7
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.644