-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kB_M8.txt
2605 lines (2605 loc) · 190 KB
/
HCQ_MSRVTT_1kB_M8.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8
Preparing the dataloaders ...
Loading dataset MSRVTT_miech_trainval in ram ...
Finish loading dataset MSRVTT_miech_trainval in ram, taking 747.5879929065704 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 118.44615030288696 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 62.703160762786865 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch0.pth ...
Done in 1.563s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch0.pth ...
Done in 3.074s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_miech_test/t2v_metrics/R1: 0.0
MSRVTT_miech_test/t2v_metrics/R5: 0.4
MSRVTT_miech_test/t2v_metrics/R10: 0.7
MSRVTT_miech_test/t2v_metrics/R50: 5.2
MSRVTT_miech_test/t2v_metrics/MedR: 509.5
MSRVTT_miech_test/t2v_metrics/MeanR: 501.004
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_miech_test/v2t_metrics/R1: 0.0
MSRVTT_miech_test/v2t_metrics/R5: 0.4
MSRVTT_miech_test/v2t_metrics/R10: 1.0
MSRVTT_miech_test/v2t_metrics/R50: 5.3
MSRVTT_miech_test/v2t_metrics/MedR: 505.0
MSRVTT_miech_test/v2t_metrics/MeanR: 503.0955
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.80898 (QuantReg: 10.41390) QuantErr: 10.41390 batch_time=29.88012
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.57001 (QuantReg: 10.44960) QuantErr: 10.44960 batch_time=0.42585
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.21768 (QuantReg: 10.46850) QuantErr: 10.46850 batch_time=0.42510
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.44603 (QuantReg: 10.46797) QuantErr: 10.46797 batch_time=0.43107
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.26020 (QuantReg: 10.46943) QuantErr: 10.46943 batch_time=0.42627
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.11453 (QuantReg: 10.48271) QuantErr: 10.48271 batch_time=0.43330
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.87836 (QuantReg: 10.48118) QuantErr: 10.48118 batch_time=0.70113
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.52940 (QuantReg: 10.49059) QuantErr: 10.49059 batch_time=0.83816
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.12815 (QuantReg: 10.48790) QuantErr: 10.48790 batch_time=0.42838
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.15058 (QuantReg: 10.47846) QuantErr: 10.47846 batch_time=0.43237
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.07477 (QuantReg: 10.47028) QuantErr: 10.47028 batch_time=0.42436
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.10933 (QuantReg: 10.48070) QuantErr: 10.48070 batch_time=0.44096
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.53603 (QuantReg: 10.49473) QuantErr: 10.49473 batch_time=0.42169
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.74741 (QuantReg: 10.49493) QuantErr: 10.49493 batch_time=0.46718
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.59129 (QuantReg: 10.49523) QuantErr: 10.49523 batch_time=0.42406
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.36816 (QuantReg: 10.48308) QuantErr: 10.48308 batch_time=0.41989
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.19201 (QuantReg: 10.48907) QuantErr: 10.48907 batch_time=0.44195
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.61051 (QuantReg: 10.47758) QuantErr: 10.47758 batch_time=0.43034
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.47559 (QuantReg: 10.48243) QuantErr: 10.48243 batch_time=0.44082
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.62004 (QuantReg: 10.47756) QuantErr: 10.47756 batch_time=0.42628
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.45312 (QuantReg: 10.48300) QuantErr: 10.48300 batch_time=0.43027
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.74971 (QuantReg: 10.47337) QuantErr: 10.47337 batch_time=0.42527
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.99374 (QuantReg: 10.46851) QuantErr: 10.46851 batch_time=0.42914
Train Epoch: 1 codebook_update_time=0.69101
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch1.pth ...
Done in 6.675s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch1.pth ...
Done in 11.621s
epoch : 1
loss : 5.37263491821289
quant_reg : 10.477305290222167
quant_err : 10.477305290222167
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_miech_test/t2v_metrics/R1: 6.9
MSRVTT_miech_test/t2v_metrics/R5: 25.6
MSRVTT_miech_test/t2v_metrics/R10: 39.1
MSRVTT_miech_test/t2v_metrics/R50: 73.0
MSRVTT_miech_test/t2v_metrics/MedR: 17.0
MSRVTT_miech_test/t2v_metrics/MeanR: 54.2945
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.04387276164142
MSRVTT_miech_test/v2t_metrics/R1: 9.2
MSRVTT_miech_test/v2t_metrics/R5: 27.9
MSRVTT_miech_test/v2t_metrics/R10: 41.6
MSRVTT_miech_test/v2t_metrics/R50: 73.1
MSRVTT_miech_test/v2t_metrics/MedR: 16.0
MSRVTT_miech_test/v2t_metrics/MeanR: 56.3425
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.020564792835017
mnt_best : 19.04387276164142
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.00193 (QuantReg: 3.77009) QuantErr: 3.77009 batch_time=28.06857
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.81597 (QuantReg: 3.89824) QuantErr: 3.89824 batch_time=0.45106
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.02143 (QuantReg: 3.96063) QuantErr: 3.96063 batch_time=1.54400
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.96521 (QuantReg: 3.97925) QuantErr: 3.97925 batch_time=0.43096
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.53997 (QuantReg: 3.97111) QuantErr: 3.97111 batch_time=0.42849
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.42666 (QuantReg: 4.07205) QuantErr: 4.07205 batch_time=0.43243
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.80884 (QuantReg: 4.16555) QuantErr: 4.16555 batch_time=0.42650
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.56908 (QuantReg: 4.06568) QuantErr: 4.06568 batch_time=0.42804
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.49561 (QuantReg: 4.19499) QuantErr: 4.19499 batch_time=0.42293
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.73121 (QuantReg: 4.44973) QuantErr: 4.44973 batch_time=0.43426
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.71577 (QuantReg: 4.40102) QuantErr: 4.40102 batch_time=0.44298
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.23718 (QuantReg: 4.41493) QuantErr: 4.41493 batch_time=0.43064
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.50235 (QuantReg: 4.56316) QuantErr: 4.56316 batch_time=0.43880
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.79670 (QuantReg: 4.65881) QuantErr: 4.65881 batch_time=0.58110
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.34973 (QuantReg: 4.51975) QuantErr: 4.51975 batch_time=0.44113
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 4.07678 (QuantReg: 4.49498) QuantErr: 4.49498 batch_time=0.45134
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.54442 (QuantReg: 4.50415) QuantErr: 4.50415 batch_time=0.42970
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 2.99845 (QuantReg: 4.71041) QuantErr: 4.71041 batch_time=0.44610
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.94350 (QuantReg: 4.69301) QuantErr: 4.69301 batch_time=0.43246
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.65008 (QuantReg: 4.74617) QuantErr: 4.74617 batch_time=1.59802
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.48040 (QuantReg: 4.90033) QuantErr: 4.90033 batch_time=0.42643
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.30516 (QuantReg: 4.95542) QuantErr: 4.95542 batch_time=0.42866
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.04663 (QuantReg: 4.99807) QuantErr: 4.99807 batch_time=0.43224
Train Epoch: 2 codebook_update_time=0.55095
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch2.pth ...
Done in 4.056s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch2.pth ...
Done in 8.083s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.04s]
epoch : 2
loss : 3.6516632890701293
quant_reg : 4.430675397872925
quant_err : 4.430675397872925
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_miech_test/t2v_metrics/R1: 9.5
MSRVTT_miech_test/t2v_metrics/R5: 32.6
MSRVTT_miech_test/t2v_metrics/R10: 46.1
MSRVTT_miech_test/t2v_metrics/R50: 79.5
MSRVTT_miech_test/t2v_metrics/MedR: 13.0
MSRVTT_miech_test/t2v_metrics/MeanR: 46.712
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.259436579639267
MSRVTT_miech_test/v2t_metrics/R1: 9.4
MSRVTT_miech_test/v2t_metrics/R5: 32.5
MSRVTT_miech_test/v2t_metrics/R10: 48.9
MSRVTT_miech_test/v2t_metrics/R50: 79.8
MSRVTT_miech_test/v2t_metrics/MedR: 11.0
MSRVTT_miech_test/v2t_metrics/MeanR: 45.747
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.628616971518564
mnt_best : 24.259436579639267
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.65859 (QuantReg: 4.05234) QuantErr: 4.05234 batch_time=33.58154
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.17881 (QuantReg: 3.96312) QuantErr: 3.96312 batch_time=0.43715
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.23804 (QuantReg: 4.00693) QuantErr: 4.00693 batch_time=0.43063
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.79220 (QuantReg: 3.99095) QuantErr: 3.99095 batch_time=0.43617
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.10412 (QuantReg: 4.31348) QuantErr: 4.31348 batch_time=0.43368
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.13996 (QuantReg: 4.11096) QuantErr: 4.11096 batch_time=0.45382
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.94390 (QuantReg: 4.16939) QuantErr: 4.16939 batch_time=0.42106
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 2.79937 (QuantReg: 4.05359) QuantErr: 4.05359 batch_time=0.42152
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.06228 (QuantReg: 4.25239) QuantErr: 4.25239 batch_time=0.43025
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.04975 (QuantReg: 4.26237) QuantErr: 4.26237 batch_time=0.43014
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.91739 (QuantReg: 4.13796) QuantErr: 4.13796 batch_time=0.43551
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.04036 (QuantReg: 4.22328) QuantErr: 4.22328 batch_time=0.42697
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.32738 (QuantReg: 4.37343) QuantErr: 4.37343 batch_time=0.46729
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.18599 (QuantReg: 4.33478) QuantErr: 4.33478 batch_time=0.43467
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 2.96490 (QuantReg: 4.21308) QuantErr: 4.21308 batch_time=0.41819
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.89315 (QuantReg: 4.30381) QuantErr: 4.30381 batch_time=0.42072
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.66076 (QuantReg: 4.34978) QuantErr: 4.34978 batch_time=0.47886
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.15591 (QuantReg: 4.40829) QuantErr: 4.40829 batch_time=0.44623
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.27323 (QuantReg: 4.55668) QuantErr: 4.55668 batch_time=0.43010
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.84491 (QuantReg: 4.41994) QuantErr: 4.41994 batch_time=0.42082
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.33144 (QuantReg: 4.48311) QuantErr: 4.48311 batch_time=0.54276
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.57241 (QuantReg: 4.52804) QuantErr: 4.52804 batch_time=0.44553
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.53294 (QuantReg: 4.49330) QuantErr: 4.49330 batch_time=0.44381
Train Epoch: 3 codebook_update_time=0.58801
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch3.pth ...
Done in 4.082s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch3.pth ...
Done in 8.124s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.1016547622680664
quant_reg : 4.259480820655822
quant_err : 4.259480820655822
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_miech_test/t2v_metrics/R1: 10.4
MSRVTT_miech_test/t2v_metrics/R5: 34.4
MSRVTT_miech_test/t2v_metrics/R10: 49.9
MSRVTT_miech_test/t2v_metrics/R50: 82.7
MSRVTT_miech_test/t2v_metrics/MedR: 11.0
MSRVTT_miech_test/t2v_metrics/MeanR: 41.9645
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.135497761404565
MSRVTT_miech_test/v2t_metrics/R1: 11.6
MSRVTT_miech_test/v2t_metrics/R5: 35.6
MSRVTT_miech_test/v2t_metrics/R10: 50.6
MSRVTT_miech_test/v2t_metrics/R50: 81.0
MSRVTT_miech_test/v2t_metrics/MedR: 10.0
MSRVTT_miech_test/v2t_metrics/MeanR: 42.173
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.543523806505323
mnt_best : 26.135497761404565
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.75811 (QuantReg: 4.07675) QuantErr: 4.07675 batch_time=34.44164
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.81058 (QuantReg: 4.19466) QuantErr: 4.19466 batch_time=0.43009
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.66622 (QuantReg: 3.95952) QuantErr: 3.95952 batch_time=0.44740
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.05777 (QuantReg: 4.17517) QuantErr: 4.17517 batch_time=0.45014
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.82813 (QuantReg: 4.18841) QuantErr: 4.18841 batch_time=0.41960
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.39626 (QuantReg: 4.24004) QuantErr: 4.24004 batch_time=0.44953
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.02340 (QuantReg: 4.15804) QuantErr: 4.15804 batch_time=0.44045
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.38334 (QuantReg: 4.17742) QuantErr: 4.17742 batch_time=0.42693
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.69948 (QuantReg: 4.25769) QuantErr: 4.25769 batch_time=0.44205
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.57376 (QuantReg: 4.35976) QuantErr: 4.35976 batch_time=0.42007
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.13454 (QuantReg: 4.30167) QuantErr: 4.30167 batch_time=0.43296
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.03030 (QuantReg: 4.22105) QuantErr: 4.22105 batch_time=0.44287
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.45443 (QuantReg: 4.36666) QuantErr: 4.36666 batch_time=0.43074
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.65314 (QuantReg: 4.09828) QuantErr: 4.09828 batch_time=0.43069
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.76105 (QuantReg: 4.31423) QuantErr: 4.31423 batch_time=0.42744
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.90399 (QuantReg: 4.31373) QuantErr: 4.31373 batch_time=0.42853
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.55777 (QuantReg: 4.48978) QuantErr: 4.48978 batch_time=0.42492
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.84284 (QuantReg: 4.24510) QuantErr: 4.24510 batch_time=0.42174
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.81850 (QuantReg: 4.53199) QuantErr: 4.53199 batch_time=0.42404
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.45539 (QuantReg: 4.21750) QuantErr: 4.21750 batch_time=0.43301
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.43199 (QuantReg: 4.46574) QuantErr: 4.46574 batch_time=0.41432
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.39784 (QuantReg: 4.47025) QuantErr: 4.47025 batch_time=0.42918
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.41533 (QuantReg: 4.36105) QuantErr: 4.36105 batch_time=0.54090
Train Epoch: 4 codebook_update_time=0.61227
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch4.pth ...
Done in 3.994s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch4.pth ...
Done in 7.969s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 2.7154566230773924
quant_reg : 4.290366445541382
quant_err : 4.290366445541382
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_miech_test/t2v_metrics/R1: 11.7
MSRVTT_miech_test/t2v_metrics/R5: 35.9
MSRVTT_miech_test/t2v_metrics/R10: 51.1
MSRVTT_miech_test/t2v_metrics/R50: 83.3
MSRVTT_miech_test/t2v_metrics/MedR: 10.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.5235
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.79075870741037
MSRVTT_miech_test/v2t_metrics/R1: 11.3
MSRVTT_miech_test/v2t_metrics/R5: 37.4
MSRVTT_miech_test/v2t_metrics/R10: 52.7
MSRVTT_miech_test/v2t_metrics/R50: 82.2
MSRVTT_miech_test/v2t_metrics/MedR: 9.5
MSRVTT_miech_test/v2t_metrics/MeanR: 40.431
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.135429784449105
mnt_best : 27.79075870741037
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.23315 (QuantReg: 4.11209) QuantErr: 4.11209 batch_time=29.80838
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.23594 (QuantReg: 4.28555) QuantErr: 4.28555 batch_time=0.42721
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.40540 (QuantReg: 4.18696) QuantErr: 4.18696 batch_time=3.93010
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.50996 (QuantReg: 4.27357) QuantErr: 4.27357 batch_time=0.45488
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.79097 (QuantReg: 4.24041) QuantErr: 4.24041 batch_time=0.44367
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.15124 (QuantReg: 4.29965) QuantErr: 4.29965 batch_time=0.46591
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.51500 (QuantReg: 4.29774) QuantErr: 4.29774 batch_time=0.43489
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.54646 (QuantReg: 4.29348) QuantErr: 4.29348 batch_time=0.45819
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.27319 (QuantReg: 4.16777) QuantErr: 4.16777 batch_time=0.45126
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.30459 (QuantReg: 4.45169) QuantErr: 4.45169 batch_time=0.42956
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.27885 (QuantReg: 4.35404) QuantErr: 4.35404 batch_time=0.50789
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.68350 (QuantReg: 4.35747) QuantErr: 4.35747 batch_time=0.42493
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.61885 (QuantReg: 4.48252) QuantErr: 4.48252 batch_time=0.47728
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.43662 (QuantReg: 4.26368) QuantErr: 4.26368 batch_time=0.99583
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.39417 (QuantReg: 4.30972) QuantErr: 4.30972 batch_time=0.42991
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 1.99576 (QuantReg: 4.49683) QuantErr: 4.49683 batch_time=0.43243
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.50149 (QuantReg: 4.46502) QuantErr: 4.46502 batch_time=0.41870
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.44820 (QuantReg: 4.37567) QuantErr: 4.37567 batch_time=0.47652
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.27904 (QuantReg: 4.45693) QuantErr: 4.45693 batch_time=0.42042
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.79704 (QuantReg: 4.35631) QuantErr: 4.35631 batch_time=0.43937
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.42891 (QuantReg: 4.34891) QuantErr: 4.34891 batch_time=0.44097
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 1.65686 (QuantReg: 4.43118) QuantErr: 4.43118 batch_time=0.43060
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.19311 (QuantReg: 4.59270) QuantErr: 4.59270 batch_time=0.46144
Train Epoch: 5 codebook_update_time=0.62361
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch5.pth ...
Done in 4.007s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch5.pth ...
Done in 7.927s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.486507229804993
quant_reg : 4.331526572227478
quant_err : 4.331526572227478
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_miech_test/t2v_metrics/R1: 10.6
MSRVTT_miech_test/t2v_metrics/R5: 38.3
MSRVTT_miech_test/t2v_metrics/R10: 53.9
MSRVTT_miech_test/t2v_metrics/R50: 83.8
MSRVTT_miech_test/t2v_metrics/MedR: 9.0
MSRVTT_miech_test/t2v_metrics/MeanR: 39.5695
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.970343600302403
MSRVTT_miech_test/v2t_metrics/R1: 10.8
MSRVTT_miech_test/v2t_metrics/R5: 37.6
MSRVTT_miech_test/v2t_metrics/R10: 53.3
MSRVTT_miech_test/v2t_metrics/R50: 83.9
MSRVTT_miech_test/v2t_metrics/MedR: 9.0
MSRVTT_miech_test/v2t_metrics/MeanR: 38.382
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.868457821068574
mnt_best : 27.970343600302403
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.23661 (QuantReg: 4.32160) QuantErr: 4.32160 batch_time=29.63229
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.20314 (QuantReg: 4.43854) QuantErr: 4.43854 batch_time=0.42942
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.45156 (QuantReg: 4.13767) QuantErr: 4.13767 batch_time=0.42175
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.51257 (QuantReg: 4.31879) QuantErr: 4.31879 batch_time=0.41960
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.18688 (QuantReg: 4.13960) QuantErr: 4.13960 batch_time=0.42015
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.32553 (QuantReg: 4.38776) QuantErr: 4.38776 batch_time=0.42041
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.43146 (QuantReg: 4.40105) QuantErr: 4.40105 batch_time=0.43771
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.43083 (QuantReg: 4.25693) QuantErr: 4.25693 batch_time=0.41734
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.29935 (QuantReg: 4.25086) QuantErr: 4.25086 batch_time=0.42212
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.31802 (QuantReg: 4.29678) QuantErr: 4.29678 batch_time=0.42506
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 1.90962 (QuantReg: 4.42925) QuantErr: 4.42925 batch_time=0.42372
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.33964 (QuantReg: 4.55508) QuantErr: 4.55508 batch_time=0.41521
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.37269 (QuantReg: 4.33349) QuantErr: 4.33349 batch_time=0.42412
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.58933 (QuantReg: 4.52879) QuantErr: 4.52879 batch_time=0.43295
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.45150 (QuantReg: 4.34434) QuantErr: 4.34434 batch_time=0.44184
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.36325 (QuantReg: 4.46081) QuantErr: 4.46081 batch_time=0.43462
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.48894 (QuantReg: 4.39091) QuantErr: 4.39091 batch_time=0.42164
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.61493 (QuantReg: 4.51317) QuantErr: 4.51317 batch_time=0.44024
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 1.82649 (QuantReg: 4.46213) QuantErr: 4.46213 batch_time=0.43633
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.43896 (QuantReg: 4.43615) QuantErr: 4.43615 batch_time=0.50841
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.38151 (QuantReg: 4.47141) QuantErr: 4.47141 batch_time=0.43293
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.09173 (QuantReg: 4.49934) QuantErr: 4.49934 batch_time=0.66673
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.17023 (QuantReg: 4.33776) QuantErr: 4.33776 batch_time=0.42373
Train Epoch: 6 codebook_update_time=0.62103
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch6.pth ...
Done in 3.774s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch6.pth ...
Done in 7.610s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 2.2699023065567014
quant_reg : 4.405037113189698
quant_err : 4.405037113189698
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_miech_test/t2v_metrics/R1: 12.6
MSRVTT_miech_test/t2v_metrics/R5: 37.3
MSRVTT_miech_test/t2v_metrics/R10: 54.5
MSRVTT_miech_test/t2v_metrics/R50: 84.2
MSRVTT_miech_test/t2v_metrics/MedR: 9.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.5335
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.477589030736123
MSRVTT_miech_test/v2t_metrics/R1: 12.8
MSRVTT_miech_test/v2t_metrics/R5: 40.8
MSRVTT_miech_test/v2t_metrics/R10: 55.3
MSRVTT_miech_test/v2t_metrics/R50: 83.3
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 39.236
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.680687560482163
mnt_best : 29.477589030736123
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.32002 (QuantReg: 4.40455) QuantErr: 4.40455 batch_time=36.58017
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.21563 (QuantReg: 4.27203) QuantErr: 4.27203 batch_time=0.41935
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.44318 (QuantReg: 4.55221) QuantErr: 4.55221 batch_time=0.42926
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.27648 (QuantReg: 4.23453) QuantErr: 4.23453 batch_time=0.42813
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.03216 (QuantReg: 4.42587) QuantErr: 4.42587 batch_time=0.43253
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.10458 (QuantReg: 4.37999) QuantErr: 4.37999 batch_time=0.47271
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.23848 (QuantReg: 4.56007) QuantErr: 4.56007 batch_time=0.42584
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.17791 (QuantReg: 4.35537) QuantErr: 4.35537 batch_time=0.42340
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 1.98174 (QuantReg: 4.55573) QuantErr: 4.55573 batch_time=0.69606
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.05370 (QuantReg: 4.53755) QuantErr: 4.53755 batch_time=0.42349
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.25599 (QuantReg: 4.36879) QuantErr: 4.36879 batch_time=0.44908
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.29132 (QuantReg: 4.44155) QuantErr: 4.44155 batch_time=0.43355
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.81227 (QuantReg: 4.37790) QuantErr: 4.37790 batch_time=0.47956
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.01294 (QuantReg: 4.52841) QuantErr: 4.52841 batch_time=1.06183
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.04208 (QuantReg: 4.57486) QuantErr: 4.57486 batch_time=0.43454
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.27730 (QuantReg: 4.61733) QuantErr: 4.61733 batch_time=0.42892
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.71985 (QuantReg: 4.59014) QuantErr: 4.59014 batch_time=0.43182
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.13112 (QuantReg: 4.40802) QuantErr: 4.40802 batch_time=0.50584
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.02918 (QuantReg: 4.46531) QuantErr: 4.46531 batch_time=0.44539
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.89228 (QuantReg: 4.56842) QuantErr: 4.56842 batch_time=0.43186
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.19027 (QuantReg: 4.64372) QuantErr: 4.64372 batch_time=0.43298
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.02867 (QuantReg: 4.52082) QuantErr: 4.52082 batch_time=0.43991
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 1.88689 (QuantReg: 4.58005) QuantErr: 4.58005 batch_time=1.43822
Train Epoch: 7 codebook_update_time=0.69591
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch7.pth ...
Done in 3.955s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch7.pth ...
Done in 7.786s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 2.1074639439582823
quant_reg : 4.45799931716919
quant_err : 4.45799931716919
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_miech_test/t2v_metrics/R1: 13.5
MSRVTT_miech_test/t2v_metrics/R5: 39.7
MSRVTT_miech_test/t2v_metrics/R10: 55.4
MSRVTT_miech_test/t2v_metrics/R50: 85.4
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 36.675
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.9654940373115
MSRVTT_miech_test/v2t_metrics/R1: 12.2
MSRVTT_miech_test/v2t_metrics/R5: 41.6
MSRVTT_miech_test/v2t_metrics/R10: 55.7
MSRVTT_miech_test/v2t_metrics/R50: 84.3
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 37.522
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.462774255992226
mnt_best : 30.9654940373115
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.11194 (QuantReg: 4.46915) QuantErr: 4.46915 batch_time=33.32511
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.08282 (QuantReg: 4.57946) QuantErr: 4.57946 batch_time=0.41885
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.26284 (QuantReg: 4.47690) QuantErr: 4.47690 batch_time=0.42413
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.06978 (QuantReg: 4.47202) QuantErr: 4.47202 batch_time=0.45240
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.16288 (QuantReg: 4.62749) QuantErr: 4.62749 batch_time=0.44730
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.25354 (QuantReg: 4.45022) QuantErr: 4.45022 batch_time=0.43747
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.81814 (QuantReg: 4.50953) QuantErr: 4.50953 batch_time=0.45553
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.16263 (QuantReg: 4.38856) QuantErr: 4.38856 batch_time=0.42334
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.80598 (QuantReg: 4.47599) QuantErr: 4.47599 batch_time=0.42275
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.61990 (QuantReg: 4.66652) QuantErr: 4.66652 batch_time=0.41882
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.25106 (QuantReg: 4.68774) QuantErr: 4.68774 batch_time=0.43434
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 1.84330 (QuantReg: 4.45009) QuantErr: 4.45009 batch_time=0.44667
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.84941 (QuantReg: 4.54656) QuantErr: 4.54656 batch_time=0.43098
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.01709 (QuantReg: 4.52773) QuantErr: 4.52773 batch_time=0.42315
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 1.62934 (QuantReg: 4.37259) QuantErr: 4.37259 batch_time=0.45327
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.93906 (QuantReg: 4.46876) QuantErr: 4.46876 batch_time=0.42563
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.22358 (QuantReg: 4.51761) QuantErr: 4.51761 batch_time=0.42437
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 1.92516 (QuantReg: 4.61936) QuantErr: 4.61936 batch_time=0.43143
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.72406 (QuantReg: 4.38010) QuantErr: 4.38010 batch_time=0.63768
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.74162 (QuantReg: 4.62251) QuantErr: 4.62251 batch_time=0.44895
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.85151 (QuantReg: 4.46250) QuantErr: 4.46250 batch_time=0.44475
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.96852 (QuantReg: 4.56731) QuantErr: 4.56731 batch_time=0.42909
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.31157 (QuantReg: 4.62521) QuantErr: 4.62521 batch_time=0.42600
Train Epoch: 8 codebook_update_time=0.54268
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch8.pth ...
Done in 3.827s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch8.pth ...
Done in 8.032s
removing stale ckpt [epoch 7] [took 0.03s]
epoch : 8
loss : 1.9812725558280946
quant_reg : 4.502874179840088
quant_err : 4.502874179840088
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_miech_test/t2v_metrics/R1: 13.4
MSRVTT_miech_test/t2v_metrics/R5: 40.6
MSRVTT_miech_test/t2v_metrics/R10: 55.4
MSRVTT_miech_test/t2v_metrics/R50: 85.1
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.347
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.120521465479033
MSRVTT_miech_test/v2t_metrics/R1: 13.6
MSRVTT_miech_test/v2t_metrics/R5: 40.9
MSRVTT_miech_test/v2t_metrics/R10: 55.8
MSRVTT_miech_test/v2t_metrics/R50: 85.9
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.2785
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.4267018329018
mnt_best : 31.120521465479033
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.21345 (QuantReg: 4.46061) QuantErr: 4.46061 batch_time=30.32742
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.96630 (QuantReg: 4.28671) QuantErr: 4.28671 batch_time=0.41748
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.76862 (QuantReg: 4.38031) QuantErr: 4.38031 batch_time=0.46404
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 1.94785 (QuantReg: 4.63353) QuantErr: 4.63353 batch_time=0.42601
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.79871 (QuantReg: 4.43182) QuantErr: 4.43182 batch_time=0.44686
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.87666 (QuantReg: 4.54194) QuantErr: 4.54194 batch_time=0.46119
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.93919 (QuantReg: 4.59780) QuantErr: 4.59780 batch_time=0.46071
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.72798 (QuantReg: 4.48421) QuantErr: 4.48421 batch_time=0.45305
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.73222 (QuantReg: 4.39326) QuantErr: 4.39326 batch_time=0.89324
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.84069 (QuantReg: 4.44999) QuantErr: 4.44999 batch_time=0.42718
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.86197 (QuantReg: 4.52509) QuantErr: 4.52509 batch_time=0.42897
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.74375 (QuantReg: 4.75593) QuantErr: 4.75593 batch_time=0.42892
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.48552 (QuantReg: 4.51933) QuantErr: 4.51933 batch_time=0.47327
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.73873 (QuantReg: 4.71011) QuantErr: 4.71011 batch_time=0.57547
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.48914 (QuantReg: 4.58371) QuantErr: 4.58371 batch_time=0.47842
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.94196 (QuantReg: 4.60605) QuantErr: 4.60605 batch_time=0.47051
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.73052 (QuantReg: 4.60000) QuantErr: 4.60000 batch_time=0.44123
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.28162 (QuantReg: 4.60455) QuantErr: 4.60455 batch_time=0.56596
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.52515 (QuantReg: 4.52377) QuantErr: 4.52377 batch_time=0.46129
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.14797 (QuantReg: 4.62576) QuantErr: 4.62576 batch_time=0.42466
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.06906 (QuantReg: 4.56332) QuantErr: 4.56332 batch_time=0.46837
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.67380 (QuantReg: 4.60125) QuantErr: 4.60125 batch_time=0.43256
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.62471 (QuantReg: 4.62870) QuantErr: 4.62870 batch_time=0.44332
Train Epoch: 9 codebook_update_time=0.56936
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch9.pth ...
Done in 3.974s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 1.8711464467048644
quant_reg : 4.561088243484497
quant_err : 4.561088243484497
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_miech_test/t2v_metrics/R1: 13.3
MSRVTT_miech_test/t2v_metrics/R5: 40.4
MSRVTT_miech_test/t2v_metrics/R10: 55.3
MSRVTT_miech_test/t2v_metrics/R50: 85.8
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 36.6085
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.973197784213635
MSRVTT_miech_test/v2t_metrics/R1: 11.9
MSRVTT_miech_test/v2t_metrics/R5: 40.3
MSRVTT_miech_test/v2t_metrics/R10: 56.5
MSRVTT_miech_test/v2t_metrics/R50: 85.4
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.928
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.035404497245537
mnt_best : 31.120521465479033
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.91175 (QuantReg: 4.49440) QuantErr: 4.49440 batch_time=27.25540
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.75700 (QuantReg: 4.77874) QuantErr: 4.77874 batch_time=0.45842
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.20982 (QuantReg: 4.40747) QuantErr: 4.40747 batch_time=0.42989
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.81878 (QuantReg: 4.57203) QuantErr: 4.57203 batch_time=0.53823
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.51588 (QuantReg: 4.64464) QuantErr: 4.64464 batch_time=0.44114
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.77236 (QuantReg: 4.64932) QuantErr: 4.64932 batch_time=0.42158
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.62250 (QuantReg: 4.65273) QuantErr: 4.65273 batch_time=1.69351
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.70142 (QuantReg: 4.51522) QuantErr: 4.51522 batch_time=0.43904
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.50521 (QuantReg: 4.58401) QuantErr: 4.58401 batch_time=0.90604
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.85786 (QuantReg: 4.78275) QuantErr: 4.78275 batch_time=0.42213
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.09198 (QuantReg: 4.65883) QuantErr: 4.65883 batch_time=0.42822
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.09093 (QuantReg: 4.53993) QuantErr: 4.53993 batch_time=0.44143
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.48579 (QuantReg: 4.62026) QuantErr: 4.62026 batch_time=0.42156
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.16402 (QuantReg: 4.61475) QuantErr: 4.61475 batch_time=3.31686
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.82569 (QuantReg: 4.66055) QuantErr: 4.66055 batch_time=0.42581
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.93535 (QuantReg: 4.59005) QuantErr: 4.59005 batch_time=0.76148
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.01745 (QuantReg: 4.46858) QuantErr: 4.46858 batch_time=0.42380
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.55033 (QuantReg: 4.72789) QuantErr: 4.72789 batch_time=0.42457
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.06664 (QuantReg: 4.66335) QuantErr: 4.66335 batch_time=0.43074
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.82499 (QuantReg: 4.73640) QuantErr: 4.73640 batch_time=1.35723
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.85041 (QuantReg: 4.71007) QuantErr: 4.71007 batch_time=0.42079
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.67606 (QuantReg: 4.50952) QuantErr: 4.50952 batch_time=0.44403
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.84863 (QuantReg: 4.49092) QuantErr: 4.49092 batch_time=0.45152
Train Epoch: 10 codebook_update_time=0.62529
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch10.pth ...
Done in 5.166s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch10.pth ...
Done in 9.029s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 1.78173499250412
quant_reg : 4.584010875701904
quant_err : 4.584010875701904
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_miech_test/t2v_metrics/R1: 14.1
MSRVTT_miech_test/t2v_metrics/R5: 42.1
MSRVTT_miech_test/t2v_metrics/R10: 56.5
MSRVTT_miech_test/t2v_metrics/R50: 84.7
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 36.3965
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.249022264691206
MSRVTT_miech_test/v2t_metrics/R1: 13.0
MSRVTT_miech_test/v2t_metrics/R5: 41.9
MSRVTT_miech_test/v2t_metrics/R10: 57.6
MSRVTT_miech_test/v2t_metrics/R50: 86.2
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 33.9385
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.53987391896321
mnt_best : 32.249022264691206
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.86889 (QuantReg: 4.56156) QuantErr: 4.56156 batch_time=27.03686
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.55697 (QuantReg: 4.70739) QuantErr: 4.70739 batch_time=0.43098
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.97162 (QuantReg: 4.65287) QuantErr: 4.65287 batch_time=0.43302
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.10408 (QuantReg: 4.40115) QuantErr: 4.40115 batch_time=0.48747
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.68919 (QuantReg: 4.72455) QuantErr: 4.72455 batch_time=0.42239
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.77292 (QuantReg: 4.47179) QuantErr: 4.47179 batch_time=0.42867
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.74275 (QuantReg: 4.60477) QuantErr: 4.60477 batch_time=0.54656
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.88577 (QuantReg: 4.58212) QuantErr: 4.58212 batch_time=0.41736
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.79604 (QuantReg: 4.66534) QuantErr: 4.66534 batch_time=0.41907
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.60267 (QuantReg: 4.49667) QuantErr: 4.49667 batch_time=0.43983
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.00579 (QuantReg: 4.66225) QuantErr: 4.66225 batch_time=0.43104
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.61963 (QuantReg: 4.68833) QuantErr: 4.68833 batch_time=0.43365
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.02384 (QuantReg: 4.63146) QuantErr: 4.63146 batch_time=1.20877
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.90092 (QuantReg: 4.60434) QuantErr: 4.60434 batch_time=0.65370
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.79306 (QuantReg: 4.47789) QuantErr: 4.47789 batch_time=0.42322
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.73362 (QuantReg: 4.53378) QuantErr: 4.53378 batch_time=0.42699
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.66609 (QuantReg: 4.62031) QuantErr: 4.62031 batch_time=0.42607
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.50178 (QuantReg: 4.53464) QuantErr: 4.53464 batch_time=0.42068
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.87793 (QuantReg: 4.51567) QuantErr: 4.51567 batch_time=0.41960
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.35724 (QuantReg: 4.72176) QuantErr: 4.72176 batch_time=3.74172
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.67698 (QuantReg: 4.55459) QuantErr: 4.55459 batch_time=0.42145
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.89317 (QuantReg: 4.67094) QuantErr: 4.67094 batch_time=0.42713
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.71909 (QuantReg: 4.64037) QuantErr: 4.64037 batch_time=0.44221
Train Epoch: 11 codebook_update_time=0.50730
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch11.pth ...
Done in 3.923s
removing stale ckpt [epoch 10] [took 0.01s]
epoch : 11
loss : 1.729197380065918
quant_reg : 4.624332691192627
quant_err : 4.624332691192627
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_miech_test/t2v_metrics/R1: 13.1
MSRVTT_miech_test/t2v_metrics/R5: 39.9
MSRVTT_miech_test/t2v_metrics/R10: 56.5
MSRVTT_miech_test/t2v_metrics/R50: 85.4
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 38.138
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.909896181966516
MSRVTT_miech_test/v2t_metrics/R1: 13.6
MSRVTT_miech_test/v2t_metrics/R5: 42.5
MSRVTT_miech_test/v2t_metrics/R10: 57.6
MSRVTT_miech_test/v2t_metrics/R50: 84.8
MSRVTT_miech_test/v2t_metrics/MedR: 7.5
MSRVTT_miech_test/v2t_metrics/MeanR: 34.3135
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.16992936142123
mnt_best : 32.249022264691206
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.78721 (QuantReg: 4.50477) QuantErr: 4.50477 batch_time=32.32668
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.49101 (QuantReg: 4.59774) QuantErr: 4.59774 batch_time=0.41701
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.66384 (QuantReg: 4.59844) QuantErr: 4.59844 batch_time=0.44627
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.66154 (QuantReg: 4.63979) QuantErr: 4.63979 batch_time=0.42174
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.46534 (QuantReg: 4.59746) QuantErr: 4.59746 batch_time=0.45122
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.78209 (QuantReg: 4.71221) QuantErr: 4.71221 batch_time=0.46889
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.86352 (QuantReg: 4.51759) QuantErr: 4.51759 batch_time=0.43276
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.44646 (QuantReg: 4.69915) QuantErr: 4.69915 batch_time=0.42250
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.58349 (QuantReg: 4.38852) QuantErr: 4.38852 batch_time=0.46155
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.49683 (QuantReg: 4.71448) QuantErr: 4.71448 batch_time=0.59592
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.59053 (QuantReg: 4.71810) QuantErr: 4.71810 batch_time=0.42884
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.78178 (QuantReg: 4.60657) QuantErr: 4.60657 batch_time=0.46112
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.58175 (QuantReg: 4.49285) QuantErr: 4.49285 batch_time=0.42427
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.60447 (QuantReg: 4.67511) QuantErr: 4.67511 batch_time=0.45447
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.62911 (QuantReg: 4.67493) QuantErr: 4.67493 batch_time=0.42897
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.65950 (QuantReg: 4.49303) QuantErr: 4.49303 batch_time=0.45885
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.87024 (QuantReg: 4.68681) QuantErr: 4.68681 batch_time=0.43623
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.85553 (QuantReg: 4.68080) QuantErr: 4.68080 batch_time=0.44247
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.52523 (QuantReg: 4.79026) QuantErr: 4.79026 batch_time=0.43647
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.78314 (QuantReg: 4.69786) QuantErr: 4.69786 batch_time=0.42103
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.76620 (QuantReg: 4.48684) QuantErr: 4.48684 batch_time=0.42040
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.79770 (QuantReg: 4.69027) QuantErr: 4.69027 batch_time=0.42140
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.70157 (QuantReg: 4.70984) QuantErr: 4.70984 batch_time=0.42389
Train Epoch: 12 codebook_update_time=1.17028
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch12.pth ...
Done in 4.526s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch12.pth ...
Done in 9.513s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 1.6549262657165527
quant_reg : 4.642446262359619
quant_err : 4.642446262359619
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_miech_test/t2v_metrics/R1: 14.0
MSRVTT_miech_test/t2v_metrics/R5: 42.0
MSRVTT_miech_test/t2v_metrics/R10: 57.1
MSRVTT_miech_test/t2v_metrics/R50: 85.1
MSRVTT_miech_test/t2v_metrics/MedR: 7.75
MSRVTT_miech_test/t2v_metrics/MeanR: 37.992
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.260503759634204
MSRVTT_miech_test/v2t_metrics/R1: 12.6
MSRVTT_miech_test/v2t_metrics/R5: 40.8
MSRVTT_miech_test/v2t_metrics/R10: 58.0
MSRVTT_miech_test/v2t_metrics/R50: 86.6
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.0565
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.008890963481992
mnt_best : 32.260503759634204
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.56706 (QuantReg: 4.64412) QuantErr: 4.64412 batch_time=33.71169
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.63042 (QuantReg: 4.61508) QuantErr: 4.61508 batch_time=0.43001
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.58602 (QuantReg: 4.74708) QuantErr: 4.74708 batch_time=0.42873
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 2.00740 (QuantReg: 4.70730) QuantErr: 4.70730 batch_time=0.67974
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.71836 (QuantReg: 4.73387) QuantErr: 4.73387 batch_time=0.43639
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.86431 (QuantReg: 4.58587) QuantErr: 4.58587 batch_time=0.42893
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.48779 (QuantReg: 4.68602) QuantErr: 4.68602 batch_time=0.41759
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.60786 (QuantReg: 4.74088) QuantErr: 4.74088 batch_time=0.43226
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.29431 (QuantReg: 4.66039) QuantErr: 4.66039 batch_time=0.42931
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.73346 (QuantReg: 4.50804) QuantErr: 4.50804 batch_time=0.44322
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.70820 (QuantReg: 4.63203) QuantErr: 4.63203 batch_time=0.43147
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.47483 (QuantReg: 4.81290) QuantErr: 4.81290 batch_time=0.43299
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.92126 (QuantReg: 4.48518) QuantErr: 4.48518 batch_time=0.42522
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.93073 (QuantReg: 4.71093) QuantErr: 4.71093 batch_time=0.46481
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.65070 (QuantReg: 4.79797) QuantErr: 4.79797 batch_time=0.44068
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.69706 (QuantReg: 4.57641) QuantErr: 4.57641 batch_time=0.74747
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.52139 (QuantReg: 4.88016) QuantErr: 4.88016 batch_time=0.42981
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.66861 (QuantReg: 4.66329) QuantErr: 4.66329 batch_time=0.42560
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.24526 (QuantReg: 4.68977) QuantErr: 4.68977 batch_time=0.42685
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.45839 (QuantReg: 4.90771) QuantErr: 4.90771 batch_time=0.49865
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.28710 (QuantReg: 4.69643) QuantErr: 4.69643 batch_time=0.45352
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.33914 (QuantReg: 4.58099) QuantErr: 4.58099 batch_time=1.60090
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.05825 (QuantReg: 4.66147) QuantErr: 4.66147 batch_time=0.41800
Train Epoch: 13 codebook_update_time=0.69400
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch13.pth ...
Done in 5.955s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch13.pth ...
Done in 11.291s
removing stale ckpt [epoch 12] [took 0.06s]
epoch : 13
loss : 1.599399610042572
quant_reg : 4.685497716903686
quant_err : 4.685497716903686
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_miech_test/t2v_metrics/R1: 14.7
MSRVTT_miech_test/t2v_metrics/R5: 42.2
MSRVTT_miech_test/t2v_metrics/R10: 57.2
MSRVTT_miech_test/t2v_metrics/R50: 85.0
MSRVTT_miech_test/t2v_metrics/MedR: 7.5
MSRVTT_miech_test/t2v_metrics/MeanR: 37.817
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.86058354823226
MSRVTT_miech_test/v2t_metrics/R1: 13.4
MSRVTT_miech_test/v2t_metrics/R5: 42.5
MSRVTT_miech_test/v2t_metrics/R10: 59.9
MSRVTT_miech_test/v2t_metrics/R50: 86.0
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 32.823
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.43198398810617
mnt_best : 32.86058354823226
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.86541 (QuantReg: 4.56817) QuantErr: 4.56817 batch_time=27.05864
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.58342 (QuantReg: 4.66088) QuantErr: 4.66088 batch_time=0.44437
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.37762 (QuantReg: 4.54992) QuantErr: 4.54992 batch_time=0.47623
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.54599 (QuantReg: 4.83902) QuantErr: 4.83902 batch_time=0.53999
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.68068 (QuantReg: 4.64622) QuantErr: 4.64622 batch_time=0.43933
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.40363 (QuantReg: 4.79236) QuantErr: 4.79236 batch_time=0.42289
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.59294 (QuantReg: 4.65517) QuantErr: 4.65517 batch_time=0.43476
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.99603 (QuantReg: 4.57786) QuantErr: 4.57786 batch_time=0.46336
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.68318 (QuantReg: 4.80365) QuantErr: 4.80365 batch_time=0.42246
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.55765 (QuantReg: 4.73230) QuantErr: 4.73230 batch_time=0.43196
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.60433 (QuantReg: 4.71430) QuantErr: 4.71430 batch_time=0.43048
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.46907 (QuantReg: 4.64518) QuantErr: 4.64518 batch_time=0.43684
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.61612 (QuantReg: 4.66809) QuantErr: 4.66809 batch_time=0.44930
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.52877 (QuantReg: 4.66154) QuantErr: 4.66154 batch_time=0.42076
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.43978 (QuantReg: 4.64861) QuantErr: 4.64861 batch_time=0.42783
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.31854 (QuantReg: 4.72603) QuantErr: 4.72603 batch_time=0.42584
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.69531 (QuantReg: 4.61581) QuantErr: 4.61581 batch_time=0.43170
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.54908 (QuantReg: 4.83296) QuantErr: 4.83296 batch_time=0.42846
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.16462 (QuantReg: 4.77915) QuantErr: 4.77915 batch_time=0.53943
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.87058 (QuantReg: 4.84757) QuantErr: 4.84757 batch_time=0.42231
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.28629 (QuantReg: 4.81677) QuantErr: 4.81677 batch_time=0.42752
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.49225 (QuantReg: 4.61478) QuantErr: 4.61478 batch_time=0.43549
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.49099 (QuantReg: 4.64184) QuantErr: 4.64184 batch_time=0.44821
Train Epoch: 14 codebook_update_time=0.63754
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch14.pth ...
Done in 5.504s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 1.54595095205307
quant_reg : 4.721134189605713
quant_err : 4.721134189605713
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_miech_test/t2v_metrics/R1: 14.3
MSRVTT_miech_test/t2v_metrics/R5: 43.2
MSRVTT_miech_test/t2v_metrics/R10: 56.5
MSRVTT_miech_test/t2v_metrics/R50: 84.7
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 34.694
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.680554047178006
MSRVTT_miech_test/v2t_metrics/R1: 13.9
MSRVTT_miech_test/v2t_metrics/R5: 42.4
MSRVTT_miech_test/v2t_metrics/R10: 59.7
MSRVTT_miech_test/v2t_metrics/R50: 86.1
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 33.1175
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.768130357064784
mnt_best : 32.86058354823226
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.60624 (QuantReg: 4.75539) QuantErr: 4.75539 batch_time=28.81849
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.69425 (QuantReg: 4.60692) QuantErr: 4.60692 batch_time=0.42153
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.69573 (QuantReg: 4.67151) QuantErr: 4.67151 batch_time=0.44034
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.98603 (QuantReg: 4.62405) QuantErr: 4.62405 batch_time=0.53418
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.69342 (QuantReg: 4.71948) QuantErr: 4.71948 batch_time=0.42677
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.16748 (QuantReg: 4.72802) QuantErr: 4.72802 batch_time=0.41801
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.64393 (QuantReg: 4.61596) QuantErr: 4.61596 batch_time=0.43049
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.20344 (QuantReg: 4.76711) QuantErr: 4.76711 batch_time=0.42916
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.84334 (QuantReg: 4.77073) QuantErr: 4.77073 batch_time=0.41875
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.44772 (QuantReg: 4.68371) QuantErr: 4.68371 batch_time=0.67583
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.32431 (QuantReg: 4.81036) QuantErr: 4.81036 batch_time=0.42580
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.50818 (QuantReg: 4.60828) QuantErr: 4.60828 batch_time=0.43674
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.85340 (QuantReg: 4.84613) QuantErr: 4.84613 batch_time=1.08629
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.51449 (QuantReg: 4.71902) QuantErr: 4.71902 batch_time=0.45012
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.53071 (QuantReg: 4.68708) QuantErr: 4.68708 batch_time=0.44602
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.37455 (QuantReg: 4.72041) QuantErr: 4.72041 batch_time=0.43627
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.33849 (QuantReg: 4.74790) QuantErr: 4.74790 batch_time=0.41261
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.72802 (QuantReg: 4.81382) QuantErr: 4.81382 batch_time=0.41528
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.17414 (QuantReg: 4.70207) QuantErr: 4.70207 batch_time=0.41536
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.27012 (QuantReg: 4.85445) QuantErr: 4.85445 batch_time=0.56527
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.42277 (QuantReg: 4.92049) QuantErr: 4.92049 batch_time=0.45452
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.64192 (QuantReg: 4.88221) QuantErr: 4.88221 batch_time=0.44730
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.60694 (QuantReg: 4.93864) QuantErr: 4.93864 batch_time=0.41814
Train Epoch: 15 codebook_update_time=0.54008
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch15.pth ...
Done in 5.064s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 1.5208620681762695
quant_reg : 4.7406089382171634
quant_err : 4.7406089382171634
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_miech_test/t2v_metrics/R1: 14.4
MSRVTT_miech_test/t2v_metrics/R5: 40.8
MSRVTT_miech_test/t2v_metrics/R10: 56.8
MSRVTT_miech_test/t2v_metrics/R50: 84.5
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.2475
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.195140915126906
MSRVTT_miech_test/v2t_metrics/R1: 12.8
MSRVTT_miech_test/v2t_metrics/R5: 43.2
MSRVTT_miech_test/v2t_metrics/R10: 58.4
MSRVTT_miech_test/v2t_metrics/R50: 86.9
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 33.0395
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.844579696963976
mnt_best : 32.86058354823226
not_improved_count: 2
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.51064 (QuantReg: 4.53435) QuantErr: 4.53435 batch_time=26.60641
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.69460 (QuantReg: 4.57749) QuantErr: 4.57749 batch_time=0.46614
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.15424 (QuantReg: 4.65817) QuantErr: 4.65817 batch_time=0.42249
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.79663 (QuantReg: 4.61368) QuantErr: 4.61368 batch_time=0.43279
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.50139 (QuantReg: 4.71042) QuantErr: 4.71042 batch_time=0.45061
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.73524 (QuantReg: 4.80849) QuantErr: 4.80849 batch_time=0.45574
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.73715 (QuantReg: 4.76944) QuantErr: 4.76944 batch_time=0.99566
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.51725 (QuantReg: 4.98551) QuantErr: 4.98551 batch_time=0.44131
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.52176 (QuantReg: 4.58521) QuantErr: 4.58521 batch_time=0.41658
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.19478 (QuantReg: 4.79493) QuantErr: 4.79493 batch_time=0.42301
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.34225 (QuantReg: 4.86838) QuantErr: 4.86838 batch_time=0.42266
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.40566 (QuantReg: 4.73512) QuantErr: 4.73512 batch_time=0.43533
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.55916 (QuantReg: 4.70301) QuantErr: 4.70301 batch_time=0.43196
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.57441 (QuantReg: 4.77242) QuantErr: 4.77242 batch_time=0.43177
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.56879 (QuantReg: 4.80652) QuantErr: 4.80652 batch_time=0.42525
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.51097 (QuantReg: 4.77268) QuantErr: 4.77268 batch_time=0.42793
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.52074 (QuantReg: 4.92029) QuantErr: 4.92029 batch_time=0.42773
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.59198 (QuantReg: 4.61460) QuantErr: 4.61460 batch_time=0.41874
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.44905 (QuantReg: 4.73245) QuantErr: 4.73245 batch_time=0.44148
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.40140 (QuantReg: 4.78404) QuantErr: 4.78404 batch_time=0.42973
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.49037 (QuantReg: 4.77598) QuantErr: 4.77598 batch_time=1.15984
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.32120 (QuantReg: 4.84524) QuantErr: 4.84524 batch_time=0.45181
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.62541 (QuantReg: 4.72883) QuantErr: 4.72883 batch_time=0.45930
Train Epoch: 16 codebook_update_time=0.97737
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch16.pth ...
Done in 6.521s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch16.pth ...
Done in 11.729s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 1.4626229755878448
quant_reg : 4.775290916442871
quant_err : 4.775290916442871
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_miech_test/t2v_metrics/R1: 14.8
MSRVTT_miech_test/t2v_metrics/R5: 43.3
MSRVTT_miech_test/t2v_metrics/R10: 58.9
MSRVTT_miech_test/t2v_metrics/R50: 86.4
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 34.3445
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.54452414956441
MSRVTT_miech_test/v2t_metrics/R1: 13.9
MSRVTT_miech_test/v2t_metrics/R5: 43.2
MSRVTT_miech_test/v2t_metrics/R10: 59.9
MSRVTT_miech_test/v2t_metrics/R50: 86.6
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 32.316500000000005
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.00971614727104
mnt_best : 33.54452414956441
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.33715 (QuantReg: 4.63231) QuantErr: 4.63231 batch_time=30.74836
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.30556 (QuantReg: 4.82410) QuantErr: 4.82410 batch_time=0.42290
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.48308 (QuantReg: 4.77046) QuantErr: 4.77046 batch_time=0.42619
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.60497 (QuantReg: 4.68611) QuantErr: 4.68611 batch_time=0.43007
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.55983 (QuantReg: 4.83832) QuantErr: 4.83832 batch_time=0.42452
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.46687 (QuantReg: 4.85750) QuantErr: 4.85750 batch_time=0.44146
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.81442 (QuantReg: 4.66180) QuantErr: 4.66180 batch_time=0.43652
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.41721 (QuantReg: 4.86841) QuantErr: 4.86841 batch_time=0.42389
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.46615 (QuantReg: 4.82679) QuantErr: 4.82679 batch_time=0.43112
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.15964 (QuantReg: 4.93045) QuantErr: 4.93045 batch_time=0.42166
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.22121 (QuantReg: 4.93018) QuantErr: 4.93018 batch_time=1.25509
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.23767 (QuantReg: 4.78773) QuantErr: 4.78773 batch_time=0.42612
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.70926 (QuantReg: 4.78999) QuantErr: 4.78999 batch_time=0.44881
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.50452 (QuantReg: 4.85040) QuantErr: 4.85040 batch_time=0.41781
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.68336 (QuantReg: 4.67432) QuantErr: 4.67432 batch_time=0.41858
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.56258 (QuantReg: 4.71704) QuantErr: 4.71704 batch_time=0.43088
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.18047 (QuantReg: 4.71123) QuantErr: 4.71123 batch_time=0.42517
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.31474 (QuantReg: 4.72066) QuantErr: 4.72066 batch_time=0.42248
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.71207 (QuantReg: 4.88131) QuantErr: 4.88131 batch_time=1.10177
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.54555 (QuantReg: 4.84029) QuantErr: 4.84029 batch_time=0.43095
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.14721 (QuantReg: 4.82863) QuantErr: 4.82863 batch_time=0.47132
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.40780 (QuantReg: 4.85063) QuantErr: 4.85063 batch_time=0.43221
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.48215 (QuantReg: 4.80201) QuantErr: 4.80201 batch_time=0.42224
Train Epoch: 17 codebook_update_time=0.54051
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch17.pth ...
Done in 4.929s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 1.4423200497627258
quant_reg : 4.793364002227783
quant_err : 4.793364002227783
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_miech_test/t2v_metrics/R1: 14.9
MSRVTT_miech_test/t2v_metrics/R5: 42.7
MSRVTT_miech_test/t2v_metrics/R10: 57.8
MSRVTT_miech_test/t2v_metrics/R50: 85.0
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 36.7545
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.25426300703426
MSRVTT_miech_test/v2t_metrics/R1: 12.9
MSRVTT_miech_test/v2t_metrics/R5: 44.2
MSRVTT_miech_test/v2t_metrics/R10: 60.5
MSRVTT_miech_test/v2t_metrics/R50: 85.9
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.302
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.55285757626559
mnt_best : 33.54452414956441
not_improved_count: 1
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.48212 (QuantReg: 4.68163) QuantErr: 4.68163 batch_time=27.84729
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.55281 (QuantReg: 4.72909) QuantErr: 4.72909 batch_time=0.42616
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.58886 (QuantReg: 4.73870) QuantErr: 4.73870 batch_time=0.43171
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.92045 (QuantReg: 4.80787) QuantErr: 4.80787 batch_time=0.42620
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.38090 (QuantReg: 4.88481) QuantErr: 4.88481 batch_time=0.44032
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.37002 (QuantReg: 4.80846) QuantErr: 4.80846 batch_time=0.43092
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.16954 (QuantReg: 4.83490) QuantErr: 4.83490 batch_time=0.43533
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.46861 (QuantReg: 4.90380) QuantErr: 4.90380 batch_time=0.42979
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.50540 (QuantReg: 4.95870) QuantErr: 4.95870 batch_time=0.90121
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.56775 (QuantReg: 4.80416) QuantErr: 4.80416 batch_time=0.54250
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.16723 (QuantReg: 4.88597) QuantErr: 4.88597 batch_time=0.42404
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.31024 (QuantReg: 5.04344) QuantErr: 5.04344 batch_time=0.41737
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.61956 (QuantReg: 4.83489) QuantErr: 4.83489 batch_time=0.43380
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.32511 (QuantReg: 4.77983) QuantErr: 4.77983 batch_time=0.55426
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.30884 (QuantReg: 4.73900) QuantErr: 4.73900 batch_time=0.42917
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.47172 (QuantReg: 4.84888) QuantErr: 4.84888 batch_time=0.42363
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.76450 (QuantReg: 4.97016) QuantErr: 4.97016 batch_time=0.44893
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.37519 (QuantReg: 4.82704) QuantErr: 4.82704 batch_time=0.42400
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.55699 (QuantReg: 4.82915) QuantErr: 4.82915 batch_time=0.45959
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.45369 (QuantReg: 4.88921) QuantErr: 4.88921 batch_time=0.44594
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.42256 (QuantReg: 4.69740) QuantErr: 4.69740 batch_time=0.45623
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.69174 (QuantReg: 4.88665) QuantErr: 4.88665 batch_time=0.45262
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.41955 (QuantReg: 4.80961) QuantErr: 4.80961 batch_time=6.45790
Train Epoch: 18 codebook_update_time=0.51456
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch18.pth ...
Done in 5.505s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch18.pth ...
Done in 10.751s
removing stale ckpt [epoch 17] [took 0.51s]
epoch : 18
loss : 1.4231473026275634
quant_reg : 4.834310945510865
quant_err : 4.834310945510865
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_miech_test/t2v_metrics/R1: 15.0
MSRVTT_miech_test/t2v_metrics/R5: 43.9
MSRVTT_miech_test/t2v_metrics/R10: 59.1
MSRVTT_miech_test/t2v_metrics/R50: 84.5
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 37.663
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.888141622070485
MSRVTT_miech_test/v2t_metrics/R1: 13.9
MSRVTT_miech_test/v2t_metrics/R5: 44.6
MSRVTT_miech_test/v2t_metrics/R10: 60.3
MSRVTT_miech_test/v2t_metrics/R50: 86.0
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.2285
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.43661646950674
mnt_best : 33.888141622070485
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.57296 (QuantReg: 4.72635) QuantErr: 4.72635 batch_time=29.39996
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.22354 (QuantReg: 4.90810) QuantErr: 4.90810 batch_time=0.43576
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.41017 (QuantReg: 4.59104) QuantErr: 4.59104 batch_time=0.42725
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.44097 (QuantReg: 4.77336) QuantErr: 4.77336 batch_time=0.42875
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.48113 (QuantReg: 4.85722) QuantErr: 4.85722 batch_time=0.41296
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.57216 (QuantReg: 4.87653) QuantErr: 4.87653 batch_time=0.41922
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.52095 (QuantReg: 4.75518) QuantErr: 4.75518 batch_time=0.83995
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.20481 (QuantReg: 4.88987) QuantErr: 4.88987 batch_time=0.41778
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.62926 (QuantReg: 4.83828) QuantErr: 4.83828 batch_time=0.44749
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.17924 (QuantReg: 4.65422) QuantErr: 4.65422 batch_time=0.43509
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.27645 (QuantReg: 4.87978) QuantErr: 4.87978 batch_time=0.43939
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.41793 (QuantReg: 4.78898) QuantErr: 4.78898 batch_time=0.44331
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.35893 (QuantReg: 4.83866) QuantErr: 4.83866 batch_time=0.42127
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.34344 (QuantReg: 4.82206) QuantErr: 4.82206 batch_time=0.41990
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.69256 (QuantReg: 4.86047) QuantErr: 4.86047 batch_time=0.44169
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.13565 (QuantReg: 4.96740) QuantErr: 4.96740 batch_time=0.45320
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.29327 (QuantReg: 4.82358) QuantErr: 4.82358 batch_time=0.42804
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.29120 (QuantReg: 4.92711) QuantErr: 4.92711 batch_time=0.43297
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.58887 (QuantReg: 4.81864) QuantErr: 4.81864 batch_time=0.42933
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.46326 (QuantReg: 4.74813) QuantErr: 4.74813 batch_time=0.43081
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.39548 (QuantReg: 4.87908) QuantErr: 4.87908 batch_time=0.45206
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.27748 (QuantReg: 4.79667) QuantErr: 4.79667 batch_time=0.55665
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.55999 (QuantReg: 4.65336) QuantErr: 4.65336 batch_time=0.46416
Train Epoch: 19 codebook_update_time=0.56778
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_M8/checkpoint-epoch19.pth ...
Done in 5.176s
removing stale ckpt [epoch 18] [took 0.05s]
epoch : 19
loss : 1.391771224975586
quant_reg : 4.85915325164795
quant_err : 4.85915325164795
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_miech_test/t2v_metrics/R1: 15.3
MSRVTT_miech_test/t2v_metrics/R5: 43.0
MSRVTT_miech_test/t2v_metrics/R10: 59.0
MSRVTT_miech_test/t2v_metrics/R50: 85.2
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 34.783