-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kB_t0.03.txt
2597 lines (2597 loc) · 192 KB
/
HCQ_MSRVTT_1kB_t0.03.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03
Preparing the dataloaders ...
Loading dataset MSRVTT_miech_trainval in ram ...
Finish loading dataset MSRVTT_miech_trainval in ram, taking 447.1428039073944 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 70.19137334823608 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 43.24365544319153 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch0.pth ...
Done in 8.513s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch0.pth ...
Done in 10.415s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_miech_test/t2v_metrics/R1: 0.1
MSRVTT_miech_test/t2v_metrics/R5: 0.6
MSRVTT_miech_test/t2v_metrics/R10: 1.0
MSRVTT_miech_test/t2v_metrics/R50: 5.0
MSRVTT_miech_test/t2v_metrics/MedR: 503.0
MSRVTT_miech_test/t2v_metrics/MeanR: 505.193
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3914867641168864
MSRVTT_miech_test/v2t_metrics/R1: 0.1
MSRVTT_miech_test/v2t_metrics/R5: 0.4
MSRVTT_miech_test/v2t_metrics/R10: 1.0
MSRVTT_miech_test/v2t_metrics/R50: 5.4
MSRVTT_miech_test/v2t_metrics/MedR: 511.5
MSRVTT_miech_test/v2t_metrics/MeanR: 499.894
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.3419951893353394
mnt_best : 0.3914867641168864
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 10.01629 (QuantReg: 22.48155) QuantErr: 22.48155 batch_time=27.06849
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.03574 (QuantReg: 22.45665) QuantErr: 22.45665 batch_time=0.53175
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.58162 (QuantReg: 22.56716) QuantErr: 22.56716 batch_time=0.50858
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.52799 (QuantReg: 22.54119) QuantErr: 22.54119 batch_time=0.52307
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.57883 (QuantReg: 22.56215) QuantErr: 22.56215 batch_time=0.54877
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.16593 (QuantReg: 22.55748) QuantErr: 22.55748 batch_time=1.25020
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 5.96807 (QuantReg: 22.50377) QuantErr: 22.50377 batch_time=3.16744
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.58561 (QuantReg: 22.58183) QuantErr: 22.58183 batch_time=0.55017
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.14263 (QuantReg: 22.54270) QuantErr: 22.54270 batch_time=0.54530
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.13350 (QuantReg: 22.54306) QuantErr: 22.54306 batch_time=0.58380
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.10087 (QuantReg: 22.56702) QuantErr: 22.56702 batch_time=0.68326
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.20384 (QuantReg: 22.56317) QuantErr: 22.56317 batch_time=0.53434
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.53969 (QuantReg: 22.60503) QuantErr: 22.60503 batch_time=0.58485
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.76511 (QuantReg: 22.63193) QuantErr: 22.63193 batch_time=1.66588
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.60330 (QuantReg: 22.57634) QuantErr: 22.57634 batch_time=0.58515
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.23298 (QuantReg: 22.58491) QuantErr: 22.58491 batch_time=0.59665
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.12115 (QuantReg: 22.57723) QuantErr: 22.57723 batch_time=0.54219
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.55521 (QuantReg: 22.54339) QuantErr: 22.54339 batch_time=0.58526
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.43391 (QuantReg: 22.59194) QuantErr: 22.59194 batch_time=0.77017
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.61137 (QuantReg: 22.61544) QuantErr: 22.61544 batch_time=0.56780
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.40297 (QuantReg: 22.57730) QuantErr: 22.57730 batch_time=0.55421
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.70002 (QuantReg: 22.60411) QuantErr: 22.60411 batch_time=0.55909
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.07472 (QuantReg: 22.58558) QuantErr: 22.58558 batch_time=0.57496
Train Epoch: 1 codebook_update_time=1.97468
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch1.pth ...
Done in 4.527s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch1.pth ...
Done in 9.365s
epoch : 1
loss : 5.425920726776123
quant_reg : 22.569019508361816
quant_err : 22.569019508361816
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_miech_test/t2v_metrics/R1: 9.7
MSRVTT_miech_test/t2v_metrics/R5: 31.2
MSRVTT_miech_test/t2v_metrics/R10: 44.8
MSRVTT_miech_test/t2v_metrics/R50: 78.6
MSRVTT_miech_test/t2v_metrics/MedR: 14.0
MSRVTT_miech_test/t2v_metrics/MeanR: 45.448
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.845226246368956
MSRVTT_miech_test/v2t_metrics/R1: 9.4
MSRVTT_miech_test/v2t_metrics/R5: 30.5
MSRVTT_miech_test/v2t_metrics/R10: 43.5
MSRVTT_miech_test/v2t_metrics/R50: 76.8
MSRVTT_miech_test/v2t_metrics/MedR: 14.0
MSRVTT_miech_test/v2t_metrics/MeanR: 48.381
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.190261717490923
mnt_best : 23.845226246368956
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 3.85600 (QuantReg: 8.91778) QuantErr: 8.91778 batch_time=28.77807
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.80645 (QuantReg: 9.26033) QuantErr: 9.26033 batch_time=4.42898
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 3.98032 (QuantReg: 9.45576) QuantErr: 9.45576 batch_time=0.54055
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.86483 (QuantReg: 9.63824) QuantErr: 9.63824 batch_time=0.54345
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.62015 (QuantReg: 9.48011) QuantErr: 9.48011 batch_time=0.57919
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.44891 (QuantReg: 10.00760) QuantErr: 10.00760 batch_time=0.53181
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.81193 (QuantReg: 10.28760) QuantErr: 10.28760 batch_time=1.32208
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.51071 (QuantReg: 10.14121) QuantErr: 10.14121 batch_time=0.64278
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 3.49145 (QuantReg: 10.24588) QuantErr: 10.24588 batch_time=0.52888
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.81223 (QuantReg: 10.69806) QuantErr: 10.69806 batch_time=0.53606
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.56833 (QuantReg: 10.57654) QuantErr: 10.57654 batch_time=0.60289
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.24316 (QuantReg: 11.02117) QuantErr: 11.02117 batch_time=0.54610
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.47001 (QuantReg: 11.24026) QuantErr: 11.24026 batch_time=0.52466
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.76074 (QuantReg: 11.31875) QuantErr: 11.31875 batch_time=0.58491
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.26336 (QuantReg: 11.20105) QuantErr: 11.20105 batch_time=0.52226
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.91882 (QuantReg: 11.25947) QuantErr: 11.25947 batch_time=0.55066
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.57047 (QuantReg: 11.19430) QuantErr: 11.19430 batch_time=0.54155
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 2.95017 (QuantReg: 11.88293) QuantErr: 11.88293 batch_time=0.55820
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.97127 (QuantReg: 11.43949) QuantErr: 11.43949 batch_time=0.57331
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.67645 (QuantReg: 11.91835) QuantErr: 11.91835 batch_time=3.02836
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.46155 (QuantReg: 12.24524) QuantErr: 12.24524 batch_time=0.53687
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.34442 (QuantReg: 12.23865) QuantErr: 12.23865 batch_time=0.56713
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 2.90208 (QuantReg: 12.40091) QuantErr: 12.40091 batch_time=0.54412
Train Epoch: 2 codebook_update_time=1.90170
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch2.pth ...
Done in 4.712s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch2.pth ...
Done in 9.485s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.6206131267547605
quant_reg : 10.86943547821045
quant_err : 10.86943547821045
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_miech_test/t2v_metrics/R1: 12.9
MSRVTT_miech_test/t2v_metrics/R5: 35.4
MSRVTT_miech_test/t2v_metrics/R10: 50.8
MSRVTT_miech_test/t2v_metrics/R50: 82.1
MSRVTT_miech_test/t2v_metrics/MedR: 10.0
MSRVTT_miech_test/t2v_metrics/MeanR: 40.113
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.520177770025366
MSRVTT_miech_test/v2t_metrics/R1: 12.9
MSRVTT_miech_test/v2t_metrics/R5: 36.2
MSRVTT_miech_test/v2t_metrics/R10: 50.2
MSRVTT_miech_test/v2t_metrics/R50: 83.3
MSRVTT_miech_test/v2t_metrics/MedR: 10.25
MSRVTT_miech_test/v2t_metrics/MeanR: 39.398
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.619848643645863
mnt_best : 28.520177770025366
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.68440 (QuantReg: 10.07666) QuantErr: 10.07666 batch_time=42.12026
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.07461 (QuantReg: 9.78725) QuantErr: 9.78725 batch_time=0.54213
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.21105 (QuantReg: 10.16128) QuantErr: 10.16128 batch_time=0.56944
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.87474 (QuantReg: 9.93586) QuantErr: 9.93586 batch_time=1.21152
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.00010 (QuantReg: 10.61899) QuantErr: 10.61899 batch_time=0.57264
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.21620 (QuantReg: 10.53321) QuantErr: 10.53321 batch_time=0.53676
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.92088 (QuantReg: 10.32094) QuantErr: 10.32094 batch_time=0.54707
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 2.69776 (QuantReg: 10.37938) QuantErr: 10.37938 batch_time=0.51732
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.01492 (QuantReg: 10.51896) QuantErr: 10.51896 batch_time=0.66415
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.88943 (QuantReg: 10.67862) QuantErr: 10.67862 batch_time=0.52002
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.83559 (QuantReg: 10.71831) QuantErr: 10.71831 batch_time=0.56332
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 2.97560 (QuantReg: 10.62367) QuantErr: 10.62367 batch_time=0.53238
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.28391 (QuantReg: 11.06875) QuantErr: 11.06875 batch_time=0.53620
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.18563 (QuantReg: 11.03397) QuantErr: 11.03397 batch_time=0.53878
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 2.91402 (QuantReg: 10.78171) QuantErr: 10.78171 batch_time=0.56649
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.75121 (QuantReg: 11.45294) QuantErr: 11.45294 batch_time=0.62184
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.55700 (QuantReg: 11.24988) QuantErr: 11.24988 batch_time=0.57727
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.05577 (QuantReg: 11.47010) QuantErr: 11.47010 batch_time=0.56779
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.14609 (QuantReg: 11.38097) QuantErr: 11.38097 batch_time=0.57984
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.79708 (QuantReg: 11.33771) QuantErr: 11.33771 batch_time=0.59108
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.29458 (QuantReg: 11.42232) QuantErr: 11.42232 batch_time=0.58046
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.47016 (QuantReg: 11.55323) QuantErr: 11.55323 batch_time=0.57134
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.50148 (QuantReg: 11.79823) QuantErr: 11.79823 batch_time=0.54440
Train Epoch: 3 codebook_update_time=1.78104
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch3.pth ...
Done in 11.304s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch3.pth ...
Done in 15.930s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 3.0537627735137938
quant_reg : 10.850633438110352
quant_err : 10.850633438110352
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_miech_test/t2v_metrics/R1: 13.9
MSRVTT_miech_test/t2v_metrics/R5: 39.0
MSRVTT_miech_test/t2v_metrics/R10: 54.3
MSRVTT_miech_test/t2v_metrics/R50: 85.0
MSRVTT_miech_test/t2v_metrics/MedR: 8.5
MSRVTT_miech_test/t2v_metrics/MeanR: 35.131
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 30.87638249367421
MSRVTT_miech_test/v2t_metrics/R1: 15.1
MSRVTT_miech_test/v2t_metrics/R5: 40.1
MSRVTT_miech_test/v2t_metrics/R10: 54.5
MSRVTT_miech_test/v2t_metrics/R50: 85.4
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 34.4555
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.07543887771663
mnt_best : 30.87638249367421
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 2.62441 (QuantReg: 10.53843) QuantErr: 10.53843 batch_time=30.78522
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.71036 (QuantReg: 10.85877) QuantErr: 10.85877 batch_time=0.56973
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.61153 (QuantReg: 10.72877) QuantErr: 10.72877 batch_time=0.54358
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.99397 (QuantReg: 10.85203) QuantErr: 10.85203 batch_time=0.56982
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.70085 (QuantReg: 10.87995) QuantErr: 10.87995 batch_time=0.57875
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.38381 (QuantReg: 11.11650) QuantErr: 11.11650 batch_time=0.59442
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.04751 (QuantReg: 11.13597) QuantErr: 11.13597 batch_time=2.21960
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.27239 (QuantReg: 10.83708) QuantErr: 10.83708 batch_time=0.56565
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.65928 (QuantReg: 11.17748) QuantErr: 11.17748 batch_time=1.08238
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.61762 (QuantReg: 11.39109) QuantErr: 11.39109 batch_time=0.53222
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.13443 (QuantReg: 11.32573) QuantErr: 11.32573 batch_time=0.56489
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.96671 (QuantReg: 11.12508) QuantErr: 11.12508 batch_time=0.51771
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.30757 (QuantReg: 11.39337) QuantErr: 11.39337 batch_time=0.53323
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.55839 (QuantReg: 11.10812) QuantErr: 11.10812 batch_time=0.57604
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.76108 (QuantReg: 11.77444) QuantErr: 11.77444 batch_time=0.53369
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.76954 (QuantReg: 11.61194) QuantErr: 11.61194 batch_time=0.50732
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.41857 (QuantReg: 11.76661) QuantErr: 11.76661 batch_time=0.58078
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.79448 (QuantReg: 11.41328) QuantErr: 11.41328 batch_time=0.55245
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.72353 (QuantReg: 12.08356) QuantErr: 12.08356 batch_time=0.52049
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.49800 (QuantReg: 11.37979) QuantErr: 11.37979 batch_time=0.56980
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.45954 (QuantReg: 11.95286) QuantErr: 11.95286 batch_time=0.54984
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.41385 (QuantReg: 11.97277) QuantErr: 11.97277 batch_time=0.58233
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.34433 (QuantReg: 12.00060) QuantErr: 12.00060 batch_time=0.68213
Train Epoch: 4 codebook_update_time=1.96084
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch4.pth ...
Done in 4.802s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch4.pth ...
Done in 28.176s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 2.665831320762634
quant_reg : 11.381974342346192
quant_err : 11.381974342346192
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_miech_test/t2v_metrics/R1: 14.7
MSRVTT_miech_test/t2v_metrics/R5: 41.5
MSRVTT_miech_test/t2v_metrics/R10: 55.3
MSRVTT_miech_test/t2v_metrics/R50: 85.9
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.861000000000004
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.31197624817158
MSRVTT_miech_test/v2t_metrics/R1: 16.9
MSRVTT_miech_test/v2t_metrics/R5: 42.5
MSRVTT_miech_test/v2t_metrics/R10: 56.6
MSRVTT_miech_test/v2t_metrics/R50: 86.4
MSRVTT_miech_test/v2t_metrics/MedR: 8.0
MSRVTT_miech_test/v2t_metrics/MeanR: 31.53
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.38460359771384
mnt_best : 32.31197624817158
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.11088 (QuantReg: 10.87063) QuantErr: 10.87063 batch_time=35.32762
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.14546 (QuantReg: 11.11597) QuantErr: 11.11597 batch_time=0.55193
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.27809 (QuantReg: 11.24243) QuantErr: 11.24243 batch_time=0.53960
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.50535 (QuantReg: 11.36381) QuantErr: 11.36381 batch_time=0.54816
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.78006 (QuantReg: 11.46172) QuantErr: 11.46172 batch_time=0.55687
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.01986 (QuantReg: 11.67622) QuantErr: 11.67622 batch_time=0.59477
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.44454 (QuantReg: 11.55895) QuantErr: 11.55895 batch_time=0.51228
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.36836 (QuantReg: 11.61576) QuantErr: 11.61576 batch_time=0.55553
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.22849 (QuantReg: 11.32385) QuantErr: 11.32385 batch_time=0.53351
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.28574 (QuantReg: 11.90907) QuantErr: 11.90907 batch_time=0.56177
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.28432 (QuantReg: 11.81127) QuantErr: 11.81127 batch_time=0.53885
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.68792 (QuantReg: 11.95370) QuantErr: 11.95370 batch_time=0.53838
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.60395 (QuantReg: 11.97501) QuantErr: 11.97501 batch_time=0.53214
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.46198 (QuantReg: 11.86271) QuantErr: 11.86271 batch_time=0.56578
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.32809 (QuantReg: 11.73225) QuantErr: 11.73225 batch_time=0.52668
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 1.97302 (QuantReg: 12.31950) QuantErr: 12.31950 batch_time=0.55655
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.44981 (QuantReg: 12.16006) QuantErr: 12.16006 batch_time=0.51856
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.40940 (QuantReg: 11.92402) QuantErr: 11.92402 batch_time=0.57605
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.31456 (QuantReg: 11.97248) QuantErr: 11.97248 batch_time=0.53615
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.61661 (QuantReg: 11.90801) QuantErr: 11.90801 batch_time=6.55738
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.19842 (QuantReg: 11.83955) QuantErr: 11.83955 batch_time=0.58475
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 1.55659 (QuantReg: 12.33188) QuantErr: 12.33188 batch_time=0.52477
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.01502 (QuantReg: 12.54850) QuantErr: 12.54850 batch_time=0.53384
Train Epoch: 5 codebook_update_time=1.93948
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch5.pth ...
Done in 22.565s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch5.pth ...
Done in 27.600s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 2.4223988003730774
quant_reg : 11.764623645782471
quant_err : 11.764623645782471
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_miech_test/t2v_metrics/R1: 15.0
MSRVTT_miech_test/t2v_metrics/R5: 44.1
MSRVTT_miech_test/t2v_metrics/R10: 58.5
MSRVTT_miech_test/t2v_metrics/R50: 87.1
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.64
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.82428083878629
MSRVTT_miech_test/v2t_metrics/R1: 15.9
MSRVTT_miech_test/v2t_metrics/R5: 43.6
MSRVTT_miech_test/v2t_metrics/R10: 57.8
MSRVTT_miech_test/v2t_metrics/R50: 86.8
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 32.1395
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.21924979024399
mnt_best : 33.82428083878629
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.12193 (QuantReg: 11.63402) QuantErr: 11.63402 batch_time=36.53738
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.00347 (QuantReg: 12.25530) QuantErr: 12.25530 batch_time=0.56118
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.48679 (QuantReg: 11.45045) QuantErr: 11.45045 batch_time=0.52325
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.47732 (QuantReg: 11.78026) QuantErr: 11.78026 batch_time=0.52598
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.18940 (QuantReg: 11.57633) QuantErr: 11.57633 batch_time=0.89879
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.27778 (QuantReg: 11.98918) QuantErr: 11.98918 batch_time=0.57073
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.38891 (QuantReg: 11.90557) QuantErr: 11.90557 batch_time=0.53879
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.40543 (QuantReg: 11.88996) QuantErr: 11.88996 batch_time=0.79752
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.26827 (QuantReg: 11.66278) QuantErr: 11.66278 batch_time=0.59242
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.10447 (QuantReg: 11.93745) QuantErr: 11.93745 batch_time=0.80560
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 1.85312 (QuantReg: 12.32332) QuantErr: 12.32332 batch_time=0.53756
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.35435 (QuantReg: 12.40609) QuantErr: 12.40609 batch_time=0.61044
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.25947 (QuantReg: 11.87222) QuantErr: 11.87222 batch_time=0.54241
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.47066 (QuantReg: 12.44074) QuantErr: 12.44074 batch_time=0.51975
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.37447 (QuantReg: 12.23373) QuantErr: 12.23373 batch_time=0.54664
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.37492 (QuantReg: 12.27790) QuantErr: 12.27790 batch_time=0.50184
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.39574 (QuantReg: 12.28693) QuantErr: 12.28693 batch_time=0.49565
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.49393 (QuantReg: 12.34203) QuantErr: 12.34203 batch_time=0.53435
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 1.80880 (QuantReg: 12.49205) QuantErr: 12.49205 batch_time=0.55358
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.40333 (QuantReg: 12.50883) QuantErr: 12.50883 batch_time=0.53469
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.25691 (QuantReg: 12.22946) QuantErr: 12.22946 batch_time=0.53800
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.04770 (QuantReg: 12.52818) QuantErr: 12.52818 batch_time=0.53930
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.05513 (QuantReg: 12.05653) QuantErr: 12.05653 batch_time=0.52851
Train Epoch: 6 codebook_update_time=1.73287
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch6.pth ...
Done in 18.313s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch6.pth ...
Done in 22.884s
removing stale ckpt [epoch 5] [took 0.02s]
epoch : 6
loss : 2.2016145343780518
quant_reg : 12.163523468017578
quant_err : 12.163523468017578
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_miech_test/t2v_metrics/R1: 16.2
MSRVTT_miech_test/t2v_metrics/R5: 44.5
MSRVTT_miech_test/t2v_metrics/R10: 59.0
MSRVTT_miech_test/t2v_metrics/R50: 86.8
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.944
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.90671758989479
MSRVTT_miech_test/v2t_metrics/R1: 16.6
MSRVTT_miech_test/v2t_metrics/R5: 44.5
MSRVTT_miech_test/v2t_metrics/R10: 57.5
MSRVTT_miech_test/v2t_metrics/R50: 86.9
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 30.7
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.890884667424444
mnt_best : 34.90671758989479
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.13021 (QuantReg: 11.88316) QuantErr: 11.88316 batch_time=44.42145
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.12733 (QuantReg: 11.91540) QuantErr: 11.91540 batch_time=0.53968
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.31873 (QuantReg: 12.56998) QuantErr: 12.56998 batch_time=0.61907
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.17244 (QuantReg: 11.84343) QuantErr: 11.84343 batch_time=0.53981
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.07469 (QuantReg: 12.44809) QuantErr: 12.44809 batch_time=0.58407
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.12948 (QuantReg: 12.27956) QuantErr: 12.27956 batch_time=0.57890
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.03059 (QuantReg: 12.37292) QuantErr: 12.37292 batch_time=0.53315
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.08866 (QuantReg: 12.23391) QuantErr: 12.23391 batch_time=0.56007
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 1.82318 (QuantReg: 12.51853) QuantErr: 12.51853 batch_time=0.53868
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.91621 (QuantReg: 12.55233) QuantErr: 12.55233 batch_time=0.53570
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.17699 (QuantReg: 12.33565) QuantErr: 12.33565 batch_time=0.58356
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.18533 (QuantReg: 12.44072) QuantErr: 12.44072 batch_time=0.58634
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.72871 (QuantReg: 12.43701) QuantErr: 12.43701 batch_time=0.53943
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 1.89999 (QuantReg: 12.58588) QuantErr: 12.58588 batch_time=0.53446
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 1.77920 (QuantReg: 12.74693) QuantErr: 12.74693 batch_time=0.53397
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.11083 (QuantReg: 12.85304) QuantErr: 12.85304 batch_time=0.55867
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.63847 (QuantReg: 12.72488) QuantErr: 12.72488 batch_time=0.57032
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.08104 (QuantReg: 12.58729) QuantErr: 12.58729 batch_time=0.54635
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.03752 (QuantReg: 12.93228) QuantErr: 12.93228 batch_time=0.55050
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.86642 (QuantReg: 12.95970) QuantErr: 12.95970 batch_time=0.55829
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.04039 (QuantReg: 12.72535) QuantErr: 12.72535 batch_time=0.54351
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 1.96651 (QuantReg: 12.96476) QuantErr: 12.96476 batch_time=0.53918
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 1.75312 (QuantReg: 12.84923) QuantErr: 12.84923 batch_time=0.57042
Train Epoch: 7 codebook_update_time=1.99170
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch7.pth ...
Done in 4.763s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch7.pth ...
Done in 9.540s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.020132586956024
quant_reg : 12.482637882232666
quant_err : 12.482637882232666
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_miech_test/t2v_metrics/R1: 17.2
MSRVTT_miech_test/t2v_metrics/R5: 45.6
MSRVTT_miech_test/t2v_metrics/R10: 59.4
MSRVTT_miech_test/t2v_metrics/R50: 88.1
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.135
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.982658314284066
MSRVTT_miech_test/v2t_metrics/R1: 16.9
MSRVTT_miech_test/v2t_metrics/R5: 45.3
MSRVTT_miech_test/v2t_metrics/R10: 60.6
MSRVTT_miech_test/v2t_metrics/R50: 87.8
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 28.382
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.93236839342976
mnt_best : 35.982658314284066
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.02503 (QuantReg: 12.35638) QuantErr: 12.35638 batch_time=29.80195
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 1.93341 (QuantReg: 12.65919) QuantErr: 12.65919 batch_time=0.54077
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.16381 (QuantReg: 12.45764) QuantErr: 12.45764 batch_time=5.64418
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 1.90281 (QuantReg: 12.55178) QuantErr: 12.55178 batch_time=0.59846
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.05795 (QuantReg: 12.88922) QuantErr: 12.88922 batch_time=0.55807
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.10686 (QuantReg: 12.59936) QuantErr: 12.59936 batch_time=0.59413
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.85376 (QuantReg: 12.32983) QuantErr: 12.32983 batch_time=2.42672
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.15169 (QuantReg: 12.30577) QuantErr: 12.30577 batch_time=0.61715
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.68597 (QuantReg: 12.62751) QuantErr: 12.62751 batch_time=0.53754
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.46728 (QuantReg: 13.24274) QuantErr: 13.24274 batch_time=0.54574
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.21439 (QuantReg: 13.30241) QuantErr: 13.30241 batch_time=0.60760
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 1.70338 (QuantReg: 12.58814) QuantErr: 12.58814 batch_time=0.61001
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.78264 (QuantReg: 12.79222) QuantErr: 12.79222 batch_time=0.65694
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.98994 (QuantReg: 13.09957) QuantErr: 13.09957 batch_time=0.53826
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 1.49021 (QuantReg: 12.60328) QuantErr: 12.60328 batch_time=0.55790
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.94595 (QuantReg: 12.70008) QuantErr: 12.70008 batch_time=0.54276
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.01872 (QuantReg: 12.80363) QuantErr: 12.80363 batch_time=0.54174
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 1.79935 (QuantReg: 13.41809) QuantErr: 13.41809 batch_time=0.68228
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.53357 (QuantReg: 12.97297) QuantErr: 12.97297 batch_time=0.52531
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.62458 (QuantReg: 13.13970) QuantErr: 13.13970 batch_time=0.54274
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.71679 (QuantReg: 12.86459) QuantErr: 12.86459 batch_time=0.55342
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.86100 (QuantReg: 12.93657) QuantErr: 12.93657 batch_time=0.54471
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.16749 (QuantReg: 13.26278) QuantErr: 13.26278 batch_time=0.54879
Train Epoch: 8 codebook_update_time=1.89656
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch8.pth ...
Done in 4.770s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch8.pth ...
Done in 9.718s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 1.8866573858261109
quant_reg : 12.768176544189453
quant_err : 12.768176544189453
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_miech_test/t2v_metrics/R1: 17.2
MSRVTT_miech_test/t2v_metrics/R5: 44.7
MSRVTT_miech_test/t2v_metrics/R10: 61.4
MSRVTT_miech_test/t2v_metrics/R50: 88.0
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.201
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.14110668517954
MSRVTT_miech_test/v2t_metrics/R1: 16.1
MSRVTT_miech_test/v2t_metrics/R5: 45.5
MSRVTT_miech_test/v2t_metrics/R10: 60.2
MSRVTT_miech_test/v2t_metrics/R50: 87.9
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 28.3095
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.33007732743194
mnt_best : 36.14110668517954
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.14973 (QuantReg: 12.78100) QuantErr: 12.78100 batch_time=35.76450
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.89522 (QuantReg: 12.23965) QuantErr: 12.23965 batch_time=0.56138
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 1.75965 (QuantReg: 12.51944) QuantErr: 12.51944 batch_time=1.22195
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 1.90994 (QuantReg: 13.03279) QuantErr: 13.03279 batch_time=0.53934
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 1.73025 (QuantReg: 12.61783) QuantErr: 12.61783 batch_time=0.58287
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 1.81652 (QuantReg: 12.69414) QuantErr: 12.69414 batch_time=0.54335
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.83094 (QuantReg: 13.17490) QuantErr: 13.17490 batch_time=0.53357
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.51641 (QuantReg: 12.95541) QuantErr: 12.95541 batch_time=0.52638
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 1.68014 (QuantReg: 12.53806) QuantErr: 12.53806 batch_time=0.54644
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.70141 (QuantReg: 12.88973) QuantErr: 12.88973 batch_time=0.60082
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.75224 (QuantReg: 12.97015) QuantErr: 12.97015 batch_time=0.54723
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.69130 (QuantReg: 13.64001) QuantErr: 13.64001 batch_time=0.57708
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.52756 (QuantReg: 12.93207) QuantErr: 12.93207 batch_time=0.54850
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.60635 (QuantReg: 13.53013) QuantErr: 13.53013 batch_time=0.55126
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.39210 (QuantReg: 13.29494) QuantErr: 13.29494 batch_time=0.54610
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.88701 (QuantReg: 13.19917) QuantErr: 13.19917 batch_time=0.55411
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.60082 (QuantReg: 13.25119) QuantErr: 13.25119 batch_time=0.54338
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.27565 (QuantReg: 13.27381) QuantErr: 13.27381 batch_time=0.55322
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.51266 (QuantReg: 13.20462) QuantErr: 13.20462 batch_time=3.18900
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.03911 (QuantReg: 13.25721) QuantErr: 13.25721 batch_time=0.54104
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.99495 (QuantReg: 13.20915) QuantErr: 13.20915 batch_time=0.55710
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.58352 (QuantReg: 13.19569) QuantErr: 13.19569 batch_time=1.49974
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.62248 (QuantReg: 13.41078) QuantErr: 13.41078 batch_time=1.16709
Train Epoch: 9 codebook_update_time=1.80671
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch9.pth ...
Done in 4.704s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch9.pth ...
Done in 9.359s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 1.7734655442237854
quant_reg : 13.10824639892578
quant_err : 13.10824639892578
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_miech_test/t2v_metrics/R1: 17.6
MSRVTT_miech_test/t2v_metrics/R5: 45.5
MSRVTT_miech_test/t2v_metrics/R10: 61.4
MSRVTT_miech_test/t2v_metrics/R50: 88.5
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.265
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.63510819941375
MSRVTT_miech_test/v2t_metrics/R1: 16.4
MSRVTT_miech_test/v2t_metrics/R5: 46.2
MSRVTT_miech_test/v2t_metrics/R10: 61.5
MSRVTT_miech_test/v2t_metrics/R50: 88.6
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.8255
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.98490107558392
mnt_best : 36.63510819941375
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.89312 (QuantReg: 12.95668) QuantErr: 12.95668 batch_time=39.54363
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.60601 (QuantReg: 13.52022) QuantErr: 13.52022 batch_time=0.53631
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.07963 (QuantReg: 12.78451) QuantErr: 12.78451 batch_time=0.54107
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.79562 (QuantReg: 13.10943) QuantErr: 13.10943 batch_time=0.51730
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.33538 (QuantReg: 13.32486) QuantErr: 13.32486 batch_time=0.52958
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 1.70861 (QuantReg: 13.22169) QuantErr: 13.22169 batch_time=0.56666
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.62924 (QuantReg: 13.33693) QuantErr: 13.33693 batch_time=0.54238
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.70649 (QuantReg: 13.12914) QuantErr: 13.12914 batch_time=0.57062
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.31005 (QuantReg: 13.44900) QuantErr: 13.44900 batch_time=0.60267
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 1.63996 (QuantReg: 13.54635) QuantErr: 13.54635 batch_time=0.55550
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.81910 (QuantReg: 13.29919) QuantErr: 13.29919 batch_time=0.52291
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.93837 (QuantReg: 13.27064) QuantErr: 13.27064 batch_time=0.59227
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.36617 (QuantReg: 13.38308) QuantErr: 13.38308 batch_time=3.22565
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.06922 (QuantReg: 13.41716) QuantErr: 13.41716 batch_time=0.58952
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.74838 (QuantReg: 13.42320) QuantErr: 13.42320 batch_time=0.56049
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.81810 (QuantReg: 13.25381) QuantErr: 13.25381 batch_time=0.56053
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 1.80098 (QuantReg: 13.16685) QuantErr: 13.16685 batch_time=0.56495
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.42736 (QuantReg: 13.72375) QuantErr: 13.72375 batch_time=0.60335
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.89136 (QuantReg: 13.31165) QuantErr: 13.31165 batch_time=0.66282
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.78339 (QuantReg: 13.78749) QuantErr: 13.78749 batch_time=0.53349
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.95526 (QuantReg: 13.75788) QuantErr: 13.75788 batch_time=0.58437
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 1.56820 (QuantReg: 13.39506) QuantErr: 13.39506 batch_time=0.54719
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.74426 (QuantReg: 13.31391) QuantErr: 13.31391 batch_time=0.59009
Train Epoch: 10 codebook_update_time=1.90401
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch10.pth ...
Done in 18.338s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch10.pth ...
Done in 22.637s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 1.6656816964149475
quant_reg : 13.312392517089844
quant_err : 13.312392517089844
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_miech_test/t2v_metrics/R1: 19.2
MSRVTT_miech_test/t2v_metrics/R5: 47.6
MSRVTT_miech_test/t2v_metrics/R10: 60.3
MSRVTT_miech_test/t2v_metrics/R50: 87.4
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.191
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.05471711091533
MSRVTT_miech_test/v2t_metrics/R1: 19.3
MSRVTT_miech_test/v2t_metrics/R5: 47.0
MSRVTT_miech_test/v2t_metrics/R10: 60.8
MSRVTT_miech_test/v2t_metrics/R50: 88.4
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.2305
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.06445202424545
mnt_best : 38.05471711091533
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.81950 (QuantReg: 13.26481) QuantErr: 13.26481 batch_time=33.87299
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.41501 (QuantReg: 13.57368) QuantErr: 13.57368 batch_time=0.53624
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.81800 (QuantReg: 13.40409) QuantErr: 13.40409 batch_time=0.54973
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.12446 (QuantReg: 12.75865) QuantErr: 12.75865 batch_time=0.52996
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.53178 (QuantReg: 13.65595) QuantErr: 13.65595 batch_time=0.59955
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.57887 (QuantReg: 13.25064) QuantErr: 13.25064 batch_time=0.54938
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.58533 (QuantReg: 13.50012) QuantErr: 13.50012 batch_time=2.18164
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.73249 (QuantReg: 13.15415) QuantErr: 13.15415 batch_time=0.52889
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.74965 (QuantReg: 13.59444) QuantErr: 13.59444 batch_time=0.55411
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.55373 (QuantReg: 13.14288) QuantErr: 13.14288 batch_time=0.55815
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.85006 (QuantReg: 13.49994) QuantErr: 13.49994 batch_time=0.94367
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.47936 (QuantReg: 13.64652) QuantErr: 13.64652 batch_time=0.56814
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.92604 (QuantReg: 13.40137) QuantErr: 13.40137 batch_time=1.58225
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.77116 (QuantReg: 13.29082) QuantErr: 13.29082 batch_time=0.54062
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.67755 (QuantReg: 13.08231) QuantErr: 13.08231 batch_time=0.56195
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.50437 (QuantReg: 13.27046) QuantErr: 13.27046 batch_time=0.56105
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.44554 (QuantReg: 13.48660) QuantErr: 13.48660 batch_time=0.61030
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.36391 (QuantReg: 13.23386) QuantErr: 13.23386 batch_time=0.61187
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.83701 (QuantReg: 13.16753) QuantErr: 13.16753 batch_time=0.53504
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.17822 (QuantReg: 13.79572) QuantErr: 13.79572 batch_time=0.53457
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.52453 (QuantReg: 13.32272) QuantErr: 13.32272 batch_time=0.52509
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.77251 (QuantReg: 13.70113) QuantErr: 13.70113 batch_time=0.53354
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.58124 (QuantReg: 13.78475) QuantErr: 13.78475 batch_time=0.54961
Train Epoch: 11 codebook_update_time=2.14596
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch11.pth ...
Done in 17.891s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 1.6061825108528138
quant_reg : 13.477547245025635
quant_err : 13.477547245025635
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_miech_test/t2v_metrics/R1: 17.6
MSRVTT_miech_test/t2v_metrics/R5: 47.2
MSRVTT_miech_test/t2v_metrics/R10: 60.9
MSRVTT_miech_test/t2v_metrics/R50: 87.6
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.44
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.984860618985834
MSRVTT_miech_test/v2t_metrics/R1: 16.7
MSRVTT_miech_test/v2t_metrics/R5: 46.3
MSRVTT_miech_test/v2t_metrics/R10: 62.0
MSRVTT_miech_test/v2t_metrics/R50: 87.7
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.5205
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.32701533289096
mnt_best : 38.05471711091533
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.54356 (QuantReg: 13.02055) QuantErr: 13.02055 batch_time=35.51439
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.37477 (QuantReg: 13.42034) QuantErr: 13.42034 batch_time=0.54007
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.52629 (QuantReg: 13.43815) QuantErr: 13.43815 batch_time=0.54630
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.50507 (QuantReg: 13.28459) QuantErr: 13.28459 batch_time=0.55072
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.36864 (QuantReg: 13.52251) QuantErr: 13.52251 batch_time=0.58821
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.58893 (QuantReg: 13.58555) QuantErr: 13.58555 batch_time=0.50826
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.67197 (QuantReg: 13.31247) QuantErr: 13.31247 batch_time=0.53471
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.30131 (QuantReg: 13.66665) QuantErr: 13.66665 batch_time=0.55683
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.45240 (QuantReg: 13.30680) QuantErr: 13.30680 batch_time=0.69949
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.43441 (QuantReg: 13.64821) QuantErr: 13.64821 batch_time=0.53291
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.62130 (QuantReg: 13.81687) QuantErr: 13.81687 batch_time=0.52776
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.58014 (QuantReg: 13.45270) QuantErr: 13.45270 batch_time=0.54090
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.45201 (QuantReg: 13.44705) QuantErr: 13.44705 batch_time=0.60804
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.58683 (QuantReg: 13.58153) QuantErr: 13.58153 batch_time=0.54424
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.65110 (QuantReg: 13.58571) QuantErr: 13.58571 batch_time=0.54045
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.54271 (QuantReg: 13.37390) QuantErr: 13.37390 batch_time=0.58522
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.78294 (QuantReg: 13.63976) QuantErr: 13.63976 batch_time=0.57741
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.81676 (QuantReg: 13.68805) QuantErr: 13.68805 batch_time=0.53942
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.31064 (QuantReg: 13.90179) QuantErr: 13.90179 batch_time=0.55206
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.68580 (QuantReg: 13.75165) QuantErr: 13.75165 batch_time=0.55191
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.65811 (QuantReg: 13.48713) QuantErr: 13.48713 batch_time=0.58637
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.75765 (QuantReg: 13.65344) QuantErr: 13.65344 batch_time=0.53027
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.65364 (QuantReg: 13.85681) QuantErr: 13.85681 batch_time=0.54820
Train Epoch: 12 codebook_update_time=2.10143
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch12.pth ...
Done in 5.683s
removing stale ckpt [epoch 11] [took 0.19s]
epoch : 12
loss : 1.5228573675155639
quant_reg : 13.599921356201172
quant_err : 13.599921356201172
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_miech_test/t2v_metrics/R1: 17.9
MSRVTT_miech_test/t2v_metrics/R5: 47.9
MSRVTT_miech_test/t2v_metrics/R10: 63.0
MSRVTT_miech_test/t2v_metrics/R50: 87.9
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.643
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.801557842909524
MSRVTT_miech_test/v2t_metrics/R1: 16.8
MSRVTT_miech_test/v2t_metrics/R5: 47.4
MSRVTT_miech_test/v2t_metrics/R10: 61.7
MSRVTT_miech_test/v2t_metrics/R50: 88.9
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.08
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.62612127945115
mnt_best : 38.05471711091533
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.38493 (QuantReg: 13.71579) QuantErr: 13.71579 batch_time=40.33111
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.49384 (QuantReg: 13.57120) QuantErr: 13.57120 batch_time=0.53777
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.35524 (QuantReg: 13.82145) QuantErr: 13.82145 batch_time=0.53088
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.69401 (QuantReg: 13.77486) QuantErr: 13.77486 batch_time=0.53608
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.61717 (QuantReg: 13.83309) QuantErr: 13.83309 batch_time=0.57939
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.74806 (QuantReg: 13.46585) QuantErr: 13.46585 batch_time=2.41041
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.26632 (QuantReg: 13.83571) QuantErr: 13.83571 batch_time=0.59323
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.47245 (QuantReg: 13.94946) QuantErr: 13.94946 batch_time=0.54988
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.23949 (QuantReg: 13.77031) QuantErr: 13.77031 batch_time=0.61127
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.54905 (QuantReg: 13.33493) QuantErr: 13.33493 batch_time=0.57840
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.58727 (QuantReg: 13.67439) QuantErr: 13.67439 batch_time=0.53975
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.32985 (QuantReg: 14.14386) QuantErr: 14.14386 batch_time=0.55351
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.71301 (QuantReg: 13.36604) QuantErr: 13.36604 batch_time=0.49865
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.83314 (QuantReg: 13.91732) QuantErr: 13.91732 batch_time=2.00669
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.51562 (QuantReg: 14.18040) QuantErr: 14.18040 batch_time=0.56234
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.52527 (QuantReg: 13.68724) QuantErr: 13.68724 batch_time=0.55081
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.43164 (QuantReg: 14.14571) QuantErr: 14.14571 batch_time=0.55633
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.56785 (QuantReg: 13.75737) QuantErr: 13.75737 batch_time=0.55419
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.09387 (QuantReg: 13.94243) QuantErr: 13.94243 batch_time=0.58129
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.27364 (QuantReg: 14.33006) QuantErr: 14.33006 batch_time=2.78479
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.22829 (QuantReg: 13.95529) QuantErr: 13.95529 batch_time=0.64410
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.17524 (QuantReg: 13.83167) QuantErr: 13.83167 batch_time=0.53527
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.04585 (QuantReg: 13.83354) QuantErr: 13.83354 batch_time=0.53610
Train Epoch: 13 codebook_update_time=1.81747
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch13.pth ...
Done in 6.074s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch13.pth ...
Done in 12.056s
removing stale ckpt [epoch 12] [took 0.24s]
epoch : 13
loss : 1.4600150163173675
quant_reg : 13.811403430938721
quant_err : 13.811403430938721
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_miech_test/t2v_metrics/R1: 19.3
MSRVTT_miech_test/t2v_metrics/R5: 47.1
MSRVTT_miech_test/t2v_metrics/R10: 62.0
MSRVTT_miech_test/t2v_metrics/R50: 88.6
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.809
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.34039962151047
MSRVTT_miech_test/v2t_metrics/R1: 17.9
MSRVTT_miech_test/v2t_metrics/R5: 48.3
MSRVTT_miech_test/v2t_metrics/R10: 63.0
MSRVTT_miech_test/v2t_metrics/R50: 89.3
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.3185
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.90648983946275
mnt_best : 38.34039962151047
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.71959 (QuantReg: 13.51162) QuantErr: 13.51162 batch_time=33.76060
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.42300 (QuantReg: 13.86672) QuantErr: 13.86672 batch_time=0.54094
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.33776 (QuantReg: 13.67053) QuantErr: 13.67053 batch_time=0.63394
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.38658 (QuantReg: 14.05453) QuantErr: 14.05453 batch_time=0.54852
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.67536 (QuantReg: 13.66040) QuantErr: 13.66040 batch_time=0.52662
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.27834 (QuantReg: 14.02400) QuantErr: 14.02400 batch_time=0.57912
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.55558 (QuantReg: 13.73598) QuantErr: 13.73598 batch_time=0.54944
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.94695 (QuantReg: 13.53996) QuantErr: 13.53996 batch_time=0.54387
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.52647 (QuantReg: 13.97395) QuantErr: 13.97395 batch_time=0.56843
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.48211 (QuantReg: 14.01818) QuantErr: 14.01818 batch_time=0.53365
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.43189 (QuantReg: 13.97998) QuantErr: 13.97998 batch_time=0.55981
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.36806 (QuantReg: 13.80006) QuantErr: 13.80006 batch_time=0.56414
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.37535 (QuantReg: 13.94133) QuantErr: 13.94133 batch_time=0.58219
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.41630 (QuantReg: 13.71564) QuantErr: 13.71564 batch_time=0.58327
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.17827 (QuantReg: 13.88634) QuantErr: 13.88634 batch_time=0.53232
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.14406 (QuantReg: 14.03536) QuantErr: 14.03536 batch_time=0.54403
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.47352 (QuantReg: 13.77070) QuantErr: 13.77070 batch_time=0.55336
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.38642 (QuantReg: 14.25397) QuantErr: 14.25397 batch_time=0.53655
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.03111 (QuantReg: 14.37737) QuantErr: 14.37737 batch_time=0.57192
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.77478 (QuantReg: 14.34958) QuantErr: 14.34958 batch_time=0.56113
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 0.98395 (QuantReg: 14.25855) QuantErr: 14.25855 batch_time=0.54534
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.38133 (QuantReg: 13.85973) QuantErr: 13.85973 batch_time=0.55584
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.29506 (QuantReg: 14.10602) QuantErr: 14.10602 batch_time=0.60178
Train Epoch: 14 codebook_update_time=2.06369
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch14.pth ...
Done in 6.610s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 1.4046821224689483
quant_reg : 13.974115398406983
quant_err : 13.974115398406983
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_miech_test/t2v_metrics/R1: 17.6
MSRVTT_miech_test/t2v_metrics/R5: 47.3
MSRVTT_miech_test/t2v_metrics/R10: 61.9
MSRVTT_miech_test/t2v_metrics/R50: 87.6
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.821
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.21244043780525
MSRVTT_miech_test/v2t_metrics/R1: 18.9
MSRVTT_miech_test/v2t_metrics/R5: 47.3
MSRVTT_miech_test/v2t_metrics/R10: 62.1
MSRVTT_miech_test/v2t_metrics/R50: 88.6
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 25.565
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.1479772555705
mnt_best : 38.34039962151047
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.51073 (QuantReg: 14.10802) QuantErr: 14.10802 batch_time=38.84701
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.48777 (QuantReg: 13.76672) QuantErr: 13.76672 batch_time=0.49888
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.57329 (QuantReg: 13.77047) QuantErr: 13.77047 batch_time=0.54780
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.79217 (QuantReg: 13.62339) QuantErr: 13.62339 batch_time=0.54389
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.55652 (QuantReg: 13.94825) QuantErr: 13.94825 batch_time=0.54377
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 0.99913 (QuantReg: 14.20334) QuantErr: 14.20334 batch_time=0.53219
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.39121 (QuantReg: 13.67310) QuantErr: 13.67310 batch_time=1.50914
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 0.99032 (QuantReg: 14.31069) QuantErr: 14.31069 batch_time=0.57730
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.81544 (QuantReg: 14.04857) QuantErr: 14.04857 batch_time=0.59182
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.34530 (QuantReg: 13.95898) QuantErr: 13.95898 batch_time=0.53600
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.17541 (QuantReg: 14.19398) QuantErr: 14.19398 batch_time=0.59596
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.29526 (QuantReg: 13.70377) QuantErr: 13.70377 batch_time=0.52591
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.74418 (QuantReg: 14.12654) QuantErr: 14.12654 batch_time=2.51188
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.32990 (QuantReg: 14.20127) QuantErr: 14.20127 batch_time=0.58694
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.40082 (QuantReg: 14.18129) QuantErr: 14.18129 batch_time=0.62756
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.22729 (QuantReg: 14.18360) QuantErr: 14.18360 batch_time=0.52742
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.12654 (QuantReg: 14.38302) QuantErr: 14.38302 batch_time=0.53834
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.67354 (QuantReg: 14.32589) QuantErr: 14.32589 batch_time=0.63478
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.06276 (QuantReg: 14.09441) QuantErr: 14.09441 batch_time=0.54384
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.11419 (QuantReg: 14.49886) QuantErr: 14.49886 batch_time=0.51564
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.18064 (QuantReg: 14.71662) QuantErr: 14.71662 batch_time=0.57120
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.43620 (QuantReg: 14.49319) QuantErr: 14.49319 batch_time=0.52004
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.42473 (QuantReg: 14.68954) QuantErr: 14.68954 batch_time=0.54922
Train Epoch: 15 codebook_update_time=1.81035
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch15.pth ...
Done in 5.787s
removing stale ckpt [epoch 14] [took 0.13s]
epoch : 15
loss : 1.3698881025314331
quant_reg : 14.126482299804687
quant_err : 14.126482299804687
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_miech_test/t2v_metrics/R1: 17.7
MSRVTT_miech_test/t2v_metrics/R5: 48.3
MSRVTT_miech_test/t2v_metrics/R10: 62.6
MSRVTT_miech_test/t2v_metrics/R50: 87.7
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.043
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.684686794458926
MSRVTT_miech_test/v2t_metrics/R1: 18.1
MSRVTT_miech_test/v2t_metrics/R5: 48.5
MSRVTT_miech_test/v2t_metrics/R10: 63.0
MSRVTT_miech_test/v2t_metrics/R50: 89.0
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.875
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.09958872807131
mnt_best : 38.34039962151047
not_improved_count: 2
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.24605 (QuantReg: 13.59291) QuantErr: 13.59291 batch_time=39.13596
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.48836 (QuantReg: 13.77159) QuantErr: 13.77159 batch_time=0.53656
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.04806 (QuantReg: 13.95226) QuantErr: 13.95226 batch_time=0.52843
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.68106 (QuantReg: 13.92839) QuantErr: 13.92839 batch_time=0.56063
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.36951 (QuantReg: 14.15701) QuantErr: 14.15701 batch_time=0.61500
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.55415 (QuantReg: 14.34663) QuantErr: 14.34663 batch_time=0.52227
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.53446 (QuantReg: 14.19092) QuantErr: 14.19092 batch_time=0.56905
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.33896 (QuantReg: 14.55329) QuantErr: 14.55329 batch_time=0.55942
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.41358 (QuantReg: 13.94269) QuantErr: 13.94269 batch_time=0.60384
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.02416 (QuantReg: 14.37736) QuantErr: 14.37736 batch_time=0.53833
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.27588 (QuantReg: 14.44684) QuantErr: 14.44684 batch_time=0.54427
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.13898 (QuantReg: 14.11161) QuantErr: 14.11161 batch_time=0.57191
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.24441 (QuantReg: 14.10211) QuantErr: 14.10211 batch_time=0.54527
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.50136 (QuantReg: 14.17365) QuantErr: 14.17365 batch_time=0.57015
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.42979 (QuantReg: 14.29449) QuantErr: 14.29449 batch_time=0.60784
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.33058 (QuantReg: 14.38432) QuantErr: 14.38432 batch_time=0.55049
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.32413 (QuantReg: 14.63406) QuantErr: 14.63406 batch_time=0.59057
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.43397 (QuantReg: 14.02812) QuantErr: 14.02812 batch_time=0.54989
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.21541 (QuantReg: 14.32923) QuantErr: 14.32923 batch_time=0.53386
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.26139 (QuantReg: 14.50807) QuantErr: 14.50807 batch_time=0.58880
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.31993 (QuantReg: 14.38381) QuantErr: 14.38381 batch_time=0.54432
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.07948 (QuantReg: 14.63577) QuantErr: 14.63577 batch_time=0.53134
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.42631 (QuantReg: 14.15175) QuantErr: 14.15175 batch_time=0.57951
Train Epoch: 16 codebook_update_time=1.80051
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch16.pth ...
Done in 6.610s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch16.pth ...
Done in 12.351s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 1.29578204703331
quant_reg : 14.300463813781738
quant_err : 14.300463813781738
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_miech_test/t2v_metrics/R1: 19.3
MSRVTT_miech_test/t2v_metrics/R5: 49.1
MSRVTT_miech_test/t2v_metrics/R10: 63.4
MSRVTT_miech_test/t2v_metrics/R50: 89.0
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.962
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.166012032718264
MSRVTT_miech_test/v2t_metrics/R1: 17.4
MSRVTT_miech_test/v2t_metrics/R5: 48.6
MSRVTT_miech_test/v2t_metrics/R10: 64.1
MSRVTT_miech_test/v2t_metrics/R50: 88.6
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.74
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.845523384454694
mnt_best : 39.166012032718264
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.13239 (QuantReg: 14.08412) QuantErr: 14.08412 batch_time=33.36726
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.11561 (QuantReg: 14.30495) QuantErr: 14.30495 batch_time=0.54056
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.32056 (QuantReg: 14.36386) QuantErr: 14.36386 batch_time=0.56579
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.48850 (QuantReg: 14.05688) QuantErr: 14.05688 batch_time=0.52393
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.42732 (QuantReg: 14.41476) QuantErr: 14.41476 batch_time=1.04041
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.21565 (QuantReg: 14.47464) QuantErr: 14.47464 batch_time=0.56209
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.53590 (QuantReg: 14.03780) QuantErr: 14.03780 batch_time=0.54352
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.27220 (QuantReg: 14.51941) QuantErr: 14.51941 batch_time=0.54216
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.35368 (QuantReg: 14.33345) QuantErr: 14.33345 batch_time=0.53393
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 0.88418 (QuantReg: 14.73146) QuantErr: 14.73146 batch_time=0.53885
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 0.99527 (QuantReg: 14.53552) QuantErr: 14.53552 batch_time=0.56549
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.03635 (QuantReg: 14.37182) QuantErr: 14.37182 batch_time=0.53787
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.52141 (QuantReg: 14.46053) QuantErr: 14.46053 batch_time=0.52131
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.27260 (QuantReg: 14.46097) QuantErr: 14.46097 batch_time=0.59492
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.59596 (QuantReg: 14.23824) QuantErr: 14.23824 batch_time=0.57475
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.35080 (QuantReg: 14.22895) QuantErr: 14.22895 batch_time=0.52788
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 0.97836 (QuantReg: 14.32390) QuantErr: 14.32390 batch_time=0.53683
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.21193 (QuantReg: 14.39600) QuantErr: 14.39600 batch_time=0.55619
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.43493 (QuantReg: 14.59484) QuantErr: 14.59484 batch_time=3.98682
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.44888 (QuantReg: 14.39639) QuantErr: 14.39639 batch_time=0.52764
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.01591 (QuantReg: 14.50755) QuantErr: 14.50755 batch_time=0.55604
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.27456 (QuantReg: 14.64532) QuantErr: 14.64532 batch_time=0.53758
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.35242 (QuantReg: 14.55985) QuantErr: 14.55985 batch_time=0.55220
Train Epoch: 17 codebook_update_time=1.79195
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch17.pth ...
Done in 5.900s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch17.pth ...
Done in 11.041s
removing stale ckpt [epoch 16] [took 0.06s]
epoch : 17
loss : 1.2613397839069367
quant_reg : 14.383664207458496
quant_err : 14.383664207458496
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_miech_test/t2v_metrics/R1: 19.8
MSRVTT_miech_test/t2v_metrics/R5: 48.6
MSRVTT_miech_test/t2v_metrics/R10: 63.5
MSRVTT_miech_test/t2v_metrics/R50: 87.2
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.217
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.387498073365805
MSRVTT_miech_test/v2t_metrics/R1: 17.0
MSRVTT_miech_test/v2t_metrics/R5: 48.3
MSRVTT_miech_test/v2t_metrics/R10: 64.2
MSRVTT_miech_test/v2t_metrics/R50: 88.4
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 28.0845
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.49531674848182
mnt_best : 39.387498073365805
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.25739 (QuantReg: 14.11591) QuantErr: 14.11591 batch_time=38.91952
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.45858 (QuantReg: 14.14635) QuantErr: 14.14635 batch_time=0.55350
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.48375 (QuantReg: 14.03888) QuantErr: 14.03888 batch_time=0.51265
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.63300 (QuantReg: 14.21944) QuantErr: 14.21944 batch_time=0.55343
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.19977 (QuantReg: 14.69156) QuantErr: 14.69156 batch_time=0.52974
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.27344 (QuantReg: 14.29839) QuantErr: 14.29839 batch_time=0.51896
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.02310 (QuantReg: 14.51132) QuantErr: 14.51132 batch_time=0.53825
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.48129 (QuantReg: 14.69449) QuantErr: 14.69449 batch_time=0.53316
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.23440 (QuantReg: 14.64754) QuantErr: 14.64754 batch_time=0.54344
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.42497 (QuantReg: 14.42582) QuantErr: 14.42582 batch_time=0.54475
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 0.99751 (QuantReg: 14.64713) QuantErr: 14.64713 batch_time=0.55575
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.08619 (QuantReg: 14.85472) QuantErr: 14.85472 batch_time=0.55316
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.38545 (QuantReg: 14.46636) QuantErr: 14.46636 batch_time=0.55709
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.12951 (QuantReg: 14.52754) QuantErr: 14.52754 batch_time=0.52403
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.17194 (QuantReg: 14.50673) QuantErr: 14.50673 batch_time=0.52495
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.28926 (QuantReg: 14.58140) QuantErr: 14.58140 batch_time=0.54441
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.63426 (QuantReg: 14.65995) QuantErr: 14.65995 batch_time=0.51833
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.30172 (QuantReg: 14.44260) QuantErr: 14.44260 batch_time=0.56740
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.32973 (QuantReg: 14.69100) QuantErr: 14.69100 batch_time=0.52836
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.35187 (QuantReg: 14.70183) QuantErr: 14.70183 batch_time=0.54265
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.33108 (QuantReg: 14.17793) QuantErr: 14.17793 batch_time=0.60766
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.40876 (QuantReg: 14.63061) QuantErr: 14.63061 batch_time=2.37000
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.18637 (QuantReg: 14.63934) QuantErr: 14.63934 batch_time=0.66272
Train Epoch: 18 codebook_update_time=1.72076
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch18.pth ...
Done in 6.913s
removing stale ckpt [epoch 17] [took 0.02s]
epoch : 18
loss : 1.2411517052650451
quant_reg : 14.529673149108886
quant_err : 14.529673149108886
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_miech_test/t2v_metrics/R1: 19.4
MSRVTT_miech_test/t2v_metrics/R5: 48.9
MSRVTT_miech_test/t2v_metrics/R10: 63.2
MSRVTT_miech_test/t2v_metrics/R50: 87.0
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 32.256
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.138954686311514
MSRVTT_miech_test/v2t_metrics/R1: 18.1
MSRVTT_miech_test/v2t_metrics/R5: 49.2
MSRVTT_miech_test/v2t_metrics/R10: 63.4
MSRVTT_miech_test/v2t_metrics/R50: 88.1
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.543
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.36286014085668
mnt_best : 39.387498073365805
not_improved_count: 1
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.45293 (QuantReg: 14.26774) QuantErr: 14.26774 batch_time=36.67811
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 0.99513 (QuantReg: 14.70875) QuantErr: 14.70875 batch_time=0.54399
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.16480 (QuantReg: 13.95113) QuantErr: 13.95113 batch_time=0.52942
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.12521 (QuantReg: 14.35215) QuantErr: 14.35215 batch_time=0.53129
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.35338 (QuantReg: 14.60676) QuantErr: 14.60676 batch_time=0.56562
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.32899 (QuantReg: 14.49524) QuantErr: 14.49524 batch_time=0.55757
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.35684 (QuantReg: 14.38442) QuantErr: 14.38442 batch_time=4.51546
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.05265 (QuantReg: 14.75764) QuantErr: 14.75764 batch_time=0.52371
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.36619 (QuantReg: 14.71912) QuantErr: 14.71912 batch_time=0.56366
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 0.99255 (QuantReg: 14.23200) QuantErr: 14.23200 batch_time=0.54364
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.11972 (QuantReg: 14.65079) QuantErr: 14.65079 batch_time=0.52119
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.12424 (QuantReg: 14.53530) QuantErr: 14.53530 batch_time=0.54461
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.21210 (QuantReg: 14.69564) QuantErr: 14.69564 batch_time=0.55071
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.11628 (QuantReg: 14.63860) QuantErr: 14.63860 batch_time=0.55653
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.49132 (QuantReg: 14.50729) QuantErr: 14.50729 batch_time=0.51616
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.01925 (QuantReg: 14.88712) QuantErr: 14.88712 batch_time=0.53441
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.01214 (QuantReg: 14.80153) QuantErr: 14.80153 batch_time=0.55769
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.09110 (QuantReg: 14.93447) QuantErr: 14.93447 batch_time=0.57762
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.42014 (QuantReg: 14.68776) QuantErr: 14.68776 batch_time=0.57035
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.21696 (QuantReg: 14.55127) QuantErr: 14.55127 batch_time=0.54844
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.21654 (QuantReg: 14.63702) QuantErr: 14.63702 batch_time=0.55329
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.04680 (QuantReg: 14.67159) QuantErr: 14.67159 batch_time=1.42518
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.37985 (QuantReg: 14.14711) QuantErr: 14.14711 batch_time=0.54985
Train Epoch: 19 codebook_update_time=1.84964
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_t0.03/checkpoint-epoch19.pth ...
Done in 7.639s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 1.1970782694816589
quant_reg : 14.669836727142334
quant_err : 14.669836727142334
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_miech_test/t2v_metrics/R1: 19.4
MSRVTT_miech_test/t2v_metrics/R5: 48.6
MSRVTT_miech_test/t2v_metrics/R10: 62.5
MSRVTT_miech_test/t2v_metrics/R50: 87.4
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.721