-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_1kB_L15.txt
2583 lines (2583 loc) · 191 KB
/
HCQ_MSRVTT_1kB_L15.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15
Preparing the dataloaders ...
Loading dataset MSRVTT_miech_trainval in ram ...
Finish loading dataset MSRVTT_miech_trainval in ram, taking 773.1326208114624 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 114.67448949813843 s.
Loading dataset MSRVTT_miech_test in ram ...
Finish loading dataset MSRVTT_miech_test in ram, taking 82.21067357063293 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch0.pth ...
Done in 1.832s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch0.pth ...
Done in 3.617s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_miech_test/t2v_metrics/R1: 0.0
MSRVTT_miech_test/t2v_metrics/R5: 0.4
MSRVTT_miech_test/t2v_metrics/R10: 1.3
MSRVTT_miech_test/t2v_metrics/R50: 5.1
MSRVTT_miech_test/t2v_metrics/MedR: 509.0
MSRVTT_miech_test/t2v_metrics/MeanR: 503.299
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_miech_test/v2t_metrics/R1: 0.2
MSRVTT_miech_test/v2t_metrics/R5: 0.6
MSRVTT_miech_test/v2t_metrics/R10: 1.0
MSRVTT_miech_test/v2t_metrics/R50: 4.5
MSRVTT_miech_test/v2t_metrics/MedR: 515.5
MSRVTT_miech_test/v2t_metrics/MeanR: 503.4175
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.493242414866094
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 30.17577 (QuantReg: 22.46971) QuantErr: 22.46971 batch_time=38.90253
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 28.56202 (QuantReg: 22.46958) QuantErr: 22.46958 batch_time=2.23519
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 24.89168 (QuantReg: 22.56749) QuantErr: 22.56749 batch_time=0.63773
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 22.56904 (QuantReg: 22.62825) QuantErr: 22.62825 batch_time=0.65254
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 20.26789 (QuantReg: 22.63294) QuantErr: 22.63294 batch_time=0.65602
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 19.52416 (QuantReg: 22.66749) QuantErr: 22.66749 batch_time=0.68930
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 19.47158 (QuantReg: 22.66861) QuantErr: 22.66861 batch_time=0.69660
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 18.43093 (QuantReg: 22.63848) QuantErr: 22.63848 batch_time=0.64974
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 17.19742 (QuantReg: 22.65280) QuantErr: 22.65280 batch_time=0.63393
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 18.15055 (QuantReg: 22.64924) QuantErr: 22.64924 batch_time=0.63081
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 16.77019 (QuantReg: 22.67354) QuantErr: 22.67354 batch_time=0.65615
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 17.17690 (QuantReg: 22.65629) QuantErr: 22.65629 batch_time=0.63690
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 15.35367 (QuantReg: 22.64768) QuantErr: 22.64768 batch_time=0.65020
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 14.11423 (QuantReg: 22.64898) QuantErr: 22.64898 batch_time=0.62580
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 15.23379 (QuantReg: 22.65808) QuantErr: 22.65808 batch_time=0.64305
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 15.21022 (QuantReg: 22.65609) QuantErr: 22.65609 batch_time=0.68575
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 14.44549 (QuantReg: 22.68740) QuantErr: 22.68740 batch_time=0.63313
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 14.33575 (QuantReg: 22.64818) QuantErr: 22.64818 batch_time=0.64096
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 12.80243 (QuantReg: 22.68117) QuantErr: 22.68117 batch_time=0.63539
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 12.82430 (QuantReg: 22.67882) QuantErr: 22.67882 batch_time=0.65083
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 13.36164 (QuantReg: 22.64316) QuantErr: 22.64316 batch_time=0.67784
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 13.11747 (QuantReg: 22.66771) QuantErr: 22.66771 batch_time=0.63409
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 14.26314 (QuantReg: 22.66678) QuantErr: 22.66678 batch_time=0.63349
Train Epoch: 1 codebook_update_time=5.40122
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch1.pth ...
Done in 4.273s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch1.pth ...
Done in 8.582s
epoch : 1
loss : 17.437395236968992
quant_reg : 22.638286781311034
quant_err : 22.638286781311034
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_miech_test/t2v_metrics/R1: 10.8
MSRVTT_miech_test/t2v_metrics/R5: 31.3
MSRVTT_miech_test/t2v_metrics/R10: 43.7
MSRVTT_miech_test/t2v_metrics/R50: 78.7
MSRVTT_miech_test/t2v_metrics/MedR: 14.0
MSRVTT_miech_test/t2v_metrics/MeanR: 44.526
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.53672016950174
MSRVTT_miech_test/v2t_metrics/R1: 11.2
MSRVTT_miech_test/v2t_metrics/R5: 33.4
MSRVTT_miech_test/v2t_metrics/R10: 46.6
MSRVTT_miech_test/v2t_metrics/R50: 78.0
MSRVTT_miech_test/v2t_metrics/MedR: 12.0
MSRVTT_miech_test/v2t_metrics/MeanR: 45.063
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.928862741740023
mnt_best : 24.53672016950174
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 12.95839 (QuantReg: 10.86897) QuantErr: 10.86897 batch_time=32.41242
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 12.59032 (QuantReg: 11.16109) QuantErr: 11.16109 batch_time=0.64984
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 13.37641 (QuantReg: 11.22874) QuantErr: 11.22874 batch_time=0.63743
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 13.48369 (QuantReg: 11.60754) QuantErr: 11.60754 batch_time=0.66524
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 10.84408 (QuantReg: 11.84893) QuantErr: 11.84893 batch_time=0.63740
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 10.43327 (QuantReg: 11.99653) QuantErr: 11.99653 batch_time=0.65877
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 11.54054 (QuantReg: 12.18454) QuantErr: 12.18454 batch_time=2.19244
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 12.00328 (QuantReg: 12.51004) QuantErr: 12.51004 batch_time=0.64617
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 12.48836 (QuantReg: 12.58160) QuantErr: 12.58160 batch_time=0.67316
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 12.13318 (QuantReg: 12.31262) QuantErr: 12.31262 batch_time=0.64379
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 10.35017 (QuantReg: 12.81479) QuantErr: 12.81479 batch_time=0.64208
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 12.53417 (QuantReg: 13.14528) QuantErr: 13.14528 batch_time=0.67342
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 9.98257 (QuantReg: 13.13459) QuantErr: 13.13459 batch_time=0.62341
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 11.67390 (QuantReg: 13.45081) QuantErr: 13.45081 batch_time=2.56405
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 12.41555 (QuantReg: 13.30147) QuantErr: 13.30147 batch_time=0.65337
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 12.19769 (QuantReg: 13.33105) QuantErr: 13.33105 batch_time=1.82264
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 9.39062 (QuantReg: 13.34625) QuantErr: 13.34625 batch_time=0.64263
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 11.26876 (QuantReg: 14.24580) QuantErr: 14.24580 batch_time=0.63377
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 11.29153 (QuantReg: 14.08286) QuantErr: 14.08286 batch_time=0.63112
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 11.86164 (QuantReg: 13.72496) QuantErr: 13.72496 batch_time=0.72604
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 10.82703 (QuantReg: 14.25244) QuantErr: 14.25244 batch_time=0.64275
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 12.19363 (QuantReg: 14.03917) QuantErr: 14.03917 batch_time=0.71774
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 10.83663 (QuantReg: 14.14044) QuantErr: 14.14044 batch_time=0.65454
Train Epoch: 2 codebook_update_time=3.65493
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch2.pth ...
Done in 4.657s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch2.pth ...
Done in 9.157s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 11.664813583374023
quant_reg : 12.887033843994141
quant_err : 12.887033843994141
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_miech_test/t2v_metrics/R1: 11.8
MSRVTT_miech_test/t2v_metrics/R5: 36.0
MSRVTT_miech_test/t2v_metrics/R10: 49.8
MSRVTT_miech_test/t2v_metrics/R50: 82.9
MSRVTT_miech_test/t2v_metrics/MedR: 11.0
MSRVTT_miech_test/t2v_metrics/MeanR: 40.994
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.656971167635263
MSRVTT_miech_test/v2t_metrics/R1: 13.4
MSRVTT_miech_test/v2t_metrics/R5: 38.0
MSRVTT_miech_test/v2t_metrics/R10: 51.8
MSRVTT_miech_test/v2t_metrics/R50: 82.9
MSRVTT_miech_test/v2t_metrics/MedR: 10.0
MSRVTT_miech_test/v2t_metrics/MeanR: 40.388
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.767295923839406
mnt_best : 27.656971167635263
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 9.49545 (QuantReg: 11.39944) QuantErr: 11.39944 batch_time=26.93728
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 11.05875 (QuantReg: 11.28277) QuantErr: 11.28277 batch_time=0.61810
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 11.00043 (QuantReg: 11.38139) QuantErr: 11.38139 batch_time=0.64511
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 10.87314 (QuantReg: 11.48408) QuantErr: 11.48408 batch_time=0.65558
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 11.14023 (QuantReg: 11.20611) QuantErr: 11.20611 batch_time=0.65368
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 9.83345 (QuantReg: 11.38654) QuantErr: 11.38654 batch_time=0.71611
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 9.91453 (QuantReg: 11.95096) QuantErr: 11.95096 batch_time=2.59774
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 8.89412 (QuantReg: 11.94427) QuantErr: 11.94427 batch_time=0.64662
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 9.71626 (QuantReg: 11.80034) QuantErr: 11.80034 batch_time=0.64002
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 8.74204 (QuantReg: 11.90429) QuantErr: 11.90429 batch_time=0.65492
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 9.86039 (QuantReg: 12.27766) QuantErr: 12.27766 batch_time=0.64684
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 10.13871 (QuantReg: 12.25556) QuantErr: 12.25556 batch_time=0.63686
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 9.73166 (QuantReg: 11.64230) QuantErr: 11.64230 batch_time=0.63435
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 10.28997 (QuantReg: 12.57906) QuantErr: 12.57906 batch_time=2.78622
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 9.05457 (QuantReg: 12.23904) QuantErr: 12.23904 batch_time=0.64270
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 8.81621 (QuantReg: 12.17571) QuantErr: 12.17571 batch_time=0.64344
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 9.42125 (QuantReg: 12.12571) QuantErr: 12.12571 batch_time=0.70177
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 8.15825 (QuantReg: 12.53174) QuantErr: 12.53174 batch_time=0.65352
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 9.38417 (QuantReg: 12.21573) QuantErr: 12.21573 batch_time=0.63213
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 9.74692 (QuantReg: 12.18688) QuantErr: 12.18688 batch_time=0.62755
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 8.66065 (QuantReg: 12.59224) QuantErr: 12.59224 batch_time=0.64293
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 9.15616 (QuantReg: 12.28885) QuantErr: 12.28885 batch_time=0.64316
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 8.38474 (QuantReg: 12.61662) QuantErr: 12.61662 batch_time=0.63248
Train Epoch: 3 codebook_update_time=4.37660
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch3.pth ...
Done in 4.479s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch3.pth ...
Done in 9.017s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 9.815468349456786
quant_reg : 12.028659381866454
quant_err : 12.028659381866454
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_miech_test/t2v_metrics/R1: 15.6
MSRVTT_miech_test/t2v_metrics/R5: 41.3
MSRVTT_miech_test/t2v_metrics/R10: 55.3
MSRVTT_miech_test/t2v_metrics/R50: 85.6
MSRVTT_miech_test/t2v_metrics/MedR: 8.0
MSRVTT_miech_test/t2v_metrics/MeanR: 35.469
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.90535600141167
MSRVTT_miech_test/v2t_metrics/R1: 14.5
MSRVTT_miech_test/v2t_metrics/R5: 40.8
MSRVTT_miech_test/v2t_metrics/R10: 53.7
MSRVTT_miech_test/v2t_metrics/R50: 85.1
MSRVTT_miech_test/v2t_metrics/MedR: 9.0
MSRVTT_miech_test/v2t_metrics/MeanR: 35.033
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.671416215885245
mnt_best : 32.90535600141167
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 9.95586 (QuantReg: 11.43717) QuantErr: 11.43717 batch_time=34.44657
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 9.45663 (QuantReg: 11.81666) QuantErr: 11.81666 batch_time=0.63390
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 8.77247 (QuantReg: 11.46492) QuantErr: 11.46492 batch_time=0.65352
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 9.75022 (QuantReg: 11.52619) QuantErr: 11.52619 batch_time=0.67600
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 9.31474 (QuantReg: 11.61248) QuantErr: 11.61248 batch_time=0.64558
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 8.64071 (QuantReg: 11.87337) QuantErr: 11.87337 batch_time=0.68841
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 8.20164 (QuantReg: 11.76822) QuantErr: 11.76822 batch_time=0.70795
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 8.10677 (QuantReg: 11.72428) QuantErr: 11.72428 batch_time=0.66093
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 8.93019 (QuantReg: 12.19901) QuantErr: 12.19901 batch_time=0.67939
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 9.09739 (QuantReg: 11.78308) QuantErr: 11.78308 batch_time=0.66521
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 8.99038 (QuantReg: 11.81912) QuantErr: 11.81912 batch_time=0.72688
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 8.36622 (QuantReg: 12.16789) QuantErr: 12.16789 batch_time=0.62798
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 10.06931 (QuantReg: 11.68462) QuantErr: 11.68462 batch_time=0.64895
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 9.52937 (QuantReg: 12.00682) QuantErr: 12.00682 batch_time=0.62785
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 8.46386 (QuantReg: 12.23695) QuantErr: 12.23695 batch_time=0.65241
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 8.66775 (QuantReg: 11.96685) QuantErr: 11.96685 batch_time=0.63390
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 9.32411 (QuantReg: 12.32828) QuantErr: 12.32828 batch_time=0.65090
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 8.62910 (QuantReg: 12.44669) QuantErr: 12.44669 batch_time=0.67237
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 8.15224 (QuantReg: 12.10043) QuantErr: 12.10043 batch_time=0.62940
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 7.59330 (QuantReg: 12.20562) QuantErr: 12.20562 batch_time=0.73591
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 8.61191 (QuantReg: 12.25133) QuantErr: 12.25133 batch_time=0.71401
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 8.70949 (QuantReg: 12.42079) QuantErr: 12.42079 batch_time=0.63205
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 9.33224 (QuantReg: 12.04047) QuantErr: 12.04047 batch_time=0.64380
Train Epoch: 4 codebook_update_time=3.47142
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch4.pth ...
Done in 4.279s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch4.pth ...
Done in 8.411s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 8.750454620361328
quant_reg : 11.919601528167725
quant_err : 11.919601528167725
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_miech_test/t2v_metrics/R1: 15.7
MSRVTT_miech_test/t2v_metrics/R5: 43.4
MSRVTT_miech_test/t2v_metrics/R10: 58.6
MSRVTT_miech_test/t2v_metrics/R50: 86.8
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.707
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.17923457004377
MSRVTT_miech_test/v2t_metrics/R1: 17.4
MSRVTT_miech_test/v2t_metrics/R5: 44.9
MSRVTT_miech_test/v2t_metrics/R10: 58.4
MSRVTT_miech_test/v2t_metrics/R50: 86.7
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 32.358000000000004
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.73299995357393
mnt_best : 34.17923457004377
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 8.43488 (QuantReg: 11.75522) QuantErr: 11.75522 batch_time=32.28847
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 8.65754 (QuantReg: 11.65528) QuantErr: 11.65528 batch_time=0.63740
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 9.18955 (QuantReg: 11.45765) QuantErr: 11.45765 batch_time=0.71788
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 7.24284 (QuantReg: 11.51961) QuantErr: 11.51961 batch_time=0.65006
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 9.68097 (QuantReg: 11.81750) QuantErr: 11.81750 batch_time=0.66365
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 6.27230 (QuantReg: 12.30429) QuantErr: 12.30429 batch_time=0.67240
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 8.62126 (QuantReg: 12.34266) QuantErr: 12.34266 batch_time=0.64543
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 7.43684 (QuantReg: 11.91767) QuantErr: 11.91767 batch_time=0.66184
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 7.39171 (QuantReg: 11.96817) QuantErr: 11.96817 batch_time=0.63790
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 8.06460 (QuantReg: 11.84412) QuantErr: 11.84412 batch_time=0.86159
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 9.90400 (QuantReg: 11.83829) QuantErr: 11.83829 batch_time=0.62329
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 7.69014 (QuantReg: 12.03321) QuantErr: 12.03321 batch_time=0.68050
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 7.58702 (QuantReg: 12.09043) QuantErr: 12.09043 batch_time=0.71378
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 9.41243 (QuantReg: 11.95747) QuantErr: 11.95747 batch_time=0.64176
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 6.92982 (QuantReg: 12.30404) QuantErr: 12.30404 batch_time=0.67506
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 8.48488 (QuantReg: 12.36776) QuantErr: 12.36776 batch_time=0.62834
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 9.22915 (QuantReg: 11.99240) QuantErr: 11.99240 batch_time=0.63097
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 8.99968 (QuantReg: 12.14054) QuantErr: 12.14054 batch_time=0.91075
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 6.82002 (QuantReg: 12.64826) QuantErr: 12.64826 batch_time=0.67835
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 7.32150 (QuantReg: 12.62496) QuantErr: 12.62496 batch_time=0.64046
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 7.24389 (QuantReg: 12.16343) QuantErr: 12.16343 batch_time=0.65416
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 7.92756 (QuantReg: 12.22746) QuantErr: 12.22746 batch_time=0.84962
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 6.41315 (QuantReg: 12.24972) QuantErr: 12.24972 batch_time=0.65333
Train Epoch: 5 codebook_update_time=3.57820
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch5.pth ...
Done in 4.533s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch5.pth ...
Done in 8.629s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 7.96667342376709
quant_reg : 12.042834522247315
quant_err : 12.042834522247315
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_miech_test/t2v_metrics/R1: 16.4
MSRVTT_miech_test/t2v_metrics/R5: 44.3
MSRVTT_miech_test/t2v_metrics/R10: 59.2
MSRVTT_miech_test/t2v_metrics/R50: 86.5
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 33.119
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.03669186118934
MSRVTT_miech_test/v2t_metrics/R1: 16.8
MSRVTT_miech_test/v2t_metrics/R5: 44.9
MSRVTT_miech_test/v2t_metrics/R10: 57.7
MSRVTT_miech_test/v2t_metrics/R50: 86.4
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 31.495
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.17578611926641
mnt_best : 35.03669186118934
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 8.01857 (QuantReg: 11.36134) QuantErr: 11.36134 batch_time=29.19497
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 7.34834 (QuantReg: 11.48024) QuantErr: 11.48024 batch_time=0.65888
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 8.52067 (QuantReg: 12.06248) QuantErr: 12.06248 batch_time=0.69900
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 7.88142 (QuantReg: 11.79549) QuantErr: 11.79549 batch_time=0.63614
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 8.67953 (QuantReg: 12.15673) QuantErr: 12.15673 batch_time=0.63069
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 7.24355 (QuantReg: 12.20405) QuantErr: 12.20405 batch_time=1.00368
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 6.30516 (QuantReg: 12.12824) QuantErr: 12.12824 batch_time=0.63026
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 7.92305 (QuantReg: 12.03600) QuantErr: 12.03600 batch_time=0.63303
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 8.73173 (QuantReg: 11.64845) QuantErr: 11.64845 batch_time=0.62404
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 7.68117 (QuantReg: 11.89384) QuantErr: 11.89384 batch_time=0.97010
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 7.66547 (QuantReg: 12.24794) QuantErr: 12.24794 batch_time=0.63638
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 9.36050 (QuantReg: 12.09120) QuantErr: 12.09120 batch_time=0.64386
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 7.85491 (QuantReg: 11.89460) QuantErr: 11.89460 batch_time=0.64173
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 7.84455 (QuantReg: 12.07318) QuantErr: 12.07318 batch_time=0.64559
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 7.78391 (QuantReg: 12.14619) QuantErr: 12.14619 batch_time=0.63206
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 8.02300 (QuantReg: 12.29819) QuantErr: 12.29819 batch_time=0.72707
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 7.33575 (QuantReg: 12.55387) QuantErr: 12.55387 batch_time=0.67476
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 7.35269 (QuantReg: 12.51917) QuantErr: 12.51917 batch_time=0.65406
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 7.00634 (QuantReg: 12.50040) QuantErr: 12.50040 batch_time=0.67520
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 6.43492 (QuantReg: 12.28601) QuantErr: 12.28601 batch_time=0.62759
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 7.25512 (QuantReg: 11.88626) QuantErr: 11.88626 batch_time=0.66608
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 6.77237 (QuantReg: 12.12461) QuantErr: 12.12461 batch_time=0.64822
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 6.93786 (QuantReg: 12.45413) QuantErr: 12.45413 batch_time=0.65232
Train Epoch: 6 codebook_update_time=4.65074
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch6.pth ...
Done in 4.235s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch6.pth ...
Done in 8.437s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 7.371531064987183
quant_reg : 12.14830372619629
quant_err : 12.14830372619629
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_miech_test/t2v_metrics/R1: 17.9
MSRVTT_miech_test/t2v_metrics/R5: 45.8
MSRVTT_miech_test/t2v_metrics/R10: 60.3
MSRVTT_miech_test/t2v_metrics/R50: 87.6
MSRVTT_miech_test/t2v_metrics/MedR: 7.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.705
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.701059942114526
MSRVTT_miech_test/v2t_metrics/R1: 16.9
MSRVTT_miech_test/v2t_metrics/R5: 45.6
MSRVTT_miech_test/v2t_metrics/R10: 60.4
MSRVTT_miech_test/v2t_metrics/R50: 87.3
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 30.5595
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.97185454431453
mnt_best : 36.701059942114526
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 7.38761 (QuantReg: 11.68119) QuantErr: 11.68119 batch_time=30.47343
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 7.34708 (QuantReg: 11.74147) QuantErr: 11.74147 batch_time=0.63154
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 6.10911 (QuantReg: 11.78413) QuantErr: 11.78413 batch_time=0.64700
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 7.20609 (QuantReg: 12.18957) QuantErr: 12.18957 batch_time=0.64432
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 6.12544 (QuantReg: 12.25819) QuantErr: 12.25819 batch_time=0.91258
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 7.49805 (QuantReg: 12.00539) QuantErr: 12.00539 batch_time=0.67943
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 7.30511 (QuantReg: 12.22197) QuantErr: 12.22197 batch_time=0.64231
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 7.36280 (QuantReg: 12.30165) QuantErr: 12.30165 batch_time=1.96831
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 5.79154 (QuantReg: 12.31665) QuantErr: 12.31665 batch_time=0.64654
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 6.48982 (QuantReg: 12.37169) QuantErr: 12.37169 batch_time=0.63871
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 7.21611 (QuantReg: 12.70041) QuantErr: 12.70041 batch_time=0.69091
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 5.19376 (QuantReg: 12.27162) QuantErr: 12.27162 batch_time=0.68144
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 8.00310 (QuantReg: 11.99634) QuantErr: 11.99634 batch_time=0.65765
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 7.45063 (QuantReg: 12.70155) QuantErr: 12.70155 batch_time=0.62417
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 6.08488 (QuantReg: 12.34080) QuantErr: 12.34080 batch_time=0.63242
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 6.85597 (QuantReg: 12.35958) QuantErr: 12.35958 batch_time=0.63990
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 6.87615 (QuantReg: 12.34423) QuantErr: 12.34423 batch_time=0.68316
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 5.86950 (QuantReg: 12.26682) QuantErr: 12.26682 batch_time=0.69273
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 5.51728 (QuantReg: 12.32967) QuantErr: 12.32967 batch_time=0.63860
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 5.26252 (QuantReg: 12.13042) QuantErr: 12.13042 batch_time=1.44542
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 6.69096 (QuantReg: 12.53367) QuantErr: 12.53367 batch_time=0.65612
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 6.53334 (QuantReg: 12.54523) QuantErr: 12.54523 batch_time=0.62745
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 6.83718 (QuantReg: 12.51678) QuantErr: 12.51678 batch_time=0.66200
Train Epoch: 7 codebook_update_time=3.67097
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch7.pth ...
Done in 5.258s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch7.pth ...
Done in 10.011s
removing stale ckpt [epoch 6] [took 0.02s]
epoch : 7
loss : 6.74201238822937
quant_reg : 12.275982173919678
quant_err : 12.275982173919678
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_miech_test/t2v_metrics/R1: 18.3
MSRVTT_miech_test/t2v_metrics/R5: 46.0
MSRVTT_miech_test/t2v_metrics/R10: 59.2
MSRVTT_miech_test/t2v_metrics/R50: 88.4
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.26
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.79963767759212
MSRVTT_miech_test/v2t_metrics/R1: 19.1
MSRVTT_miech_test/v2t_metrics/R5: 47.5
MSRVTT_miech_test/v2t_metrics/R10: 58.8
MSRVTT_miech_test/v2t_metrics/R50: 89.4
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 29.0235
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.64449143318185
mnt_best : 36.79963767759212
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 5.40632 (QuantReg: 12.02589) QuantErr: 12.02589 batch_time=31.67717
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 5.73264 (QuantReg: 12.16480) QuantErr: 12.16480 batch_time=0.64074
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 7.47606 (QuantReg: 12.20769) QuantErr: 12.20769 batch_time=0.65323
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 6.46140 (QuantReg: 12.11577) QuantErr: 12.11577 batch_time=0.65134
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 5.51199 (QuantReg: 12.14913) QuantErr: 12.14913 batch_time=0.64378
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 5.52029 (QuantReg: 12.19058) QuantErr: 12.19058 batch_time=0.63750
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 6.32179 (QuantReg: 12.36197) QuantErr: 12.36197 batch_time=0.63280
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 8.39770 (QuantReg: 12.29679) QuantErr: 12.29679 batch_time=0.64045
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 7.18571 (QuantReg: 11.92758) QuantErr: 11.92758 batch_time=0.66062
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 7.89364 (QuantReg: 12.32843) QuantErr: 12.32843 batch_time=0.64293
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 6.83182 (QuantReg: 12.27356) QuantErr: 12.27356 batch_time=0.64677
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 6.39071 (QuantReg: 12.32953) QuantErr: 12.32953 batch_time=0.69575
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 6.18856 (QuantReg: 12.69085) QuantErr: 12.69085 batch_time=2.21858
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 6.40136 (QuantReg: 12.24430) QuantErr: 12.24430 batch_time=0.63712
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 6.44095 (QuantReg: 12.23380) QuantErr: 12.23380 batch_time=0.64904
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 5.16478 (QuantReg: 12.66831) QuantErr: 12.66831 batch_time=2.42129
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 4.79673 (QuantReg: 12.41941) QuantErr: 12.41941 batch_time=0.64509
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 6.82483 (QuantReg: 12.52168) QuantErr: 12.52168 batch_time=1.14131
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 7.26264 (QuantReg: 12.31507) QuantErr: 12.31507 batch_time=0.63677
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 7.25464 (QuantReg: 12.69147) QuantErr: 12.69147 batch_time=0.62856
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 6.98725 (QuantReg: 12.98051) QuantErr: 12.98051 batch_time=0.94162
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 5.78148 (QuantReg: 12.28283) QuantErr: 12.28283 batch_time=0.64206
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 5.90814 (QuantReg: 12.65209) QuantErr: 12.65209 batch_time=0.62555
Train Epoch: 8 codebook_update_time=3.83647
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch8.pth ...
Done in 4.295s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 6.381667308807373
quant_reg : 12.33930772781372
quant_err : 12.33930772781372
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_miech_test/t2v_metrics/R1: 17.5
MSRVTT_miech_test/t2v_metrics/R5: 46.5
MSRVTT_miech_test/t2v_metrics/R10: 60.2
MSRVTT_miech_test/t2v_metrics/R50: 87.0
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 31.09
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.59000742465094
MSRVTT_miech_test/v2t_metrics/R1: 19.8
MSRVTT_miech_test/v2t_metrics/R5: 46.3
MSRVTT_miech_test/v2t_metrics/R10: 58.9
MSRVTT_miech_test/v2t_metrics/R50: 87.7
MSRVTT_miech_test/v2t_metrics/MedR: 7.0
MSRVTT_miech_test/v2t_metrics/MeanR: 29.0315
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.79669493232607
mnt_best : 36.79963767759212
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 5.88398 (QuantReg: 12.24390) QuantErr: 12.24390 batch_time=27.70824
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 6.38424 (QuantReg: 12.23958) QuantErr: 12.23958 batch_time=0.65391
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 5.61115 (QuantReg: 12.00058) QuantErr: 12.00058 batch_time=0.62437
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 6.97017 (QuantReg: 12.35796) QuantErr: 12.35796 batch_time=0.68653
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 5.52793 (QuantReg: 12.60757) QuantErr: 12.60757 batch_time=0.65838
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 6.30147 (QuantReg: 12.68894) QuantErr: 12.68894 batch_time=0.63272
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 5.46373 (QuantReg: 12.40987) QuantErr: 12.40987 batch_time=0.63335
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 5.70680 (QuantReg: 12.74856) QuantErr: 12.74856 batch_time=0.63910
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 6.30557 (QuantReg: 12.16518) QuantErr: 12.16518 batch_time=3.00530
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 6.12711 (QuantReg: 12.21216) QuantErr: 12.21216 batch_time=0.71697
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 5.46004 (QuantReg: 11.85958) QuantErr: 11.85958 batch_time=0.76633
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 5.89902 (QuantReg: 12.66330) QuantErr: 12.66330 batch_time=0.65883
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 6.27093 (QuantReg: 12.45356) QuantErr: 12.45356 batch_time=0.64713
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 6.36871 (QuantReg: 12.55412) QuantErr: 12.55412 batch_time=0.65987
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 4.95003 (QuantReg: 12.66613) QuantErr: 12.66613 batch_time=0.71599
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 5.60659 (QuantReg: 12.94933) QuantErr: 12.94933 batch_time=1.44366
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 5.88619 (QuantReg: 12.73657) QuantErr: 12.73657 batch_time=0.69037
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 6.62656 (QuantReg: 12.15806) QuantErr: 12.15806 batch_time=0.64864
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 6.12839 (QuantReg: 12.00251) QuantErr: 12.00251 batch_time=0.65871
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 6.15741 (QuantReg: 12.51546) QuantErr: 12.51546 batch_time=0.63860
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 6.88172 (QuantReg: 12.61511) QuantErr: 12.61511 batch_time=0.66614
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 6.72433 (QuantReg: 12.46230) QuantErr: 12.46230 batch_time=0.66243
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 5.30616 (QuantReg: 12.48683) QuantErr: 12.48683 batch_time=0.65761
Train Epoch: 9 codebook_update_time=3.48899
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch9.pth ...
Done in 5.610s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch9.pth ...
Done in 10.733s
removing stale ckpt [epoch 8] [took 0.02s]
epoch : 9
loss : 5.981542764663696
quant_reg : 12.434980407714844
quant_err : 12.434980407714844
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_miech_test/t2v_metrics/R1: 19.3
MSRVTT_miech_test/t2v_metrics/R5: 47.9
MSRVTT_miech_test/t2v_metrics/R10: 61.8
MSRVTT_miech_test/t2v_metrics/R50: 88.6
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.238
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.51475141698836
MSRVTT_miech_test/v2t_metrics/R1: 20.0
MSRVTT_miech_test/v2t_metrics/R5: 48.7
MSRVTT_miech_test/v2t_metrics/R10: 62.5
MSRVTT_miech_test/v2t_metrics/R50: 88.7
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.663
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.338064801564244
mnt_best : 38.51475141698836
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 5.06531 (QuantReg: 12.18162) QuantErr: 12.18162 batch_time=30.73231
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 4.70044 (QuantReg: 12.34738) QuantErr: 12.34738 batch_time=0.63608
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 6.20942 (QuantReg: 12.09101) QuantErr: 12.09101 batch_time=0.92081
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 6.75641 (QuantReg: 12.03175) QuantErr: 12.03175 batch_time=0.65525
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 5.91520 (QuantReg: 12.07951) QuantErr: 12.07951 batch_time=1.52150
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 6.23966 (QuantReg: 12.25438) QuantErr: 12.25438 batch_time=0.63582
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 5.32675 (QuantReg: 12.33780) QuantErr: 12.33780 batch_time=0.62681
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 6.84765 (QuantReg: 12.52526) QuantErr: 12.52526 batch_time=0.64302
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 5.48064 (QuantReg: 12.56569) QuantErr: 12.56569 batch_time=0.64169
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 5.22060 (QuantReg: 12.69082) QuantErr: 12.69082 batch_time=0.63023
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 5.63156 (QuantReg: 12.49084) QuantErr: 12.49084 batch_time=0.63475
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 5.40071 (QuantReg: 12.82939) QuantErr: 12.82939 batch_time=0.66678
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 5.06102 (QuantReg: 12.56808) QuantErr: 12.56808 batch_time=0.62858
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 6.44689 (QuantReg: 12.85540) QuantErr: 12.85540 batch_time=0.66503
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 5.24281 (QuantReg: 12.06362) QuantErr: 12.06362 batch_time=0.64863
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 5.49830 (QuantReg: 12.44875) QuantErr: 12.44875 batch_time=0.64368
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 5.32168 (QuantReg: 12.65851) QuantErr: 12.65851 batch_time=0.66500
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 6.00701 (QuantReg: 12.27586) QuantErr: 12.27586 batch_time=0.63643
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 5.53831 (QuantReg: 12.78490) QuantErr: 12.78490 batch_time=0.63057
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 4.81142 (QuantReg: 12.57209) QuantErr: 12.57209 batch_time=0.67170
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 6.04675 (QuantReg: 12.51268) QuantErr: 12.51268 batch_time=0.65312
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.95560 (QuantReg: 12.38401) QuantErr: 12.38401 batch_time=0.64939
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 6.63661 (QuantReg: 12.51961) QuantErr: 12.51961 batch_time=0.63459
Train Epoch: 10 codebook_update_time=4.05802
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch10.pth ...
Done in 4.818s
removing stale ckpt [epoch 9] [took 0.02s]
epoch : 10
loss : 5.75596855545044
quant_reg : 12.47993877029419
quant_err : 12.47993877029419
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_miech_test/t2v_metrics/R1: 18.5
MSRVTT_miech_test/t2v_metrics/R5: 47.5
MSRVTT_miech_test/t2v_metrics/R10: 61.5
MSRVTT_miech_test/t2v_metrics/R50: 89.0
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.603
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.807690689022714
MSRVTT_miech_test/v2t_metrics/R1: 19.0
MSRVTT_miech_test/v2t_metrics/R5: 48.1
MSRVTT_miech_test/v2t_metrics/R10: 60.4
MSRVTT_miech_test/v2t_metrics/R50: 88.8
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 27.114
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.07546407203648
mnt_best : 38.51475141698836
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 4.72782 (QuantReg: 12.53468) QuantErr: 12.53468 batch_time=30.68171
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 6.74400 (QuantReg: 12.09239) QuantErr: 12.09239 batch_time=0.73137
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 5.32017 (QuantReg: 12.59477) QuantErr: 12.59477 batch_time=0.63851
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 5.17171 (QuantReg: 12.36275) QuantErr: 12.36275 batch_time=0.63961
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 5.26304 (QuantReg: 12.56711) QuantErr: 12.56711 batch_time=0.72445
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 5.60902 (QuantReg: 12.37270) QuantErr: 12.37270 batch_time=0.64078
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 5.44188 (QuantReg: 12.34559) QuantErr: 12.34559 batch_time=0.63319
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 6.71578 (QuantReg: 12.62985) QuantErr: 12.62985 batch_time=2.05710
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 6.95625 (QuantReg: 12.46245) QuantErr: 12.46245 batch_time=0.64043
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 6.42116 (QuantReg: 12.43430) QuantErr: 12.43430 batch_time=0.66884
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 5.85555 (QuantReg: 12.57446) QuantErr: 12.57446 batch_time=0.64654
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 5.86463 (QuantReg: 12.55360) QuantErr: 12.55360 batch_time=0.63743
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 5.27140 (QuantReg: 12.48515) QuantErr: 12.48515 batch_time=0.63206
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 5.95411 (QuantReg: 12.39728) QuantErr: 12.39728 batch_time=1.85505
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 4.36812 (QuantReg: 12.54506) QuantErr: 12.54506 batch_time=0.66567
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 6.41982 (QuantReg: 12.15127) QuantErr: 12.15127 batch_time=0.68903
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 5.04852 (QuantReg: 12.72804) QuantErr: 12.72804 batch_time=0.63805
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 5.68386 (QuantReg: 12.62802) QuantErr: 12.62802 batch_time=0.63778
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 5.29692 (QuantReg: 12.65523) QuantErr: 12.65523 batch_time=2.13724
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 6.20204 (QuantReg: 12.39035) QuantErr: 12.39035 batch_time=0.84226
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 5.25991 (QuantReg: 12.70519) QuantErr: 12.70519 batch_time=0.64618
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 6.50590 (QuantReg: 12.40167) QuantErr: 12.40167 batch_time=0.70509
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 5.47666 (QuantReg: 12.27514) QuantErr: 12.27514 batch_time=0.70344
Train Epoch: 11 codebook_update_time=3.52193
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch11.pth ...
Done in 5.952s
removing stale ckpt [epoch 10] [took 0.23s]
epoch : 11
loss : 5.508341112136841
quant_reg : 12.47500581741333
quant_err : 12.47500581741333
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_miech_test/t2v_metrics/R1: 18.8
MSRVTT_miech_test/t2v_metrics/R5: 46.5
MSRVTT_miech_test/t2v_metrics/R10: 61.6
MSRVTT_miech_test/t2v_metrics/R50: 89.1
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.1115
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.762769534963525
MSRVTT_miech_test/v2t_metrics/R1: 19.1
MSRVTT_miech_test/v2t_metrics/R5: 47.9
MSRVTT_miech_test/v2t_metrics/R10: 62.6
MSRVTT_miech_test/v2t_metrics/R50: 88.9
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.8425
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.54615569717977
mnt_best : 38.51475141698836
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 5.31581 (QuantReg: 12.49293) QuantErr: 12.49293 batch_time=28.69542
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 6.13011 (QuantReg: 12.70043) QuantErr: 12.70043 batch_time=0.64719
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 5.73037 (QuantReg: 12.32647) QuantErr: 12.32647 batch_time=0.65350
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 5.05145 (QuantReg: 12.67412) QuantErr: 12.67412 batch_time=0.62502
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 6.03077 (QuantReg: 12.18010) QuantErr: 12.18010 batch_time=2.48776
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 5.03432 (QuantReg: 12.44452) QuantErr: 12.44452 batch_time=0.65113
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 4.69367 (QuantReg: 12.72635) QuantErr: 12.72635 batch_time=2.74082
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 6.16388 (QuantReg: 12.42298) QuantErr: 12.42298 batch_time=0.65534
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 5.51160 (QuantReg: 12.80596) QuantErr: 12.80596 batch_time=0.64190
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 6.20075 (QuantReg: 12.75586) QuantErr: 12.75586 batch_time=0.63069
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 4.53026 (QuantReg: 12.88389) QuantErr: 12.88389 batch_time=0.66397
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 4.64195 (QuantReg: 12.67775) QuantErr: 12.67775 batch_time=0.64960
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 5.80877 (QuantReg: 12.31502) QuantErr: 12.31502 batch_time=1.21383
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 5.10717 (QuantReg: 12.43227) QuantErr: 12.43227 batch_time=0.62819
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 4.51483 (QuantReg: 12.79267) QuantErr: 12.79267 batch_time=0.69208
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 4.18677 (QuantReg: 12.75775) QuantErr: 12.75775 batch_time=0.63758
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 4.97196 (QuantReg: 12.59170) QuantErr: 12.59170 batch_time=0.63579
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 4.67821 (QuantReg: 12.72661) QuantErr: 12.72661 batch_time=0.70094
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 5.24079 (QuantReg: 12.29863) QuantErr: 12.29863 batch_time=0.64661
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 5.84827 (QuantReg: 12.62643) QuantErr: 12.62643 batch_time=0.67699
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 5.65791 (QuantReg: 12.90989) QuantErr: 12.90989 batch_time=0.63909
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 4.39569 (QuantReg: 12.87168) QuantErr: 12.87168 batch_time=0.67036
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 5.96714 (QuantReg: 12.75036) QuantErr: 12.75036 batch_time=0.69740
Train Epoch: 12 codebook_update_time=4.08440
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch12.pth ...
Done in 6.610s
removing stale ckpt [epoch 11] [took 0.03s]
epoch : 12
loss : 5.252197081565857
quant_reg : 12.569703632354736
quant_err : 12.569703632354736
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_miech_test/t2v_metrics/R1: 19.5
MSRVTT_miech_test/t2v_metrics/R5: 46.0
MSRVTT_miech_test/t2v_metrics/R10: 60.9
MSRVTT_miech_test/t2v_metrics/R50: 88.1
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.525
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.94342921331847
MSRVTT_miech_test/v2t_metrics/R1: 20.4
MSRVTT_miech_test/v2t_metrics/R5: 49.2
MSRVTT_miech_test/v2t_metrics/R10: 64.1
MSRVTT_miech_test/v2t_metrics/R50: 88.8
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.2715
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.06985460401014
mnt_best : 38.51475141698836
not_improved_count: 3
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 5.32856 (QuantReg: 12.43824) QuantErr: 12.43824 batch_time=29.47369
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 4.96941 (QuantReg: 12.55625) QuantErr: 12.55625 batch_time=0.62594
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 4.77570 (QuantReg: 12.40489) QuantErr: 12.40489 batch_time=1.11902
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 4.57266 (QuantReg: 12.44008) QuantErr: 12.44008 batch_time=0.63409
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 5.89126 (QuantReg: 12.51172) QuantErr: 12.51172 batch_time=0.65928
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 5.01783 (QuantReg: 12.72822) QuantErr: 12.72822 batch_time=0.64444
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 4.44453 (QuantReg: 12.85304) QuantErr: 12.85304 batch_time=0.71253
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 4.78217 (QuantReg: 12.42903) QuantErr: 12.42903 batch_time=1.11187
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 5.83799 (QuantReg: 12.50689) QuantErr: 12.50689 batch_time=0.62791
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 5.41102 (QuantReg: 12.29621) QuantErr: 12.29621 batch_time=0.63283
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 5.12669 (QuantReg: 12.37909) QuantErr: 12.37909 batch_time=0.66719
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 4.61088 (QuantReg: 12.40455) QuantErr: 12.40455 batch_time=0.67168
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 3.95842 (QuantReg: 12.85599) QuantErr: 12.85599 batch_time=0.63815
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 4.89192 (QuantReg: 12.71562) QuantErr: 12.71562 batch_time=2.41609
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 4.85639 (QuantReg: 12.67607) QuantErr: 12.67607 batch_time=0.63005
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 5.34308 (QuantReg: 12.82755) QuantErr: 12.82755 batch_time=0.63985
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 5.10771 (QuantReg: 12.79169) QuantErr: 12.79169 batch_time=0.63118
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 4.47230 (QuantReg: 12.67674) QuantErr: 12.67674 batch_time=0.66322
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 3.99017 (QuantReg: 12.67794) QuantErr: 12.67794 batch_time=0.72149
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 4.91208 (QuantReg: 12.36406) QuantErr: 12.36406 batch_time=0.62774
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 5.49836 (QuantReg: 12.60361) QuantErr: 12.60361 batch_time=0.72035
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 5.06477 (QuantReg: 12.63212) QuantErr: 12.63212 batch_time=0.63387
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 4.71028 (QuantReg: 12.65011) QuantErr: 12.65011 batch_time=0.68340
Train Epoch: 13 codebook_update_time=3.64766
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch13.pth ...
Done in 5.141s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch13.pth ...
Done in 10.599s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 5.069659412384033
quant_reg : 12.603925762176514
quant_err : 12.603925762176514
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_miech_test/t2v_metrics/R1: 20.1
MSRVTT_miech_test/t2v_metrics/R5: 48.5
MSRVTT_miech_test/t2v_metrics/R10: 61.2
MSRVTT_miech_test/t2v_metrics/R50: 88.6
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.959
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.07476781188362
MSRVTT_miech_test/v2t_metrics/R1: 20.1
MSRVTT_miech_test/v2t_metrics/R5: 48.5
MSRVTT_miech_test/v2t_metrics/R10: 63.4
MSRVTT_miech_test/v2t_metrics/R50: 88.8
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 25.8705
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.53748297985253
mnt_best : 39.07476781188362
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 5.93341 (QuantReg: 12.51245) QuantErr: 12.51245 batch_time=28.42174
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 4.67323 (QuantReg: 12.45599) QuantErr: 12.45599 batch_time=0.63816
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 4.43546 (QuantReg: 12.73703) QuantErr: 12.73703 batch_time=0.67221
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 5.82549 (QuantReg: 12.56698) QuantErr: 12.56698 batch_time=1.65522
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 4.33368 (QuantReg: 12.23907) QuantErr: 12.23907 batch_time=0.72769
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 5.10248 (QuantReg: 12.64645) QuantErr: 12.64645 batch_time=0.64510
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 6.14907 (QuantReg: 12.64393) QuantErr: 12.64393 batch_time=1.39908
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 4.67687 (QuantReg: 12.60045) QuantErr: 12.60045 batch_time=1.52242
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 4.56038 (QuantReg: 12.71849) QuantErr: 12.71849 batch_time=0.63428
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 4.18950 (QuantReg: 12.73294) QuantErr: 12.73294 batch_time=0.66538
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 4.77307 (QuantReg: 12.73256) QuantErr: 12.73256 batch_time=0.63677
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 4.50929 (QuantReg: 12.74086) QuantErr: 12.74086 batch_time=0.62678
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 5.78560 (QuantReg: 12.41432) QuantErr: 12.41432 batch_time=0.62373
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 4.55487 (QuantReg: 13.02633) QuantErr: 13.02633 batch_time=0.63917
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 4.25727 (QuantReg: 12.53586) QuantErr: 12.53586 batch_time=0.65065
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 5.29713 (QuantReg: 12.39441) QuantErr: 12.39441 batch_time=0.64222
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 4.19551 (QuantReg: 13.00608) QuantErr: 13.00608 batch_time=0.64275
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 5.49174 (QuantReg: 12.67465) QuantErr: 12.67465 batch_time=0.63363
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 5.24111 (QuantReg: 12.73209) QuantErr: 12.73209 batch_time=0.63344
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 3.75944 (QuantReg: 12.78055) QuantErr: 12.78055 batch_time=0.63832
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 5.05081 (QuantReg: 12.92829) QuantErr: 12.92829 batch_time=0.66464
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 4.09971 (QuantReg: 12.95844) QuantErr: 12.95844 batch_time=0.62720
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 4.47421 (QuantReg: 12.59157) QuantErr: 12.59157 batch_time=0.63271
Train Epoch: 14 codebook_update_time=3.95059
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch14.pth ...
Done in 6.331s
removing stale ckpt [epoch 13] [took 0.22s]
epoch : 14
loss : 4.81339208316803
quant_reg : 12.64384955215454
quant_err : 12.64384955215454
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_miech_test/t2v_metrics/R1: 19.3
MSRVTT_miech_test/t2v_metrics/R5: 48.0
MSRVTT_miech_test/t2v_metrics/R10: 61.5
MSRVTT_miech_test/t2v_metrics/R50: 87.5
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.57
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.47906884963452
MSRVTT_miech_test/v2t_metrics/R1: 20.5
MSRVTT_miech_test/v2t_metrics/R5: 49.6
MSRVTT_miech_test/v2t_metrics/R10: 62.1
MSRVTT_miech_test/v2t_metrics/R50: 88.2
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.305
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.820714283052226
mnt_best : 39.07476781188362
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 3.62053 (QuantReg: 12.60906) QuantErr: 12.60906 batch_time=28.57110
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 6.13073 (QuantReg: 12.52476) QuantErr: 12.52476 batch_time=1.10477
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 4.84467 (QuantReg: 12.64946) QuantErr: 12.64946 batch_time=0.69158
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 4.50730 (QuantReg: 12.23284) QuantErr: 12.23284 batch_time=0.62999
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 3.28436 (QuantReg: 13.00257) QuantErr: 13.00257 batch_time=0.64138
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 4.68180 (QuantReg: 12.58694) QuantErr: 12.58694 batch_time=0.63831
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 3.79511 (QuantReg: 13.03285) QuantErr: 13.03285 batch_time=1.04970
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 5.03548 (QuantReg: 12.36406) QuantErr: 12.36406 batch_time=0.63017
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 5.62128 (QuantReg: 12.78338) QuantErr: 12.78338 batch_time=0.62811
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 4.64559 (QuantReg: 12.76362) QuantErr: 12.76362 batch_time=0.72510
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 5.33640 (QuantReg: 12.66886) QuantErr: 12.66886 batch_time=0.63949
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 5.09169 (QuantReg: 12.96065) QuantErr: 12.96065 batch_time=0.62735
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 4.40499 (QuantReg: 12.60138) QuantErr: 12.60138 batch_time=0.63568
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 4.75489 (QuantReg: 12.95210) QuantErr: 12.95210 batch_time=3.30652
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 4.44041 (QuantReg: 12.74977) QuantErr: 12.74977 batch_time=0.69469
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 5.97204 (QuantReg: 12.53591) QuantErr: 12.53591 batch_time=0.64016
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 4.55170 (QuantReg: 12.64732) QuantErr: 12.64732 batch_time=1.01979
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 5.16792 (QuantReg: 12.62196) QuantErr: 12.62196 batch_time=0.63815
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 3.38058 (QuantReg: 12.79797) QuantErr: 12.79797 batch_time=0.63636
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 4.40670 (QuantReg: 13.01795) QuantErr: 13.01795 batch_time=0.63808
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 6.25501 (QuantReg: 12.49385) QuantErr: 12.49385 batch_time=0.62426
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 5.58575 (QuantReg: 12.39656) QuantErr: 12.39656 batch_time=0.65656
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 5.53962 (QuantReg: 12.78735) QuantErr: 12.78735 batch_time=0.66956
Train Epoch: 15 codebook_update_time=3.84417
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch15.pth ...
Done in 4.860s
removing stale ckpt [epoch 14] [took 0.04s]
epoch : 15
loss : 4.705581805229187
quant_reg : 12.682234455108643
quant_err : 12.682234455108643
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_miech_test/t2v_metrics/R1: 19.8
MSRVTT_miech_test/t2v_metrics/R5: 47.4
MSRVTT_miech_test/t2v_metrics/R10: 61.4
MSRVTT_miech_test/t2v_metrics/R50: 87.9
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.415
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.62519049769372
MSRVTT_miech_test/v2t_metrics/R1: 19.5
MSRVTT_miech_test/v2t_metrics/R5: 50.6
MSRVTT_miech_test/v2t_metrics/R10: 62.8
MSRVTT_miech_test/v2t_metrics/R50: 88.3
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.5015
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.57141596055972
mnt_best : 39.07476781188362
not_improved_count: 2
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 3.62637 (QuantReg: 12.89250) QuantErr: 12.89250 batch_time=31.38013
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 4.31736 (QuantReg: 12.75611) QuantErr: 12.75611 batch_time=0.62670
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 4.42129 (QuantReg: 11.98025) QuantErr: 11.98025 batch_time=0.66218
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 4.58147 (QuantReg: 12.95976) QuantErr: 12.95976 batch_time=0.62782
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 3.56435 (QuantReg: 12.42188) QuantErr: 12.42188 batch_time=0.65321
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 5.69275 (QuantReg: 12.73623) QuantErr: 12.73623 batch_time=0.71891
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 3.89432 (QuantReg: 12.91975) QuantErr: 12.91975 batch_time=0.63384
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 3.23462 (QuantReg: 12.68320) QuantErr: 12.68320 batch_time=0.63756
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 4.44312 (QuantReg: 12.92489) QuantErr: 12.92489 batch_time=0.62845
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 4.23854 (QuantReg: 12.90034) QuantErr: 12.90034 batch_time=0.63904
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 5.21797 (QuantReg: 12.67069) QuantErr: 12.67069 batch_time=0.64584
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 3.72398 (QuantReg: 12.87188) QuantErr: 12.87188 batch_time=0.65542
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 4.61379 (QuantReg: 12.75956) QuantErr: 12.75956 batch_time=0.64858
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 4.12380 (QuantReg: 12.99292) QuantErr: 12.99292 batch_time=2.15255
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 4.79948 (QuantReg: 12.42040) QuantErr: 12.42040 batch_time=0.64926
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 4.06979 (QuantReg: 13.03985) QuantErr: 13.03985 batch_time=0.64459
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 4.75918 (QuantReg: 12.40113) QuantErr: 12.40113 batch_time=0.62775
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 3.65214 (QuantReg: 12.79558) QuantErr: 12.79558 batch_time=0.64950
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 5.81156 (QuantReg: 12.67713) QuantErr: 12.67713 batch_time=0.64984
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 4.84594 (QuantReg: 12.36116) QuantErr: 12.36116 batch_time=0.62897
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 5.33588 (QuantReg: 12.72148) QuantErr: 12.72148 batch_time=0.71660
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 4.46725 (QuantReg: 12.61105) QuantErr: 12.61105 batch_time=0.65476
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 4.29431 (QuantReg: 12.82110) QuantErr: 12.82110 batch_time=0.64098
Train Epoch: 16 codebook_update_time=3.73073
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch16.pth ...
Done in 4.777s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch16.pth ...
Done in 9.901s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 4.538355688095093
quant_reg : 12.694468570709228
quant_err : 12.694468570709228
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
MSRVTT_miech_test/t2v_metrics/R1: 21.0
MSRVTT_miech_test/t2v_metrics/R5: 49.1
MSRVTT_miech_test/t2v_metrics/R10: 63.1
MSRVTT_miech_test/t2v_metrics/R50: 88.0
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.563
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 40.2201218538813
MSRVTT_miech_test/v2t_metrics/R1: 20.9
MSRVTT_miech_test/v2t_metrics/R5: 49.9
MSRVTT_miech_test/v2t_metrics/R10: 64.2
MSRVTT_miech_test/v2t_metrics/R50: 88.1
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 25.8495
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 40.606349965726835
mnt_best : 40.2201218538813
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 4.61035 (QuantReg: 12.48800) QuantErr: 12.48800 batch_time=29.77256
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 4.88521 (QuantReg: 12.34406) QuantErr: 12.34406 batch_time=0.66711
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 3.35129 (QuantReg: 12.83429) QuantErr: 12.83429 batch_time=0.67253
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 4.56746 (QuantReg: 12.41560) QuantErr: 12.41560 batch_time=0.64074
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 4.05545 (QuantReg: 12.73616) QuantErr: 12.73616 batch_time=0.64600
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 3.97810 (QuantReg: 12.75148) QuantErr: 12.75148 batch_time=0.62420
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 4.01075 (QuantReg: 12.23201) QuantErr: 12.23201 batch_time=0.78816
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 4.81825 (QuantReg: 12.69054) QuantErr: 12.69054 batch_time=0.69248
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 3.70709 (QuantReg: 12.95908) QuantErr: 12.95908 batch_time=0.64979
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 2.95922 (QuantReg: 12.71634) QuantErr: 12.71634 batch_time=0.91694
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 4.48582 (QuantReg: 12.54893) QuantErr: 12.54893 batch_time=0.65126
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 4.74044 (QuantReg: 12.84804) QuantErr: 12.84804 batch_time=0.77820
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 4.01564 (QuantReg: 12.57850) QuantErr: 12.57850 batch_time=1.32268
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 4.31147 (QuantReg: 12.64573) QuantErr: 12.64573 batch_time=0.64629
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 4.78072 (QuantReg: 12.70600) QuantErr: 12.70600 batch_time=0.72458
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 4.06429 (QuantReg: 12.88638) QuantErr: 12.88638 batch_time=0.71096
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 4.58363 (QuantReg: 12.85068) QuantErr: 12.85068 batch_time=0.68568
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 5.10985 (QuantReg: 12.53713) QuantErr: 12.53713 batch_time=0.64545
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 3.55786 (QuantReg: 13.24629) QuantErr: 13.24629 batch_time=0.67165
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 5.00631 (QuantReg: 12.67585) QuantErr: 12.67585 batch_time=0.63684
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 4.21502 (QuantReg: 13.13503) QuantErr: 13.13503 batch_time=0.65093
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 3.55049 (QuantReg: 13.05528) QuantErr: 13.05528 batch_time=0.67706
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 4.59384 (QuantReg: 12.94961) QuantErr: 12.94961 batch_time=0.67892
Train Epoch: 17 codebook_update_time=3.53193
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch17.pth ...
Done in 15.045s
removing stale ckpt [epoch 16] [took 0.22s]
epoch : 17
loss : 4.42098865032196
quant_reg : 12.748867542266845
quant_err : 12.748867542266845
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
MSRVTT_miech_test/t2v_metrics/R1: 20.6
MSRVTT_miech_test/t2v_metrics/R5: 47.5
MSRVTT_miech_test/t2v_metrics/R10: 61.6
MSRVTT_miech_test/t2v_metrics/R50: 87.8
MSRVTT_miech_test/t2v_metrics/MedR: 6.0
MSRVTT_miech_test/t2v_metrics/MeanR: 30.065
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.20852584232216
MSRVTT_miech_test/v2t_metrics/R1: 21.7
MSRVTT_miech_test/v2t_metrics/R5: 49.6
MSRVTT_miech_test/v2t_metrics/R10: 64.6
MSRVTT_miech_test/v2t_metrics/R50: 89.0
MSRVTT_miech_test/v2t_metrics/MedR: 6.0
MSRVTT_miech_test/v2t_metrics/MeanR: 26.263
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.12046111712713
mnt_best : 40.2201218538813
not_improved_count: 1
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 3.43388 (QuantReg: 12.74228) QuantErr: 12.74228 batch_time=31.11161
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 4.75773 (QuantReg: 12.54190) QuantErr: 12.54190 batch_time=0.62459
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 3.81986 (QuantReg: 12.61132) QuantErr: 12.61132 batch_time=1.51573
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 4.69233 (QuantReg: 12.57895) QuantErr: 12.57895 batch_time=0.66487
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 4.15854 (QuantReg: 13.02694) QuantErr: 13.02694 batch_time=0.67055
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 4.18954 (QuantReg: 12.81702) QuantErr: 12.81702 batch_time=0.69864
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 3.47977 (QuantReg: 12.69895) QuantErr: 12.69895 batch_time=0.65364
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 4.66935 (QuantReg: 12.60414) QuantErr: 12.60414 batch_time=0.64562
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 3.99438 (QuantReg: 12.62622) QuantErr: 12.62622 batch_time=0.68632
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 3.86936 (QuantReg: 12.56248) QuantErr: 12.56248 batch_time=0.66219
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 3.98743 (QuantReg: 12.91360) QuantErr: 12.91360 batch_time=0.69073
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 5.15712 (QuantReg: 12.69938) QuantErr: 12.69938 batch_time=0.72488
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 4.21499 (QuantReg: 12.99757) QuantErr: 12.99757 batch_time=0.63065
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 4.05365 (QuantReg: 12.54843) QuantErr: 12.54843 batch_time=0.65864
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 4.24542 (QuantReg: 13.15803) QuantErr: 13.15803 batch_time=0.64188
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 3.02285 (QuantReg: 12.92432) QuantErr: 12.92432 batch_time=0.63751
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 5.12970 (QuantReg: 12.65219) QuantErr: 12.65219 batch_time=1.86652
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 4.95461 (QuantReg: 12.69836) QuantErr: 12.69836 batch_time=1.02548
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 4.43411 (QuantReg: 13.06217) QuantErr: 13.06217 batch_time=0.64728
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 4.38202 (QuantReg: 12.57960) QuantErr: 12.57960 batch_time=0.69136
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 3.91329 (QuantReg: 13.06259) QuantErr: 13.06259 batch_time=0.62653
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 5.98290 (QuantReg: 12.72244) QuantErr: 12.72244 batch_time=0.63473
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 3.88286 (QuantReg: 13.04071) QuantErr: 13.04071 batch_time=0.64568
Train Epoch: 18 codebook_update_time=3.40908
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch18.pth ...
Done in 6.304s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch18.pth ...
Done in 11.398s
removing stale ckpt [epoch 17] [took 0.30s]
epoch : 18
loss : 4.248186477661132
quant_reg : 12.766760643005371
quant_err : 12.766760643005371
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
MSRVTT_miech_test/t2v_metrics/R1: 21.3
MSRVTT_miech_test/t2v_metrics/R5: 51.0
MSRVTT_miech_test/t2v_metrics/R10: 63.6
MSRVTT_miech_test/t2v_metrics/R50: 88.1
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 27.948
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 41.03322312094189
MSRVTT_miech_test/v2t_metrics/R1: 20.7
MSRVTT_miech_test/v2t_metrics/R5: 51.5
MSRVTT_miech_test/v2t_metrics/R10: 65.0
MSRVTT_miech_test/v2t_metrics/R50: 88.8
MSRVTT_miech_test/v2t_metrics/MedR: 5.0
MSRVTT_miech_test/v2t_metrics/MeanR: 25.3085
MSRVTT_miech_test/v2t_metrics/geometric_mean_R1-R5-R10: 41.07368269201994
mnt_best : 41.03322312094189
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 3.76589 (QuantReg: 12.54078) QuantErr: 12.54078 batch_time=28.00093
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 4.73227 (QuantReg: 12.68414) QuantErr: 12.68414 batch_time=0.64061
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 3.72612 (QuantReg: 12.70545) QuantErr: 12.70545 batch_time=0.63989
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 3.83625 (QuantReg: 12.88420) QuantErr: 12.88420 batch_time=0.62338
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 3.87545 (QuantReg: 12.79934) QuantErr: 12.79934 batch_time=0.65758
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 4.71023 (QuantReg: 12.64417) QuantErr: 12.64417 batch_time=0.64114
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 4.27211 (QuantReg: 12.71882) QuantErr: 12.71882 batch_time=0.76916
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 4.29737 (QuantReg: 13.12984) QuantErr: 13.12984 batch_time=0.64921
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 4.01244 (QuantReg: 12.99491) QuantErr: 12.99491 batch_time=0.62838
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 5.45765 (QuantReg: 12.72354) QuantErr: 12.72354 batch_time=0.69121
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 4.41321 (QuantReg: 12.88611) QuantErr: 12.88611 batch_time=0.72710
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 3.91231 (QuantReg: 12.64323) QuantErr: 12.64323 batch_time=0.63576
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 3.61619 (QuantReg: 13.20971) QuantErr: 13.20971 batch_time=3.46084
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 3.80130 (QuantReg: 12.97746) QuantErr: 12.97746 batch_time=1.24894
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 5.24595 (QuantReg: 13.00883) QuantErr: 13.00883 batch_time=0.65759
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 3.69979 (QuantReg: 13.27557) QuantErr: 13.27557 batch_time=0.65197
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 3.91005 (QuantReg: 13.02543) QuantErr: 13.02543 batch_time=0.67014
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 4.82946 (QuantReg: 13.02523) QuantErr: 13.02523 batch_time=0.64320
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 4.59847 (QuantReg: 12.77975) QuantErr: 12.77975 batch_time=0.64733
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 3.91535 (QuantReg: 12.69115) QuantErr: 12.69115 batch_time=0.63711
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 4.38228 (QuantReg: 12.80083) QuantErr: 12.80083 batch_time=0.65606
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 4.70900 (QuantReg: 12.61564) QuantErr: 12.61564 batch_time=0.64035
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 4.40238 (QuantReg: 12.52071) QuantErr: 12.52071 batch_time=0.63396
Train Epoch: 19 codebook_update_time=3.87235
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_1kB_L15/checkpoint-epoch19.pth ...
Done in 6.343s
removing stale ckpt [epoch 18] [took 0.04s]
epoch : 19
loss : 4.204873432159424
quant_reg : 12.79885461807251
quant_err : 12.79885461807251
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
MSRVTT_miech_test/t2v_metrics/R1: 20.1
MSRVTT_miech_test/t2v_metrics/R5: 50.2
MSRVTT_miech_test/t2v_metrics/R10: 62.8
MSRVTT_miech_test/t2v_metrics/R50: 88.0
MSRVTT_miech_test/t2v_metrics/MedR: 5.0
MSRVTT_miech_test/t2v_metrics/MeanR: 29.264
MSRVTT_miech_test/t2v_metrics/geometric_mean_R1-R5-R10: 39.86757373257613
MSRVTT_miech_test/v2t_metrics/R1: 20.3
MSRVTT_miech_test/v2t_metrics/R5: 50.8
MSRVTT_miech_test/v2t_metrics/R10: 64.5