-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_LSMDC_t0.12.txt
2585 lines (2585 loc) · 189 KB
/
HCQ_LSMDC_t0.12.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12
Preparing the dataloaders ...
Loading dataset LSMDC_full_trainval in ram ...
Finish loading dataset LSMDC_full_trainval in ram, taking 9645.226111650467 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 30.443146467208862 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 25.307016134262085 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch0.pth ...
Done in 1.519s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch0.pth ...
Done in 3.027s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
LSMDC_full_test/t2v_metrics/R1: 0.0
LSMDC_full_test/t2v_metrics/R5: 0.9
LSMDC_full_test/t2v_metrics/R10: 1.6
LSMDC_full_test/t2v_metrics/R50: 4.4
LSMDC_full_test/t2v_metrics/MedR: 508.5
LSMDC_full_test/t2v_metrics/MeanR: 502.992
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
LSMDC_full_test/v2t_metrics/R1: 0.0
LSMDC_full_test/v2t_metrics/R5: 0.3
LSMDC_full_test/v2t_metrics/R10: 0.9
LSMDC_full_test/v2t_metrics/R50: 5.1
LSMDC_full_test/v2t_metrics/MedR: 510.0
LSMDC_full_test/v2t_metrics/MeanR: 501.125
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.72362 (QuantReg: 22.49566) QuantErr: 22.49566 batch_time=27.96865
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.11445 (QuantReg: 22.62115) QuantErr: 22.62115 batch_time=0.49833
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 8.69297 (QuantReg: 22.68786) QuantErr: 22.68786 batch_time=0.50064
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 8.23291 (QuantReg: 22.70568) QuantErr: 22.70568 batch_time=0.50229
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.98705 (QuantReg: 22.68086) QuantErr: 22.68086 batch_time=0.51067
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 7.59013 (QuantReg: 22.70340) QuantErr: 22.70340 batch_time=0.49721
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 7.46586 (QuantReg: 22.67672) QuantErr: 22.67672 batch_time=0.54066
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 7.14290 (QuantReg: 22.66494) QuantErr: 22.66494 batch_time=1.23633
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 7.20798 (QuantReg: 22.71169) QuantErr: 22.71169 batch_time=0.50091
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.85072 (QuantReg: 22.69464) QuantErr: 22.69464 batch_time=0.52234
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 7.43558 (QuantReg: 22.68770) QuantErr: 22.68770 batch_time=0.50472
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 6.45766 (QuantReg: 22.68358) QuantErr: 22.68358 batch_time=0.51437
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 6.97539 (QuantReg: 22.71204) QuantErr: 22.71204 batch_time=0.49975
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 7.16080 (QuantReg: 22.69394) QuantErr: 22.69394 batch_time=0.54327
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 6.72704 (QuantReg: 22.69358) QuantErr: 22.69358 batch_time=0.52368
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 6.37423 (QuantReg: 22.68254) QuantErr: 22.68254 batch_time=0.51493
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 7.15222 (QuantReg: 22.67807) QuantErr: 22.67807 batch_time=0.52661
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 6.69626 (QuantReg: 22.69733) QuantErr: 22.69733 batch_time=0.49254
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 6.47346 (QuantReg: 22.71409) QuantErr: 22.71409 batch_time=0.54169
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 6.69583 (QuantReg: 22.67667) QuantErr: 22.67667 batch_time=0.50290
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 6.24786 (QuantReg: 22.68953) QuantErr: 22.68953 batch_time=0.50373
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 6.13932 (QuantReg: 22.69000) QuantErr: 22.69000 batch_time=0.50169
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 7.21452 (QuantReg: 22.71579) QuantErr: 22.71579 batch_time=0.49967
Train Epoch: 1 codebook_update_time=1.90886
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch1.pth ...
Done in 4.265s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch1.pth ...
Done in 7.982s
epoch : 1
loss : 7.236667749404908
quant_reg : 22.6831598815918
quant_err : 22.6831598815918
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
LSMDC_full_test/t2v_metrics/R1: 6.7
LSMDC_full_test/t2v_metrics/R5: 18.3
LSMDC_full_test/t2v_metrics/R10: 26.7
LSMDC_full_test/t2v_metrics/R50: 55.3
LSMDC_full_test/t2v_metrics/MedR: 40.0
LSMDC_full_test/t2v_metrics/MeanR: 105.227
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 14.84837924017963
LSMDC_full_test/v2t_metrics/R1: 5.8
LSMDC_full_test/v2t_metrics/R5: 17.4
LSMDC_full_test/v2t_metrics/R10: 26.3
LSMDC_full_test/v2t_metrics/R50: 53.7
LSMDC_full_test/v2t_metrics/MedR: 42.5
LSMDC_full_test/v2t_metrics/MeanR: 108.03
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 13.845575009040477
mnt_best : 14.84837924017963
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 6.30474 (QuantReg: 12.20597) QuantErr: 12.20597 batch_time=24.04655
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 6.49617 (QuantReg: 12.05335) QuantErr: 12.05335 batch_time=0.73164
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 6.45596 (QuantReg: 12.41306) QuantErr: 12.41306 batch_time=0.49457
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 6.17195 (QuantReg: 12.56668) QuantErr: 12.56668 batch_time=0.48793
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 6.44196 (QuantReg: 12.06492) QuantErr: 12.06492 batch_time=0.49213
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 6.71848 (QuantReg: 12.53135) QuantErr: 12.53135 batch_time=0.48775
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 6.62172 (QuantReg: 12.53166) QuantErr: 12.53166 batch_time=0.61658
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 5.77459 (QuantReg: 12.65086) QuantErr: 12.65086 batch_time=0.48692
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 6.28803 (QuantReg: 13.04433) QuantErr: 13.04433 batch_time=0.51525
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 5.34749 (QuantReg: 12.90741) QuantErr: 12.90741 batch_time=0.49055
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 5.87714 (QuantReg: 12.93500) QuantErr: 12.93500 batch_time=0.52395
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 6.00991 (QuantReg: 13.00739) QuantErr: 13.00739 batch_time=0.51182
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 6.25258 (QuantReg: 13.13971) QuantErr: 13.13971 batch_time=0.49679
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 6.22244 (QuantReg: 13.00310) QuantErr: 13.00310 batch_time=0.49580
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 6.11940 (QuantReg: 13.40109) QuantErr: 13.40109 batch_time=0.88849
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 5.91117 (QuantReg: 13.74812) QuantErr: 13.74812 batch_time=0.51416
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 5.69976 (QuantReg: 13.22986) QuantErr: 13.22986 batch_time=0.48730
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 6.36583 (QuantReg: 13.71671) QuantErr: 13.71671 batch_time=0.49613
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 6.41270 (QuantReg: 14.15243) QuantErr: 14.15243 batch_time=1.98887
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 5.99626 (QuantReg: 14.02162) QuantErr: 14.02162 batch_time=0.56462
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 5.67124 (QuantReg: 13.93363) QuantErr: 13.93363 batch_time=0.52173
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 6.20167 (QuantReg: 14.17896) QuantErr: 14.17896 batch_time=0.49965
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 6.16980 (QuantReg: 14.25941) QuantErr: 14.25941 batch_time=0.49415
Train Epoch: 2 codebook_update_time=1.65274
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch2.pth ...
Done in 12.753s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch2.pth ...
Done in 17.800s
removing stale ckpt [epoch 1] [took 0.02s]
removing stale ckpt [epoch 0] [took 0.03s]
epoch : 2
loss : 6.1161640281677245
quant_reg : 13.1454606590271
quant_err : 13.1454606590271
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
LSMDC_full_test/t2v_metrics/R1: 7.8
LSMDC_full_test/t2v_metrics/R5: 20.5
LSMDC_full_test/t2v_metrics/R10: 30.3
LSMDC_full_test/t2v_metrics/R50: 57.9
LSMDC_full_test/t2v_metrics/MedR: 32.0
LSMDC_full_test/t2v_metrics/MeanR: 91.926
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 16.921169042174505
LSMDC_full_test/v2t_metrics/R1: 7.1
LSMDC_full_test/v2t_metrics/R5: 19.6
LSMDC_full_test/v2t_metrics/R10: 31.4
LSMDC_full_test/v2t_metrics/R50: 57.6
LSMDC_full_test/v2t_metrics/MedR: 33.0
LSMDC_full_test/v2t_metrics/MeanR: 94.013
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.34862966129563
mnt_best : 16.921169042174505
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 6.10530 (QuantReg: 11.88315) QuantErr: 11.88315 batch_time=19.29476
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 5.53601 (QuantReg: 11.73952) QuantErr: 11.73952 batch_time=0.50843
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 5.98567 (QuantReg: 11.81912) QuantErr: 11.81912 batch_time=0.50189
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 5.67120 (QuantReg: 12.13159) QuantErr: 12.13159 batch_time=0.50527
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 5.55725 (QuantReg: 11.72030) QuantErr: 11.72030 batch_time=0.56467
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 5.68255 (QuantReg: 11.64512) QuantErr: 11.64512 batch_time=0.50298
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 5.75752 (QuantReg: 11.98401) QuantErr: 11.98401 batch_time=0.49254
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 6.02944 (QuantReg: 11.93339) QuantErr: 11.93339 batch_time=0.49368
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 5.67472 (QuantReg: 11.91257) QuantErr: 11.91257 batch_time=0.53754
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 5.67602 (QuantReg: 12.26346) QuantErr: 12.26346 batch_time=0.49278
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 5.64211 (QuantReg: 11.86089) QuantErr: 11.86089 batch_time=0.54278
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 5.61904 (QuantReg: 12.20752) QuantErr: 12.20752 batch_time=0.50575
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 5.61700 (QuantReg: 12.38062) QuantErr: 12.38062 batch_time=0.49871
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 5.75670 (QuantReg: 12.37800) QuantErr: 12.37800 batch_time=0.53797
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 5.38169 (QuantReg: 12.19075) QuantErr: 12.19075 batch_time=0.51461
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 5.83766 (QuantReg: 12.25371) QuantErr: 12.25371 batch_time=0.52560
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 5.35819 (QuantReg: 12.49711) QuantErr: 12.49711 batch_time=0.50584
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 5.81429 (QuantReg: 12.32780) QuantErr: 12.32780 batch_time=0.50368
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 5.43386 (QuantReg: 12.51363) QuantErr: 12.51363 batch_time=0.49480
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 5.79225 (QuantReg: 12.44119) QuantErr: 12.44119 batch_time=0.48830
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 5.36322 (QuantReg: 12.28864) QuantErr: 12.28864 batch_time=0.52419
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 5.42512 (QuantReg: 12.43619) QuantErr: 12.43619 batch_time=0.54925
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 5.45112 (QuantReg: 12.36484) QuantErr: 12.36484 batch_time=0.50920
Train Epoch: 3 codebook_update_time=1.67352
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch3.pth ...
Done in 4.234s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch3.pth ...
Done in 8.205s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 5.697446174621582
quant_reg : 12.14298168182373
quant_err : 12.14298168182373
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
LSMDC_full_test/t2v_metrics/R1: 9.8
LSMDC_full_test/t2v_metrics/R5: 22.4
LSMDC_full_test/t2v_metrics/R10: 31.2
LSMDC_full_test/t2v_metrics/R50: 60.0
LSMDC_full_test/t2v_metrics/MedR: 31.0
LSMDC_full_test/t2v_metrics/MeanR: 87.719
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.990784080879333
LSMDC_full_test/v2t_metrics/R1: 8.8
LSMDC_full_test/v2t_metrics/R5: 22.5
LSMDC_full_test/v2t_metrics/R10: 29.6
LSMDC_full_test/v2t_metrics/R50: 58.3
LSMDC_full_test/v2t_metrics/MedR: 32.0
LSMDC_full_test/v2t_metrics/MeanR: 95.898
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 18.029580989945906
mnt_best : 18.990784080879333
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 5.59395 (QuantReg: 11.62666) QuantErr: 11.62666 batch_time=16.52006
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 5.44974 (QuantReg: 12.17366) QuantErr: 12.17366 batch_time=0.49867
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 5.61251 (QuantReg: 11.88919) QuantErr: 11.88919 batch_time=0.50106
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 5.51620 (QuantReg: 11.88573) QuantErr: 11.88573 batch_time=0.49466
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 5.09011 (QuantReg: 12.00648) QuantErr: 12.00648 batch_time=0.50657
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 5.00499 (QuantReg: 11.79345) QuantErr: 11.79345 batch_time=0.50822
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 5.44170 (QuantReg: 12.13751) QuantErr: 12.13751 batch_time=0.50126
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 5.79211 (QuantReg: 11.83358) QuantErr: 11.83358 batch_time=0.49910
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 5.01851 (QuantReg: 11.63689) QuantErr: 11.63689 batch_time=0.49709
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 5.15097 (QuantReg: 11.56922) QuantErr: 11.56922 batch_time=0.51031
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 5.92085 (QuantReg: 11.73762) QuantErr: 11.73762 batch_time=0.51193
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 4.81056 (QuantReg: 11.47915) QuantErr: 11.47915 batch_time=0.48730
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 5.40221 (QuantReg: 11.56537) QuantErr: 11.56537 batch_time=7.47981
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 5.55711 (QuantReg: 12.04520) QuantErr: 12.04520 batch_time=0.53284
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 5.51342 (QuantReg: 11.76122) QuantErr: 11.76122 batch_time=0.50443
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 5.76195 (QuantReg: 11.86727) QuantErr: 11.86727 batch_time=0.49510
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 5.35126 (QuantReg: 11.97012) QuantErr: 11.97012 batch_time=0.50040
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 5.89392 (QuantReg: 12.01109) QuantErr: 12.01109 batch_time=0.50709
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 5.12265 (QuantReg: 12.19785) QuantErr: 12.19785 batch_time=0.50018
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 5.11230 (QuantReg: 11.89250) QuantErr: 11.89250 batch_time=1.66282
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 4.54959 (QuantReg: 11.95795) QuantErr: 11.95795 batch_time=0.50035
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 4.95541 (QuantReg: 11.93426) QuantErr: 11.93426 batch_time=0.50428
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 5.33924 (QuantReg: 12.05219) QuantErr: 12.05219 batch_time=0.62733
Train Epoch: 4 codebook_update_time=1.66682
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch4.pth ...
Done in 4.800s
removing stale ckpt [epoch 3] [took 0.01s]
epoch : 4
loss : 5.404772598266602
quant_reg : 11.901963722229004
quant_err : 11.901963722229004
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
LSMDC_full_test/t2v_metrics/R1: 8.6
LSMDC_full_test/t2v_metrics/R5: 23.2
LSMDC_full_test/t2v_metrics/R10: 33.3
LSMDC_full_test/t2v_metrics/R50: 61.7
LSMDC_full_test/t2v_metrics/MedR: 29.0
LSMDC_full_test/t2v_metrics/MeanR: 86.885
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.79938129848611
LSMDC_full_test/v2t_metrics/R1: 8.8
LSMDC_full_test/v2t_metrics/R5: 21.9
LSMDC_full_test/v2t_metrics/R10: 31.3
LSMDC_full_test/v2t_metrics/R50: 60.1
LSMDC_full_test/v2t_metrics/MedR: 29.75
LSMDC_full_test/v2t_metrics/MeanR: 90.039
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 18.203589840524494
mnt_best : 18.990784080879333
not_improved_count: 1
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 6.02462 (QuantReg: 11.75027) QuantErr: 11.75027 batch_time=22.11508
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 5.03293 (QuantReg: 11.87025) QuantErr: 11.87025 batch_time=0.52216
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 5.16915 (QuantReg: 11.47761) QuantErr: 11.47761 batch_time=0.50871
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 5.62406 (QuantReg: 11.69540) QuantErr: 11.69540 batch_time=0.51214
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 5.38225 (QuantReg: 12.08714) QuantErr: 12.08714 batch_time=0.49525
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 4.84760 (QuantReg: 12.06400) QuantErr: 12.06400 batch_time=0.49836
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 5.09148 (QuantReg: 11.97313) QuantErr: 11.97313 batch_time=0.50029
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 5.27649 (QuantReg: 11.84863) QuantErr: 11.84863 batch_time=1.41949
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 5.80030 (QuantReg: 11.80895) QuantErr: 11.80895 batch_time=0.53856
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 5.40223 (QuantReg: 11.55962) QuantErr: 11.55962 batch_time=1.91437
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 5.36200 (QuantReg: 11.68115) QuantErr: 11.68115 batch_time=0.55740
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 5.11158 (QuantReg: 11.89879) QuantErr: 11.89879 batch_time=0.50301
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 4.90249 (QuantReg: 11.81152) QuantErr: 11.81152 batch_time=3.25563
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 5.13837 (QuantReg: 12.12841) QuantErr: 12.12841 batch_time=0.49124
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 5.50998 (QuantReg: 12.11050) QuantErr: 12.11050 batch_time=0.53025
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 5.48599 (QuantReg: 11.92832) QuantErr: 11.92832 batch_time=0.52877
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 4.95158 (QuantReg: 11.89963) QuantErr: 11.89963 batch_time=0.52631
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 5.04082 (QuantReg: 12.16164) QuantErr: 12.16164 batch_time=0.51236
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 4.93395 (QuantReg: 11.80136) QuantErr: 11.80136 batch_time=0.51035
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 5.08728 (QuantReg: 11.66404) QuantErr: 11.66404 batch_time=0.52647
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 4.94418 (QuantReg: 11.90202) QuantErr: 11.90202 batch_time=0.52485
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 5.13910 (QuantReg: 11.81968) QuantErr: 11.81968 batch_time=0.49902
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 4.72618 (QuantReg: 12.01891) QuantErr: 12.01891 batch_time=0.50345
Train Epoch: 5 codebook_update_time=2.13201
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch5.pth ...
Done in 4.263s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch5.pth ...
Done in 8.894s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 5.179115818023682
quant_reg : 11.86327512359619
quant_err : 11.86327512359619
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
LSMDC_full_test/t2v_metrics/R1: 9.8
LSMDC_full_test/t2v_metrics/R5: 23.2
LSMDC_full_test/t2v_metrics/R10: 33.4
LSMDC_full_test/t2v_metrics/R50: 62.3
LSMDC_full_test/t2v_metrics/MedR: 29.0
LSMDC_full_test/t2v_metrics/MeanR: 86.025
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.655624303227874
LSMDC_full_test/v2t_metrics/R1: 9.4
LSMDC_full_test/v2t_metrics/R5: 24.8
LSMDC_full_test/v2t_metrics/R10: 33.9
LSMDC_full_test/v2t_metrics/R50: 61.5
LSMDC_full_test/v2t_metrics/MedR: 28.0
LSMDC_full_test/v2t_metrics/MeanR: 88.331
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.918642832651194
mnt_best : 19.655624303227874
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 5.52621 (QuantReg: 11.70088) QuantErr: 11.70088 batch_time=21.67551
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 5.21086 (QuantReg: 11.75568) QuantErr: 11.75568 batch_time=0.52310
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 5.28266 (QuantReg: 11.79358) QuantErr: 11.79358 batch_time=0.50154
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 4.89873 (QuantReg: 11.66260) QuantErr: 11.66260 batch_time=0.53293
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 4.82312 (QuantReg: 11.66192) QuantErr: 11.66192 batch_time=0.49728
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 5.15590 (QuantReg: 11.68042) QuantErr: 11.68042 batch_time=0.56444
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 4.85862 (QuantReg: 11.57850) QuantErr: 11.57850 batch_time=0.50993
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 5.04003 (QuantReg: 11.72828) QuantErr: 11.72828 batch_time=0.52794
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 5.38807 (QuantReg: 11.79830) QuantErr: 11.79830 batch_time=0.49703
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 5.23147 (QuantReg: 11.96992) QuantErr: 11.96992 batch_time=0.51523
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 4.70049 (QuantReg: 11.93154) QuantErr: 11.93154 batch_time=0.51128
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 5.08094 (QuantReg: 11.69785) QuantErr: 11.69785 batch_time=0.49667
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 5.07434 (QuantReg: 12.06415) QuantErr: 12.06415 batch_time=0.49870
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 5.58732 (QuantReg: 11.73112) QuantErr: 11.73112 batch_time=0.50792
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 4.97893 (QuantReg: 11.98847) QuantErr: 11.98847 batch_time=0.49838
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 4.36877 (QuantReg: 11.68383) QuantErr: 11.68383 batch_time=0.49609
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 4.76699 (QuantReg: 11.79385) QuantErr: 11.79385 batch_time=0.51267
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 5.03286 (QuantReg: 12.34276) QuantErr: 12.34276 batch_time=0.54675
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 5.32728 (QuantReg: 11.57550) QuantErr: 11.57550 batch_time=0.49859
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 4.97435 (QuantReg: 11.80398) QuantErr: 11.80398 batch_time=0.50371
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 4.82552 (QuantReg: 12.11320) QuantErr: 12.11320 batch_time=0.51731
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 5.38758 (QuantReg: 11.92711) QuantErr: 11.92711 batch_time=0.50263
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 5.24254 (QuantReg: 12.05624) QuantErr: 12.05624 batch_time=0.53397
Train Epoch: 6 codebook_update_time=1.73197
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch6.pth ...
Done in 4.293s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch6.pth ...
Done in 15.276s
removing stale ckpt [epoch 5] [took 0.04s]
epoch : 6
loss : 5.046317289352417
quant_reg : 11.829826763153076
quant_err : 11.829826763153076
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
LSMDC_full_test/t2v_metrics/R1: 9.9
LSMDC_full_test/t2v_metrics/R5: 24.2
LSMDC_full_test/t2v_metrics/R10: 34.7
LSMDC_full_test/t2v_metrics/R50: 63.1
LSMDC_full_test/t2v_metrics/MedR: 26.0
LSMDC_full_test/t2v_metrics/MeanR: 83.025
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.257849723081687
LSMDC_full_test/v2t_metrics/R1: 9.1
LSMDC_full_test/v2t_metrics/R5: 23.1
LSMDC_full_test/v2t_metrics/R10: 32.9
LSMDC_full_test/v2t_metrics/R50: 61.4
LSMDC_full_test/v2t_metrics/MedR: 27.0
LSMDC_full_test/v2t_metrics/MeanR: 87.941
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.05240289059154
mnt_best : 20.257849723081687
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 5.09147 (QuantReg: 11.72171) QuantErr: 11.72171 batch_time=18.73376
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 4.94928 (QuantReg: 11.87058) QuantErr: 11.87058 batch_time=0.50487
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 4.74885 (QuantReg: 12.05197) QuantErr: 12.05197 batch_time=0.50919
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 4.69680 (QuantReg: 11.70213) QuantErr: 11.70213 batch_time=0.49551
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 5.28093 (QuantReg: 12.08214) QuantErr: 12.08214 batch_time=0.53685
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 5.00230 (QuantReg: 11.93108) QuantErr: 11.93108 batch_time=0.50216
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 4.84723 (QuantReg: 11.73550) QuantErr: 11.73550 batch_time=1.40618
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 5.08742 (QuantReg: 11.72573) QuantErr: 11.72573 batch_time=0.52236
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 4.78246 (QuantReg: 11.71984) QuantErr: 11.71984 batch_time=0.51981
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 4.76779 (QuantReg: 11.77139) QuantErr: 11.77139 batch_time=0.50266
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 4.71071 (QuantReg: 11.67258) QuantErr: 11.67258 batch_time=0.53757
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 4.91470 (QuantReg: 11.56962) QuantErr: 11.56962 batch_time=0.53412
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 4.89933 (QuantReg: 11.65607) QuantErr: 11.65607 batch_time=0.49508
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 4.72042 (QuantReg: 11.68863) QuantErr: 11.68863 batch_time=3.04017
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 4.84271 (QuantReg: 12.11451) QuantErr: 12.11451 batch_time=0.50146
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 4.53705 (QuantReg: 11.96960) QuantErr: 11.96960 batch_time=0.60931
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 4.92848 (QuantReg: 11.78110) QuantErr: 11.78110 batch_time=0.54226
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 4.96945 (QuantReg: 11.82174) QuantErr: 11.82174 batch_time=0.51891
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 4.74801 (QuantReg: 11.68496) QuantErr: 11.68496 batch_time=0.51224
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 4.74356 (QuantReg: 12.00546) QuantErr: 12.00546 batch_time=0.51293
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 4.69718 (QuantReg: 11.73154) QuantErr: 11.73154 batch_time=0.51215
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 4.71621 (QuantReg: 11.76329) QuantErr: 11.76329 batch_time=0.51614
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 4.70259 (QuantReg: 11.97707) QuantErr: 11.97707 batch_time=0.55147
Train Epoch: 7 codebook_update_time=1.72396
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch7.pth ...
Done in 4.468s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch7.pth ...
Done in 9.457s
removing stale ckpt [epoch 6] [took 0.04s]
epoch : 7
loss : 4.871375328063965
quant_reg : 11.841005916595458
quant_err : 11.841005916595458
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
LSMDC_full_test/t2v_metrics/R1: 9.8
LSMDC_full_test/t2v_metrics/R5: 26.4
LSMDC_full_test/t2v_metrics/R10: 36.4
LSMDC_full_test/t2v_metrics/R50: 64.9
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 81.139
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.117562850088174
LSMDC_full_test/v2t_metrics/R1: 9.8
LSMDC_full_test/v2t_metrics/R5: 25.3
LSMDC_full_test/v2t_metrics/R10: 36.0
LSMDC_full_test/v2t_metrics/R50: 64.0
LSMDC_full_test/v2t_metrics/MedR: 24.0
LSMDC_full_test/v2t_metrics/MeanR: 83.418
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.74354761395199
mnt_best : 21.117562850088174
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 4.75030 (QuantReg: 11.78132) QuantErr: 11.78132 batch_time=21.13723
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 4.67927 (QuantReg: 11.75678) QuantErr: 11.75678 batch_time=0.50158
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 4.85624 (QuantReg: 12.01593) QuantErr: 12.01593 batch_time=0.50384
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 5.21047 (QuantReg: 11.96178) QuantErr: 11.96178 batch_time=0.49801
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 4.49520 (QuantReg: 11.78119) QuantErr: 11.78119 batch_time=0.50096
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 4.84075 (QuantReg: 12.13333) QuantErr: 12.13333 batch_time=0.50672
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 4.73381 (QuantReg: 11.81869) QuantErr: 11.81869 batch_time=0.49778
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 5.10472 (QuantReg: 11.88929) QuantErr: 11.88929 batch_time=0.63064
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 4.63814 (QuantReg: 11.95074) QuantErr: 11.95074 batch_time=0.50420
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 4.82229 (QuantReg: 11.64340) QuantErr: 11.64340 batch_time=0.50336
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 5.12282 (QuantReg: 12.11659) QuantErr: 12.11659 batch_time=0.52345
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 4.73715 (QuantReg: 12.03800) QuantErr: 12.03800 batch_time=0.50572
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 4.86775 (QuantReg: 11.93728) QuantErr: 11.93728 batch_time=0.51946
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 4.51594 (QuantReg: 12.29663) QuantErr: 12.29663 batch_time=0.56047
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 4.97997 (QuantReg: 12.01195) QuantErr: 12.01195 batch_time=1.24733
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 4.42249 (QuantReg: 11.71428) QuantErr: 11.71428 batch_time=0.49844
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 4.90525 (QuantReg: 12.13457) QuantErr: 12.13457 batch_time=0.49424
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 4.64748 (QuantReg: 12.03160) QuantErr: 12.03160 batch_time=0.50380
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 4.51421 (QuantReg: 11.55910) QuantErr: 11.55910 batch_time=0.54340
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 4.89090 (QuantReg: 12.02026) QuantErr: 12.02026 batch_time=0.49648
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 4.46061 (QuantReg: 11.88970) QuantErr: 11.88970 batch_time=0.59796
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 4.45873 (QuantReg: 12.01528) QuantErr: 12.01528 batch_time=0.50920
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 4.71648 (QuantReg: 11.79207) QuantErr: 11.79207 batch_time=0.53136
Train Epoch: 8 codebook_update_time=1.71909
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch8.pth ...
Done in 4.428s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 4.743138902664184
quant_reg : 11.888781074523926
quant_err : 11.888781074523926
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
LSMDC_full_test/t2v_metrics/R1: 10.2
LSMDC_full_test/t2v_metrics/R5: 25.7
LSMDC_full_test/t2v_metrics/R10: 34.9
LSMDC_full_test/t2v_metrics/R50: 64.9
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 81.686
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.914761059462297
LSMDC_full_test/v2t_metrics/R1: 10.4
LSMDC_full_test/v2t_metrics/R5: 25.9
LSMDC_full_test/v2t_metrics/R10: 34.8
LSMDC_full_test/v2t_metrics/R50: 62.8
LSMDC_full_test/v2t_metrics/MedR: 25.0
LSMDC_full_test/v2t_metrics/MeanR: 84.546
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.084862948210233
mnt_best : 21.117562850088174
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 4.67241 (QuantReg: 11.68302) QuantErr: 11.68302 batch_time=19.96515
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 4.72759 (QuantReg: 11.97001) QuantErr: 11.97001 batch_time=0.65228
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 4.65776 (QuantReg: 11.92726) QuantErr: 11.92726 batch_time=0.50304
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 4.94363 (QuantReg: 12.11322) QuantErr: 12.11322 batch_time=0.54134
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 4.85931 (QuantReg: 12.00270) QuantErr: 12.00270 batch_time=0.54121
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 4.34270 (QuantReg: 12.19249) QuantErr: 12.19249 batch_time=0.53563
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 4.56015 (QuantReg: 11.74883) QuantErr: 11.74883 batch_time=0.47793
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 5.01304 (QuantReg: 11.54925) QuantErr: 11.54925 batch_time=0.51142
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 4.53798 (QuantReg: 11.71795) QuantErr: 11.71795 batch_time=0.49981
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 4.37429 (QuantReg: 11.89322) QuantErr: 11.89322 batch_time=0.61858
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 5.22307 (QuantReg: 12.05666) QuantErr: 12.05666 batch_time=0.52596
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 4.79278 (QuantReg: 12.11725) QuantErr: 12.11725 batch_time=0.49842
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 4.44330 (QuantReg: 11.73132) QuantErr: 11.73132 batch_time=0.49561
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 4.65034 (QuantReg: 12.20146) QuantErr: 12.20146 batch_time=2.21008
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 4.60578 (QuantReg: 11.97665) QuantErr: 11.97665 batch_time=0.51073
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 4.81318 (QuantReg: 12.14903) QuantErr: 12.14903 batch_time=0.55274
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 4.34409 (QuantReg: 11.91816) QuantErr: 11.91816 batch_time=0.49309
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 4.47697 (QuantReg: 11.43192) QuantErr: 11.43192 batch_time=0.49447
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 4.57882 (QuantReg: 11.85285) QuantErr: 11.85285 batch_time=0.50724
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 4.44508 (QuantReg: 12.08766) QuantErr: 12.08766 batch_time=0.50549
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 5.09174 (QuantReg: 12.10295) QuantErr: 12.10295 batch_time=0.49779
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 5.14613 (QuantReg: 12.10365) QuantErr: 12.10365 batch_time=0.51293
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 4.58234 (QuantReg: 11.62977) QuantErr: 11.62977 batch_time=0.49688
Train Epoch: 9 codebook_update_time=1.88991
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch9.pth ...
Done in 5.889s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch9.pth ...
Done in 11.116s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 4.623035400390625
quant_reg : 11.929827339172363
quant_err : 11.929827339172363
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
LSMDC_full_test/t2v_metrics/R1: 11.2
LSMDC_full_test/t2v_metrics/R5: 25.3
LSMDC_full_test/t2v_metrics/R10: 35.6
LSMDC_full_test/t2v_metrics/R50: 64.5
LSMDC_full_test/t2v_metrics/MedR: 25.0
LSMDC_full_test/t2v_metrics/MeanR: 79.061
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.607085010041548
LSMDC_full_test/v2t_metrics/R1: 10.3
LSMDC_full_test/v2t_metrics/R5: 25.2
LSMDC_full_test/v2t_metrics/R10: 35.6
LSMDC_full_test/v2t_metrics/R50: 63.7
LSMDC_full_test/v2t_metrics/MedR: 26.0
LSMDC_full_test/v2t_metrics/MeanR: 80.876
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.984369321071096
mnt_best : 21.607085010041548
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 4.44868 (QuantReg: 11.76305) QuantErr: 11.76305 batch_time=18.40216
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 4.58163 (QuantReg: 11.55250) QuantErr: 11.55250 batch_time=0.95341
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 4.40397 (QuantReg: 11.81058) QuantErr: 11.81058 batch_time=0.50962
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 4.34905 (QuantReg: 11.86965) QuantErr: 11.86965 batch_time=0.50256
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 4.60322 (QuantReg: 12.08620) QuantErr: 12.08620 batch_time=0.49630
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 4.49253 (QuantReg: 12.08866) QuantErr: 12.08866 batch_time=0.70303
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 4.47577 (QuantReg: 11.43694) QuantErr: 11.43694 batch_time=1.06500
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 4.29453 (QuantReg: 11.79679) QuantErr: 11.79679 batch_time=1.89751
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 4.57760 (QuantReg: 11.99819) QuantErr: 11.99819 batch_time=0.50826
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 4.66146 (QuantReg: 12.20336) QuantErr: 12.20336 batch_time=0.50421
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 4.43182 (QuantReg: 11.99289) QuantErr: 11.99289 batch_time=0.49815
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 4.45078 (QuantReg: 11.78019) QuantErr: 11.78019 batch_time=0.61188
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 4.13612 (QuantReg: 11.78141) QuantErr: 11.78141 batch_time=0.50109
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 4.79238 (QuantReg: 12.19659) QuantErr: 12.19659 batch_time=0.51066
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 4.96259 (QuantReg: 12.22303) QuantErr: 12.22303 batch_time=0.50080
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 4.72351 (QuantReg: 12.15959) QuantErr: 12.15959 batch_time=0.49778
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 4.59929 (QuantReg: 11.76943) QuantErr: 11.76943 batch_time=0.50556
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 4.65764 (QuantReg: 12.08813) QuantErr: 12.08813 batch_time=0.62136
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 4.27862 (QuantReg: 12.05158) QuantErr: 12.05158 batch_time=0.96378
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 4.24375 (QuantReg: 11.79910) QuantErr: 11.79910 batch_time=0.51094
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 4.35835 (QuantReg: 12.04326) QuantErr: 12.04326 batch_time=0.79808
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 5.01880 (QuantReg: 12.10949) QuantErr: 12.10949 batch_time=0.54280
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 4.60703 (QuantReg: 11.93453) QuantErr: 11.93453 batch_time=0.52960
Train Epoch: 10 codebook_update_time=1.67754
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch10.pth ...
Done in 4.810s
removing stale ckpt [epoch 9] [took 0.02s]
epoch : 10
loss : 4.5262621965408325
quant_reg : 11.937867965698242
quant_err : 11.937867965698242
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
LSMDC_full_test/t2v_metrics/R1: 10.3
LSMDC_full_test/t2v_metrics/R5: 25.4
LSMDC_full_test/t2v_metrics/R10: 34.9
LSMDC_full_test/t2v_metrics/R50: 64.1
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 81.394
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.900922557202218
LSMDC_full_test/v2t_metrics/R1: 10.2
LSMDC_full_test/v2t_metrics/R5: 25.2
LSMDC_full_test/v2t_metrics/R10: 34.4
LSMDC_full_test/v2t_metrics/R50: 61.7
LSMDC_full_test/v2t_metrics/MedR: 25.0
LSMDC_full_test/v2t_metrics/MeanR: 82.391
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.678532670027792
mnt_best : 21.607085010041548
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 4.01896 (QuantReg: 11.99370) QuantErr: 11.99370 batch_time=19.76474
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 4.59974 (QuantReg: 12.07438) QuantErr: 12.07438 batch_time=0.78610
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 4.33638 (QuantReg: 11.75749) QuantErr: 11.75749 batch_time=4.56555
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 4.39364 (QuantReg: 11.76582) QuantErr: 11.76582 batch_time=0.50558
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 4.49738 (QuantReg: 11.77303) QuantErr: 11.77303 batch_time=0.51685
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 4.82886 (QuantReg: 12.07043) QuantErr: 12.07043 batch_time=0.49482
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 4.49929 (QuantReg: 11.99274) QuantErr: 11.99274 batch_time=0.50261
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 4.41463 (QuantReg: 11.99429) QuantErr: 11.99429 batch_time=0.51117
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 4.59813 (QuantReg: 12.07091) QuantErr: 12.07091 batch_time=0.52005
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 4.21207 (QuantReg: 12.25923) QuantErr: 12.25923 batch_time=0.52458
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 4.47969 (QuantReg: 12.04251) QuantErr: 12.04251 batch_time=0.51655
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 3.91578 (QuantReg: 11.95231) QuantErr: 11.95231 batch_time=0.51673
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 4.53176 (QuantReg: 11.87950) QuantErr: 11.87950 batch_time=0.51011
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 3.95582 (QuantReg: 11.92799) QuantErr: 11.92799 batch_time=2.73451
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 4.41032 (QuantReg: 11.63372) QuantErr: 11.63372 batch_time=0.51292
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 4.86059 (QuantReg: 12.04574) QuantErr: 12.04574 batch_time=0.49744
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 4.15409 (QuantReg: 11.91108) QuantErr: 11.91108 batch_time=0.51015
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 4.25481 (QuantReg: 11.77370) QuantErr: 11.77370 batch_time=0.51865
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 4.47364 (QuantReg: 11.85595) QuantErr: 11.85595 batch_time=0.50390
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 4.75919 (QuantReg: 12.02121) QuantErr: 12.02121 batch_time=0.49508
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 4.29031 (QuantReg: 11.74873) QuantErr: 11.74873 batch_time=0.54851
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 4.06361 (QuantReg: 11.95508) QuantErr: 11.95508 batch_time=0.49670
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 4.49337 (QuantReg: 11.74084) QuantErr: 11.74084 batch_time=0.55630
Train Epoch: 11 codebook_update_time=1.93545
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch11.pth ...
Done in 6.168s
removing stale ckpt [epoch 10] [took 0.02s]
epoch : 11
loss : 4.410327227592468
quant_reg : 11.952526866912843
quant_err : 11.952526866912843
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
LSMDC_full_test/t2v_metrics/R1: 10.7
LSMDC_full_test/t2v_metrics/R5: 25.0
LSMDC_full_test/t2v_metrics/R10: 35.1
LSMDC_full_test/t2v_metrics/R50: 63.6
LSMDC_full_test/t2v_metrics/MedR: 23.0
LSMDC_full_test/t2v_metrics/MeanR: 80.407
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.09649470457379
LSMDC_full_test/v2t_metrics/R1: 10.7
LSMDC_full_test/v2t_metrics/R5: 24.5
LSMDC_full_test/v2t_metrics/R10: 34.8
LSMDC_full_test/v2t_metrics/R50: 63.4
LSMDC_full_test/v2t_metrics/MedR: 25.0
LSMDC_full_test/v2t_metrics/MeanR: 84.671
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.895031747185975
mnt_best : 21.607085010041548
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 4.52178 (QuantReg: 11.78248) QuantErr: 11.78248 batch_time=18.01075
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 4.26101 (QuantReg: 11.96506) QuantErr: 11.96506 batch_time=0.53909
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 4.76121 (QuantReg: 11.92509) QuantErr: 11.92509 batch_time=0.49655
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 3.92469 (QuantReg: 11.56425) QuantErr: 11.56425 batch_time=0.54654
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 4.28494 (QuantReg: 12.01463) QuantErr: 12.01463 batch_time=0.50083
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 4.19676 (QuantReg: 12.00030) QuantErr: 12.00030 batch_time=0.52203
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 4.46453 (QuantReg: 12.06420) QuantErr: 12.06420 batch_time=0.80912
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 4.43369 (QuantReg: 11.73345) QuantErr: 11.73345 batch_time=0.51985
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 4.39665 (QuantReg: 12.10234) QuantErr: 12.10234 batch_time=0.59312
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 4.16445 (QuantReg: 11.76913) QuantErr: 11.76913 batch_time=0.51016
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 4.22290 (QuantReg: 11.84678) QuantErr: 11.84678 batch_time=0.49419
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 4.35599 (QuantReg: 12.04028) QuantErr: 12.04028 batch_time=0.50996
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 4.52549 (QuantReg: 11.97266) QuantErr: 11.97266 batch_time=0.51237
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 4.40918 (QuantReg: 12.08093) QuantErr: 12.08093 batch_time=0.52172
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 4.28815 (QuantReg: 11.92552) QuantErr: 11.92552 batch_time=0.50647
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 4.27796 (QuantReg: 11.92499) QuantErr: 11.92499 batch_time=0.50387
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 3.79623 (QuantReg: 11.88358) QuantErr: 11.88358 batch_time=0.51863
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 4.61540 (QuantReg: 11.82000) QuantErr: 11.82000 batch_time=0.51330
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 4.07087 (QuantReg: 11.86418) QuantErr: 11.86418 batch_time=0.81042
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 4.44224 (QuantReg: 11.79712) QuantErr: 11.79712 batch_time=0.52377
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 4.08742 (QuantReg: 11.96201) QuantErr: 11.96201 batch_time=0.51197
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 4.00988 (QuantReg: 12.02903) QuantErr: 12.02903 batch_time=0.55267
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 4.27117 (QuantReg: 11.82552) QuantErr: 11.82552 batch_time=0.49150
Train Epoch: 12 codebook_update_time=1.72139
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch12.pth ...
Done in 6.159s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch12.pth ...
Done in 10.838s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 4.333999530792236
quant_reg : 11.92297611618042
quant_err : 11.92297611618042
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
LSMDC_full_test/t2v_metrics/R1: 10.8
LSMDC_full_test/t2v_metrics/R5: 26.4
LSMDC_full_test/t2v_metrics/R10: 36.6
LSMDC_full_test/t2v_metrics/R50: 65.7
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 80.105
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.85259025299446
LSMDC_full_test/v2t_metrics/R1: 9.7
LSMDC_full_test/v2t_metrics/R5: 26.0
LSMDC_full_test/v2t_metrics/R10: 35.7
LSMDC_full_test/v2t_metrics/R50: 63.3
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 81.798
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.803565094040625
mnt_best : 21.85259025299446
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 4.06461 (QuantReg: 11.75046) QuantErr: 11.75046 batch_time=19.37581
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 4.41505 (QuantReg: 11.67640) QuantErr: 11.67640 batch_time=0.55123
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 4.36052 (QuantReg: 12.04288) QuantErr: 12.04288 batch_time=0.50676
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 3.95069 (QuantReg: 11.84291) QuantErr: 11.84291 batch_time=0.52175
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 4.01342 (QuantReg: 11.91876) QuantErr: 11.91876 batch_time=0.49733
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 4.57338 (QuantReg: 12.08838) QuantErr: 12.08838 batch_time=0.54298
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 4.29877 (QuantReg: 12.02925) QuantErr: 12.02925 batch_time=0.50102
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 4.11055 (QuantReg: 11.97939) QuantErr: 11.97939 batch_time=0.53652
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 3.98513 (QuantReg: 12.07332) QuantErr: 12.07332 batch_time=0.49718
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 4.38252 (QuantReg: 11.89661) QuantErr: 11.89661 batch_time=0.49755
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 4.20299 (QuantReg: 11.96416) QuantErr: 11.96416 batch_time=0.50643
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 4.10462 (QuantReg: 12.40769) QuantErr: 12.40769 batch_time=0.50668
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 4.53817 (QuantReg: 12.02900) QuantErr: 12.02900 batch_time=0.50184
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 4.51134 (QuantReg: 11.79172) QuantErr: 11.79172 batch_time=2.80610
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 4.06825 (QuantReg: 12.20646) QuantErr: 12.20646 batch_time=0.49862
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 4.09541 (QuantReg: 12.36800) QuantErr: 12.36800 batch_time=0.50822
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 3.93795 (QuantReg: 11.73664) QuantErr: 11.73664 batch_time=0.69673
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 4.46709 (QuantReg: 12.04806) QuantErr: 12.04806 batch_time=0.50156
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 4.38192 (QuantReg: 12.31105) QuantErr: 12.31105 batch_time=0.62371
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 4.18467 (QuantReg: 11.69731) QuantErr: 11.69731 batch_time=0.49165
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 4.31237 (QuantReg: 11.78423) QuantErr: 11.78423 batch_time=0.51762
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 4.17830 (QuantReg: 12.09984) QuantErr: 12.09984 batch_time=0.59323
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 4.07498 (QuantReg: 11.84955) QuantErr: 11.84955 batch_time=0.50139
Train Epoch: 13 codebook_update_time=1.71517
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch13.pth ...
Done in 20.023s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch13.pth ...
Done in 25.140s
removing stale ckpt [epoch 12] [took 0.30s]
epoch : 13
loss : 4.235946066856385
quant_reg : 11.972372653961182
quant_err : 11.972372653961182
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
LSMDC_full_test/t2v_metrics/R1: 11.3
LSMDC_full_test/t2v_metrics/R5: 26.8
LSMDC_full_test/t2v_metrics/R10: 37.3
LSMDC_full_test/t2v_metrics/R50: 64.3
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 79.517
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.437477122283305
LSMDC_full_test/v2t_metrics/R1: 9.8
LSMDC_full_test/v2t_metrics/R5: 25.9
LSMDC_full_test/v2t_metrics/R10: 38.0
LSMDC_full_test/v2t_metrics/R50: 64.0
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 80.384
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.286445415997097
mnt_best : 22.437477122283305
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 4.44755 (QuantReg: 11.97222) QuantErr: 11.97222 batch_time=25.01749
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 3.97771 (QuantReg: 11.81408) QuantErr: 11.81408 batch_time=0.56250
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 4.35669 (QuantReg: 11.72222) QuantErr: 11.72222 batch_time=0.49959
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 4.04272 (QuantReg: 11.91272) QuantErr: 11.91272 batch_time=0.49536
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 4.01153 (QuantReg: 12.11363) QuantErr: 12.11363 batch_time=0.50481
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 4.07129 (QuantReg: 12.19171) QuantErr: 12.19171 batch_time=0.55760
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 4.03887 (QuantReg: 11.83594) QuantErr: 11.83594 batch_time=0.52912
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 4.72347 (QuantReg: 11.76344) QuantErr: 11.76344 batch_time=0.55652
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 4.18560 (QuantReg: 12.16926) QuantErr: 12.16926 batch_time=0.49514
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 4.25456 (QuantReg: 12.07762) QuantErr: 12.07762 batch_time=0.51378
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 4.00941 (QuantReg: 11.91133) QuantErr: 11.91133 batch_time=0.51330
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 4.33774 (QuantReg: 12.06314) QuantErr: 12.06314 batch_time=0.49468
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 4.59102 (QuantReg: 12.12167) QuantErr: 12.12167 batch_time=0.51107
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 4.10303 (QuantReg: 12.00677) QuantErr: 12.00677 batch_time=0.50467
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 4.45250 (QuantReg: 12.23416) QuantErr: 12.23416 batch_time=1.05663
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 4.26987 (QuantReg: 11.89603) QuantErr: 11.89603 batch_time=1.44473
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 3.95218 (QuantReg: 12.20283) QuantErr: 12.20283 batch_time=0.50143
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 4.40172 (QuantReg: 12.22583) QuantErr: 12.22583 batch_time=0.52742
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 4.57883 (QuantReg: 11.84424) QuantErr: 11.84424 batch_time=0.50064
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 4.41950 (QuantReg: 12.12370) QuantErr: 12.12370 batch_time=0.50118
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 4.13147 (QuantReg: 12.36630) QuantErr: 12.36630 batch_time=0.49831
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 4.03480 (QuantReg: 12.15351) QuantErr: 12.15351 batch_time=0.70766
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 4.09063 (QuantReg: 11.98042) QuantErr: 11.98042 batch_time=0.49517
Train Epoch: 14 codebook_update_time=1.67602
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch14.pth ...
Done in 5.380s
removing stale ckpt [epoch 13] [took 0.02s]
epoch : 14
loss : 4.2020436401367185
quant_reg : 11.973605308532715
quant_err : 11.973605308532715
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
LSMDC_full_test/t2v_metrics/R1: 10.9
LSMDC_full_test/t2v_metrics/R5: 27.4
LSMDC_full_test/t2v_metrics/R10: 37.0
LSMDC_full_test/t2v_metrics/R50: 65.7
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 80.89
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.27372884065203
LSMDC_full_test/v2t_metrics/R1: 10.6
LSMDC_full_test/v2t_metrics/R5: 27.0
LSMDC_full_test/v2t_metrics/R10: 38.3
LSMDC_full_test/v2t_metrics/R50: 64.3
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 83.661
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.213797121013616
mnt_best : 22.437477122283305
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 3.71171 (QuantReg: 11.73104) QuantErr: 11.73104 batch_time=23.12100
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 3.98548 (QuantReg: 11.68129) QuantErr: 11.68129 batch_time=0.53465
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 4.17313 (QuantReg: 12.10297) QuantErr: 12.10297 batch_time=0.49861
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 4.30261 (QuantReg: 12.27151) QuantErr: 12.27151 batch_time=0.88894
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 3.72819 (QuantReg: 12.17077) QuantErr: 12.17077 batch_time=0.49782
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 4.36729 (QuantReg: 12.08445) QuantErr: 12.08445 batch_time=0.50377
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 3.61827 (QuantReg: 12.10411) QuantErr: 12.10411 batch_time=0.49711
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 4.00373 (QuantReg: 11.90976) QuantErr: 11.90976 batch_time=0.49694
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 4.28359 (QuantReg: 12.03613) QuantErr: 12.03613 batch_time=2.48690
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 4.34797 (QuantReg: 12.04231) QuantErr: 12.04231 batch_time=0.50403
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 3.90901 (QuantReg: 11.78219) QuantErr: 11.78219 batch_time=0.51356
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 4.01903 (QuantReg: 12.07748) QuantErr: 12.07748 batch_time=0.49271
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 4.07368 (QuantReg: 12.05181) QuantErr: 12.05181 batch_time=0.50614
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 4.21290 (QuantReg: 11.96358) QuantErr: 11.96358 batch_time=0.49831
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 3.83170 (QuantReg: 11.86279) QuantErr: 11.86279 batch_time=0.51418
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 4.19644 (QuantReg: 11.98440) QuantErr: 11.98440 batch_time=0.49877
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 4.38866 (QuantReg: 11.88161) QuantErr: 11.88161 batch_time=0.49390
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 4.11887 (QuantReg: 11.95141) QuantErr: 11.95141 batch_time=0.50263
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 4.29678 (QuantReg: 11.93588) QuantErr: 11.93588 batch_time=0.51321
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 3.93602 (QuantReg: 11.75589) QuantErr: 11.75589 batch_time=0.55056
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 4.22989 (QuantReg: 12.17871) QuantErr: 12.17871 batch_time=0.49458
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 4.04178 (QuantReg: 12.08093) QuantErr: 12.08093 batch_time=0.54205
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 4.08751 (QuantReg: 12.27526) QuantErr: 12.27526 batch_time=0.51667
Train Epoch: 15 codebook_update_time=1.63966
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch15.pth ...
Done in 5.082s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 4.117335777282715
quant_reg : 11.963492938995362
quant_err : 11.963492938995362
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
LSMDC_full_test/t2v_metrics/R1: 10.9
LSMDC_full_test/t2v_metrics/R5: 27.1
LSMDC_full_test/t2v_metrics/R10: 37.0
LSMDC_full_test/t2v_metrics/R50: 65.9
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 79.695
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.19213935915197
LSMDC_full_test/v2t_metrics/R1: 9.5
LSMDC_full_test/v2t_metrics/R5: 27.2
LSMDC_full_test/v2t_metrics/R10: 37.1
LSMDC_full_test/v2t_metrics/R50: 64.1
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 82.87
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.24330769652278
mnt_best : 22.437477122283305
not_improved_count: 2
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 4.20495 (QuantReg: 12.03353) QuantErr: 12.03353 batch_time=24.34456
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 3.90055 (QuantReg: 11.92511) QuantErr: 11.92511 batch_time=0.88908
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 3.87797 (QuantReg: 11.72136) QuantErr: 11.72136 batch_time=0.49891
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 4.33631 (QuantReg: 11.84010) QuantErr: 11.84010 batch_time=0.50040
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 4.43774 (QuantReg: 11.89795) QuantErr: 11.89795 batch_time=0.53884
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 4.35641 (QuantReg: 12.16141) QuantErr: 12.16141 batch_time=0.49739
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 4.12612 (QuantReg: 11.97172) QuantErr: 11.97172 batch_time=0.50543
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 4.35191 (QuantReg: 12.06970) QuantErr: 12.06970 batch_time=0.52492
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 3.81631 (QuantReg: 11.95790) QuantErr: 11.95790 batch_time=0.53058
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 4.40757 (QuantReg: 12.19230) QuantErr: 12.19230 batch_time=0.60459
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 4.06729 (QuantReg: 11.71492) QuantErr: 11.71492 batch_time=1.65610
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 4.46765 (QuantReg: 11.94655) QuantErr: 11.94655 batch_time=0.49720
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 4.05581 (QuantReg: 12.01272) QuantErr: 12.01272 batch_time=0.49932
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 4.15880 (QuantReg: 12.10262) QuantErr: 12.10262 batch_time=0.50158
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 3.64701 (QuantReg: 11.94151) QuantErr: 11.94151 batch_time=0.50765
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 4.03533 (QuantReg: 12.13558) QuantErr: 12.13558 batch_time=0.50705
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 3.97583 (QuantReg: 11.90761) QuantErr: 11.90761 batch_time=0.52319
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 4.03509 (QuantReg: 12.15423) QuantErr: 12.15423 batch_time=0.63589
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 3.99970 (QuantReg: 11.93654) QuantErr: 11.93654 batch_time=0.54262
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 4.19525 (QuantReg: 12.31611) QuantErr: 12.31611 batch_time=0.54322
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 3.87315 (QuantReg: 12.22471) QuantErr: 12.22471 batch_time=0.49935
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 3.72462 (QuantReg: 12.08218) QuantErr: 12.08218 batch_time=0.54413
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 4.06040 (QuantReg: 12.07014) QuantErr: 12.07014 batch_time=0.49418
Train Epoch: 16 codebook_update_time=1.64498
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch16.pth ...
Done in 5.272s
removing stale ckpt [epoch 15] [took 0.01s]
epoch : 16
loss : 4.070005658149719
quant_reg : 12.002485385894776
quant_err : 12.002485385894776
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
LSMDC_full_test/t2v_metrics/R1: 10.6
LSMDC_full_test/t2v_metrics/R5: 27.3
LSMDC_full_test/t2v_metrics/R10: 37.3
LSMDC_full_test/t2v_metrics/R50: 65.4
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 79.571
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.10000887232365
LSMDC_full_test/v2t_metrics/R1: 9.8
LSMDC_full_test/v2t_metrics/R5: 26.8
LSMDC_full_test/v2t_metrics/R10: 37.7
LSMDC_full_test/v2t_metrics/R50: 64.3
LSMDC_full_test/v2t_metrics/MedR: 20.5
LSMDC_full_test/v2t_metrics/MeanR: 83.089
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.473396335773998
mnt_best : 22.437477122283305
not_improved_count: 3
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 3.96003 (QuantReg: 11.90515) QuantErr: 11.90515 batch_time=20.35737
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 4.45535 (QuantReg: 12.08990) QuantErr: 12.08990 batch_time=0.49873
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 4.25426 (QuantReg: 12.03670) QuantErr: 12.03670 batch_time=0.49999
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 3.74002 (QuantReg: 11.96632) QuantErr: 11.96632 batch_time=0.48922
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 4.16386 (QuantReg: 11.83254) QuantErr: 11.83254 batch_time=0.58156
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 4.11859 (QuantReg: 11.96561) QuantErr: 11.96561 batch_time=0.49446
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 3.72191 (QuantReg: 12.07261) QuantErr: 12.07261 batch_time=0.50640
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 3.91276 (QuantReg: 12.11623) QuantErr: 12.11623 batch_time=0.51912
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 4.23970 (QuantReg: 12.04876) QuantErr: 12.04876 batch_time=0.50257
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 3.93453 (QuantReg: 12.30109) QuantErr: 12.30109 batch_time=0.50508
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 3.70506 (QuantReg: 12.06694) QuantErr: 12.06694 batch_time=0.50713
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 4.16510 (QuantReg: 11.93209) QuantErr: 11.93209 batch_time=0.50538
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 3.88610 (QuantReg: 11.89376) QuantErr: 11.89376 batch_time=4.52044
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 4.08232 (QuantReg: 11.95085) QuantErr: 11.95085 batch_time=0.50029
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 3.94699 (QuantReg: 11.92467) QuantErr: 11.92467 batch_time=0.49127
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 3.59270 (QuantReg: 12.17082) QuantErr: 12.17082 batch_time=0.51602
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 3.99073 (QuantReg: 11.74934) QuantErr: 11.74934 batch_time=0.50304
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 4.22070 (QuantReg: 12.01875) QuantErr: 12.01875 batch_time=0.60948
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 3.88031 (QuantReg: 12.18427) QuantErr: 12.18427 batch_time=0.51243
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 4.08631 (QuantReg: 11.84928) QuantErr: 11.84928 batch_time=0.49701
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 3.95554 (QuantReg: 12.10820) QuantErr: 12.10820 batch_time=0.54224
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 4.42714 (QuantReg: 12.06501) QuantErr: 12.06501 batch_time=0.50337
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 3.67393 (QuantReg: 12.08446) QuantErr: 12.08446 batch_time=0.51413
Train Epoch: 17 codebook_update_time=2.02026
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch17.pth ...
Done in 4.117s
removing stale ckpt [epoch 16] [took 0.06s]
epoch : 17
loss : 4.007415914535523
quant_reg : 11.976138214111328
quant_err : 11.976138214111328
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
LSMDC_full_test/t2v_metrics/R1: 10.7
LSMDC_full_test/t2v_metrics/R5: 26.7
LSMDC_full_test/t2v_metrics/R10: 36.3
LSMDC_full_test/t2v_metrics/R50: 64.5
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 82.385
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.807232541151457
LSMDC_full_test/v2t_metrics/R1: 10.7
LSMDC_full_test/v2t_metrics/R5: 26.4
LSMDC_full_test/v2t_metrics/R10: 37.3
LSMDC_full_test/v2t_metrics/R50: 64.8
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 83.427
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.922942533920608
mnt_best : 22.437477122283305
not_improved_count: 4
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 4.21455 (QuantReg: 12.08488) QuantErr: 12.08488 batch_time=26.15019
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 3.86522 (QuantReg: 12.19403) QuantErr: 12.19403 batch_time=0.53794
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 3.95218 (QuantReg: 12.13994) QuantErr: 12.13994 batch_time=0.49128
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 3.86950 (QuantReg: 11.93473) QuantErr: 11.93473 batch_time=0.51273
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 3.83629 (QuantReg: 12.10142) QuantErr: 12.10142 batch_time=0.49736
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 3.92143 (QuantReg: 11.98852) QuantErr: 11.98852 batch_time=0.52891
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 3.84241 (QuantReg: 11.90667) QuantErr: 11.90667 batch_time=0.50979
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 4.10821 (QuantReg: 12.08186) QuantErr: 12.08186 batch_time=0.53744
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 4.31218 (QuantReg: 11.75980) QuantErr: 11.75980 batch_time=0.49453
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 3.79579 (QuantReg: 12.07803) QuantErr: 12.07803 batch_time=0.50130
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 3.84287 (QuantReg: 12.04675) QuantErr: 12.04675 batch_time=0.51267
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 3.85859 (QuantReg: 12.14530) QuantErr: 12.14530 batch_time=0.49315
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 3.80167 (QuantReg: 11.83311) QuantErr: 11.83311 batch_time=3.60379
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 3.97540 (QuantReg: 11.93445) QuantErr: 11.93445 batch_time=0.50467
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 3.85539 (QuantReg: 12.10394) QuantErr: 12.10394 batch_time=0.49419
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 4.60678 (QuantReg: 12.05780) QuantErr: 12.05780 batch_time=0.50338
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 4.01843 (QuantReg: 12.09620) QuantErr: 12.09620 batch_time=0.49482
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 4.05036 (QuantReg: 11.98699) QuantErr: 11.98699 batch_time=0.49294
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 3.81495 (QuantReg: 12.17223) QuantErr: 12.17223 batch_time=0.49376
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 3.94315 (QuantReg: 11.96177) QuantErr: 11.96177 batch_time=0.49872
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 4.60372 (QuantReg: 12.14105) QuantErr: 12.14105 batch_time=0.49899
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 3.98197 (QuantReg: 11.97108) QuantErr: 11.97108 batch_time=0.50794
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 3.88547 (QuantReg: 11.52924) QuantErr: 11.52924 batch_time=0.50228
Train Epoch: 18 codebook_update_time=2.03894
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch18.pth ...
Done in 4.675s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch18.pth ...
Done in 8.750s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 3.9447631912231444
quant_reg : 11.998945713043213
quant_err : 11.998945713043213
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
LSMDC_full_test/t2v_metrics/R1: 11.5
LSMDC_full_test/t2v_metrics/R5: 27.3
LSMDC_full_test/t2v_metrics/R10: 37.6
LSMDC_full_test/t2v_metrics/R50: 65.1
LSMDC_full_test/t2v_metrics/MedR: 20.5
LSMDC_full_test/t2v_metrics/MeanR: 80.303
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.769287672405596
LSMDC_full_test/v2t_metrics/R1: 11.3
LSMDC_full_test/v2t_metrics/R5: 27.1
LSMDC_full_test/v2t_metrics/R10: 38.4
LSMDC_full_test/v2t_metrics/R50: 64.0
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 83.399
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.7401322579715
mnt_best : 22.769287672405596
not_improved_count: 0
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 3.65397 (QuantReg: 11.83205) QuantErr: 11.83205 batch_time=22.15298
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 4.07344 (QuantReg: 11.71856) QuantErr: 11.71856 batch_time=0.51172
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 4.27591 (QuantReg: 11.90775) QuantErr: 11.90775 batch_time=0.49705
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 3.92706 (QuantReg: 11.90639) QuantErr: 11.90639 batch_time=0.49397
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 3.80346 (QuantReg: 11.95644) QuantErr: 11.95644 batch_time=0.50147
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 4.23230 (QuantReg: 12.12631) QuantErr: 12.12631 batch_time=0.49975
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 3.93563 (QuantReg: 12.08845) QuantErr: 12.08845 batch_time=0.64590
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 3.83375 (QuantReg: 12.00493) QuantErr: 12.00493 batch_time=0.59255
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 3.57604 (QuantReg: 12.17501) QuantErr: 12.17501 batch_time=0.50567
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 3.89693 (QuantReg: 12.12657) QuantErr: 12.12657 batch_time=0.49738
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 4.03295 (QuantReg: 11.93300) QuantErr: 11.93300 batch_time=0.55071
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 3.80319 (QuantReg: 11.81885) QuantErr: 11.81885 batch_time=0.52067
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 3.70696 (QuantReg: 12.19654) QuantErr: 12.19654 batch_time=0.49982
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 3.80370 (QuantReg: 12.14981) QuantErr: 12.14981 batch_time=1.45933
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 4.05115 (QuantReg: 12.04823) QuantErr: 12.04823 batch_time=0.50929
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 3.77993 (QuantReg: 12.31752) QuantErr: 12.31752 batch_time=0.50390
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 3.86746 (QuantReg: 11.84582) QuantErr: 11.84582 batch_time=0.52399
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 3.80812 (QuantReg: 12.03674) QuantErr: 12.03674 batch_time=0.52600
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 3.66607 (QuantReg: 12.12889) QuantErr: 12.12889 batch_time=0.52579
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 4.03679 (QuantReg: 11.98263) QuantErr: 11.98263 batch_time=0.54021
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 3.61258 (QuantReg: 11.92834) QuantErr: 11.92834 batch_time=0.51000
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 3.82548 (QuantReg: 12.16972) QuantErr: 12.16972 batch_time=1.66439
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 3.67567 (QuantReg: 11.97469) QuantErr: 11.97469 batch_time=0.50296
Train Epoch: 19 codebook_update_time=1.67954
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.12/checkpoint-epoch19.pth ...
Done in 5.340s
removing stale ckpt [epoch 18] [took 0.01s]
epoch : 19
loss : 3.8900009508132936
quant_reg : 12.01949309539795
quant_err : 12.01949309539795
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
LSMDC_full_test/t2v_metrics/R1: 10.9
LSMDC_full_test/t2v_metrics/R5: 28.0
LSMDC_full_test/t2v_metrics/R10: 37.8
LSMDC_full_test/t2v_metrics/R50: 67.2
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 77.709
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.595681426113433
LSMDC_full_test/v2t_metrics/R1: 10.8
LSMDC_full_test/v2t_metrics/R5: 27.2
LSMDC_full_test/v2t_metrics/R10: 39.0
LSMDC_full_test/v2t_metrics/R50: 63.9
LSMDC_full_test/v2t_metrics/MedR: 22.0