-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_LSMDC_t0.03.txt
2593 lines (2593 loc) · 190 KB
/
HCQ_LSMDC_t0.03.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03
Preparing the dataloaders ...
Loading dataset LSMDC_full_trainval in ram ...
Finish loading dataset LSMDC_full_trainval in ram, taking 4760.364514827728 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 141.9911825656891 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 74.27371382713318 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch0.pth ...
Done in 2.998s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch0.pth ...
Done in 6.191s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
LSMDC_full_test/t2v_metrics/R1: 0.0
LSMDC_full_test/t2v_metrics/R5: 0.9
LSMDC_full_test/t2v_metrics/R10: 1.6
LSMDC_full_test/t2v_metrics/R50: 4.4
LSMDC_full_test/t2v_metrics/MedR: 508.5
LSMDC_full_test/t2v_metrics/MeanR: 502.992
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
LSMDC_full_test/v2t_metrics/R1: 0.0
LSMDC_full_test/v2t_metrics/R5: 0.3
LSMDC_full_test/v2t_metrics/R10: 0.9
LSMDC_full_test/v2t_metrics/R50: 5.1
LSMDC_full_test/v2t_metrics/MedR: 510.0
LSMDC_full_test/v2t_metrics/MeanR: 501.125
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
mnt_best : 0.0
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.98514 (QuantReg: 22.49566) QuantErr: 22.49566 batch_time=21.91914
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.48910 (QuantReg: 22.70084) QuantErr: 22.70084 batch_time=0.52391
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 8.93816 (QuantReg: 22.78124) QuantErr: 22.78124 batch_time=0.53800
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 8.30606 (QuantReg: 22.74493) QuantErr: 22.74493 batch_time=0.54080
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.99376 (QuantReg: 22.73358) QuantErr: 22.73358 batch_time=0.55327
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 7.56413 (QuantReg: 22.70661) QuantErr: 22.70661 batch_time=0.49225
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 7.37572 (QuantReg: 22.82521) QuantErr: 22.82521 batch_time=0.51497
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 7.04182 (QuantReg: 22.77296) QuantErr: 22.77296 batch_time=0.54121
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 7.10345 (QuantReg: 22.68091) QuantErr: 22.68091 batch_time=0.50533
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.74948 (QuantReg: 22.66408) QuantErr: 22.66408 batch_time=0.49888
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 7.16199 (QuantReg: 22.69939) QuantErr: 22.69939 batch_time=0.50002
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 6.28310 (QuantReg: 22.75757) QuantErr: 22.75757 batch_time=0.49195
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 6.71331 (QuantReg: 22.73063) QuantErr: 22.73063 batch_time=0.50863
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 7.05694 (QuantReg: 22.69620) QuantErr: 22.69620 batch_time=0.55725
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 6.58470 (QuantReg: 22.69415) QuantErr: 22.69415 batch_time=0.50642
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 6.22092 (QuantReg: 22.70195) QuantErr: 22.70195 batch_time=0.51615
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 6.95294 (QuantReg: 22.65427) QuantErr: 22.65427 batch_time=0.62338
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 6.37097 (QuantReg: 22.64941) QuantErr: 22.64941 batch_time=0.50990
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 6.09913 (QuantReg: 22.62158) QuantErr: 22.62158 batch_time=0.79666
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 6.46845 (QuantReg: 22.65393) QuantErr: 22.65393 batch_time=0.53439
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 5.78129 (QuantReg: 22.72780) QuantErr: 22.72780 batch_time=0.49858
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 5.81527 (QuantReg: 22.70182) QuantErr: 22.70182 batch_time=0.49669
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 7.06640 (QuantReg: 22.69108) QuantErr: 22.69108 batch_time=0.51602
Train Epoch: 1 codebook_update_time=1.94815
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch1.pth ...
Done in 4.843s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch1.pth ...
Done in 9.984s
epoch : 1
loss : 7.092259418487549
quant_reg : 22.70752062225342
quant_err : 22.70752062225342
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
LSMDC_full_test/t2v_metrics/R1: 7.6
LSMDC_full_test/t2v_metrics/R5: 20.0
LSMDC_full_test/t2v_metrics/R10: 27.5
LSMDC_full_test/t2v_metrics/R50: 56.2
LSMDC_full_test/t2v_metrics/MedR: 36.0
LSMDC_full_test/t2v_metrics/MeanR: 101.252
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 16.10863572309328
LSMDC_full_test/v2t_metrics/R1: 6.4
LSMDC_full_test/v2t_metrics/R5: 16.9
LSMDC_full_test/v2t_metrics/R10: 24.8
LSMDC_full_test/v2t_metrics/R50: 56.4
LSMDC_full_test/v2t_metrics/MedR: 40.0
LSMDC_full_test/v2t_metrics/MeanR: 104.073
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 13.894388984895713
mnt_best : 16.10863572309328
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 5.85796 (QuantReg: 7.25308) QuantErr: 7.25308 batch_time=18.86814
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 6.04242 (QuantReg: 7.27219) QuantErr: 7.27219 batch_time=0.50285
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 6.19086 (QuantReg: 7.79906) QuantErr: 7.79906 batch_time=0.51258
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 5.87618 (QuantReg: 7.99230) QuantErr: 7.99230 batch_time=0.49892
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 6.20215 (QuantReg: 7.49506) QuantErr: 7.49506 batch_time=0.50367
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 6.62935 (QuantReg: 8.32617) QuantErr: 8.32617 batch_time=0.50235
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 6.20273 (QuantReg: 7.93802) QuantErr: 7.93802 batch_time=0.48818
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 5.42480 (QuantReg: 8.37970) QuantErr: 8.37970 batch_time=0.54873
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 5.84970 (QuantReg: 8.87826) QuantErr: 8.87826 batch_time=0.51897
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.79398 (QuantReg: 9.53203) QuantErr: 9.53203 batch_time=0.50350
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 5.35992 (QuantReg: 8.67562) QuantErr: 8.67562 batch_time=0.51590
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 5.69155 (QuantReg: 9.19067) QuantErr: 9.19067 batch_time=0.52479
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 5.96735 (QuantReg: 8.86368) QuantErr: 8.86368 batch_time=0.51338
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 5.85022 (QuantReg: 9.32692) QuantErr: 9.32692 batch_time=1.15576
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 5.86422 (QuantReg: 9.06000) QuantErr: 9.06000 batch_time=0.54242
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 5.35152 (QuantReg: 9.44207) QuantErr: 9.44207 batch_time=0.51519
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 5.24609 (QuantReg: 9.37177) QuantErr: 9.37177 batch_time=0.52488
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 6.24805 (QuantReg: 9.97486) QuantErr: 9.97486 batch_time=0.51632
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 6.27312 (QuantReg: 9.72537) QuantErr: 9.72537 batch_time=0.50519
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 5.60582 (QuantReg: 10.09572) QuantErr: 10.09572 batch_time=0.51568
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 5.37810 (QuantReg: 9.83288) QuantErr: 9.83288 batch_time=0.50879
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 5.68556 (QuantReg: 10.38179) QuantErr: 10.38179 batch_time=0.49250
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 5.63055 (QuantReg: 9.80499) QuantErr: 9.80499 batch_time=0.53680
Train Epoch: 2 codebook_update_time=1.82449
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch2.pth ...
Done in 7.122s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch2.pth ...
Done in 12.001s
removing stale ckpt [epoch 1] [took 0.03s]
removing stale ckpt [epoch 0] [took 0.04s]
epoch : 2
loss : 5.724356561660767
quant_reg : 8.95188332939148
quant_err : 8.95188332939148
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
LSMDC_full_test/t2v_metrics/R1: 8.9
LSMDC_full_test/t2v_metrics/R5: 22.0
LSMDC_full_test/t2v_metrics/R10: 30.4
LSMDC_full_test/t2v_metrics/R50: 61.2
LSMDC_full_test/t2v_metrics/MedR: 32.0
LSMDC_full_test/t2v_metrics/MeanR: 88.651
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.12294435664251
LSMDC_full_test/v2t_metrics/R1: 7.4
LSMDC_full_test/v2t_metrics/R5: 20.0
LSMDC_full_test/v2t_metrics/R10: 29.9
LSMDC_full_test/v2t_metrics/R50: 58.8
LSMDC_full_test/v2t_metrics/MedR: 32.0
LSMDC_full_test/v2t_metrics/MeanR: 91.443
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.417649054676367
mnt_best : 18.12294435664251
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 5.66993 (QuantReg: 8.57466) QuantErr: 8.57466 batch_time=16.66592
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 5.12393 (QuantReg: 8.33496) QuantErr: 8.33496 batch_time=0.51031
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 5.67581 (QuantReg: 8.26792) QuantErr: 8.26792 batch_time=0.49946
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 5.11093 (QuantReg: 8.61757) QuantErr: 8.61757 batch_time=0.75288
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 5.09600 (QuantReg: 9.18409) QuantErr: 9.18409 batch_time=0.94599
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 5.23700 (QuantReg: 8.91786) QuantErr: 8.91786 batch_time=0.53305
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 5.09757 (QuantReg: 9.25711) QuantErr: 9.25711 batch_time=0.51697
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 5.72224 (QuantReg: 8.72482) QuantErr: 8.72482 batch_time=0.51812
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 5.17528 (QuantReg: 9.13321) QuantErr: 9.13321 batch_time=0.56856
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 4.88626 (QuantReg: 9.61572) QuantErr: 9.61572 batch_time=0.49939
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 5.13235 (QuantReg: 9.11867) QuantErr: 9.11867 batch_time=0.49586
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 5.02928 (QuantReg: 9.38129) QuantErr: 9.38129 batch_time=0.50290
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 5.16264 (QuantReg: 8.98436) QuantErr: 8.98436 batch_time=1.15353
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 5.35187 (QuantReg: 9.55695) QuantErr: 9.55695 batch_time=0.50277
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.81150 (QuantReg: 9.40491) QuantErr: 9.40491 batch_time=0.52853
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 5.22300 (QuantReg: 9.30166) QuantErr: 9.30166 batch_time=0.54446
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 4.88367 (QuantReg: 9.69522) QuantErr: 9.69522 batch_time=0.56793
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 5.37034 (QuantReg: 8.99964) QuantErr: 8.99964 batch_time=0.51057
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 4.71199 (QuantReg: 9.96626) QuantErr: 9.96626 batch_time=0.49373
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 5.21502 (QuantReg: 9.76937) QuantErr: 9.76937 batch_time=0.52039
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 4.86005 (QuantReg: 9.29659) QuantErr: 9.29659 batch_time=0.49999
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 4.99518 (QuantReg: 10.06645) QuantErr: 10.06645 batch_time=0.51903
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 5.05738 (QuantReg: 9.90371) QuantErr: 9.90371 batch_time=0.50786
Train Epoch: 3 codebook_update_time=1.68347
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch3.pth ...
Done in 4.852s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch3.pth ...
Done in 11.201s
removing stale ckpt [epoch 2] [took 0.71s]
epoch : 3
loss : 5.212393970489502
quant_reg : 9.19029257774353
quant_err : 9.19029257774353
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
LSMDC_full_test/t2v_metrics/R1: 9.0
LSMDC_full_test/t2v_metrics/R5: 24.6
LSMDC_full_test/t2v_metrics/R10: 33.1
LSMDC_full_test/t2v_metrics/R50: 62.3
LSMDC_full_test/t2v_metrics/MedR: 26.0
LSMDC_full_test/t2v_metrics/MeanR: 82.297
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.423844984319455
LSMDC_full_test/v2t_metrics/R1: 8.4
LSMDC_full_test/v2t_metrics/R5: 23.6
LSMDC_full_test/v2t_metrics/R10: 34.1
LSMDC_full_test/v2t_metrics/R50: 61.2
LSMDC_full_test/v2t_metrics/MedR: 29.0
LSMDC_full_test/v2t_metrics/MeanR: 85.204
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 18.908128974286086
mnt_best : 19.423844984319455
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 5.21241 (QuantReg: 8.59566) QuantErr: 8.59566 batch_time=21.14341
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 4.85766 (QuantReg: 9.16453) QuantErr: 9.16453 batch_time=0.50008
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 5.23485 (QuantReg: 9.16356) QuantErr: 9.16356 batch_time=0.52414
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 4.86192 (QuantReg: 9.31500) QuantErr: 9.31500 batch_time=0.50870
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 4.42637 (QuantReg: 9.86557) QuantErr: 9.86557 batch_time=0.50966
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 4.38862 (QuantReg: 9.75531) QuantErr: 9.75531 batch_time=0.49321
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 5.01486 (QuantReg: 9.54045) QuantErr: 9.54045 batch_time=0.49794
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 5.24407 (QuantReg: 9.28552) QuantErr: 9.28552 batch_time=0.52147
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 4.32244 (QuantReg: 9.35811) QuantErr: 9.35811 batch_time=0.50115
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 4.92305 (QuantReg: 9.25592) QuantErr: 9.25592 batch_time=0.57022
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 5.49514 (QuantReg: 9.16078) QuantErr: 9.16078 batch_time=0.53428
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 4.26159 (QuantReg: 9.88977) QuantErr: 9.88977 batch_time=0.50576
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 5.04031 (QuantReg: 9.88949) QuantErr: 9.88949 batch_time=0.52528
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 5.00679 (QuantReg: 9.53678) QuantErr: 9.53678 batch_time=0.50073
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 5.14073 (QuantReg: 9.67732) QuantErr: 9.67732 batch_time=0.49829
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 5.30803 (QuantReg: 9.74582) QuantErr: 9.74582 batch_time=0.52594
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 4.51567 (QuantReg: 9.87143) QuantErr: 9.87143 batch_time=0.53323
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 5.22786 (QuantReg: 9.82255) QuantErr: 9.82255 batch_time=0.51255
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 4.33789 (QuantReg: 10.47891) QuantErr: 10.47891 batch_time=0.55362
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 4.59808 (QuantReg: 10.20259) QuantErr: 10.20259 batch_time=0.51056
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 3.72073 (QuantReg: 10.75281) QuantErr: 10.75281 batch_time=0.50767
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 4.49928 (QuantReg: 10.78371) QuantErr: 10.78371 batch_time=0.51428
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 4.67988 (QuantReg: 10.10650) QuantErr: 10.10650 batch_time=0.51718
Train Epoch: 4 codebook_update_time=1.79246
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch4.pth ...
Done in 7.512s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch4.pth ...
Done in 12.374s
removing stale ckpt [epoch 3] [took 0.02s]
epoch : 4
loss : 4.837412541389465
quant_reg : 9.729822257995606
quant_err : 9.729822257995606
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
LSMDC_full_test/t2v_metrics/R1: 10.0
LSMDC_full_test/t2v_metrics/R5: 24.5
LSMDC_full_test/t2v_metrics/R10: 34.2
LSMDC_full_test/t2v_metrics/R50: 62.2
LSMDC_full_test/t2v_metrics/MedR: 27.0
LSMDC_full_test/t2v_metrics/MeanR: 82.627
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.310973060761608
LSMDC_full_test/v2t_metrics/R1: 8.7
LSMDC_full_test/v2t_metrics/R5: 24.1
LSMDC_full_test/v2t_metrics/R10: 33.4
LSMDC_full_test/v2t_metrics/R50: 61.7
LSMDC_full_test/v2t_metrics/MedR: 27.0
LSMDC_full_test/v2t_metrics/MeanR: 81.214
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.132024161726545
mnt_best : 20.310973060761608
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 5.59983 (QuantReg: 9.13127) QuantErr: 9.13127 batch_time=24.92952
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 4.34591 (QuantReg: 9.87340) QuantErr: 9.87340 batch_time=0.65228
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 4.64010 (QuantReg: 10.14518) QuantErr: 10.14518 batch_time=0.49431
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 4.93454 (QuantReg: 9.97691) QuantErr: 9.97691 batch_time=0.49978
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 4.69998 (QuantReg: 10.28778) QuantErr: 10.28778 batch_time=0.52610
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 4.19892 (QuantReg: 10.37155) QuantErr: 10.37155 batch_time=0.50864
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 4.47638 (QuantReg: 10.34951) QuantErr: 10.34951 batch_time=0.50361
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 4.90215 (QuantReg: 10.36512) QuantErr: 10.36512 batch_time=0.50630
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 5.22257 (QuantReg: 10.05035) QuantErr: 10.05035 batch_time=0.49593
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 4.96352 (QuantReg: 10.36400) QuantErr: 10.36400 batch_time=0.55892
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 4.72436 (QuantReg: 10.24490) QuantErr: 10.24490 batch_time=0.48727
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 4.47746 (QuantReg: 10.65119) QuantErr: 10.65119 batch_time=0.49884
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 4.21539 (QuantReg: 10.44792) QuantErr: 10.44792 batch_time=0.50316
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 4.42428 (QuantReg: 10.49244) QuantErr: 10.49244 batch_time=0.51197
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 4.86631 (QuantReg: 10.48986) QuantErr: 10.48986 batch_time=0.53642
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 4.84623 (QuantReg: 10.18270) QuantErr: 10.18270 batch_time=0.49819
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 4.05163 (QuantReg: 11.04078) QuantErr: 11.04078 batch_time=0.53615
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 4.20136 (QuantReg: 10.61311) QuantErr: 10.61311 batch_time=1.06911
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 4.24178 (QuantReg: 10.95977) QuantErr: 10.95977 batch_time=0.51780
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 4.45167 (QuantReg: 10.83725) QuantErr: 10.83725 batch_time=0.52030
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 4.10153 (QuantReg: 10.92734) QuantErr: 10.92734 batch_time=0.50021
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 4.33257 (QuantReg: 11.36714) QuantErr: 11.36714 batch_time=0.49847
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.82053 (QuantReg: 11.10239) QuantErr: 11.10239 batch_time=0.49222
Train Epoch: 5 codebook_update_time=2.01245
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch5.pth ...
Done in 5.090s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch5.pth ...
Done in 10.564s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 4.499166204452514
quant_reg : 10.506342262268067
quant_err : 10.506342262268067
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
LSMDC_full_test/t2v_metrics/R1: 11.6
LSMDC_full_test/t2v_metrics/R5: 27.4
LSMDC_full_test/t2v_metrics/R10: 36.4
LSMDC_full_test/t2v_metrics/R50: 64.4
LSMDC_full_test/t2v_metrics/MedR: 23.5
LSMDC_full_test/t2v_metrics/MeanR: 77.553
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.61708578234258
LSMDC_full_test/v2t_metrics/R1: 10.7
LSMDC_full_test/v2t_metrics/R5: 27.2
LSMDC_full_test/v2t_metrics/R10: 36.1
LSMDC_full_test/v2t_metrics/R50: 62.9
LSMDC_full_test/v2t_metrics/MedR: 25.0
LSMDC_full_test/v2t_metrics/MeanR: 78.271
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.902143893312374
mnt_best : 22.61708578234258
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 4.92775 (QuantReg: 10.40559) QuantErr: 10.40559 batch_time=17.25102
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 4.35566 (QuantReg: 10.42138) QuantErr: 10.42138 batch_time=0.51006
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 4.71130 (QuantReg: 10.54586) QuantErr: 10.54586 batch_time=0.49294
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 4.16861 (QuantReg: 10.94672) QuantErr: 10.94672 batch_time=0.67309
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 4.10506 (QuantReg: 10.94990) QuantErr: 10.94990 batch_time=0.49951
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 4.50014 (QuantReg: 10.55670) QuantErr: 10.55670 batch_time=0.49819
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 4.25427 (QuantReg: 10.78158) QuantErr: 10.78158 batch_time=0.52259
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 4.26807 (QuantReg: 11.07476) QuantErr: 11.07476 batch_time=0.51505
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 4.79834 (QuantReg: 10.52943) QuantErr: 10.52943 batch_time=0.51845
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 4.45844 (QuantReg: 10.81887) QuantErr: 10.81887 batch_time=0.50477
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 3.79251 (QuantReg: 11.34404) QuantErr: 11.34404 batch_time=0.50505
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 4.21643 (QuantReg: 10.77279) QuantErr: 10.77279 batch_time=0.50965
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 4.09066 (QuantReg: 11.36747) QuantErr: 11.36747 batch_time=0.51168
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 5.01391 (QuantReg: 11.10165) QuantErr: 11.10165 batch_time=0.50562
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 3.96706 (QuantReg: 11.58521) QuantErr: 11.58521 batch_time=0.57573
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 3.48161 (QuantReg: 11.36058) QuantErr: 11.36058 batch_time=0.53939
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 3.86988 (QuantReg: 11.55095) QuantErr: 11.55095 batch_time=0.50367
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 4.12991 (QuantReg: 11.71652) QuantErr: 11.71652 batch_time=0.50317
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 4.75764 (QuantReg: 11.21321) QuantErr: 11.21321 batch_time=0.52989
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 4.16933 (QuantReg: 11.54354) QuantErr: 11.54354 batch_time=0.49843
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 3.86918 (QuantReg: 11.82799) QuantErr: 11.82799 batch_time=0.50988
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 4.75584 (QuantReg: 10.68860) QuantErr: 10.68860 batch_time=0.52882
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 4.48516 (QuantReg: 11.08255) QuantErr: 11.08255 batch_time=0.52343
Train Epoch: 6 codebook_update_time=1.79143
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch6.pth ...
Done in 4.864s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch6.pth ...
Done in 10.266s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 4.292045039176941
quant_reg : 11.10253269958496
quant_err : 11.10253269958496
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
LSMDC_full_test/t2v_metrics/R1: 12.1
LSMDC_full_test/t2v_metrics/R5: 28.0
LSMDC_full_test/t2v_metrics/R10: 37.0
LSMDC_full_test/t2v_metrics/R50: 64.0
LSMDC_full_test/t2v_metrics/MedR: 23.5
LSMDC_full_test/t2v_metrics/MeanR: 76.18
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.22995536030829
LSMDC_full_test/v2t_metrics/R1: 10.8
LSMDC_full_test/v2t_metrics/R5: 27.7
LSMDC_full_test/v2t_metrics/R10: 36.4
LSMDC_full_test/v2t_metrics/R50: 64.1
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 80.026
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.16502894289201
mnt_best : 23.22995536030829
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 4.27519 (QuantReg: 11.19657) QuantErr: 11.19657 batch_time=18.25062
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 4.00196 (QuantReg: 11.28275) QuantErr: 11.28275 batch_time=0.50722
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 3.98518 (QuantReg: 10.98679) QuantErr: 10.98679 batch_time=0.50255
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 3.86824 (QuantReg: 11.40745) QuantErr: 11.40745 batch_time=0.49841
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 4.48059 (QuantReg: 11.10724) QuantErr: 11.10724 batch_time=0.49687
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 4.02998 (QuantReg: 11.58745) QuantErr: 11.58745 batch_time=0.89730
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 4.09439 (QuantReg: 11.32643) QuantErr: 11.32643 batch_time=0.50335
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 4.49210 (QuantReg: 11.29639) QuantErr: 11.29639 batch_time=1.44615
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 3.93772 (QuantReg: 11.01928) QuantErr: 11.01928 batch_time=0.49900
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 4.04468 (QuantReg: 11.41882) QuantErr: 11.41882 batch_time=0.49356
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 4.01500 (QuantReg: 11.96430) QuantErr: 11.96430 batch_time=0.52590
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 4.36456 (QuantReg: 11.46616) QuantErr: 11.46616 batch_time=0.51709
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 4.00231 (QuantReg: 11.51459) QuantErr: 11.51459 batch_time=0.50695
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 3.74251 (QuantReg: 11.78836) QuantErr: 11.78836 batch_time=1.12277
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 3.93490 (QuantReg: 12.00490) QuantErr: 12.00490 batch_time=0.53269
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 3.67835 (QuantReg: 11.52709) QuantErr: 11.52709 batch_time=0.49276
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 4.18690 (QuantReg: 11.83943) QuantErr: 11.83943 batch_time=0.49804
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 4.01805 (QuantReg: 12.04802) QuantErr: 12.04802 batch_time=0.50860
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 3.68288 (QuantReg: 11.77297) QuantErr: 11.77297 batch_time=0.54903
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 3.87602 (QuantReg: 11.90742) QuantErr: 11.90742 batch_time=0.51426
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 3.68986 (QuantReg: 12.01134) QuantErr: 12.01134 batch_time=0.51140
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 3.83951 (QuantReg: 12.15225) QuantErr: 12.15225 batch_time=0.53746
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 3.69156 (QuantReg: 12.09214) QuantErr: 12.09214 batch_time=0.49465
Train Epoch: 7 codebook_update_time=1.72572
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch7.pth ...
Done in 6.444s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch7.pth ...
Done in 11.531s
removing stale ckpt [epoch 6] [took 0.06s]
epoch : 7
loss : 4.053584080696106
quant_reg : 11.623634479522705
quant_err : 11.623634479522705
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
LSMDC_full_test/t2v_metrics/R1: 11.5
LSMDC_full_test/t2v_metrics/R5: 28.3
LSMDC_full_test/t2v_metrics/R10: 38.8
LSMDC_full_test/t2v_metrics/R50: 65.9
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 73.347
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.286559816165195
LSMDC_full_test/v2t_metrics/R1: 10.2
LSMDC_full_test/v2t_metrics/R5: 29.9
LSMDC_full_test/v2t_metrics/R10: 38.5
LSMDC_full_test/v2t_metrics/R50: 65.4
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 72.428
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.7288448027701
mnt_best : 23.286559816165195
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 4.01951 (QuantReg: 11.79506) QuantErr: 11.79506 batch_time=25.36421
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.66304 (QuantReg: 11.90194) QuantErr: 11.90194 batch_time=0.50843
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 4.08554 (QuantReg: 11.91057) QuantErr: 11.91057 batch_time=0.52684
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 4.44190 (QuantReg: 11.72733) QuantErr: 11.72733 batch_time=0.52284
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 3.52054 (QuantReg: 11.95941) QuantErr: 11.95941 batch_time=0.49495
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 3.87633 (QuantReg: 11.82593) QuantErr: 11.82593 batch_time=0.48428
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 3.73821 (QuantReg: 11.63395) QuantErr: 11.63395 batch_time=0.49099
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 4.20356 (QuantReg: 11.97766) QuantErr: 11.97766 batch_time=0.51074
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 3.55593 (QuantReg: 12.19468) QuantErr: 12.19468 batch_time=0.52431
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 3.84392 (QuantReg: 11.73869) QuantErr: 11.73869 batch_time=0.49500
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 4.28808 (QuantReg: 11.92125) QuantErr: 11.92125 batch_time=0.50263
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 3.80930 (QuantReg: 12.26013) QuantErr: 12.26013 batch_time=0.50855
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 3.78979 (QuantReg: 12.16919) QuantErr: 12.16919 batch_time=0.50490
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 3.49371 (QuantReg: 12.78449) QuantErr: 12.78449 batch_time=0.50374
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 4.04159 (QuantReg: 12.37179) QuantErr: 12.37179 batch_time=0.52054
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 3.54789 (QuantReg: 12.39912) QuantErr: 12.39912 batch_time=0.50941
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 3.84693 (QuantReg: 12.10783) QuantErr: 12.10783 batch_time=0.51325
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 3.67639 (QuantReg: 12.29393) QuantErr: 12.29393 batch_time=0.50662
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 3.77308 (QuantReg: 12.23176) QuantErr: 12.23176 batch_time=3.17189
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 3.89129 (QuantReg: 12.23362) QuantErr: 12.23362 batch_time=0.50942
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 3.51556 (QuantReg: 12.78162) QuantErr: 12.78162 batch_time=0.52595
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 3.59163 (QuantReg: 12.88474) QuantErr: 12.88474 batch_time=0.48721
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 3.71723 (QuantReg: 12.37965) QuantErr: 12.37965 batch_time=0.49077
Train Epoch: 8 codebook_update_time=1.68643
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch8.pth ...
Done in 4.634s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 3.854290849685669
quant_reg : 12.125828372955322
quant_err : 12.125828372955322
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
LSMDC_full_test/t2v_metrics/R1: 11.2
LSMDC_full_test/t2v_metrics/R5: 28.0
LSMDC_full_test/t2v_metrics/R10: 37.4
LSMDC_full_test/t2v_metrics/R50: 64.3
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 73.92
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.720395417306463
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 29.1
LSMDC_full_test/v2t_metrics/R10: 37.7
LSMDC_full_test/v2t_metrics/R50: 65.2
LSMDC_full_test/v2t_metrics/MedR: 22.0
LSMDC_full_test/v2t_metrics/MeanR: 74.84
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.279700740712105
mnt_best : 23.286559816165195
not_improved_count: 1
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 3.80164 (QuantReg: 11.72135) QuantErr: 11.72135 batch_time=18.89641
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 3.78352 (QuantReg: 12.34028) QuantErr: 12.34028 batch_time=0.50300
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 3.41287 (QuantReg: 12.40570) QuantErr: 12.40570 batch_time=0.50540
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 4.08132 (QuantReg: 12.80232) QuantErr: 12.80232 batch_time=0.58282
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 4.06146 (QuantReg: 12.55924) QuantErr: 12.55924 batch_time=0.49875
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 3.37241 (QuantReg: 13.10691) QuantErr: 13.10691 batch_time=0.50060
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 3.70851 (QuantReg: 12.40464) QuantErr: 12.40464 batch_time=1.74726
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 4.11947 (QuantReg: 12.22077) QuantErr: 12.22077 batch_time=0.53997
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 3.60145 (QuantReg: 12.38742) QuantErr: 12.38742 batch_time=0.51283
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 3.52052 (QuantReg: 12.78042) QuantErr: 12.78042 batch_time=0.50620
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 4.11031 (QuantReg: 12.45193) QuantErr: 12.45193 batch_time=0.49434
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 3.50147 (QuantReg: 12.70679) QuantErr: 12.70679 batch_time=0.51281
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 3.35034 (QuantReg: 12.97891) QuantErr: 12.97891 batch_time=0.60568
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 3.49207 (QuantReg: 13.14457) QuantErr: 13.14457 batch_time=0.49209
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 3.59021 (QuantReg: 12.78537) QuantErr: 12.78537 batch_time=0.50639
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 3.80107 (QuantReg: 13.10164) QuantErr: 13.10164 batch_time=0.48655
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 3.41802 (QuantReg: 13.14157) QuantErr: 13.14157 batch_time=0.83639
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 3.30109 (QuantReg: 12.61543) QuantErr: 12.61543 batch_time=0.51587
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 3.69969 (QuantReg: 12.76726) QuantErr: 12.76726 batch_time=0.48840
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 3.59705 (QuantReg: 13.39892) QuantErr: 13.39892 batch_time=0.52092
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 4.15098 (QuantReg: 13.23306) QuantErr: 13.23306 batch_time=0.53442
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 4.28318 (QuantReg: 13.05255) QuantErr: 13.05255 batch_time=0.50146
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 3.65125 (QuantReg: 12.83609) QuantErr: 12.83609 batch_time=0.49400
Train Epoch: 9 codebook_update_time=2.01689
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch9.pth ...
Done in 4.244s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch9.pth ...
Done in 9.645s
removing stale ckpt [epoch 8] [took 0.03s]
epoch : 9
loss : 3.6587708740234377
quant_reg : 12.783386840820313
quant_err : 12.783386840820313
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
LSMDC_full_test/t2v_metrics/R1: 11.6
LSMDC_full_test/t2v_metrics/R5: 28.3
LSMDC_full_test/t2v_metrics/R10: 39.0
LSMDC_full_test/t2v_metrics/R50: 66.9
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 71.086
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.393920547916284
LSMDC_full_test/v2t_metrics/R1: 11.7
LSMDC_full_test/v2t_metrics/R5: 30.0
LSMDC_full_test/v2t_metrics/R10: 40.0
LSMDC_full_test/v2t_metrics/R50: 66.4
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 69.368
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.12435455153434
mnt_best : 23.393920547916284
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 3.55606 (QuantReg: 12.88066) QuantErr: 12.88066 batch_time=21.11158
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 3.44633 (QuantReg: 12.24036) QuantErr: 12.24036 batch_time=0.51994
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 3.51605 (QuantReg: 12.45205) QuantErr: 12.45205 batch_time=0.50734
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 3.35607 (QuantReg: 12.73322) QuantErr: 12.73322 batch_time=0.62009
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 3.80106 (QuantReg: 12.96562) QuantErr: 12.96562 batch_time=0.51741
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 3.12608 (QuantReg: 13.22292) QuantErr: 13.22292 batch_time=0.50982
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 3.61008 (QuantReg: 13.06452) QuantErr: 13.06452 batch_time=0.50286
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 3.39375 (QuantReg: 12.78998) QuantErr: 12.78998 batch_time=0.53052
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 3.50597 (QuantReg: 13.57636) QuantErr: 13.57636 batch_time=0.50281
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 3.77503 (QuantReg: 13.21147) QuantErr: 13.21147 batch_time=0.49443
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 3.50998 (QuantReg: 13.23206) QuantErr: 13.23206 batch_time=0.50006
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 3.57188 (QuantReg: 13.02238) QuantErr: 13.02238 batch_time=0.53378
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 3.18008 (QuantReg: 13.41401) QuantErr: 13.41401 batch_time=0.63643
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 3.39019 (QuantReg: 13.41992) QuantErr: 13.41992 batch_time=0.49123
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 3.85300 (QuantReg: 13.43880) QuantErr: 13.43880 batch_time=0.49722
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 3.74395 (QuantReg: 13.74537) QuantErr: 13.74537 batch_time=0.51804
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.62280 (QuantReg: 13.15547) QuantErr: 13.15547 batch_time=0.50578
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 3.53173 (QuantReg: 13.62379) QuantErr: 13.62379 batch_time=0.52450
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 3.23016 (QuantReg: 13.38816) QuantErr: 13.38816 batch_time=0.49400
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 3.18348 (QuantReg: 13.54149) QuantErr: 13.54149 batch_time=0.50070
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 3.08639 (QuantReg: 13.67076) QuantErr: 13.67076 batch_time=0.51470
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.09069 (QuantReg: 13.44936) QuantErr: 13.44936 batch_time=0.50860
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 3.53184 (QuantReg: 13.49532) QuantErr: 13.49532 batch_time=0.58610
Train Epoch: 10 codebook_update_time=1.66901
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch10.pth ...
Done in 13.047s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch10.pth ...
Done in 17.947s
removing stale ckpt [epoch 9] [took 0.09s]
epoch : 10
loss : 3.4976204643249513
quant_reg : 13.22139315032959
quant_err : 13.22139315032959
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
LSMDC_full_test/t2v_metrics/R1: 12.9
LSMDC_full_test/t2v_metrics/R5: 29.5
LSMDC_full_test/t2v_metrics/R10: 39.2
LSMDC_full_test/t2v_metrics/R50: 67.3
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 71.785
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.616856712842402
LSMDC_full_test/v2t_metrics/R1: 10.8
LSMDC_full_test/v2t_metrics/R5: 28.7
LSMDC_full_test/v2t_metrics/R10: 37.6
LSMDC_full_test/v2t_metrics/R50: 65.3
LSMDC_full_test/v2t_metrics/MedR: 22.5
LSMDC_full_test/v2t_metrics/MeanR: 71.545
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.672417610346688
mnt_best : 24.616856712842402
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.86357 (QuantReg: 13.75952) QuantErr: 13.75952 batch_time=23.06334
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 3.50204 (QuantReg: 13.52185) QuantErr: 13.52185 batch_time=0.50218
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 3.50712 (QuantReg: 13.17315) QuantErr: 13.17315 batch_time=0.57662
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 3.41228 (QuantReg: 13.48015) QuantErr: 13.48015 batch_time=0.51385
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 3.48352 (QuantReg: 13.59643) QuantErr: 13.59643 batch_time=1.08306
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 4.00875 (QuantReg: 13.28920) QuantErr: 13.28920 batch_time=0.50460
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 3.54595 (QuantReg: 13.41714) QuantErr: 13.41714 batch_time=0.52157
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 3.44038 (QuantReg: 13.54706) QuantErr: 13.54706 batch_time=0.50011
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 3.47492 (QuantReg: 13.49974) QuantErr: 13.49974 batch_time=0.49748
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.78402 (QuantReg: 13.64721) QuantErr: 13.64721 batch_time=0.52155
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 3.51870 (QuantReg: 14.02256) QuantErr: 14.02256 batch_time=0.49921
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.79120 (QuantReg: 13.94070) QuantErr: 13.94070 batch_time=0.49495
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 3.53295 (QuantReg: 13.91559) QuantErr: 13.91559 batch_time=0.51962
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.98650 (QuantReg: 13.68198) QuantErr: 13.68198 batch_time=0.50111
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 3.30641 (QuantReg: 13.68255) QuantErr: 13.68255 batch_time=0.55327
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 3.94323 (QuantReg: 13.58136) QuantErr: 13.58136 batch_time=0.50508
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.83679 (QuantReg: 13.95010) QuantErr: 13.95010 batch_time=0.49440
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 3.13211 (QuantReg: 13.86448) QuantErr: 13.86448 batch_time=0.49846
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 3.46877 (QuantReg: 13.55467) QuantErr: 13.55467 batch_time=0.52034
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 3.56813 (QuantReg: 14.00273) QuantErr: 14.00273 batch_time=0.49905
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 3.40272 (QuantReg: 13.50689) QuantErr: 13.50689 batch_time=0.50290
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.81073 (QuantReg: 13.99825) QuantErr: 13.99825 batch_time=0.49514
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 3.27048 (QuantReg: 13.66777) QuantErr: 13.66777 batch_time=1.13240
Train Epoch: 11 codebook_update_time=1.68281
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch11.pth ...
Done in 4.753s
removing stale ckpt [epoch 10] [took 0.05s]
epoch : 11
loss : 3.3261898221969606
quant_reg : 13.727463394165039
quant_err : 13.727463394165039
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
LSMDC_full_test/t2v_metrics/R1: 11.4
LSMDC_full_test/t2v_metrics/R5: 28.6
LSMDC_full_test/t2v_metrics/R10: 39.4
LSMDC_full_test/t2v_metrics/R50: 66.2
LSMDC_full_test/t2v_metrics/MedR: 21.0
LSMDC_full_test/t2v_metrics/MeanR: 75.495
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.420115656219544
LSMDC_full_test/v2t_metrics/R1: 11.1
LSMDC_full_test/v2t_metrics/R5: 28.6
LSMDC_full_test/v2t_metrics/R10: 38.5
LSMDC_full_test/v2t_metrics/R50: 66.0
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 72.967
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.03473642178669
mnt_best : 24.616856712842402
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 3.22451 (QuantReg: 13.37775) QuantErr: 13.37775 batch_time=21.09080
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 2.93296 (QuantReg: 13.75723) QuantErr: 13.75723 batch_time=0.51634
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 3.80906 (QuantReg: 13.49538) QuantErr: 13.49538 batch_time=0.52216
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.73535 (QuantReg: 13.97484) QuantErr: 13.97484 batch_time=0.49368
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.73865 (QuantReg: 14.16261) QuantErr: 14.16261 batch_time=0.50280
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 2.94472 (QuantReg: 14.03619) QuantErr: 14.03619 batch_time=0.48970
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 3.28723 (QuantReg: 14.21892) QuantErr: 14.21892 batch_time=0.49750
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 3.26058 (QuantReg: 13.79285) QuantErr: 13.79285 batch_time=0.52067
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 3.14263 (QuantReg: 14.11921) QuantErr: 14.11921 batch_time=0.49917
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 3.01054 (QuantReg: 14.14971) QuantErr: 14.14971 batch_time=0.50220
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 3.21194 (QuantReg: 14.10735) QuantErr: 14.10735 batch_time=0.49902
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 3.06683 (QuantReg: 14.11983) QuantErr: 14.11983 batch_time=0.51695
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 3.58535 (QuantReg: 13.95160) QuantErr: 13.95160 batch_time=0.51728
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 3.50392 (QuantReg: 14.03537) QuantErr: 14.03537 batch_time=0.53609
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 3.24398 (QuantReg: 13.80696) QuantErr: 13.80696 batch_time=0.52459
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 3.30961 (QuantReg: 14.20040) QuantErr: 14.20040 batch_time=0.55319
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.34872 (QuantReg: 14.38853) QuantErr: 14.38853 batch_time=0.49588
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 3.48992 (QuantReg: 13.84877) QuantErr: 13.84877 batch_time=0.75623
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.78969 (QuantReg: 14.05698) QuantErr: 14.05698 batch_time=0.82324
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 3.59828 (QuantReg: 14.06470) QuantErr: 14.06470 batch_time=1.92016
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 3.11467 (QuantReg: 14.39524) QuantErr: 14.39524 batch_time=0.51979
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.95261 (QuantReg: 14.43019) QuantErr: 14.43019 batch_time=0.51288
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 3.16527 (QuantReg: 14.23497) QuantErr: 14.23497 batch_time=0.53160
Train Epoch: 12 codebook_update_time=1.74977
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch12.pth ...
Done in 4.129s
removing stale ckpt [epoch 11] [took 1.12s]
epoch : 12
loss : 3.1955235652923584
quant_reg : 14.04886220550537
quant_err : 14.04886220550537
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
LSMDC_full_test/t2v_metrics/R1: 11.8
LSMDC_full_test/t2v_metrics/R5: 30.0
LSMDC_full_test/t2v_metrics/R10: 40.1
LSMDC_full_test/t2v_metrics/R50: 67.2
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 73.268
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.21303398228545
LSMDC_full_test/v2t_metrics/R1: 12.0
LSMDC_full_test/v2t_metrics/R5: 29.0
LSMDC_full_test/v2t_metrics/R10: 39.6
LSMDC_full_test/v2t_metrics/R50: 66.2
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 69.624
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.974973913027696
mnt_best : 24.616856712842402
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.76852 (QuantReg: 14.17772) QuantErr: 14.17772 batch_time=19.11513
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 3.18306 (QuantReg: 14.15963) QuantErr: 14.15963 batch_time=3.00938
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 3.19366 (QuantReg: 14.65181) QuantErr: 14.65181 batch_time=0.50549
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 3.00801 (QuantReg: 14.42173) QuantErr: 14.42173 batch_time=0.50366
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.89210 (QuantReg: 14.60179) QuantErr: 14.60179 batch_time=0.52231
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 3.32535 (QuantReg: 14.17934) QuantErr: 14.17934 batch_time=0.50843
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 3.05248 (QuantReg: 14.48961) QuantErr: 14.48961 batch_time=0.50156
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 3.17158 (QuantReg: 14.72756) QuantErr: 14.72756 batch_time=0.51207
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 2.57556 (QuantReg: 14.46123) QuantErr: 14.46123 batch_time=0.51854
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 3.29616 (QuantReg: 14.36668) QuantErr: 14.36668 batch_time=0.52610
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 3.11498 (QuantReg: 14.44864) QuantErr: 14.44864 batch_time=0.50972
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 2.87972 (QuantReg: 14.71186) QuantErr: 14.71186 batch_time=0.51223
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 3.20946 (QuantReg: 14.42194) QuantErr: 14.42194 batch_time=0.65957
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 3.23980 (QuantReg: 14.51396) QuantErr: 14.51396 batch_time=0.50423
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.15181 (QuantReg: 15.09681) QuantErr: 15.09681 batch_time=0.49673
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 2.97988 (QuantReg: 14.95308) QuantErr: 14.95308 batch_time=0.54802
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 2.91702 (QuantReg: 14.34507) QuantErr: 14.34507 batch_time=0.52253
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 3.35139 (QuantReg: 14.57256) QuantErr: 14.57256 batch_time=0.48908
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 3.10484 (QuantReg: 14.80713) QuantErr: 14.80713 batch_time=0.49030
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.88035 (QuantReg: 14.36793) QuantErr: 14.36793 batch_time=0.48972
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 3.12092 (QuantReg: 14.39367) QuantErr: 14.39367 batch_time=0.88104
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 2.98814 (QuantReg: 14.51327) QuantErr: 14.51327 batch_time=0.49917
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.87690 (QuantReg: 14.58293) QuantErr: 14.58293 batch_time=0.54620
Train Epoch: 13 codebook_update_time=1.72889
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch13.pth ...
Done in 4.990s
removing stale ckpt [epoch 12] [took 0.02s]
epoch : 13
loss : 3.0417536563873293
quant_reg : 14.502163482666015
quant_err : 14.502163482666015
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
LSMDC_full_test/t2v_metrics/R1: 12.0
LSMDC_full_test/t2v_metrics/R5: 30.5
LSMDC_full_test/t2v_metrics/R10: 40.4
LSMDC_full_test/t2v_metrics/R50: 66.1
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 71.147
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.54449778038944
LSMDC_full_test/v2t_metrics/R1: 12.6
LSMDC_full_test/v2t_metrics/R5: 30.3
LSMDC_full_test/v2t_metrics/R10: 40.5
LSMDC_full_test/v2t_metrics/R50: 68.0
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 69.349
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.912810942895643
mnt_best : 24.616856712842402
not_improved_count: 3
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 3.12100 (QuantReg: 14.22288) QuantErr: 14.22288 batch_time=18.27684
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.91508 (QuantReg: 14.52956) QuantErr: 14.52956 batch_time=0.50376
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 3.39674 (QuantReg: 14.30454) QuantErr: 14.30454 batch_time=0.52273
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.65358 (QuantReg: 14.41420) QuantErr: 14.41420 batch_time=0.53637
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.77832 (QuantReg: 14.82712) QuantErr: 14.82712 batch_time=0.55203
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.81779 (QuantReg: 14.63865) QuantErr: 14.63865 batch_time=0.50998
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.80779 (QuantReg: 14.66038) QuantErr: 14.66038 batch_time=0.50619
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 3.70603 (QuantReg: 14.40348) QuantErr: 14.40348 batch_time=0.53372
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.79828 (QuantReg: 14.61130) QuantErr: 14.61130 batch_time=0.55997
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 3.13634 (QuantReg: 14.82778) QuantErr: 14.82778 batch_time=0.50708
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.79250 (QuantReg: 14.95485) QuantErr: 14.95485 batch_time=0.61971
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 3.24948 (QuantReg: 14.94348) QuantErr: 14.94348 batch_time=0.51232
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 3.25723 (QuantReg: 14.42719) QuantErr: 14.42719 batch_time=0.51230
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 2.91204 (QuantReg: 14.87614) QuantErr: 14.87614 batch_time=0.54351
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 3.09942 (QuantReg: 14.84713) QuantErr: 14.84713 batch_time=0.51351
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 2.95436 (QuantReg: 14.68797) QuantErr: 14.68797 batch_time=0.50098
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 2.85898 (QuantReg: 14.91199) QuantErr: 14.91199 batch_time=0.56564
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 3.29779 (QuantReg: 15.06883) QuantErr: 15.06883 batch_time=0.51245
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 3.36845 (QuantReg: 14.70109) QuantErr: 14.70109 batch_time=0.51503
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 3.05295 (QuantReg: 14.90351) QuantErr: 14.90351 batch_time=4.82036
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.70033 (QuantReg: 15.11292) QuantErr: 15.11292 batch_time=0.65418
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.61891 (QuantReg: 15.20076) QuantErr: 15.20076 batch_time=0.51438
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.79861 (QuantReg: 15.06131) QuantErr: 15.06131 batch_time=0.48950
Train Epoch: 14 codebook_update_time=1.68657
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch14.pth ...
Done in 6.144s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch14.pth ...
Done in 11.745s
removing stale ckpt [epoch 13] [took 0.05s]
epoch : 14
loss : 2.9563326864242554
quant_reg : 14.778128837585449
quant_err : 14.778128837585449
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
LSMDC_full_test/t2v_metrics/R1: 12.7
LSMDC_full_test/t2v_metrics/R5: 31.1
LSMDC_full_test/t2v_metrics/R10: 41.0
LSMDC_full_test/t2v_metrics/R50: 66.4
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 72.681
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.299735971873204
LSMDC_full_test/v2t_metrics/R1: 11.5
LSMDC_full_test/v2t_metrics/R5: 30.8
LSMDC_full_test/v2t_metrics/R10: 41.0
LSMDC_full_test/v2t_metrics/R50: 67.2
LSMDC_full_test/v2t_metrics/MedR: 18.5
LSMDC_full_test/v2t_metrics/MeanR: 69.904
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.39743321561595
mnt_best : 25.299735971873204
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.17381 (QuantReg: 14.81111) QuantErr: 14.81111 batch_time=18.53733
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.66257 (QuantReg: 14.59810) QuantErr: 14.59810 batch_time=0.52818
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.88604 (QuantReg: 15.03105) QuantErr: 15.03105 batch_time=0.51627
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.88330 (QuantReg: 14.95950) QuantErr: 14.95950 batch_time=0.55580
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.37074 (QuantReg: 15.17113) QuantErr: 15.17113 batch_time=0.60146
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 3.04723 (QuantReg: 15.28032) QuantErr: 15.28032 batch_time=0.52372
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.21254 (QuantReg: 15.44188) QuantErr: 15.44188 batch_time=0.95494
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.58171 (QuantReg: 15.03363) QuantErr: 15.03363 batch_time=0.50245
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 3.32210 (QuantReg: 15.04419) QuantErr: 15.04419 batch_time=0.52902
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 3.16479 (QuantReg: 14.89038) QuantErr: 14.89038 batch_time=0.50998
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.48817 (QuantReg: 15.30130) QuantErr: 15.30130 batch_time=0.50453
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.66156 (QuantReg: 15.17963) QuantErr: 15.17963 batch_time=0.50057
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.77744 (QuantReg: 15.19649) QuantErr: 15.19649 batch_time=0.50197
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 3.23574 (QuantReg: 15.41255) QuantErr: 15.41255 batch_time=0.63947
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.54232 (QuantReg: 15.29571) QuantErr: 15.29571 batch_time=0.52068
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.66207 (QuantReg: 15.43553) QuantErr: 15.43553 batch_time=0.51499
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 3.14212 (QuantReg: 15.37089) QuantErr: 15.37089 batch_time=0.51816
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 3.06674 (QuantReg: 15.23132) QuantErr: 15.23132 batch_time=0.52017
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 2.75841 (QuantReg: 15.12967) QuantErr: 15.12967 batch_time=0.53067
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.65386 (QuantReg: 15.10222) QuantErr: 15.10222 batch_time=0.50714
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.73065 (QuantReg: 15.51199) QuantErr: 15.51199 batch_time=0.50815
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.76489 (QuantReg: 15.40419) QuantErr: 15.40419 batch_time=0.50670
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.81402 (QuantReg: 15.27001) QuantErr: 15.27001 batch_time=0.55227
Train Epoch: 15 codebook_update_time=1.78296
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch15.pth ...
Done in 4.025s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch15.pth ...
Done in 9.262s
removing stale ckpt [epoch 14] [took 0.01s]
epoch : 15
loss : 2.8340960664749146
quant_reg : 15.148891372680664
quant_err : 15.148891372680664
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
LSMDC_full_test/t2v_metrics/R1: 12.5
LSMDC_full_test/t2v_metrics/R5: 32.2
LSMDC_full_test/t2v_metrics/R10: 41.9
LSMDC_full_test/t2v_metrics/R50: 68.5
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 71.273
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.644444888038073
LSMDC_full_test/v2t_metrics/R1: 11.9
LSMDC_full_test/v2t_metrics/R5: 30.7
LSMDC_full_test/v2t_metrics/R10: 40.6
LSMDC_full_test/v2t_metrics/R50: 67.2
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 68.168
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.569922700600685
mnt_best : 25.644444888038073
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 2.77261 (QuantReg: 15.33324) QuantErr: 15.33324 batch_time=22.29518
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 2.44277 (QuantReg: 15.28925) QuantErr: 15.28925 batch_time=0.49134
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 2.41497 (QuantReg: 15.42460) QuantErr: 15.42460 batch_time=0.49512
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 3.22362 (QuantReg: 15.12672) QuantErr: 15.12672 batch_time=0.49764
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 3.03899 (QuantReg: 15.04262) QuantErr: 15.04262 batch_time=0.49400
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 2.65507 (QuantReg: 15.45123) QuantErr: 15.45123 batch_time=0.81228
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.55663 (QuantReg: 15.12569) QuantErr: 15.12569 batch_time=0.48357
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 2.70479 (QuantReg: 15.28614) QuantErr: 15.28614 batch_time=0.48852
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 2.60047 (QuantReg: 15.61132) QuantErr: 15.61132 batch_time=0.53811
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 2.82833 (QuantReg: 15.51375) QuantErr: 15.51375 batch_time=0.47896
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 2.97279 (QuantReg: 15.19759) QuantErr: 15.19759 batch_time=0.49441
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 3.04664 (QuantReg: 14.94159) QuantErr: 14.94159 batch_time=0.52086
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.70644 (QuantReg: 15.33579) QuantErr: 15.33579 batch_time=0.50784
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 3.04902 (QuantReg: 15.46714) QuantErr: 15.46714 batch_time=0.49868
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.34154 (QuantReg: 15.77114) QuantErr: 15.77114 batch_time=0.52713
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 2.56416 (QuantReg: 15.84330) QuantErr: 15.84330 batch_time=0.50860
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 2.73383 (QuantReg: 15.32058) QuantErr: 15.32058 batch_time=0.54614
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 2.82107 (QuantReg: 15.68111) QuantErr: 15.68111 batch_time=0.49167
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.73596 (QuantReg: 15.65185) QuantErr: 15.65185 batch_time=0.55503
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 3.04710 (QuantReg: 15.69923) QuantErr: 15.69923 batch_time=0.51854
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.46631 (QuantReg: 15.82942) QuantErr: 15.82942 batch_time=0.56054
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 2.52048 (QuantReg: 15.84833) QuantErr: 15.84833 batch_time=0.51722
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 2.53139 (QuantReg: 15.83432) QuantErr: 15.83432 batch_time=0.48497
Train Epoch: 16 codebook_update_time=1.70776
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch16.pth ...
Done in 22.817s
removing stale ckpt [epoch 15] [took 0.02s]
epoch : 16
loss : 2.727659976005554
quant_reg : 15.453642143249512
quant_err : 15.453642143249512
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
LSMDC_full_test/t2v_metrics/R1: 12.2
LSMDC_full_test/t2v_metrics/R5: 31.4
LSMDC_full_test/t2v_metrics/R10: 40.6
LSMDC_full_test/t2v_metrics/R50: 67.5
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 71.457
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.961566545059643
LSMDC_full_test/v2t_metrics/R1: 11.6
LSMDC_full_test/v2t_metrics/R5: 30.0
LSMDC_full_test/v2t_metrics/R10: 40.4
LSMDC_full_test/v2t_metrics/R50: 66.0
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 70.796
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.13534639930271
mnt_best : 25.644444888038073
not_improved_count: 1
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 2.59346 (QuantReg: 15.34087) QuantErr: 15.34087 batch_time=19.01764
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 3.09792 (QuantReg: 15.32359) QuantErr: 15.32359 batch_time=0.49667
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 2.80327 (QuantReg: 15.68697) QuantErr: 15.68697 batch_time=0.49350
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.43258 (QuantReg: 15.93682) QuantErr: 15.93682 batch_time=0.51056
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 3.00264 (QuantReg: 15.58120) QuantErr: 15.58120 batch_time=0.50053
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 2.59998 (QuantReg: 15.34070) QuantErr: 15.34070 batch_time=0.53023
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 2.38001 (QuantReg: 15.71725) QuantErr: 15.71725 batch_time=0.49598
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 2.16054 (QuantReg: 15.91565) QuantErr: 15.91565 batch_time=0.51593
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 2.82402 (QuantReg: 15.68816) QuantErr: 15.68816 batch_time=0.48995
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 2.53376 (QuantReg: 15.89071) QuantErr: 15.89071 batch_time=0.50508
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.99106 (QuantReg: 16.20169) QuantErr: 16.20169 batch_time=0.52919
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 2.72143 (QuantReg: 15.78574) QuantErr: 15.78574 batch_time=0.62040
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 2.49087 (QuantReg: 15.88121) QuantErr: 15.88121 batch_time=0.82544
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.73397 (QuantReg: 15.73943) QuantErr: 15.73943 batch_time=0.50093
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 2.81079 (QuantReg: 15.67095) QuantErr: 15.67095 batch_time=0.50474
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.38976 (QuantReg: 15.86381) QuantErr: 15.86381 batch_time=0.49755
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 2.67466 (QuantReg: 15.65937) QuantErr: 15.65937 batch_time=0.50086
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 2.95485 (QuantReg: 15.77350) QuantErr: 15.77350 batch_time=0.50545
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.34612 (QuantReg: 16.11422) QuantErr: 16.11422 batch_time=0.50731
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 2.91950 (QuantReg: 15.80094) QuantErr: 15.80094 batch_time=0.50345
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 2.44938 (QuantReg: 15.91125) QuantErr: 15.91125 batch_time=0.51725
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 3.13643 (QuantReg: 15.72406) QuantErr: 15.72406 batch_time=0.49313
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 2.23940 (QuantReg: 15.85292) QuantErr: 15.85292 batch_time=0.50100
Train Epoch: 17 codebook_update_time=1.87817
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch17.pth ...
Done in 4.197s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 2.615510983943939
quant_reg : 15.71787160873413
quant_err : 15.71787160873413
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
LSMDC_full_test/t2v_metrics/R1: 12.3
LSMDC_full_test/t2v_metrics/R5: 32.2
LSMDC_full_test/t2v_metrics/R10: 41.5
LSMDC_full_test/t2v_metrics/R50: 66.8
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 72.13
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.42551118630265
LSMDC_full_test/v2t_metrics/R1: 10.7
LSMDC_full_test/v2t_metrics/R5: 29.7
LSMDC_full_test/v2t_metrics/R10: 40.8
LSMDC_full_test/v2t_metrics/R50: 67.2
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 69.901
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.492728699515187
mnt_best : 25.644444888038073
not_improved_count: 2
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.57441 (QuantReg: 15.56168) QuantErr: 15.56168 batch_time=20.09179
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 2.30914 (QuantReg: 15.83814) QuantErr: 15.83814 batch_time=0.50923
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 2.29952 (QuantReg: 15.79907) QuantErr: 15.79907 batch_time=0.54418
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 2.40313 (QuantReg: 16.00583) QuantErr: 16.00583 batch_time=0.49009
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 2.49552 (QuantReg: 16.01347) QuantErr: 16.01347 batch_time=0.49961
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 2.41809 (QuantReg: 15.91059) QuantErr: 15.91059 batch_time=0.49527
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 2.43758 (QuantReg: 16.02126) QuantErr: 16.02126 batch_time=0.51759
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 2.67626 (QuantReg: 16.15030) QuantErr: 16.15030 batch_time=0.52209
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 2.96486 (QuantReg: 15.74280) QuantErr: 15.74280 batch_time=0.51718
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 2.36262 (QuantReg: 16.01246) QuantErr: 16.01246 batch_time=0.50487
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 2.54643 (QuantReg: 16.17530) QuantErr: 16.17530 batch_time=0.50100
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 2.26385 (QuantReg: 16.22714) QuantErr: 16.22714 batch_time=0.50747
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.18449 (QuantReg: 15.99400) QuantErr: 15.99400 batch_time=0.50216
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 2.56102 (QuantReg: 16.09100) QuantErr: 16.09100 batch_time=0.59329
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 2.43820 (QuantReg: 16.09319) QuantErr: 16.09319 batch_time=0.49440
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 2.86350 (QuantReg: 15.95969) QuantErr: 15.95969 batch_time=0.49731
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 2.59239 (QuantReg: 16.07050) QuantErr: 16.07050 batch_time=0.52037
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 2.63140 (QuantReg: 15.89696) QuantErr: 15.89696 batch_time=0.52823
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.26499 (QuantReg: 16.30750) QuantErr: 16.30750 batch_time=0.51031
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 2.61949 (QuantReg: 16.08769) QuantErr: 16.08769 batch_time=0.49732
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 3.04502 (QuantReg: 15.88647) QuantErr: 15.88647 batch_time=0.50514
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.79537 (QuantReg: 16.05619) QuantErr: 16.05619 batch_time=0.82745
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 2.57514 (QuantReg: 16.00427) QuantErr: 16.00427 batch_time=0.49514
Train Epoch: 18 codebook_update_time=1.82026
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch18.pth ...
Done in 5.341s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 2.5075857481956483
quant_reg : 15.987174938201905
quant_err : 15.987174938201905
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
LSMDC_full_test/t2v_metrics/R1: 13.0
LSMDC_full_test/t2v_metrics/R5: 30.6
LSMDC_full_test/t2v_metrics/R10: 41.9
LSMDC_full_test/t2v_metrics/R50: 67.5
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 70.948
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.54423693966927
LSMDC_full_test/v2t_metrics/R1: 13.1
LSMDC_full_test/v2t_metrics/R5: 31.1
LSMDC_full_test/v2t_metrics/R10: 40.9
LSMDC_full_test/v2t_metrics/R50: 67.5
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 68.676
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.541809663588385
mnt_best : 25.644444888038073
not_improved_count: 3
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 2.07232 (QuantReg: 16.08555) QuantErr: 16.08555 batch_time=23.41700
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 2.79237 (QuantReg: 15.85509) QuantErr: 15.85509 batch_time=0.53675
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 2.69061 (QuantReg: 16.02020) QuantErr: 16.02020 batch_time=0.54055
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 2.57500 (QuantReg: 15.99179) QuantErr: 15.99179 batch_time=0.53789
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 2.26754 (QuantReg: 16.01073) QuantErr: 16.01073 batch_time=0.52230
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 2.94746 (QuantReg: 16.16800) QuantErr: 16.16800 batch_time=0.51877
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 2.54076 (QuantReg: 16.07229) QuantErr: 16.07229 batch_time=0.72808
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 2.73976 (QuantReg: 16.26838) QuantErr: 16.26838 batch_time=0.50295
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 2.36776 (QuantReg: 16.24125) QuantErr: 16.24125 batch_time=0.49366
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 2.44253 (QuantReg: 16.22419) QuantErr: 16.22419 batch_time=0.59740
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 2.61383 (QuantReg: 16.01265) QuantErr: 16.01265 batch_time=0.54776
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 2.52654 (QuantReg: 16.26793) QuantErr: 16.26793 batch_time=0.50084
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 2.27642 (QuantReg: 16.41724) QuantErr: 16.41724 batch_time=0.49957
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 2.65856 (QuantReg: 16.23636) QuantErr: 16.23636 batch_time=0.49572
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 2.64994 (QuantReg: 16.34918) QuantErr: 16.34918 batch_time=0.49643
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 2.32196 (QuantReg: 16.73325) QuantErr: 16.73325 batch_time=0.51732
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 2.22667 (QuantReg: 16.33953) QuantErr: 16.33953 batch_time=0.54417
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 2.35614 (QuantReg: 16.19517) QuantErr: 16.19517 batch_time=0.50216
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 2.04283 (QuantReg: 16.57066) QuantErr: 16.57066 batch_time=0.51214
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 2.54821 (QuantReg: 16.30731) QuantErr: 16.30731 batch_time=0.52303
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 2.43071 (QuantReg: 16.24306) QuantErr: 16.24306 batch_time=1.53864
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 2.35950 (QuantReg: 16.43497) QuantErr: 16.43497 batch_time=0.52010
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.94015 (QuantReg: 16.47412) QuantErr: 16.47412 batch_time=0.50406
Train Epoch: 19 codebook_update_time=1.72714
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_t0.03/checkpoint-epoch19.pth ...
Done in 4.270s
removing stale ckpt [epoch 18] [took 0.02s]
epoch : 19
loss : 2.4405412940979003
quant_reg : 16.22606852722168
quant_err : 16.22606852722168
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
LSMDC_full_test/t2v_metrics/R1: 11.7
LSMDC_full_test/t2v_metrics/R5: 32.2
LSMDC_full_test/t2v_metrics/R10: 43.4
LSMDC_full_test/t2v_metrics/R50: 68.3
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 70.028
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.381102773127758
LSMDC_full_test/v2t_metrics/R1: 11.1
LSMDC_full_test/v2t_metrics/R5: 30.6
LSMDC_full_test/v2t_metrics/R10: 41.2