-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_ActivityNet_bs128.txt
2597 lines (2597 loc) · 200 KB
/
HCQ_ActivityNet_bs128.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128
Preparing the dataloaders ...
Loading dataset ActivityNet_val1_trainval in ram ...
Finish loading dataset ActivityNet_val1_trainval in ram, taking 632.6719100475311 s.
Loading dataset ActivityNet_val1_test in ram ...
Finish loading dataset ActivityNet_val1_test in ram, taking 247.49860501289368 s.
Loading dataset ActivityNet_val1_test in ram ...
Finish loading dataset ActivityNet_val1_test in ram, taking 121.92634558677673 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch0.pth ...
Done in 4.065s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch0.pth ...
Done in 5.472s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
ActivityNet_val1_test/t2v_metrics/R1: 0.04067520846044336
ActivityNet_val1_test/t2v_metrics/R5: 0.18303843807199513
ActivityNet_val1_test/t2v_metrics/R10: 0.24405125076266015
ActivityNet_val1_test/t2v_metrics/R50: 0.9558673988204189
ActivityNet_val1_test/t2v_metrics/MedR: 2484.0
ActivityNet_val1_test/t2v_metrics/MeanR: 2488.6077893024203
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12202562538133006
ActivityNet_val1_test/v2t_metrics/R1: 0.04067520846044336
ActivityNet_val1_test/v2t_metrics/R5: 0.1016880211511084
ActivityNet_val1_test/v2t_metrics/R10: 0.22371364653243847
ActivityNet_val1_test/v2t_metrics/R50: 0.9151921903599756
ActivityNet_val1_test/v2t_metrics/MedR: 2511.0
ActivityNet_val1_test/v2t_metrics/MeanR: 2492.4876957494407
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.09744600075376822
mnt_best : 0.12202562538133006
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.86133 (QuantReg: 22.40812) QuantErr: 22.40812 batch_time=25.64044
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.70169 (QuantReg: 22.50962) QuantErr: 22.50962 batch_time=0.67414
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 6.70137 (QuantReg: 22.63057) QuantErr: 22.63057 batch_time=0.66981
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 5.03630 (QuantReg: 22.70709) QuantErr: 22.70709 batch_time=0.64778
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 4.17119 (QuantReg: 22.67682) QuantErr: 22.67682 batch_time=0.64939
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 3.20043 (QuantReg: 22.70436) QuantErr: 22.70436 batch_time=0.64543
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 3.14136 (QuantReg: 22.69837) QuantErr: 22.69837 batch_time=0.64444
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 2.64793 (QuantReg: 22.67697) QuantErr: 22.67697 batch_time=0.64670
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 2.33770 (QuantReg: 22.68045) QuantErr: 22.68045 batch_time=0.64766
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 2.19518 (QuantReg: 22.68900) QuantErr: 22.68900 batch_time=0.65009
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 1.91724 (QuantReg: 22.69458) QuantErr: 22.69458 batch_time=0.66993
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 2.02643 (QuantReg: 22.70510) QuantErr: 22.70510 batch_time=0.65584
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 2.08646 (QuantReg: 22.69926) QuantErr: 22.69926 batch_time=0.65366
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 1.62319 (QuantReg: 22.69029) QuantErr: 22.69029 batch_time=0.65103
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 1.71684 (QuantReg: 22.68892) QuantErr: 22.68892 batch_time=0.66234
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 1.69842 (QuantReg: 22.66619) QuantErr: 22.66619 batch_time=0.65726
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 1.45012 (QuantReg: 22.66401) QuantErr: 22.66401 batch_time=0.65079
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 1.72999 (QuantReg: 22.65899) QuantErr: 22.65899 batch_time=0.64874
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 1.57557 (QuantReg: 22.64421) QuantErr: 22.64421 batch_time=0.95710
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 1.33979 (QuantReg: 22.68754) QuantErr: 22.68754 batch_time=0.64471
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 1.13284 (QuantReg: 22.69653) QuantErr: 22.69653 batch_time=0.65480
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 1.11405 (QuantReg: 22.66403) QuantErr: 22.66403 batch_time=0.64806
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 1.17348 (QuantReg: 22.66912) QuantErr: 22.66912 batch_time=0.65051
Train Epoch: 1 codebook_update_time=2.26754
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch1.pth ...
Done in 6.425s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch1.pth ...
Done in 10.262s
epoch : 1
loss : 2.8291447224617006
quant_reg : 22.663669891357422
quant_err : 22.663669891357422
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
ActivityNet_val1_test/t2v_metrics/R1: 10.51454138702461
ActivityNet_val1_test/t2v_metrics/R5: 33.272320520642666
ActivityNet_val1_test/t2v_metrics/R10: 49.25767744559691
ActivityNet_val1_test/t2v_metrics/R50: 85.76367703884482
ActivityNet_val1_test/t2v_metrics/MedR: 11.0
ActivityNet_val1_test/t2v_metrics/MeanR: 34.624161073825505
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.829487281219194
ActivityNet_val1_test/v2t_metrics/R1: 12.670327435428106
ActivityNet_val1_test/v2t_metrics/R5: 35.81452104942038
ActivityNet_val1_test/v2t_metrics/R10: 50.96603620093553
ActivityNet_val1_test/v2t_metrics/R50: 87.12629652226968
ActivityNet_val1_test/v2t_metrics/MedR: 10.0
ActivityNet_val1_test/v2t_metrics/MeanR: 33.17144600366077
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.491104226739452
mnt_best : 25.829487281219194
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 1.26277 (QuantReg: 12.07542) QuantErr: 12.07542 batch_time=28.18180
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 1.05394 (QuantReg: 11.82825) QuantErr: 11.82825 batch_time=0.68909
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 0.92753 (QuantReg: 12.15348) QuantErr: 12.15348 batch_time=0.69392
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 1.02575 (QuantReg: 11.85568) QuantErr: 11.85568 batch_time=0.63482
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 0.94570 (QuantReg: 12.05942) QuantErr: 12.05942 batch_time=0.64486
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 0.98588 (QuantReg: 12.44884) QuantErr: 12.44884 batch_time=0.64232
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 1.07973 (QuantReg: 12.22546) QuantErr: 12.22546 batch_time=0.64170
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 1.07099 (QuantReg: 12.24922) QuantErr: 12.24922 batch_time=0.63859
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 0.89759 (QuantReg: 12.14025) QuantErr: 12.14025 batch_time=0.67049
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 0.80837 (QuantReg: 12.60327) QuantErr: 12.60327 batch_time=0.64313
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 0.81544 (QuantReg: 12.60614) QuantErr: 12.60614 batch_time=0.64876
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 0.97419 (QuantReg: 12.44439) QuantErr: 12.44439 batch_time=0.67584
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 0.85251 (QuantReg: 12.67172) QuantErr: 12.67172 batch_time=0.65811
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 0.99064 (QuantReg: 12.92047) QuantErr: 12.92047 batch_time=0.66034
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 0.84417 (QuantReg: 12.89269) QuantErr: 12.89269 batch_time=0.64940
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 0.74639 (QuantReg: 12.88044) QuantErr: 12.88044 batch_time=0.66719
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 0.76289 (QuantReg: 13.00110) QuantErr: 13.00110 batch_time=0.64327
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 0.68438 (QuantReg: 12.89093) QuantErr: 12.89093 batch_time=0.65601
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 0.58440 (QuantReg: 12.89680) QuantErr: 12.89680 batch_time=1.24785
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 0.56219 (QuantReg: 13.12439) QuantErr: 13.12439 batch_time=0.67302
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 0.73104 (QuantReg: 12.83913) QuantErr: 12.83913 batch_time=0.64764
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 0.63786 (QuantReg: 13.16199) QuantErr: 13.16199 batch_time=0.65071
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 0.69660 (QuantReg: 13.38215) QuantErr: 13.38215 batch_time=0.67959
Train Epoch: 2 codebook_update_time=1.78371
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch2.pth ...
Done in 3.997s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch2.pth ...
Done in 7.959s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 0.8595910174846649
quant_reg : 12.590146812438965
quant_err : 12.590146812438965
learning_rate : 5e-05
n_samples : 64000
n_steps : 500
ActivityNet_val1_test/t2v_metrics/R1: 12.466951393125889
ActivityNet_val1_test/t2v_metrics/R5: 37.27882855399634
ActivityNet_val1_test/t2v_metrics/R10: 53.65059995932479
ActivityNet_val1_test/t2v_metrics/R50: 89.32275777913362
ActivityNet_val1_test/t2v_metrics/MedR: 9.0
ActivityNet_val1_test/t2v_metrics/MeanR: 29.076876143990237
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 29.214538842551203
ActivityNet_val1_test/v2t_metrics/R1: 13.971934106162294
ActivityNet_val1_test/v2t_metrics/R5: 39.35326418547895
ActivityNet_val1_test/v2t_metrics/R10: 56.49786455155583
ActivityNet_val1_test/v2t_metrics/R50: 90.35997559487492
ActivityNet_val1_test/v2t_metrics/MedR: 8.0
ActivityNet_val1_test/v2t_metrics/MeanR: 27.75859263778727
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.43569798235668
mnt_best : 29.214538842551203
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 0.79840 (QuantReg: 10.64189) QuantErr: 10.64189 batch_time=26.46939
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 0.61587 (QuantReg: 11.22693) QuantErr: 11.22693 batch_time=0.64797
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 0.57006 (QuantReg: 10.84987) QuantErr: 10.84987 batch_time=0.65293
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 0.61525 (QuantReg: 10.83387) QuantErr: 10.83387 batch_time=0.69308
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 0.62599 (QuantReg: 10.76262) QuantErr: 10.76262 batch_time=0.66666
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 0.60638 (QuantReg: 10.41826) QuantErr: 10.41826 batch_time=0.64927
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 0.50063 (QuantReg: 10.83335) QuantErr: 10.83335 batch_time=0.64289
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 0.53655 (QuantReg: 10.74238) QuantErr: 10.74238 batch_time=0.64431
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 0.62439 (QuantReg: 10.76291) QuantErr: 10.76291 batch_time=0.64385
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 0.57050 (QuantReg: 11.11535) QuantErr: 11.11535 batch_time=0.66078
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 0.56590 (QuantReg: 10.83972) QuantErr: 10.83972 batch_time=0.65019
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 0.60695 (QuantReg: 11.02952) QuantErr: 11.02952 batch_time=0.64769
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 0.56641 (QuantReg: 11.14981) QuantErr: 11.14981 batch_time=6.35427
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 0.46224 (QuantReg: 11.58760) QuantErr: 11.58760 batch_time=0.65448
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 0.49089 (QuantReg: 11.30477) QuantErr: 11.30477 batch_time=0.66432
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 0.56007 (QuantReg: 11.07203) QuantErr: 11.07203 batch_time=0.64053
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 0.50887 (QuantReg: 11.19533) QuantErr: 11.19533 batch_time=0.64936
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 0.51533 (QuantReg: 10.79175) QuantErr: 10.79175 batch_time=0.64657
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 0.44646 (QuantReg: 11.37865) QuantErr: 11.37865 batch_time=0.65227
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 0.46518 (QuantReg: 11.29146) QuantErr: 11.29146 batch_time=0.63919
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 0.44970 (QuantReg: 11.39111) QuantErr: 11.39111 batch_time=0.66001
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 0.43778 (QuantReg: 11.46351) QuantErr: 11.46351 batch_time=0.64340
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 0.40081 (QuantReg: 11.12813) QuantErr: 11.12813 batch_time=0.64844
Train Epoch: 3 codebook_update_time=1.80891
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch3.pth ...
Done in 3.773s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch3.pth ...
Done in 7.530s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 0.5574689654111862
quant_reg : 11.020709091186523
quant_err : 11.020709091186523
learning_rate : 4.25e-05
n_samples : 96000
n_steps : 750
ActivityNet_val1_test/t2v_metrics/R1: 13.91092129347163
ActivityNet_val1_test/t2v_metrics/R5: 40.044742729306485
ActivityNet_val1_test/t2v_metrics/R10: 57.189343095383364
ActivityNet_val1_test/t2v_metrics/R50: 90.82774049217002
ActivityNet_val1_test/t2v_metrics/MedR: 8.0
ActivityNet_val1_test/t2v_metrics/MeanR: 26.956579214968478
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 31.700942161997556
ActivityNet_val1_test/v2t_metrics/R1: 15.334553589587147
ActivityNet_val1_test/v2t_metrics/R5: 40.98027252389669
ActivityNet_val1_test/v2t_metrics/R10: 58.08419768151312
ActivityNet_val1_test/v2t_metrics/R50: 90.78706528370958
ActivityNet_val1_test/v2t_metrics/MedR: 8.0
ActivityNet_val1_test/v2t_metrics/MeanR: 26.170225747406956
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.17172004951286
mnt_best : 31.700942161997556
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 0.43232 (QuantReg: 10.62519) QuantErr: 10.62519 batch_time=25.25426
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 0.37508 (QuantReg: 10.44307) QuantErr: 10.44307 batch_time=0.65011
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 0.46402 (QuantReg: 10.42327) QuantErr: 10.42327 batch_time=0.65018
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 0.48230 (QuantReg: 10.39592) QuantErr: 10.39592 batch_time=0.84728
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 0.39609 (QuantReg: 10.34249) QuantErr: 10.34249 batch_time=0.64858
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 0.39161 (QuantReg: 10.40416) QuantErr: 10.40416 batch_time=0.64212
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 0.44245 (QuantReg: 10.44605) QuantErr: 10.44605 batch_time=0.64170
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 0.55722 (QuantReg: 10.60080) QuantErr: 10.60080 batch_time=0.77546
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 0.34728 (QuantReg: 10.74629) QuantErr: 10.74629 batch_time=0.65227
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 0.39124 (QuantReg: 10.87798) QuantErr: 10.87798 batch_time=0.64453
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 0.39272 (QuantReg: 10.92550) QuantErr: 10.92550 batch_time=0.65245
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 0.49167 (QuantReg: 10.75740) QuantErr: 10.75740 batch_time=0.64907
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 0.35767 (QuantReg: 11.17390) QuantErr: 11.17390 batch_time=0.64221
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 0.44120 (QuantReg: 10.73508) QuantErr: 10.73508 batch_time=5.38336
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 0.41551 (QuantReg: 10.78856) QuantErr: 10.78856 batch_time=0.65696
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 0.38478 (QuantReg: 11.26255) QuantErr: 11.26255 batch_time=0.64619
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 0.40578 (QuantReg: 11.01624) QuantErr: 11.01624 batch_time=0.65806
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 0.27030 (QuantReg: 11.08467) QuantErr: 11.08467 batch_time=0.65531
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 0.39522 (QuantReg: 10.67382) QuantErr: 10.67382 batch_time=0.63947
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 0.41277 (QuantReg: 11.10534) QuantErr: 11.10534 batch_time=0.83802
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 0.41477 (QuantReg: 10.81035) QuantErr: 10.81035 batch_time=0.64765
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 0.35054 (QuantReg: 11.08756) QuantErr: 11.08756 batch_time=0.64078
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 0.45928 (QuantReg: 10.78584) QuantErr: 10.78584 batch_time=0.64121
Train Epoch: 4 codebook_update_time=2.18522
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch4.pth ...
Done in 4.064s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch4.pth ...
Done in 7.971s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 0.41555819773674013
quant_reg : 10.778521152496339
quant_err : 10.778521152496339
learning_rate : 4.25e-05
n_samples : 128000
n_steps : 1000
ActivityNet_val1_test/t2v_metrics/R1: 14.134634940004068
ActivityNet_val1_test/t2v_metrics/R5: 41.04128533658735
ActivityNet_val1_test/t2v_metrics/R10: 57.779133618059795
ActivityNet_val1_test/t2v_metrics/R50: 91.21415497254424
ActivityNet_val1_test/t2v_metrics/MedR: 8.0
ActivityNet_val1_test/t2v_metrics/MeanR: 27.258490949766117
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 32.24226366217791
ActivityNet_val1_test/v2t_metrics/R1: 15.659955257270694
ActivityNet_val1_test/v2t_metrics/R5: 42.261541590400654
ActivityNet_val1_test/v2t_metrics/R10: 59.99593247915396
ActivityNet_val1_test/v2t_metrics/R50: 91.58023184868823
ActivityNet_val1_test/v2t_metrics/MedR: 7.0
ActivityNet_val1_test/v2t_metrics/MeanR: 25.550945698596706
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 34.11556382061522
mnt_best : 32.24226366217791
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 0.30199 (QuantReg: 10.68278) QuantErr: 10.68278 batch_time=20.90157
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 0.30570 (QuantReg: 10.81147) QuantErr: 10.81147 batch_time=0.66451
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 0.30977 (QuantReg: 10.61954) QuantErr: 10.61954 batch_time=0.64131
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 0.44453 (QuantReg: 10.64198) QuantErr: 10.64198 batch_time=0.98888
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 0.36817 (QuantReg: 10.68812) QuantErr: 10.68812 batch_time=0.64454
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 0.32842 (QuantReg: 10.73813) QuantErr: 10.73813 batch_time=0.64652
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 0.33464 (QuantReg: 10.84562) QuantErr: 10.84562 batch_time=1.95640
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 0.37644 (QuantReg: 10.68724) QuantErr: 10.68724 batch_time=0.65009
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 0.32725 (QuantReg: 10.60026) QuantErr: 10.60026 batch_time=0.64105
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 0.34655 (QuantReg: 10.73516) QuantErr: 10.73516 batch_time=0.65553
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 0.39646 (QuantReg: 10.84473) QuantErr: 10.84473 batch_time=0.70335
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 0.29523 (QuantReg: 10.92056) QuantErr: 10.92056 batch_time=0.68025
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 0.26595 (QuantReg: 10.60654) QuantErr: 10.60654 batch_time=1.46403
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 0.28061 (QuantReg: 10.41490) QuantErr: 10.41490 batch_time=2.33779
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 0.30912 (QuantReg: 10.88607) QuantErr: 10.88607 batch_time=0.64181
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 0.33144 (QuantReg: 10.93570) QuantErr: 10.93570 batch_time=1.11199
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 0.30662 (QuantReg: 10.79770) QuantErr: 10.79770 batch_time=0.66175
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 0.30640 (QuantReg: 10.78586) QuantErr: 10.78586 batch_time=0.64394
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 0.28345 (QuantReg: 11.02093) QuantErr: 11.02093 batch_time=1.57414
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 0.24664 (QuantReg: 10.97987) QuantErr: 10.97987 batch_time=0.64367
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 0.30425 (QuantReg: 10.96988) QuantErr: 10.96988 batch_time=0.63877
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 0.24305 (QuantReg: 10.93338) QuantErr: 10.93338 batch_time=0.65091
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 0.26338 (QuantReg: 10.85322) QuantErr: 10.85322 batch_time=0.66906
Train Epoch: 5 codebook_update_time=1.76750
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch5.pth ...
Done in 3.954s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch5.pth ...
Done in 7.825s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 0.3285088377594948
quant_reg : 10.775203769683838
quant_err : 10.775203769683838
learning_rate : 3.6125000000000004e-05
n_samples : 160000
n_steps : 1250
ActivityNet_val1_test/t2v_metrics/R1: 15.395566402277812
ActivityNet_val1_test/t2v_metrics/R5: 41.976815131177545
ActivityNet_val1_test/t2v_metrics/R10: 58.73500101688021
ActivityNet_val1_test/t2v_metrics/R50: 90.94976611755135
ActivityNet_val1_test/t2v_metrics/MedR: 8.0
ActivityNet_val1_test/t2v_metrics/MeanR: 27.22106975798251
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 33.60733271058537
ActivityNet_val1_test/v2t_metrics/R1: 15.151515151515152
ActivityNet_val1_test/v2t_metrics/R5: 42.50559284116331
ActivityNet_val1_test/v2t_metrics/R10: 60.52471018913972
ActivityNet_val1_test/v2t_metrics/R50: 91.80394549522066
ActivityNet_val1_test/v2t_metrics/MedR: 7.0
ActivityNet_val1_test/v2t_metrics/MeanR: 25.453731950376245
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.906135007095614
mnt_best : 33.60733271058537
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 0.38843 (QuantReg: 10.68040) QuantErr: 10.68040 batch_time=23.57521
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 0.25647 (QuantReg: 10.88374) QuantErr: 10.88374 batch_time=0.73616
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 0.30235 (QuantReg: 10.81501) QuantErr: 10.81501 batch_time=0.92176
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 0.32118 (QuantReg: 10.32021) QuantErr: 10.32021 batch_time=0.64599
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 0.27893 (QuantReg: 10.75209) QuantErr: 10.75209 batch_time=0.64730
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 0.33599 (QuantReg: 10.71778) QuantErr: 10.71778 batch_time=0.81387
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 0.27137 (QuantReg: 10.69643) QuantErr: 10.69643 batch_time=0.81934
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 0.35389 (QuantReg: 10.91389) QuantErr: 10.91389 batch_time=0.65705
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 0.31815 (QuantReg: 10.80056) QuantErr: 10.80056 batch_time=0.65238
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 0.26736 (QuantReg: 11.06353) QuantErr: 11.06353 batch_time=0.70854
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 0.22190 (QuantReg: 10.85609) QuantErr: 10.85609 batch_time=0.64976
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 0.28935 (QuantReg: 10.70486) QuantErr: 10.70486 batch_time=0.64283
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 0.28896 (QuantReg: 10.81706) QuantErr: 10.81706 batch_time=0.63972
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 0.25153 (QuantReg: 10.69076) QuantErr: 10.69076 batch_time=0.68715
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 0.21526 (QuantReg: 10.85252) QuantErr: 10.85252 batch_time=1.42898
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 0.31394 (QuantReg: 10.64979) QuantErr: 10.64979 batch_time=0.65426
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 0.25826 (QuantReg: 10.97348) QuantErr: 10.97348 batch_time=0.64921
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 0.27156 (QuantReg: 11.05665) QuantErr: 11.05665 batch_time=0.64686
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 0.24975 (QuantReg: 10.76671) QuantErr: 10.76671 batch_time=0.64324
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 0.31362 (QuantReg: 11.02974) QuantErr: 11.02974 batch_time=0.65757
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 0.21315 (QuantReg: 10.87427) QuantErr: 10.87427 batch_time=0.64795
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 0.21610 (QuantReg: 10.98182) QuantErr: 10.98182 batch_time=0.64364
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 0.25352 (QuantReg: 10.77214) QuantErr: 10.77214 batch_time=0.64421
Train Epoch: 6 codebook_update_time=1.93863
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch6.pth ...
Done in 15.099s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch6.pth ...
Done in 18.908s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 0.28429930919408797
quant_reg : 10.763820152282715
quant_err : 10.763820152282715
learning_rate : 3.6125000000000004e-05
n_samples : 192000
n_steps : 1500
ActivityNet_val1_test/t2v_metrics/R1: 15.598942444580029
ActivityNet_val1_test/t2v_metrics/R5: 43.74618669920683
ActivityNet_val1_test/t2v_metrics/R10: 60.5043725849095
ActivityNet_val1_test/t2v_metrics/R50: 91.31584299369534
ActivityNet_val1_test/t2v_metrics/MedR: 7.0
ActivityNet_val1_test/t2v_metrics/MeanR: 25.54403091315843
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 34.56267703528711
ActivityNet_val1_test/v2t_metrics/R1: 16.67683546878178
ActivityNet_val1_test/v2t_metrics/R5: 44.29530201342282
ActivityNet_val1_test/v2t_metrics/R10: 62.21273134024812
ActivityNet_val1_test/v2t_metrics/R50: 91.76327028676022
ActivityNet_val1_test/v2t_metrics/MedR: 7.0
ActivityNet_val1_test/v2t_metrics/MeanR: 24.98637380516575
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 35.81928066679331
mnt_best : 34.56267703528711
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 0.30229 (QuantReg: 10.55149) QuantErr: 10.55149 batch_time=22.30147
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 0.30615 (QuantReg: 10.48997) QuantErr: 10.48997 batch_time=1.10413
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 0.27374 (QuantReg: 10.65777) QuantErr: 10.65777 batch_time=1.54219
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 0.24293 (QuantReg: 11.03754) QuantErr: 11.03754 batch_time=0.64909
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 0.22156 (QuantReg: 10.78556) QuantErr: 10.78556 batch_time=0.74545
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 0.20372 (QuantReg: 11.07190) QuantErr: 11.07190 batch_time=0.64967
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 0.23892 (QuantReg: 10.72866) QuantErr: 10.72866 batch_time=0.84757
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 0.24414 (QuantReg: 10.57635) QuantErr: 10.57635 batch_time=7.97332
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 0.23203 (QuantReg: 10.79638) QuantErr: 10.79638 batch_time=0.65295
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 0.29016 (QuantReg: 10.71004) QuantErr: 10.71004 batch_time=0.65517
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 0.19035 (QuantReg: 10.83233) QuantErr: 10.83233 batch_time=0.64876
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 0.21100 (QuantReg: 10.89036) QuantErr: 10.89036 batch_time=0.65237
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 0.22106 (QuantReg: 10.51180) QuantErr: 10.51180 batch_time=0.64304
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 0.24024 (QuantReg: 10.97250) QuantErr: 10.97250 batch_time=0.64126
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 0.31380 (QuantReg: 10.80200) QuantErr: 10.80200 batch_time=0.65788
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 0.23145 (QuantReg: 10.87861) QuantErr: 10.87861 batch_time=0.64654
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 0.25852 (QuantReg: 10.60952) QuantErr: 10.60952 batch_time=0.67915
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 0.23383 (QuantReg: 11.04639) QuantErr: 11.04639 batch_time=0.65489
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 0.29354 (QuantReg: 10.82287) QuantErr: 10.82287 batch_time=0.64388
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 0.15268 (QuantReg: 10.92702) QuantErr: 10.92702 batch_time=0.64072
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 0.19315 (QuantReg: 10.92085) QuantErr: 10.92085 batch_time=0.64300
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 0.19035 (QuantReg: 10.93960) QuantErr: 10.93960 batch_time=0.66746
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 0.21987 (QuantReg: 10.62396) QuantErr: 10.62396 batch_time=0.66054
Train Epoch: 7 codebook_update_time=1.83978
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch7.pth ...
Done in 21.442s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch7.pth ...
Done in 25.408s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 0.24280645179748536
quant_reg : 10.767874702453613
quant_err : 10.767874702453613
learning_rate : 3.0706250000000004e-05
n_samples : 224000
n_steps : 1750
ActivityNet_val1_test/t2v_metrics/R1: 15.924344112263576
ActivityNet_val1_test/t2v_metrics/R5: 44.25462680496238
ActivityNet_val1_test/t2v_metrics/R10: 61.541590400650804
ActivityNet_val1_test/t2v_metrics/R50: 90.99044132601179
ActivityNet_val1_test/t2v_metrics/MedR: 7.0
ActivityNet_val1_test/t2v_metrics/MeanR: 26.680496237543217
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.13416614052744
ActivityNet_val1_test/v2t_metrics/R1: 17.388651616839535
ActivityNet_val1_test/v2t_metrics/R5: 45.12914378686191
ActivityNet_val1_test/v2t_metrics/R10: 63.412649989831195
ActivityNet_val1_test/v2t_metrics/R50: 91.49888143176734
ActivityNet_val1_test/v2t_metrics/MedR: 7.0
ActivityNet_val1_test/v2t_metrics/MeanR: 25.06772422208664
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.78179907256159
mnt_best : 35.13416614052744
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 0.25017 (QuantReg: 10.72681) QuantErr: 10.72681 batch_time=23.83086
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 0.18028 (QuantReg: 10.70438) QuantErr: 10.70438 batch_time=0.64441
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 0.20877 (QuantReg: 10.93380) QuantErr: 10.93380 batch_time=0.66160
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 0.21291 (QuantReg: 10.83552) QuantErr: 10.83552 batch_time=0.63939
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 0.23888 (QuantReg: 10.89232) QuantErr: 10.89232 batch_time=2.80453
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 0.27161 (QuantReg: 10.70914) QuantErr: 10.70914 batch_time=0.64833
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 0.24327 (QuantReg: 10.78460) QuantErr: 10.78460 batch_time=0.64174
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 0.24865 (QuantReg: 10.71569) QuantErr: 10.71569 batch_time=1.15614
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 0.25380 (QuantReg: 10.72511) QuantErr: 10.72511 batch_time=0.67281
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 0.21028 (QuantReg: 10.64086) QuantErr: 10.64086 batch_time=0.64134
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 0.17547 (QuantReg: 10.78639) QuantErr: 10.78639 batch_time=0.64515
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 0.20443 (QuantReg: 10.50945) QuantErr: 10.50945 batch_time=0.64169
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 0.16856 (QuantReg: 10.82636) QuantErr: 10.82636 batch_time=0.65094
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 0.17489 (QuantReg: 10.66352) QuantErr: 10.66352 batch_time=0.66192
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 0.21084 (QuantReg: 10.60990) QuantErr: 10.60990 batch_time=0.64037
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 0.16471 (QuantReg: 10.93224) QuantErr: 10.93224 batch_time=0.64321
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 0.17529 (QuantReg: 10.57649) QuantErr: 10.57649 batch_time=0.65876
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 0.17846 (QuantReg: 10.87887) QuantErr: 10.87887 batch_time=0.64393
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 0.17722 (QuantReg: 10.84530) QuantErr: 10.84530 batch_time=0.63767
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 0.19346 (QuantReg: 10.58539) QuantErr: 10.58539 batch_time=2.67068
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 0.18837 (QuantReg: 10.97056) QuantErr: 10.97056 batch_time=0.69199
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 0.25635 (QuantReg: 10.91460) QuantErr: 10.91460 batch_time=0.64546
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 0.14814 (QuantReg: 10.53818) QuantErr: 10.53818 batch_time=0.63462
Train Epoch: 8 codebook_update_time=1.76113
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch8.pth ...
Done in 4.017s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch8.pth ...
Done in 8.313s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 0.22032551497220992
quant_reg : 10.79703706741333
quant_err : 10.79703706741333
learning_rate : 3.0706250000000004e-05
n_samples : 256000
n_steps : 2000
ActivityNet_val1_test/t2v_metrics/R1: 16.107382550335572
ActivityNet_val1_test/t2v_metrics/R5: 44.86475493186902
ActivityNet_val1_test/t2v_metrics/R10: 62.02969290217612
ActivityNet_val1_test/t2v_metrics/R50: 91.25483018100468
ActivityNet_val1_test/t2v_metrics/MedR: 7.0
ActivityNet_val1_test/t2v_metrics/MeanR: 26.9538336383974
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 35.523027174004156
ActivityNet_val1_test/v2t_metrics/R1: 16.900549115314217
ActivityNet_val1_test/v2t_metrics/R5: 46.02399837299166
ActivityNet_val1_test/v2t_metrics/R10: 64.44986780557251
ActivityNet_val1_test/v2t_metrics/R50: 91.76327028676022
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 25.474069554606466
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 36.872484128282494
mnt_best : 35.523027174004156
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 0.17867 (QuantReg: 10.87741) QuantErr: 10.87741 batch_time=24.18010
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 0.17991 (QuantReg: 10.69508) QuantErr: 10.69508 batch_time=2.36209
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 0.15790 (QuantReg: 10.85445) QuantErr: 10.85445 batch_time=0.64729
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 0.19782 (QuantReg: 10.90718) QuantErr: 10.90718 batch_time=0.64366
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 0.24620 (QuantReg: 10.82522) QuantErr: 10.82522 batch_time=0.64966
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 0.13773 (QuantReg: 10.86765) QuantErr: 10.86765 batch_time=0.65037
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 0.27465 (QuantReg: 10.64122) QuantErr: 10.64122 batch_time=0.64110
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 0.23426 (QuantReg: 10.72536) QuantErr: 10.72536 batch_time=0.64462
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 0.25638 (QuantReg: 10.76872) QuantErr: 10.76872 batch_time=0.65816
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 0.21838 (QuantReg: 10.71451) QuantErr: 10.71451 batch_time=0.65108
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 0.14753 (QuantReg: 10.51871) QuantErr: 10.51871 batch_time=0.66018
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 0.22483 (QuantReg: 10.68111) QuantErr: 10.68111 batch_time=0.65718
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 0.23458 (QuantReg: 10.68837) QuantErr: 10.68837 batch_time=0.63352
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 0.26365 (QuantReg: 10.92529) QuantErr: 10.92529 batch_time=4.37017
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 0.17477 (QuantReg: 10.89073) QuantErr: 10.89073 batch_time=0.64321
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 0.17448 (QuantReg: 10.76244) QuantErr: 10.76244 batch_time=0.64624
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 0.16471 (QuantReg: 10.71885) QuantErr: 10.71885 batch_time=0.66330
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 0.15268 (QuantReg: 10.93667) QuantErr: 10.93667 batch_time=0.65269
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 0.16982 (QuantReg: 10.81541) QuantErr: 10.81541 batch_time=0.66269
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 0.15871 (QuantReg: 10.86596) QuantErr: 10.86596 batch_time=0.64536
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 0.20213 (QuantReg: 10.81937) QuantErr: 10.81937 batch_time=0.64731
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 0.22006 (QuantReg: 10.89966) QuantErr: 10.89966 batch_time=0.66651
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 0.16606 (QuantReg: 10.64421) QuantErr: 10.64421 batch_time=0.64853
Train Epoch: 9 codebook_update_time=1.77928
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch9.pth ...
Done in 6.161s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch9.pth ...
Done in 13.351s
removing stale ckpt [epoch 8] [took 0.34s]
epoch : 9
loss : 0.19809449735283852
quant_reg : 10.800171058654785
quant_err : 10.800171058654785
learning_rate : 2.6100312500000002e-05
n_samples : 288000
n_steps : 2250
ActivityNet_val1_test/t2v_metrics/R1: 16.900549115314217
ActivityNet_val1_test/t2v_metrics/R5: 45.27150701647346
ActivityNet_val1_test/t2v_metrics/R10: 63.24994915598943
ActivityNet_val1_test/t2v_metrics/R50: 91.05145413870247
ActivityNet_val1_test/t2v_metrics/MedR: 7.0
ActivityNet_val1_test/t2v_metrics/MeanR: 26.824079723408584
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.441422043083044
ActivityNet_val1_test/v2t_metrics/R1: 17.53101484645109
ActivityNet_val1_test/v2t_metrics/R5: 46.77648972950986
ActivityNet_val1_test/v2t_metrics/R10: 64.2668293675005
ActivityNet_val1_test/v2t_metrics/R50: 91.49888143176734
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 25.83577384584096
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 37.492165018122236
mnt_best : 36.441422043083044
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 0.14940 (QuantReg: 10.80495) QuantErr: 10.80495 batch_time=25.43971
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 0.18204 (QuantReg: 10.71825) QuantErr: 10.71825 batch_time=0.66463
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 0.17358 (QuantReg: 10.82188) QuantErr: 10.82188 batch_time=0.64894
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 0.19938 (QuantReg: 10.87071) QuantErr: 10.87071 batch_time=0.65078
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 0.15222 (QuantReg: 10.65945) QuantErr: 10.65945 batch_time=0.64230
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 0.16308 (QuantReg: 10.52228) QuantErr: 10.52228 batch_time=0.66845
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 0.20694 (QuantReg: 10.78585) QuantErr: 10.78585 batch_time=1.22941
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 0.18978 (QuantReg: 10.91699) QuantErr: 10.91699 batch_time=0.65420
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 0.13131 (QuantReg: 10.71752) QuantErr: 10.71752 batch_time=0.64081
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 0.15572 (QuantReg: 10.88466) QuantErr: 10.88466 batch_time=0.64601
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 0.20302 (QuantReg: 10.61178) QuantErr: 10.61178 batch_time=0.64519
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 0.19129 (QuantReg: 10.80009) QuantErr: 10.80009 batch_time=0.68255
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 0.15448 (QuantReg: 10.79744) QuantErr: 10.79744 batch_time=0.64975
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 0.18496 (QuantReg: 10.89967) QuantErr: 10.89967 batch_time=0.65231
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 0.22395 (QuantReg: 10.79202) QuantErr: 10.79202 batch_time=0.64631
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 0.17962 (QuantReg: 10.80112) QuantErr: 10.80112 batch_time=0.64188
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 0.19960 (QuantReg: 10.92002) QuantErr: 10.92002 batch_time=0.69988
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 0.13990 (QuantReg: 10.95731) QuantErr: 10.95731 batch_time=0.64376
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 0.17776 (QuantReg: 10.69244) QuantErr: 10.69244 batch_time=0.64492
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 0.19109 (QuantReg: 11.11222) QuantErr: 11.11222 batch_time=2.07713
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 0.13954 (QuantReg: 11.10796) QuantErr: 11.10796 batch_time=0.64183
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 0.13817 (QuantReg: 10.97757) QuantErr: 10.97757 batch_time=0.64123
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 0.21265 (QuantReg: 10.96767) QuantErr: 10.96767 batch_time=0.64942
Train Epoch: 10 codebook_update_time=1.83364
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch10.pth ...
Done in 4.471s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch10.pth ...
Done in 8.623s
removing stale ckpt [epoch 9] [took 0.04s]
epoch : 10
loss : 0.17955050987005233
quant_reg : 10.784574653625489
quant_err : 10.784574653625489
learning_rate : 2.6100312500000002e-05
n_samples : 320000
n_steps : 2500
ActivityNet_val1_test/t2v_metrics/R1: 16.96156192800488
ActivityNet_val1_test/t2v_metrics/R5: 46.34940004067521
ActivityNet_val1_test/t2v_metrics/R10: 63.51433801098231
ActivityNet_val1_test/t2v_metrics/R50: 91.45820622330689
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 25.9816961561928
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.823684850155196
ActivityNet_val1_test/v2t_metrics/R1: 17.67337807606264
ActivityNet_val1_test/v2t_metrics/R5: 47.79336994102095
ActivityNet_val1_test/v2t_metrics/R10: 65.36505999593248
ActivityNet_val1_test/v2t_metrics/R50: 91.62090705714867
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 25.39271913768558
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.07830680612879
mnt_best : 36.823684850155196
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 0.19712 (QuantReg: 10.83497) QuantErr: 10.83497 batch_time=21.81338
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 0.14177 (QuantReg: 10.72243) QuantErr: 10.72243 batch_time=0.64480
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 0.18439 (QuantReg: 10.60823) QuantErr: 10.60823 batch_time=0.64955
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 0.20418 (QuantReg: 10.86818) QuantErr: 10.86818 batch_time=0.64532
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 0.15245 (QuantReg: 10.77436) QuantErr: 10.77436 batch_time=0.64278
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 0.18475 (QuantReg: 10.78029) QuantErr: 10.78029 batch_time=0.65816
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 0.17446 (QuantReg: 10.69382) QuantErr: 10.69382 batch_time=0.95000
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 0.17338 (QuantReg: 10.74846) QuantErr: 10.74846 batch_time=0.65402
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 0.17007 (QuantReg: 10.81282) QuantErr: 10.81282 batch_time=0.65286
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 0.18463 (QuantReg: 11.05151) QuantErr: 11.05151 batch_time=0.65503
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 0.16374 (QuantReg: 10.61991) QuantErr: 10.61991 batch_time=0.64872
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 0.20472 (QuantReg: 10.58018) QuantErr: 10.58018 batch_time=0.65801
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 0.13782 (QuantReg: 10.58796) QuantErr: 10.58796 batch_time=0.64962
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 0.14333 (QuantReg: 10.88811) QuantErr: 10.88811 batch_time=3.68035
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 0.14503 (QuantReg: 10.89856) QuantErr: 10.89856 batch_time=0.64791
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 0.16416 (QuantReg: 10.71110) QuantErr: 10.71110 batch_time=0.64193
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 0.17862 (QuantReg: 10.80689) QuantErr: 10.80689 batch_time=0.64701
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 0.17386 (QuantReg: 10.72978) QuantErr: 10.72978 batch_time=0.65208
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 0.16780 (QuantReg: 10.72107) QuantErr: 10.72107 batch_time=0.64614
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 0.17099 (QuantReg: 10.91318) QuantErr: 10.91318 batch_time=0.91264
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 0.11934 (QuantReg: 11.06623) QuantErr: 11.06623 batch_time=0.64860
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 0.11685 (QuantReg: 10.90019) QuantErr: 10.90019 batch_time=0.67117
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 0.17394 (QuantReg: 10.96733) QuantErr: 10.96733 batch_time=0.65547
Train Epoch: 11 codebook_update_time=1.95715
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch11.pth ...
Done in 4.719s
removing stale ckpt [epoch 10] [took 0.01s]
epoch : 11
loss : 0.1643868734240532
quant_reg : 10.766478240966796
quant_err : 10.766478240966796
learning_rate : 2.2185265625e-05
n_samples : 352000
n_steps : 2750
ActivityNet_val1_test/t2v_metrics/R1: 16.67683546878178
ActivityNet_val1_test/t2v_metrics/R5: 46.308724832214764
ActivityNet_val1_test/t2v_metrics/R10: 63.63636363636363
ActivityNet_val1_test/t2v_metrics/R50: 90.92942851332113
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 27.316249745779945
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 36.62918652202238
ActivityNet_val1_test/v2t_metrics/R1: 17.714053284523082
ActivityNet_val1_test/v2t_metrics/R5: 48.03742119178361
ActivityNet_val1_test/v2t_metrics/R10: 65.05999593247915
ActivityNet_val1_test/v2t_metrics/R50: 91.35651820215578
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 26.739678665853162
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.112773551477055
mnt_best : 36.823684850155196
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 0.13010 (QuantReg: 10.62824) QuantErr: 10.62824 batch_time=24.71962
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 0.18347 (QuantReg: 10.70307) QuantErr: 10.70307 batch_time=0.64383
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 0.21578 (QuantReg: 10.54397) QuantErr: 10.54397 batch_time=0.65648
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 0.17386 (QuantReg: 10.37536) QuantErr: 10.37536 batch_time=0.93400
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 0.19022 (QuantReg: 10.83144) QuantErr: 10.83144 batch_time=0.64069
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 0.17072 (QuantReg: 10.60056) QuantErr: 10.60056 batch_time=0.65018
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 0.18524 (QuantReg: 10.65392) QuantErr: 10.65392 batch_time=0.64846
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 0.10442 (QuantReg: 10.67848) QuantErr: 10.67848 batch_time=0.64588
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 0.12898 (QuantReg: 10.78828) QuantErr: 10.78828 batch_time=0.68414
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 0.16997 (QuantReg: 10.80818) QuantErr: 10.80818 batch_time=0.70378
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 0.12258 (QuantReg: 10.93220) QuantErr: 10.93220 batch_time=0.68760
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 0.16199 (QuantReg: 10.71958) QuantErr: 10.71958 batch_time=0.78279
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 0.16382 (QuantReg: 10.86508) QuantErr: 10.86508 batch_time=0.64598
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 0.09933 (QuantReg: 10.76197) QuantErr: 10.76197 batch_time=0.64718
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 0.19178 (QuantReg: 10.64205) QuantErr: 10.64205 batch_time=0.65317
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 0.17006 (QuantReg: 10.68346) QuantErr: 10.68346 batch_time=0.64155
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 0.16550 (QuantReg: 10.69671) QuantErr: 10.69671 batch_time=0.64111
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 0.15790 (QuantReg: 10.71448) QuantErr: 10.71448 batch_time=0.64706
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 0.14067 (QuantReg: 10.87011) QuantErr: 10.87011 batch_time=0.69874
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 0.14192 (QuantReg: 10.74851) QuantErr: 10.74851 batch_time=0.64353
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 0.13308 (QuantReg: 10.66002) QuantErr: 10.66002 batch_time=0.65121
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 0.17868 (QuantReg: 10.90090) QuantErr: 10.90090 batch_time=0.63893
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 0.13530 (QuantReg: 10.78760) QuantErr: 10.78760 batch_time=0.64302
Train Epoch: 12 codebook_update_time=2.21119
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch12.pth ...
Done in 5.007s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch12.pth ...
Done in 9.752s
removing stale ckpt [epoch 11] [took 0.02s]
epoch : 12
loss : 0.15271886843442917
quant_reg : 10.733918270111085
quant_err : 10.733918270111085
learning_rate : 2.2185265625e-05
n_samples : 384000
n_steps : 3000
ActivityNet_val1_test/t2v_metrics/R1: 17.164937970307097
ActivityNet_val1_test/t2v_metrics/R5: 46.75615212527964
ActivityNet_val1_test/t2v_metrics/R10: 64.20581655480984
ActivityNet_val1_test/t2v_metrics/R50: 90.8074028879398
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 27.70002033760423
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.212181023506474
ActivityNet_val1_test/v2t_metrics/R1: 17.69371568029286
ActivityNet_val1_test/v2t_metrics/R5: 47.915395566402275
ActivityNet_val1_test/v2t_metrics/R10: 65.73113687207648
ActivityNet_val1_test/v2t_metrics/R50: 91.29550538946512
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 26.335163717714053
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 38.19634042369397
mnt_best : 37.212181023506474
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 0.14196 (QuantReg: 10.74568) QuantErr: 10.74568 batch_time=22.46467
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 0.11550 (QuantReg: 10.60409) QuantErr: 10.60409 batch_time=0.64170
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 0.18206 (QuantReg: 10.77544) QuantErr: 10.77544 batch_time=0.64326
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 0.14428 (QuantReg: 10.63940) QuantErr: 10.63940 batch_time=0.64640
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 0.13867 (QuantReg: 10.77318) QuantErr: 10.77318 batch_time=0.64240
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 0.12037 (QuantReg: 10.67719) QuantErr: 10.67719 batch_time=0.63940
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 0.11743 (QuantReg: 10.78879) QuantErr: 10.78879 batch_time=3.09935
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 0.16336 (QuantReg: 10.81190) QuantErr: 10.81190 batch_time=0.63814
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 0.15762 (QuantReg: 10.94040) QuantErr: 10.94040 batch_time=0.64778
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 0.10443 (QuantReg: 10.72621) QuantErr: 10.72621 batch_time=0.65025
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 0.09639 (QuantReg: 10.78716) QuantErr: 10.78716 batch_time=0.65185
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 0.16600 (QuantReg: 10.42938) QuantErr: 10.42938 batch_time=0.64663
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 0.15648 (QuantReg: 10.60687) QuantErr: 10.60687 batch_time=0.66268
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 0.12020 (QuantReg: 10.90066) QuantErr: 10.90066 batch_time=0.65793
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 0.19901 (QuantReg: 10.74312) QuantErr: 10.74312 batch_time=0.64654
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 0.14252 (QuantReg: 10.83706) QuantErr: 10.83706 batch_time=0.66097
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 0.15323 (QuantReg: 10.78911) QuantErr: 10.78911 batch_time=0.66426
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 0.12081 (QuantReg: 10.49823) QuantErr: 10.49823 batch_time=0.65972
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 0.23265 (QuantReg: 10.82727) QuantErr: 10.82727 batch_time=4.36063
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 0.12964 (QuantReg: 10.64284) QuantErr: 10.64284 batch_time=0.66151
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 0.15838 (QuantReg: 10.67105) QuantErr: 10.67105 batch_time=0.64530
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 0.10975 (QuantReg: 10.69004) QuantErr: 10.69004 batch_time=0.67688
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 0.10430 (QuantReg: 10.79563) QuantErr: 10.79563 batch_time=0.65269
Train Epoch: 13 codebook_update_time=1.75459
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch13.pth ...
Done in 4.305s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch13.pth ...
Done in 9.526s
removing stale ckpt [epoch 12] [took 0.01s]
epoch : 13
loss : 0.14238685286045075
quant_reg : 10.695345520019531
quant_err : 10.695345520019531
learning_rate : 1.885747578125e-05
n_samples : 416000
n_steps : 3250
ActivityNet_val1_test/t2v_metrics/R1: 17.225950782997764
ActivityNet_val1_test/t2v_metrics/R5: 47.122229001423634
ActivityNet_val1_test/t2v_metrics/R10: 64.93797030709783
ActivityNet_val1_test/t2v_metrics/R50: 90.58368924140736
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 27.93207240187106
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 37.494644989867005
ActivityNet_val1_test/v2t_metrics/R1: 18.934309538336382
ActivityNet_val1_test/v2t_metrics/R5: 48.95261338214358
ActivityNet_val1_test/v2t_metrics/R10: 66.34126499898312
ActivityNet_val1_test/v2t_metrics/R50: 91.2751677852349
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 26.86983933292658
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.4702339912904
mnt_best : 37.494644989867005
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 0.08956 (QuantReg: 10.75674) QuantErr: 10.75674 batch_time=22.86858
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 0.12829 (QuantReg: 10.84736) QuantErr: 10.84736 batch_time=0.63816
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 0.12392 (QuantReg: 10.51743) QuantErr: 10.51743 batch_time=0.64589
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 0.15534 (QuantReg: 10.52043) QuantErr: 10.52043 batch_time=0.64430
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 0.16283 (QuantReg: 10.54947) QuantErr: 10.54947 batch_time=0.63827
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 0.14608 (QuantReg: 10.49591) QuantErr: 10.49591 batch_time=0.65163
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 0.15769 (QuantReg: 10.70887) QuantErr: 10.70887 batch_time=2.33579
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 0.09575 (QuantReg: 10.64832) QuantErr: 10.64832 batch_time=0.63266
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 0.16930 (QuantReg: 10.66771) QuantErr: 10.66771 batch_time=0.63968
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 0.14935 (QuantReg: 10.49588) QuantErr: 10.49588 batch_time=0.64947
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 0.12124 (QuantReg: 10.61825) QuantErr: 10.61825 batch_time=0.65126
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 0.12946 (QuantReg: 10.94359) QuantErr: 10.94359 batch_time=0.64137
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 0.12726 (QuantReg: 10.56032) QuantErr: 10.56032 batch_time=0.65852
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 0.14425 (QuantReg: 10.74134) QuantErr: 10.74134 batch_time=1.42686
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 0.12219 (QuantReg: 10.83475) QuantErr: 10.83475 batch_time=0.65321
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 0.11768 (QuantReg: 10.63499) QuantErr: 10.63499 batch_time=0.64871
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 0.16459 (QuantReg: 10.77607) QuantErr: 10.77607 batch_time=0.64581
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 0.08750 (QuantReg: 10.52369) QuantErr: 10.52369 batch_time=0.63782
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 0.17201 (QuantReg: 10.48234) QuantErr: 10.48234 batch_time=0.64389
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 0.13147 (QuantReg: 10.70988) QuantErr: 10.70988 batch_time=4.95450
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 0.12183 (QuantReg: 10.72118) QuantErr: 10.72118 batch_time=0.65012
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 0.14064 (QuantReg: 10.63638) QuantErr: 10.63638 batch_time=0.64910
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 0.10073 (QuantReg: 10.63356) QuantErr: 10.63356 batch_time=0.64088
Train Epoch: 14 codebook_update_time=1.75636
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch14.pth ...
Done in 4.015s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch14.pth ...
Done in 8.496s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 0.13446473085880278
quant_reg : 10.651949718475342
quant_err : 10.651949718475342
learning_rate : 1.885747578125e-05
n_samples : 448000
n_steps : 3500
ActivityNet_val1_test/t2v_metrics/R1: 17.836078909904412
ActivityNet_val1_test/t2v_metrics/R5: 47.427293064876956
ActivityNet_val1_test/t2v_metrics/R10: 64.87695749440716
ActivityNet_val1_test/t2v_metrics/R50: 90.3396379906447
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 28.537929631889362
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.00196607168883
ActivityNet_val1_test/v2t_metrics/R1: 18.52755745373195
ActivityNet_val1_test/v2t_metrics/R5: 48.89160056945292
ActivityNet_val1_test/v2t_metrics/R10: 66.25991458206224
ActivityNet_val1_test/v2t_metrics/R50: 91.11246695139313
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 27.04474272930649
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.15324528445985
mnt_best : 38.00196607168883
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 0.09400 (QuantReg: 10.74126) QuantErr: 10.74126 batch_time=21.81631
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 0.13460 (QuantReg: 10.60012) QuantErr: 10.60012 batch_time=0.64589
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 0.12912 (QuantReg: 10.55435) QuantErr: 10.55435 batch_time=0.65431
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 0.09798 (QuantReg: 10.42793) QuantErr: 10.42793 batch_time=0.65182
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 0.10553 (QuantReg: 10.47609) QuantErr: 10.47609 batch_time=0.65011
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 0.14913 (QuantReg: 10.53494) QuantErr: 10.53494 batch_time=0.64973
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 0.17091 (QuantReg: 10.73915) QuantErr: 10.73915 batch_time=0.64263
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 0.12606 (QuantReg: 10.65842) QuantErr: 10.65842 batch_time=0.64360
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 0.11260 (QuantReg: 10.61797) QuantErr: 10.61797 batch_time=0.64594
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 0.13208 (QuantReg: 10.31634) QuantErr: 10.31634 batch_time=0.66168
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 0.15677 (QuantReg: 10.44209) QuantErr: 10.44209 batch_time=0.66358
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 0.18374 (QuantReg: 10.46926) QuantErr: 10.46926 batch_time=0.65534
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 0.11717 (QuantReg: 10.77343) QuantErr: 10.77343 batch_time=1.63035
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 0.09925 (QuantReg: 10.45994) QuantErr: 10.45994 batch_time=0.67734
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 0.09488 (QuantReg: 10.49052) QuantErr: 10.49052 batch_time=0.67004
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 0.14676 (QuantReg: 10.68024) QuantErr: 10.68024 batch_time=0.65029
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 0.17554 (QuantReg: 10.47120) QuantErr: 10.47120 batch_time=0.71788
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 0.08354 (QuantReg: 10.67948) QuantErr: 10.67948 batch_time=0.65136
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 0.11912 (QuantReg: 10.36714) QuantErr: 10.36714 batch_time=0.66615
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 0.10687 (QuantReg: 10.41367) QuantErr: 10.41367 batch_time=0.66130
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 0.14848 (QuantReg: 10.48195) QuantErr: 10.48195 batch_time=0.69512
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 0.12383 (QuantReg: 10.52368) QuantErr: 10.52368 batch_time=0.64728
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 0.13091 (QuantReg: 10.45149) QuantErr: 10.45149 batch_time=0.64229
Train Epoch: 15 codebook_update_time=2.24740
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch15.pth ...
Done in 4.633s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch15.pth ...
Done in 9.518s
removing stale ckpt [epoch 14] [took 0.11s]
epoch : 15
loss : 0.12723694437742233
quant_reg : 10.585981769561768
quant_err : 10.585981769561768
learning_rate : 1.6028854414062497e-05
n_samples : 480000
n_steps : 3750
ActivityNet_val1_test/t2v_metrics/R1: 17.81574130567419
ActivityNet_val1_test/t2v_metrics/R5: 47.87472035794183
ActivityNet_val1_test/t2v_metrics/R10: 64.85661989017694
ActivityNet_val1_test/t2v_metrics/R50: 90.46166361602603
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 28.783201138905838
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.1026182596942
ActivityNet_val1_test/v2t_metrics/R1: 18.568232662192393
ActivityNet_val1_test/v2t_metrics/R5: 48.830587756762256
ActivityNet_val1_test/v2t_metrics/R10: 66.64632906243645
ActivityNet_val1_test/v2t_metrics/R50: 91.1734797640838
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 27.76428716697173
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.241558868111596
mnt_best : 38.1026182596942
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 0.11937 (QuantReg: 10.56717) QuantErr: 10.56717 batch_time=22.35816
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 0.09494 (QuantReg: 10.64229) QuantErr: 10.64229 batch_time=0.68825
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 0.08349 (QuantReg: 10.54766) QuantErr: 10.54766 batch_time=1.20460
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 0.15114 (QuantReg: 10.55434) QuantErr: 10.55434 batch_time=0.64986
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 0.09151 (QuantReg: 10.76358) QuantErr: 10.76358 batch_time=0.66915
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 0.08387 (QuantReg: 10.55905) QuantErr: 10.55905 batch_time=0.66163
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 0.12246 (QuantReg: 10.61885) QuantErr: 10.61885 batch_time=0.63415
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 0.12277 (QuantReg: 10.40690) QuantErr: 10.40690 batch_time=0.65036
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 0.13384 (QuantReg: 10.55587) QuantErr: 10.55587 batch_time=0.65897
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 0.17178 (QuantReg: 10.33397) QuantErr: 10.33397 batch_time=0.65324
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 0.12512 (QuantReg: 10.45475) QuantErr: 10.45475 batch_time=0.64614
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 0.13899 (QuantReg: 10.56851) QuantErr: 10.56851 batch_time=0.66849
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 0.10341 (QuantReg: 10.32651) QuantErr: 10.32651 batch_time=0.66194
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 0.10165 (QuantReg: 10.67988) QuantErr: 10.67988 batch_time=0.64283
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 0.10439 (QuantReg: 10.51468) QuantErr: 10.51468 batch_time=0.66376
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 0.16959 (QuantReg: 10.39988) QuantErr: 10.39988 batch_time=0.65380
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 0.09180 (QuantReg: 10.39494) QuantErr: 10.39494 batch_time=0.67310
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 0.10603 (QuantReg: 10.57636) QuantErr: 10.57636 batch_time=0.64795
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 0.11025 (QuantReg: 10.56029) QuantErr: 10.56029 batch_time=0.66087
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 0.16243 (QuantReg: 10.50099) QuantErr: 10.50099 batch_time=0.66076
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 0.08698 (QuantReg: 10.43562) QuantErr: 10.43562 batch_time=0.66679
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 0.09379 (QuantReg: 10.70445) QuantErr: 10.70445 batch_time=0.65475
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 0.09670 (QuantReg: 10.60090) QuantErr: 10.60090 batch_time=0.66198
Train Epoch: 16 codebook_update_time=1.88869
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch16.pth ...
Done in 6.099s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch16.pth ...
Done in 11.814s
removing stale ckpt [epoch 15] [took 0.08s]
epoch : 16
loss : 0.12433208328485489
quant_reg : 10.54029502105713
quant_err : 10.54029502105713
learning_rate : 1.6028854414062497e-05
n_samples : 512000
n_steps : 4000
ActivityNet_val1_test/t2v_metrics/R1: 18.303843807199513
ActivityNet_val1_test/t2v_metrics/R5: 48.38316046369738
ActivityNet_val1_test/t2v_metrics/R10: 64.93797030709783
ActivityNet_val1_test/t2v_metrics/R50: 90.03457392719137
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 29.68964815944682
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.59920990028789
ActivityNet_val1_test/v2t_metrics/R1: 18.486882245271506
ActivityNet_val1_test/v2t_metrics/R5: 49.76611755135245
ActivityNet_val1_test/v2t_metrics/R10: 66.91071791742932
ActivityNet_val1_test/v2t_metrics/R50: 90.86841570063046
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 28.33109619686801
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.48489994855078
mnt_best : 38.59920990028789
not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 0.11109 (QuantReg: 10.59993) QuantErr: 10.59993 batch_time=23.02203
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 0.13267 (QuantReg: 10.57943) QuantErr: 10.57943 batch_time=0.64465
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 0.10759 (QuantReg: 10.51055) QuantErr: 10.51055 batch_time=4.67424
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 0.12218 (QuantReg: 10.55581) QuantErr: 10.55581 batch_time=0.68130
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 0.10600 (QuantReg: 10.16763) QuantErr: 10.16763 batch_time=0.65665
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 0.09400 (QuantReg: 10.46909) QuantErr: 10.46909 batch_time=0.65255
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 0.11227 (QuantReg: 10.28705) QuantErr: 10.28705 batch_time=0.64528
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 0.13798 (QuantReg: 10.44706) QuantErr: 10.44706 batch_time=0.64231
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 0.09692 (QuantReg: 10.43073) QuantErr: 10.43073 batch_time=0.65394
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 0.12683 (QuantReg: 10.41608) QuantErr: 10.41608 batch_time=0.67993
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 0.09871 (QuantReg: 10.38634) QuantErr: 10.38634 batch_time=0.68282
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 0.15572 (QuantReg: 10.31833) QuantErr: 10.31833 batch_time=0.65696
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 0.10148 (QuantReg: 10.45290) QuantErr: 10.45290 batch_time=0.66576
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 0.11193 (QuantReg: 10.60916) QuantErr: 10.60916 batch_time=1.08527
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 0.08147 (QuantReg: 10.62315) QuantErr: 10.62315 batch_time=0.63900
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 0.09474 (QuantReg: 10.60581) QuantErr: 10.60581 batch_time=0.64149
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 0.17566 (QuantReg: 10.34274) QuantErr: 10.34274 batch_time=0.65383
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 0.09833 (QuantReg: 10.27898) QuantErr: 10.27898 batch_time=0.64256
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 0.12595 (QuantReg: 10.40472) QuantErr: 10.40472 batch_time=1.42611
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 0.12424 (QuantReg: 10.44755) QuantErr: 10.44755 batch_time=0.70554
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 0.12133 (QuantReg: 10.62668) QuantErr: 10.62668 batch_time=0.65315
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 0.10771 (QuantReg: 10.43473) QuantErr: 10.43473 batch_time=0.96554
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 0.13022 (QuantReg: 10.36958) QuantErr: 10.36958 batch_time=0.67845
Train Epoch: 17 codebook_update_time=1.87759
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch17.pth ...
Done in 5.014s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch17.pth ...
Done in 10.417s
removing stale ckpt [epoch 16] [took 0.07s]
epoch : 17
loss : 0.11971532887220383
quant_reg : 10.46480778503418
quant_err : 10.46480778503418
learning_rate : 1.3624526251953122e-05
n_samples : 544000
n_steps : 4250
ActivityNet_val1_test/t2v_metrics/R1: 18.446207036811064
ActivityNet_val1_test/t2v_metrics/R5: 48.240797234085825
ActivityNet_val1_test/t2v_metrics/R10: 65.16168395363026
ActivityNet_val1_test/t2v_metrics/R50: 90.29896278218426
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 30.402481187716088
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.7053754684394
ActivityNet_val1_test/v2t_metrics/R1: 19.178360789099045
ActivityNet_val1_test/v2t_metrics/R5: 49.50172869635957
ActivityNet_val1_test/v2t_metrics/R10: 66.46329062436445
ActivityNet_val1_test/v2t_metrics/R50: 90.74639007524914
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 28.88427903193004
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.81113535562355
mnt_best : 38.7053754684394
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 0.08343 (QuantReg: 10.31499) QuantErr: 10.31499 batch_time=24.63652
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 0.13665 (QuantReg: 10.22870) QuantErr: 10.22870 batch_time=0.63906
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 0.11111 (QuantReg: 10.17742) QuantErr: 10.17742 batch_time=2.27750
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 0.11123 (QuantReg: 10.43421) QuantErr: 10.43421 batch_time=0.64327
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 0.10897 (QuantReg: 10.62097) QuantErr: 10.62097 batch_time=0.64173
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 0.11511 (QuantReg: 10.30744) QuantErr: 10.30744 batch_time=0.66012
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 0.11727 (QuantReg: 10.52980) QuantErr: 10.52980 batch_time=1.41768
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 0.09312 (QuantReg: 10.52608) QuantErr: 10.52608 batch_time=0.64254
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 0.08157 (QuantReg: 10.25451) QuantErr: 10.25451 batch_time=0.65308
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 0.07537 (QuantReg: 10.51247) QuantErr: 10.51247 batch_time=0.66148
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 0.11316 (QuantReg: 10.22009) QuantErr: 10.22009 batch_time=0.68328
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 0.11599 (QuantReg: 10.35986) QuantErr: 10.35986 batch_time=0.65247
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 0.17772 (QuantReg: 10.43071) QuantErr: 10.43071 batch_time=0.66208
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 0.13594 (QuantReg: 10.34782) QuantErr: 10.34782 batch_time=0.88538
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 0.10070 (QuantReg: 10.27641) QuantErr: 10.27641 batch_time=0.65881
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 0.12114 (QuantReg: 10.29175) QuantErr: 10.29175 batch_time=0.64477
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 0.13449 (QuantReg: 10.42731) QuantErr: 10.42731 batch_time=0.64999
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 0.13059 (QuantReg: 10.47486) QuantErr: 10.47486 batch_time=0.64651
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 0.08842 (QuantReg: 10.43710) QuantErr: 10.43710 batch_time=0.63586
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 0.13959 (QuantReg: 10.46580) QuantErr: 10.46580 batch_time=1.72910
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 0.12818 (QuantReg: 10.28916) QuantErr: 10.28916 batch_time=2.44964
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 0.10675 (QuantReg: 10.29477) QuantErr: 10.29477 batch_time=0.65000
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 0.08286 (QuantReg: 10.29215) QuantErr: 10.29215 batch_time=0.65075
Train Epoch: 18 codebook_update_time=1.74497
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch18.pth ...
Done in 4.591s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 0.11488990464806556
quant_reg : 10.397148376464843
quant_err : 10.397148376464843
learning_rate : 1.3624526251953122e-05
n_samples : 576000
n_steps : 4500
ActivityNet_val1_test/t2v_metrics/R1: 18.425869432580843
ActivityNet_val1_test/t2v_metrics/R5: 48.505186089078705
ActivityNet_val1_test/t2v_metrics/R10: 64.75493186902582
ActivityNet_val1_test/t2v_metrics/R50: 90.2379499694936
ActivityNet_val1_test/t2v_metrics/MedR: 6.0
ActivityNet_val1_test/t2v_metrics/MeanR: 30.149074639007527
ActivityNet_val1_test/t2v_metrics/geometric_mean_R1-R5-R10: 38.68087933923349
ActivityNet_val1_test/v2t_metrics/R1: 19.402074435631484
ActivityNet_val1_test/v2t_metrics/R5: 49.847467968273335
ActivityNet_val1_test/v2t_metrics/R10: 65.81248728899736
ActivityNet_val1_test/v2t_metrics/R50: 90.68537726255848
ActivityNet_val1_test/v2t_metrics/MedR: 6.0
ActivityNet_val1_test/v2t_metrics/MeanR: 29.25218629245475
ActivityNet_val1_test/v2t_metrics/geometric_mean_R1-R5-R10: 39.926985740282795
mnt_best : 38.7053754684394
not_improved_count: 1
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 0.12944 (QuantReg: 10.35297) QuantErr: 10.35297 batch_time=21.23902
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 0.14680 (QuantReg: 10.33442) QuantErr: 10.33442 batch_time=0.64920
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 0.12708 (QuantReg: 10.31906) QuantErr: 10.31906 batch_time=0.65747
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 0.13086 (QuantReg: 10.25302) QuantErr: 10.25302 batch_time=0.66904
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 0.12901 (QuantReg: 10.30415) QuantErr: 10.30415 batch_time=0.64575
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 0.12903 (QuantReg: 10.48188) QuantErr: 10.48188 batch_time=0.64089
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 0.09831 (QuantReg: 10.46824) QuantErr: 10.46824 batch_time=0.66213
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 0.09321 (QuantReg: 10.46529) QuantErr: 10.46529 batch_time=0.63914
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 0.08569 (QuantReg: 10.35979) QuantErr: 10.35979 batch_time=0.65360
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 0.13623 (QuantReg: 10.34992) QuantErr: 10.34992 batch_time=0.65480
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 0.09203 (QuantReg: 10.17803) QuantErr: 10.17803 batch_time=0.64270
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 0.08775 (QuantReg: 10.28996) QuantErr: 10.28996 batch_time=0.64603
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 0.14226 (QuantReg: 10.29826) QuantErr: 10.29826 batch_time=0.64548
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 0.12823 (QuantReg: 10.36983) QuantErr: 10.36983 batch_time=0.66819
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 0.09691 (QuantReg: 10.39418) QuantErr: 10.39418 batch_time=0.64715
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 0.07929 (QuantReg: 10.46517) QuantErr: 10.46517 batch_time=0.63900
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 0.11586 (QuantReg: 10.29710) QuantErr: 10.29710 batch_time=1.56641
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 0.11213 (QuantReg: 10.42791) QuantErr: 10.42791 batch_time=0.64159
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 0.11417 (QuantReg: 10.34027) QuantErr: 10.34027 batch_time=0.63984
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 0.14986 (QuantReg: 10.16006) QuantErr: 10.16006 batch_time=0.65157
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 0.12525 (QuantReg: 10.37715) QuantErr: 10.37715 batch_time=0.68427
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 0.09619 (QuantReg: 10.37251) QuantErr: 10.37251 batch_time=0.65898
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 0.07971 (QuantReg: 10.30872) QuantErr: 10.30872 batch_time=0.67765
Train Epoch: 19 codebook_update_time=1.79053
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_ActivityNet_bs128/checkpoint-epoch19.pth ...
Done in 4.323s
removing stale ckpt [epoch 18] [took 0.02s]
epoch : 19
loss : 0.11110387647151947
quant_reg : 10.327787311553955
quant_err : 10.327787311553955
learning_rate : 1.1580847314160154e-05
n_samples : 608000
n_steps : 4750