SCD模型在SECOND数据集上训练，验证时，出现了NaN #96
Comments
|
Please be patient and wait for a few more epochs. If it still doesn't work, please check the learning rate or the data. |
|
@ABCnutter I found SECOND dataset only have label1(valid for seg_map_path_from) and label2(valid for seg_map_path_to), the seg_map_path do not exist, i generate it with unchaneged -> 0 and changed -> 1, whether it is caused by seg_map_path label uvalid value? |
This comment was marked as resolved.
This comment was marked as resolved.
|
I have the same problem, Has it been solved?? |
|
What images should i put in my SECOND dataset format label and label1 label2 |
|
I have the same problem, Has it been solved?? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
环境:
scd_upernet在SECOND数据集上训练，但在验证的时候分数为NaN。
log日志如下：
2024/04/26 10:57:47 - mmengine - INFO - ------------------------------------------------------------ System environment: sys.platform: linux Python: 3.9.0 (default, Nov 15 2020, 14:28:56) [GCC 7.3.0] CUDA available: True MUSA available: False numpy_random_seed: 1381265969 GPU 0: NVIDIA A10 CUDA_HOME: /usr/local/cuda-11.8 NVCC: Cuda compilation tools, release 11.8, V11.8.89 GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 PyTorch: 2.0.1+cu118 PyTorch compiling details: PyTorch built with: - GCC 9.3 - C++ Version: 201703 - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX2 - CUDA Runtime 11.8 - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90 - CuDNN 8.7 - Magma 2.6.1 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, TorchVision: 0.15.2+cu118 OpenCV: 4.9.0 MMEngine: 0.10.4 Runtime environment: cudnn_benchmark: True mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: 1381265969 Distributed launcher: none Distributed training: False GPU number: 1 ------------------------------------------------------------ 2024/04/26 10:57:47 - mmengine - INFO - Config: crop_size = ( 256, 384, ) data_preprocessor = dict( bgr_to_rgb=True, mean=[ 123.675, 116.28, 103.53, 123.675, 116.28, 103.53, ], pad_val=0, seg_pad_val=255, size_divisor=32, std=[ 58.395, 57.12, 57.375, 58.395, 57.12, 57.375, ], test_cfg=dict(size_divisor=32), type='DualInputSegDataPreProcessor') data_root = '/home/teslaa10/桌面/MambaCD/data/SECOND' dataset_type = 'SECOND_Dataset' default_hooks = dict( checkpoint=dict(by_epoch=False, interval=4000, type='CheckpointHook'), logger=dict(interval=50, log_metric_by_epoch=False, type='LoggerHook'), param_scheduler=dict(type='ParamSchedulerHook'), sampler_seed=dict(type='DistSamplerSeedHook'), timer=dict(type='IterTimerHook'), visualization=dict( draw_on_from_to_img=False, interval=1, type='CDVisualizationHook')) default_scope = 'opencd' env_cfg = dict( cudnn_benchmark=True, dist_cfg=dict(backend='nccl'), mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0)) launcher = 'none' load_from = None log_level = 'INFO' log_processor = dict(by_epoch=False) model = dict( backbone=dict( contract_dilation=True, depth=18, dilations=( 1, 1, 1, 1, ), norm_cfg=dict(requires_grad=True, type='SyncBN'), norm_eval=False, num_stages=4, out_indices=( 0, 1, 2, 3, ), strides=( 1, 2, 2, 2, ), style='pytorch', type='mmseg.ResNetV1c'), data_preprocessor=dict( bgr_to_rgb=True, mean=[ 123.675, 116.28, 103.53, 123.675, 116.28, 103.53, ], pad_val=0, seg_pad_val=255, size_divisor=32, std=[ 58.395, 57.12, 57.375, 58.395, 57.12, 57.375, ], test_cfg=dict(size_divisor=32), type='DualInputSegDataPreProcessor'), decode_head=dict( binary_cd_head=dict( align_corners=False, channels=64, dropout_ratio=0.1, in_channels=[ 64, 128, 256, 512, ], in_index=[ 0, 1, 2, 3, ], loss_decode=dict( loss_weight=1.0, type='mmseg.CrossEntropyLoss', use_sigmoid=False), norm_cfg=dict(requires_grad=True, type='SyncBN'), num_classes=2, pool_scales=( 1, 2, 3, 6, ), type='mmseg.UPerHead'), binary_cd_neck=dict(policy='abs_diff', type='FeatureFusionNeck'), semantic_cd_head=dict( align_corners=False, channels=64, dropout_ratio=0.1, ignore_index=255, in_channels=[ 64, 128, 256, 512, ], in_index=[ 0, 1, 2, 3, ], loss_decode=dict( avg_non_ignore=True, loss_weight=1.0, type='mmseg.CrossEntropyLoss', use_sigmoid=False), norm_cfg=dict(requires_grad=True, type='SyncBN'), num_classes=6, pool_scales=( 1, 2, 3, 6, ), type='mmseg.UPerHead'), type='GeneralSCDHead'), postprocess_pred_and_label='cover_semantic', pretrained='open-mmlab://resnet18_v1c', test_cfg=dict(mode='whole'), train_cfg=dict(), type='SiamEncoderMultiDecoder') norm_cfg = dict(requires_grad=True, type='SyncBN') optim_wrapper = dict( optimizer=dict( betas=( 0.9, 0.999, ), lr=0.001, type='AdamW', weight_decay=0.05), paramwise_cfg=dict(custom_keys=dict(head=dict(lr_mult=10.0))), type='OptimWrapper') optimizer = dict( betas=( 0.9, 0.999, ), lr=0.001, momentum=0.9, type='AdamW', weight_decay=0.05) param_scheduler = [ dict( begin=0, by_epoch=False, end=40000, eta_min=0.0001, power=0.9, type='PolyLR'), ] resume = False test_cfg = dict(type='TestLoop') test_dataloader = dict( batch_size=1, dataset=dict( data_prefix=dict( img_path_from='val/im1', img_path_to='val/im2', seg_map_path='val/label', seg_map_path_from='val/label1', seg_map_path_to='val/label2'), data_root='/home/teslaa10/桌面/MambaCD/data/SECOND', pipeline=[ dict(type='MultiImgLoadImageFromFile'), dict(keep_ratio=True, scale=( 512, 512, ), type='MultiImgResize'), dict(type='MultiImgMultiAnnLoadAnnotations'), dict(type='MultiImgPackSegInputs'), ], type='SECOND_Dataset'), num_workers=4, persistent_workers=True, sampler=dict(shuffle=False, type='DefaultSampler')) test_evaluator = dict( cal_sek=True, iou_metrics=[ 'mFscore', 'mIoU', ], type='SCDMetric') test_pipeline = [ dict(type='MultiImgLoadImageFromFile'), dict(keep_ratio=True, scale=( 512, 512, ), type='MultiImgResize'), dict(type='MultiImgMultiAnnLoadAnnotations'), dict(type='MultiImgPackSegInputs'), ] train_cfg = dict(max_iters=40000, type='IterBasedTrainLoop', val_interval=4000) train_dataloader = dict( batch_size=32, dataset=dict( data_prefix=dict( img_path_from='train/im1', img_path_to='train/im2', seg_map_path='train/label_reverse', seg_map_path_from='train/label1', seg_map_path_to='train/label2'), data_root='/home/teslaa10/桌面/MambaCD/data/SECOND', pipeline=[ dict(type='MultiImgLoadImageFromFile'), dict(type='MultiImgMultiAnnLoadAnnotations'), dict( cat_max_ratio=0.75, crop_size=( 256, 384, ), type='MultiImgRandomCrop'), dict(prob=0.5, type='MultiImgRandomFlip'), dict(type='MultiImgPackSegInputs'), ], type='SECOND_Dataset'), num_workers=4, persistent_workers=True, sampler=dict(shuffle=True, type='InfiniteSampler')) train_pipeline = [ dict(type='MultiImgLoadImageFromFile'), dict(type='MultiImgMultiAnnLoadAnnotations'), dict( cat_max_ratio=0.75, crop_size=( 256, 384, ), type='MultiImgRandomCrop'), dict(prob=0.5, type='MultiImgRandomFlip'), dict(type='MultiImgPackSegInputs'), ] tta_model = dict(type='mmseg.SegTTAModel') val_cfg = dict(type='ValLoop') val_dataloader = dict( batch_size=1, dataset=dict( data_prefix=dict( img_path_from='train/im1', img_path_to='train/im2', seg_map_path='train/label_reverse', seg_map_path_from='train/label1', seg_map_path_to='train/label2'), data_root='/home/teslaa10/桌面/MambaCD/data/SECOND', pipeline=[ dict(type='MultiImgLoadImageFromFile'), dict(keep_ratio=True, scale=( 512, 512, ), type='MultiImgResize'), dict(type='MultiImgMultiAnnLoadAnnotations'), dict(type='MultiImgPackSegInputs'), ], type='SECOND_Dataset'), num_workers=4, persistent_workers=True, sampler=dict(shuffle=False, type='DefaultSampler')) val_evaluator = dict( cal_sek=True, iou_metrics=[ 'mFscore', 'mIoU', ], type='SCDMetric') vis_backends = [ dict(type='CDLocalVisBackend'), ] visualizer = dict( alpha=1.0, name='visualizer', type='CDLocalVisualizer', vis_backends=[ dict(type='CDLocalVisBackend'), ]) work_dir = 'runs/second_reverse' 2024/04/26 10:57:49 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used. 2024/04/26 10:57:49 - mmengine - INFO - Hooks will be executed in the following order: ........ decode_head.semantic_cd_head.fpn_bottleneck.bn.bias - torch.Size([64]): The value is the same before and after calling `init_weights` of SiamEncoderMultiDecoder 2024/04/26 10:57:51 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io 2024/04/26 10:57:51 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future. 2024/04/26 10:57:51 - mmengine - INFO - Checkpoints will be saved to /home/teslaa10/桌面/open-cd/runs/second_reverse. 2024/04/26 10:58:29 - mmengine - INFO - Iter(train) [ 50/40000] base_lr: 9.9901e-04 lr: 9.9901e-04 eta: 8:25:25 time: 0.6564 data_time: 0.0347 memory: 6374 loss: 1.6423 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.8324 decode.semantic_cd_from.acc_seg: 62.0327 decode.semantic_cd_to.loss_ce: 0.8098 decode.semantic_cd_to.acc_seg: 79.0120 2024/04/26 10:58:57 - mmengine - INFO - Exp name: scd_upernet_r18_256x512_40k_custom_second_reverse_20240426_105747 2024/04/26 10:59:01 - mmengine - INFO - Iter(train) [ 100/40000] base_lr: 9.9799e-04 lr: 9.9799e-04 eta: 7:50:14 time: 0.6519 data_time: 0.0240 memory: 6218 loss: 1.5679 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.8580 decode.semantic_cd_from.acc_seg: 78.4337 decode.semantic_cd_to.loss_ce: 0.7098 decode.semantic_cd_to.acc_seg: 74.9893 2024/04/26 10:59:34 - mmengine - INFO - Iter(train) [ 150/40000] base_lr: 9.9698e-04 lr: 9.9698e-04 eta: 7:37:30 time: 0.6555 data_time: 0.0333 memory: 6214 loss: 1.5296 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7733 decode.semantic_cd_from.acc_seg: 68.5549 decode.semantic_cd_to.loss_ce: 0.7563 decode.semantic_cd_to.acc_seg: 75.5639 2024/04/26 11:00:06 - mmengine - INFO - Iter(train) [ 200/40000] base_lr: 9.9597e-04 lr: 9.9597e-04 eta: 7:30:32 time: 0.6514 data_time: 0.0246 memory: 6220 loss: 1.4951 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7925 decode.semantic_cd_from.acc_seg: 68.0908 decode.semantic_cd_to.loss_ce: 0.7026 decode.semantic_cd_to.acc_seg: 70.9616 2024/04/26 11:00:40 - mmengine - INFO - Iter(train) [ 250/40000] base_lr: 9.9496e-04 lr: 9.9496e-04 eta: 7:27:47 time: 0.6605 data_time: 0.0336 memory: 6216 loss: 1.4491 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7545 decode.semantic_cd_from.acc_seg: 67.9426 decode.semantic_cd_to.loss_ce: 0.6945 decode.semantic_cd_to.acc_seg: 74.8863 2024/04/26 11:01:12 - mmengine - INFO - Iter(train) [ 300/40000] base_lr: 9.9394e-04 lr: 9.9394e-04 eta: 7:24:43 time: 0.6501 data_time: 0.0254 memory: 6218 loss: 1.4242 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7940 decode.semantic_cd_from.acc_seg: 73.8835 decode.semantic_cd_to.loss_ce: 0.6302 decode.semantic_cd_to.acc_seg: 73.2641 2024/04/26 11:01:45 - mmengine - INFO - Iter(train) [ 350/40000] base_lr: 9.9293e-04 lr: 9.9293e-04 eta: 7:22:15 time: 0.6375 data_time: 0.0326 memory: 6216 loss: 1.3852 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6924 decode.semantic_cd_from.acc_seg: 67.3209 decode.semantic_cd_to.loss_ce: 0.6928 decode.semantic_cd_to.acc_seg: 77.4595 2024/04/26 11:02:18 - mmengine - INFO - Iter(train) [ 400/40000] base_lr: 9.9192e-04 lr: 9.9192e-04 eta: 7:20:30 time: 0.6534 data_time: 0.0260 memory: 6217 loss: 1.4094 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7324 decode.semantic_cd_from.acc_seg: 65.1774 decode.semantic_cd_to.loss_ce: 0.6769 decode.semantic_cd_to.acc_seg: 76.5736 2024/04/26 11:02:51 - mmengine - INFO - Iter(train) [ 450/40000] base_lr: 9.9090e-04 lr: 9.9090e-04 eta: 7:19:20 time: 0.6601 data_time: 0.0319 memory: 6217 loss: 1.3805 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7229 decode.semantic_cd_from.acc_seg: 72.6983 decode.semantic_cd_to.loss_ce: 0.6576 decode.semantic_cd_to.acc_seg: 78.3995 2024/04/26 11:03:23 - mmengine - INFO - Iter(train) [ 500/40000] base_lr: 9.8989e-04 lr: 9.8989e-04 eta: 7:17:42 time: 0.6420 data_time: 0.0243 memory: 6219 loss: 1.3505 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6828 decode.semantic_cd_from.acc_seg: 77.0075 decode.semantic_cd_to.loss_ce: 0.6677 decode.semantic_cd_to.acc_seg: 78.8156 2024/04/26 11:03:56 - mmengine - INFO - Iter(train) [ 550/40000] base_lr: 9.8887e-04 lr: 9.8887e-04 eta: 7:17:04 time: 0.6610 data_time: 0.0338 memory: 6215 loss: 1.3806 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6907 decode.semantic_cd_from.acc_seg: 74.7352 decode.semantic_cd_to.loss_ce: 0.6899 decode.semantic_cd_to.acc_seg: 74.2981 2024/04/26 11:04:29 - mmengine - INFO - Iter(train) [ 600/40000] base_lr: 9.8786e-04 lr: 9.8786e-04 eta: 7:16:01 time: 0.6506 data_time: 0.0274 memory: 6217 loss: 1.2539 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6612 decode.semantic_cd_from.acc_seg: 73.2606 decode.semantic_cd_to.loss_ce: 0.5928 decode.semantic_cd_to.acc_seg: 78.2316 2024/04/26 11:05:02 - mmengine - INFO - Iter(train) [ 650/40000] base_lr: 9.8685e-04 lr: 9.8685e-04 eta: 7:15:08 time: 0.6599 data_time: 0.0338 memory: 6216 loss: 1.3703 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7442 decode.semantic_cd_from.acc_seg: 62.5257 decode.semantic_cd_to.loss_ce: 0.6260 decode.semantic_cd_to.acc_seg: 78.1242 2024/04/26 11:05:35 - mmengine - INFO - Iter(train) [ 700/40000] base_lr: 9.8583e-04 lr: 9.8583e-04 eta: 7:14:18 time: 0.6401 data_time: 0.0223 memory: 6217 loss: 1.3233 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6921 decode.semantic_cd_from.acc_seg: 63.7578 decode.semantic_cd_to.loss_ce: 0.6312 decode.semantic_cd_to.acc_seg: 73.9684 2024/04/26 11:06:08 - mmengine - INFO - Iter(train) [ 750/40000] base_lr: 9.8482e-04 lr: 9.8482e-04 eta: 7:13:35 time: 0.6524 data_time: 0.0318 memory: 6215 loss: 1.3098 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6750 decode.semantic_cd_from.acc_seg: 71.1711 decode.semantic_cd_to.loss_ce: 0.6348 decode.semantic_cd_to.acc_seg: 68.7412 2024/04/26 11:06:40 - mmengine - INFO - Iter(train) [ 800/40000] base_lr: 9.8380e-04 lr: 9.8380e-04 eta: 7:12:28 time: 0.6437 data_time: 0.0211 memory: 6217 loss: 1.2980 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7200 decode.semantic_cd_from.acc_seg: 70.3378 decode.semantic_cd_to.loss_ce: 0.5780 decode.semantic_cd_to.acc_seg: 79.4105 2024/04/26 11:07:13 - mmengine - INFO - Iter(train) [ 850/40000] base_lr: 9.8279e-04 lr: 9.8279e-04 eta: 7:11:48 time: 0.6596 data_time: 0.0279 memory: 6213 loss: 1.3866 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6956 decode.semantic_cd_from.acc_seg: 71.4475 decode.semantic_cd_to.loss_ce: 0.6909 decode.semantic_cd_to.acc_seg: 79.3616 2024/04/26 11:07:46 - mmengine - INFO - Iter(train) [ 900/40000] base_lr: 9.8177e-04 lr: 9.8177e-04 eta: 7:11:04 time: 0.6469 data_time: 0.0223 memory: 6217 loss: 1.2347 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6414 decode.semantic_cd_from.acc_seg: 84.1916 decode.semantic_cd_to.loss_ce: 0.5933 decode.semantic_cd_to.acc_seg: 73.1520 2024/04/26 11:08:19 - mmengine - INFO - Iter(train) [ 950/40000] base_lr: 9.8076e-04 lr: 9.8076e-04 eta: 7:10:22 time: 0.6631 data_time: 0.0299 memory: 6215 loss: 1.3037 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6484 decode.semantic_cd_from.acc_seg: 81.5560 decode.semantic_cd_to.loss_ce: 0.6553 decode.semantic_cd_to.acc_seg: 72.4231 2024/04/26 11:08:51 - mmengine - INFO - Exp name: scd_upernet_r18_256x512_40k_custom_second_reverse_20240426_105747 2024/04/26 11:08:51 - mmengine - INFO - Iter(train) [ 1000/40000] base_lr: 9.7974e-04 lr: 9.7974e-04 eta: 7:09:26 time: 0.6459 data_time: 0.0226 memory: 6214 loss: 1.2749 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6785 decode.semantic_cd_from.acc_seg: 69.2935 decode.semantic_cd_to.loss_ce: 0.5965 decode.semantic_cd_to.acc_seg: 79.8328 2024/04/26 11:09:24 - mmengine - INFO - Iter(train) [ 1050/40000] base_lr: 9.7873e-04 lr: 9.7873e-04 eta: 7:08:35 time: 0.6307 data_time: 0.0287 memory: 6219 loss: 1.3226 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.7077 decode.semantic_cd_from.acc_seg: 67.9155 decode.semantic_cd_to.loss_ce: 0.6149 decode.semantic_cd_to.acc_seg: 70.6348 2024/04/26 11:09:57 - mmengine - INFO - Iter(train) [ 1100/40000] base_lr: 9.7771e-04 lr: 9.7771e-04 eta: 7:07:54 time: 0.6731 data_time: 0.0346 memory: 6216 loss: 1.2177 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6522 decode.semantic_cd_from.acc_seg: 74.1093 decode.semantic_cd_to.loss_ce: 0.5655 decode.semantic_cd_to.acc_seg: 74.2081 2024/04/26 11:10:29 - mmengine - INFO - Iter(train) [ 1150/40000] base_lr: 9.7670e-04 lr: 9.7670e-04 eta: 7:07:06 time: 0.6500 data_time: 0.0270 memory: 6221 loss: 1.1561 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5893 decode.semantic_cd_from.acc_seg: 74.4151 decode.semantic_cd_to.loss_ce: 0.5669 decode.semantic_cd_to.acc_seg: 82.8591 2024/04/26 11:11:02 - mmengine - INFO - Iter(train) [ 1200/40000] base_lr: 9.7568e-04 lr: 9.7568e-04 eta: 7:06:12 time: 0.6446 data_time: 0.0241 memory: 6215 loss: 1.1951 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6409 decode.semantic_cd_from.acc_seg: 71.5545 decode.semantic_cd_to.loss_ce: 0.5542 decode.semantic_cd_to.acc_seg: 81.3648 2024/04/26 11:11:34 - mmengine - INFO - Iter(train) [ 1250/40000] base_lr: 9.7467e-04 lr: 9.7467e-04 eta: 7:05:35 time: 0.6532 data_time: 0.0283 memory: 6215 loss: 1.1937 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6298 decode.semantic_cd_from.acc_seg: 73.6095 decode.semantic_cd_to.loss_ce: 0.5638 decode.semantic_cd_to.acc_seg: 80.6597 2024/04/26 11:12:07 - mmengine - INFO - Iter(train) [ 1300/40000] base_lr: 9.7365e-04 lr: 9.7365e-04 eta: 7:04:45 time: 0.6527 data_time: 0.0259 memory: 6215 loss: 1.3329 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6645 decode.semantic_cd_from.acc_seg: 71.7276 decode.semantic_cd_to.loss_ce: 0.6684 decode.semantic_cd_to.acc_seg: 82.6563 2024/04/26 11:12:39 - mmengine - INFO - Iter(train) [ 1350/40000] base_lr: 9.7264e-04 lr: 9.7264e-04 eta: 7:04:06 time: 0.6535 data_time: 0.0310 memory: 6216 loss: 1.2894 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6578 decode.semantic_cd_from.acc_seg: 77.5528 decode.semantic_cd_to.loss_ce: 0.6316 decode.semantic_cd_to.acc_seg: 77.3752 2024/04/26 11:13:12 - mmengine - INFO - Iter(train) [ 1400/40000] base_lr: 9.7162e-04 lr: 9.7162e-04 eta: 7:03:23 time: 0.6515 data_time: 0.0245 memory: 6215 loss: 1.1613 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6348 decode.semantic_cd_from.acc_seg: 67.3040 decode.semantic_cd_to.loss_ce: 0.5266 decode.semantic_cd_to.acc_seg: 76.2559 2024/04/26 11:13:45 - mmengine - INFO - Iter(train) [ 1450/40000] base_lr: 9.7060e-04 lr: 9.7060e-04 eta: 7:02:48 time: 0.6577 data_time: 0.0281 memory: 6216 loss: 1.2746 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6874 decode.semantic_cd_from.acc_seg: 75.3978 decode.semantic_cd_to.loss_ce: 0.5872 decode.semantic_cd_to.acc_seg: 72.4715 2024/04/26 11:14:17 - mmengine - INFO - Iter(train) [ 1500/40000] base_lr: 9.6959e-04 lr: 9.6959e-04 eta: 7:02:01 time: 0.6510 data_time: 0.0217 memory: 6218 loss: 1.0869 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5267 decode.semantic_cd_from.acc_seg: 77.3848 decode.semantic_cd_to.loss_ce: 0.5602 decode.semantic_cd_to.acc_seg: 83.6625 2024/04/26 11:14:50 - mmengine - INFO - Iter(train) [ 1550/40000] base_lr: 9.6857e-04 lr: 9.6857e-04 eta: 7:01:23 time: 0.6621 data_time: 0.0290 memory: 6217 loss: 1.3156 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6806 decode.semantic_cd_from.acc_seg: 68.6981 decode.semantic_cd_to.loss_ce: 0.6350 decode.semantic_cd_to.acc_seg: 66.7464 2024/04/26 11:15:22 - mmengine - INFO - Iter(train) [ 1600/40000] base_lr: 9.6755e-04 lr: 9.6755e-04 eta: 7:00:41 time: 0.6478 data_time: 0.0233 memory: 6217 loss: 1.1073 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5764 decode.semantic_cd_from.acc_seg: 78.2243 decode.semantic_cd_to.loss_ce: 0.5309 decode.semantic_cd_to.acc_seg: 78.6301 2024/04/26 11:15:55 - mmengine - INFO - Iter(train) [ 1650/40000] base_lr: 9.6654e-04 lr: 9.6654e-04 eta: 7:00:02 time: 0.6461 data_time: 0.0309 memory: 6219 loss: 1.1745 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6594 decode.semantic_cd_from.acc_seg: 80.9841 decode.semantic_cd_to.loss_ce: 0.5150 decode.semantic_cd_to.acc_seg: 80.9291 2024/04/26 11:16:28 - mmengine - INFO - Iter(train) [ 1700/40000] base_lr: 9.6552e-04 lr: 9.6552e-04 eta: 6:59:23 time: 0.6541 data_time: 0.0254 memory: 6217 loss: 1.1096 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6078 decode.semantic_cd_from.acc_seg: 68.7960 decode.semantic_cd_to.loss_ce: 0.5018 decode.semantic_cd_to.acc_seg: 80.6704 2024/04/26 11:17:00 - mmengine - INFO - Iter(train) [ 1750/40000] base_lr: 9.6450e-04 lr: 9.6450e-04 eta: 6:58:48 time: 0.6598 data_time: 0.0308 memory: 6217 loss: 1.2709 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6921 decode.semantic_cd_from.acc_seg: 79.1489 decode.semantic_cd_to.loss_ce: 0.5788 decode.semantic_cd_to.acc_seg: 75.0460 2024/04/26 11:17:33 - mmengine - INFO - Iter(train) [ 1800/40000] base_lr: 9.6349e-04 lr: 9.6349e-04 eta: 6:58:08 time: 0.6458 data_time: 0.0232 memory: 6213 loss: 1.1975 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6631 decode.semantic_cd_from.acc_seg: 65.0049 decode.semantic_cd_to.loss_ce: 0.5344 decode.semantic_cd_to.acc_seg: 83.0443 2024/04/26 11:18:06 - mmengine - INFO - Iter(train) [ 1850/40000] base_lr: 9.6247e-04 lr: 9.6247e-04 eta: 6:57:36 time: 0.6681 data_time: 0.0315 memory: 6214 loss: 1.1375 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6105 decode.semantic_cd_from.acc_seg: 79.6435 decode.semantic_cd_to.loss_ce: 0.5270 decode.semantic_cd_to.acc_seg: 72.1409 2024/04/26 11:18:38 - mmengine - INFO - Iter(train) [ 1900/40000] base_lr: 9.6145e-04 lr: 9.6145e-04 eta: 6:56:57 time: 0.6494 data_time: 0.0221 memory: 6218 loss: 1.1581 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6188 decode.semantic_cd_from.acc_seg: 70.2349 decode.semantic_cd_to.loss_ce: 0.5393 decode.semantic_cd_to.acc_seg: 80.9136 2024/04/26 11:19:11 - mmengine - INFO - Iter(train) [ 1950/40000] base_lr: 9.6043e-04 lr: 9.6043e-04 eta: 6:56:24 time: 0.6527 data_time: 0.0269 memory: 6217 loss: 1.1501 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6142 decode.semantic_cd_from.acc_seg: 81.8326 decode.semantic_cd_to.loss_ce: 0.5359 decode.semantic_cd_to.acc_seg: 77.1848 2024/04/26 11:19:44 - mmengine - INFO - Exp name: scd_upernet_r18_256x512_40k_custom_second_reverse_20240426_105747 2024/04/26 11:19:44 - mmengine - INFO - Iter(train) [ 2000/40000] base_lr: 9.5942e-04 lr: 9.5942e-04 eta: 6:55:45 time: 0.6515 data_time: 0.0231 memory: 6216 loss: 1.0477 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5378 decode.semantic_cd_from.acc_seg: 81.2718 decode.semantic_cd_to.loss_ce: 0.5099 decode.semantic_cd_to.acc_seg: 79.8077 2024/04/26 11:20:16 - mmengine - INFO - Iter(train) [ 2050/40000] base_lr: 9.5840e-04 lr: 9.5840e-04 eta: 6:55:08 time: 0.6410 data_time: 0.0303 memory: 6215 loss: 1.0983 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5937 decode.semantic_cd_from.acc_seg: 80.7065 decode.semantic_cd_to.loss_ce: 0.5046 decode.semantic_cd_to.acc_seg: 84.9454 2024/04/26 11:20:49 - mmengine - INFO - Iter(train) [ 2100/40000] base_lr: 9.5738e-04 lr: 9.5738e-04 eta: 6:54:30 time: 0.6504 data_time: 0.0207 memory: 6213 loss: 1.0608 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5620 decode.semantic_cd_from.acc_seg: 77.9023 decode.semantic_cd_to.loss_ce: 0.4987 decode.semantic_cd_to.acc_seg: 79.8157 2024/04/26 11:21:21 - mmengine - INFO - Iter(train) [ 2150/40000] base_lr: 9.5636e-04 lr: 9.5636e-04 eta: 6:53:53 time: 0.6511 data_time: 0.0256 memory: 6217 loss: 1.0508 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5284 decode.semantic_cd_from.acc_seg: 82.4979 decode.semantic_cd_to.loss_ce: 0.5224 decode.semantic_cd_to.acc_seg: 82.7942 2024/04/26 11:21:54 - mmengine - INFO - Iter(train) [ 2200/40000] base_lr: 9.5534e-04 lr: 9.5534e-04 eta: 6:53:17 time: 0.6478 data_time: 0.0199 memory: 6220 loss: 1.1878 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6202 decode.semantic_cd_from.acc_seg: 83.5395 decode.semantic_cd_to.loss_ce: 0.5676 decode.semantic_cd_to.acc_seg: 71.6823 2024/04/26 11:22:26 - mmengine - INFO - Iter(train) [ 2250/40000] base_lr: 9.5433e-04 lr: 9.5433e-04 eta: 6:52:39 time: 0.6488 data_time: 0.0350 memory: 6215 loss: 1.1190 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5984 decode.semantic_cd_from.acc_seg: 79.7233 decode.semantic_cd_to.loss_ce: 0.5206 decode.semantic_cd_to.acc_seg: 72.1518 2024/04/26 11:22:59 - mmengine - INFO - Iter(train) [ 2300/40000] base_lr: 9.5331e-04 lr: 9.5331e-04 eta: 6:52:03 time: 0.6486 data_time: 0.0199 memory: 6223 loss: 1.1774 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6118 decode.semantic_cd_from.acc_seg: 73.8663 decode.semantic_cd_to.loss_ce: 0.5656 decode.semantic_cd_to.acc_seg: 76.5118 2024/04/26 11:23:31 - mmengine - INFO - Iter(train) [ 2350/40000] base_lr: 9.5229e-04 lr: 9.5229e-04 eta: 6:51:26 time: 0.6489 data_time: 0.0276 memory: 6215 loss: 1.1186 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5736 decode.semantic_cd_from.acc_seg: 79.9533 decode.semantic_cd_to.loss_ce: 0.5450 decode.semantic_cd_to.acc_seg: 81.6492 2024/04/26 11:24:03 - mmengine - INFO - Iter(train) [ 2400/40000] base_lr: 9.5127e-04 lr: 9.5127e-04 eta: 6:50:30 time: 0.5324 data_time: 0.0206 memory: 6215 loss: 1.1375 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6120 decode.semantic_cd_from.acc_seg: 67.0379 decode.semantic_cd_to.loss_ce: 0.5255 decode.semantic_cd_to.acc_seg: 81.3190 2024/04/26 11:24:27 - mmengine - INFO - Iter(train) [ 2450/40000] base_lr: 9.5025e-04 lr: 9.5025e-04 eta: 6:47:40 time: 0.4755 data_time: 0.0288 memory: 6220 loss: 1.0064 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5323 decode.semantic_cd_from.acc_seg: 67.7515 decode.semantic_cd_to.loss_ce: 0.4741 decode.semantic_cd_to.acc_seg: 83.8450 2024/04/26 11:24:50 - mmengine - INFO - Iter(train) [ 2500/40000] base_lr: 9.4923e-04 lr: 9.4923e-04 eta: 6:44:53 time: 0.4718 data_time: 0.0226 memory: 6216 loss: 1.0556 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5694 decode.semantic_cd_from.acc_seg: 82.0476 decode.semantic_cd_to.loss_ce: 0.4862 decode.semantic_cd_to.acc_seg: 77.6847 2024/04/26 11:25:14 - mmengine - INFO - Iter(train) [ 2550/40000] base_lr: 9.4821e-04 lr: 9.4821e-04 eta: 6:42:15 time: 0.4832 data_time: 0.0280 memory: 6214 loss: 1.0519 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5329 decode.semantic_cd_from.acc_seg: 76.5292 decode.semantic_cd_to.loss_ce: 0.5190 decode.semantic_cd_to.acc_seg: 70.9648 2024/04/26 11:25:38 - mmengine - INFO - Iter(train) [ 2600/40000] base_lr: 9.4719e-04 lr: 9.4719e-04 eta: 6:39:38 time: 0.4666 data_time: 0.0210 memory: 6218 loss: 1.0382 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5544 decode.semantic_cd_from.acc_seg: 72.7263 decode.semantic_cd_to.loss_ce: 0.4838 decode.semantic_cd_to.acc_seg: 88.4448 2024/04/26 11:26:01 - mmengine - INFO - Iter(train) [ 2650/40000] base_lr: 9.4617e-04 lr: 9.4617e-04 eta: 6:37:07 time: 0.4796 data_time: 0.0361 memory: 6215 loss: 1.0921 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5954 decode.semantic_cd_from.acc_seg: 71.0227 decode.semantic_cd_to.loss_ce: 0.4967 decode.semantic_cd_to.acc_seg: 76.5624 2024/04/26 11:26:29 - mmengine - INFO - Iter(train) [ 2700/40000] base_lr: 9.4515e-04 lr: 9.4515e-04 eta: 6:35:42 time: 0.6480 data_time: 0.0228 memory: 6217 loss: 1.0315 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5660 decode.semantic_cd_from.acc_seg: 87.4005 decode.semantic_cd_to.loss_ce: 0.4655 decode.semantic_cd_to.acc_seg: 75.5524 2024/04/26 11:27:02 - mmengine - INFO - Iter(train) [ 2750/40000] base_lr: 9.4414e-04 lr: 9.4414e-04 eta: 6:35:21 time: 0.6541 data_time: 0.0333 memory: 6215 loss: 1.0108 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5140 decode.semantic_cd_from.acc_seg: 84.7930 decode.semantic_cd_to.loss_ce: 0.4968 decode.semantic_cd_to.acc_seg: 77.3926 2024/04/26 11:27:34 - mmengine - INFO - Iter(train) [ 2800/40000] base_lr: 9.4312e-04 lr: 9.4312e-04 eta: 6:34:56 time: 0.6514 data_time: 0.0198 memory: 6217 loss: 1.0039 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5390 decode.semantic_cd_from.acc_seg: 73.8155 decode.semantic_cd_to.loss_ce: 0.4649 decode.semantic_cd_to.acc_seg: 78.2484 2024/04/26 11:28:07 - mmengine - INFO - Iter(train) [ 2850/40000] base_lr: 9.4210e-04 lr: 9.4210e-04 eta: 6:34:37 time: 0.6564 data_time: 0.0257 memory: 6216 loss: 0.9835 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5248 decode.semantic_cd_from.acc_seg: 85.5152 decode.semantic_cd_to.loss_ce: 0.4587 decode.semantic_cd_to.acc_seg: 79.3816 2024/04/26 11:28:40 - mmengine - INFO - Iter(train) [ 2900/40000] base_lr: 9.4108e-04 lr: 9.4108e-04 eta: 6:34:14 time: 0.6525 data_time: 0.0195 memory: 6213 loss: 1.0466 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5639 decode.semantic_cd_from.acc_seg: 84.4389 decode.semantic_cd_to.loss_ce: 0.4826 decode.semantic_cd_to.acc_seg: 85.4537 2024/04/26 11:29:12 - mmengine - INFO - Iter(train) [ 2950/40000] base_lr: 9.4005e-04 lr: 9.4005e-04 eta: 6:33:54 time: 0.6570 data_time: 0.0262 memory: 6217 loss: 1.1140 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5816 decode.semantic_cd_from.acc_seg: 77.4268 decode.semantic_cd_to.loss_ce: 0.5325 decode.semantic_cd_to.acc_seg: 82.7459 2024/04/26 11:29:45 - mmengine - INFO - Exp name: scd_upernet_r18_256x512_40k_custom_second_reverse_20240426_105747 2024/04/26 11:29:45 - mmengine - INFO - Iter(train) [ 3000/40000] base_lr: 9.3903e-04 lr: 9.3903e-04 eta: 6:33:30 time: 0.6511 data_time: 0.0231 memory: 6219 loss: 1.1535 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.6485 decode.semantic_cd_from.acc_seg: 65.0963 decode.semantic_cd_to.loss_ce: 0.5050 decode.semantic_cd_to.acc_seg: 86.4622 2024/04/26 11:30:18 - mmengine - INFO - Iter(train) [ 3050/40000] base_lr: 9.3801e-04 lr: 9.3801e-04 eta: 6:33:06 time: 0.6275 data_time: 0.0236 memory: 6217 loss: 0.9736 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5054 decode.semantic_cd_from.acc_seg: 71.3786 decode.semantic_cd_to.loss_ce: 0.4682 decode.semantic_cd_to.acc_seg: 83.7279 2024/04/26 11:30:50 - mmengine - INFO - Iter(train) [ 3100/40000] base_lr: 9.3699e-04 lr: 9.3699e-04 eta: 6:32:41 time: 0.6512 data_time: 0.0196 memory: 6217 loss: 1.0868 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5846 decode.semantic_cd_from.acc_seg: 71.1487 decode.semantic_cd_to.loss_ce: 0.5021 decode.semantic_cd_to.acc_seg: 81.8537 2024/04/26 11:31:23 - mmengine - INFO - Iter(train) [ 3150/40000] base_lr: 9.3597e-04 lr: 9.3597e-04 eta: 6:32:16 time: 0.6502 data_time: 0.0237 memory: 6217 loss: 0.9710 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5003 decode.semantic_cd_from.acc_seg: 72.2864 decode.semantic_cd_to.loss_ce: 0.4707 decode.semantic_cd_to.acc_seg: 84.8217 2024/04/26 11:31:55 - mmengine - INFO - Iter(train) [ 3200/40000] base_lr: 9.3495e-04 lr: 9.3495e-04 eta: 6:31:52 time: 0.6496 data_time: 0.0207 memory: 6217 loss: 1.0185 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5253 decode.semantic_cd_from.acc_seg: 75.1490 decode.semantic_cd_to.loss_ce: 0.4932 decode.semantic_cd_to.acc_seg: 81.9104 2024/04/26 11:32:28 - mmengine - INFO - Iter(train) [ 3250/40000] base_lr: 9.3393e-04 lr: 9.3393e-04 eta: 6:31:29 time: 0.6507 data_time: 0.0283 memory: 6215 loss: 1.0353 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5883 decode.semantic_cd_from.acc_seg: 74.6657 decode.semantic_cd_to.loss_ce: 0.4470 decode.semantic_cd_to.acc_seg: 80.3325 2024/04/26 11:33:00 - mmengine - INFO - Iter(train) [ 3300/40000] base_lr: 9.3291e-04 lr: 9.3291e-04 eta: 6:31:02 time: 0.6467 data_time: 0.0185 memory: 6218 loss: 0.9831 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5234 decode.semantic_cd_from.acc_seg: 82.6042 decode.semantic_cd_to.loss_ce: 0.4597 decode.semantic_cd_to.acc_seg: 78.0049 2024/04/26 11:33:33 - mmengine - INFO - Iter(train) [ 3350/40000] base_lr: 9.3189e-04 lr: 9.3189e-04 eta: 6:30:38 time: 0.6498 data_time: 0.0254 memory: 6218 loss: 1.0182 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5290 decode.semantic_cd_from.acc_seg: 77.2096 decode.semantic_cd_to.loss_ce: 0.4891 decode.semantic_cd_to.acc_seg: 84.2850 2024/04/26 11:34:06 - mmengine - INFO - Iter(train) [ 3400/40000] base_lr: 9.3087e-04 lr: 9.3087e-04 eta: 6:30:12 time: 0.6448 data_time: 0.0202 memory: 6219 loss: 1.0745 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5498 decode.semantic_cd_from.acc_seg: 77.6357 decode.semantic_cd_to.loss_ce: 0.5247 decode.semantic_cd_to.acc_seg: 77.6897 2024/04/26 11:34:38 - mmengine - INFO - Iter(train) [ 3450/40000] base_lr: 9.2984e-04 lr: 9.2984e-04 eta: 6:29:49 time: 0.6533 data_time: 0.0251 memory: 6218 loss: 0.9507 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5224 decode.semantic_cd_from.acc_seg: 70.2391 decode.semantic_cd_to.loss_ce: 0.4284 decode.semantic_cd_to.acc_seg: 87.3421 2024/04/26 11:35:11 - mmengine - INFO - Iter(train) [ 3500/40000] base_lr: 9.2882e-04 lr: 9.2882e-04 eta: 6:29:21 time: 0.6289 data_time: 0.0175 memory: 6221 loss: 1.0867 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5595 decode.semantic_cd_from.acc_seg: 70.9600 decode.semantic_cd_to.loss_ce: 0.5272 decode.semantic_cd_to.acc_seg: 81.0125 2024/04/26 11:35:44 - mmengine - INFO - Iter(train) [ 3550/40000] base_lr: 9.2780e-04 lr: 9.2780e-04 eta: 6:28:57 time: 0.6501 data_time: 0.0244 memory: 6214 loss: 0.9062 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.4600 decode.semantic_cd_from.acc_seg: 86.2910 decode.semantic_cd_to.loss_ce: 0.4462 decode.semantic_cd_to.acc_seg: 86.1902 2024/04/26 11:36:16 - mmengine - INFO - Iter(train) [ 3600/40000] base_lr: 9.2678e-04 lr: 9.2678e-04 eta: 6:28:31 time: 0.6528 data_time: 0.0180 memory: 6217 loss: 0.9360 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5000 decode.semantic_cd_from.acc_seg: 84.1638 decode.semantic_cd_to.loss_ce: 0.4360 decode.semantic_cd_to.acc_seg: 86.7319 2024/04/26 11:36:49 - mmengine - INFO - Iter(train) [ 3650/40000] base_lr: 9.2576e-04 lr: 9.2576e-04 eta: 6:28:03 time: 0.6622 data_time: 0.0348 memory: 6218 loss: 1.0109 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5406 decode.semantic_cd_from.acc_seg: 78.0553 decode.semantic_cd_to.loss_ce: 0.4703 decode.semantic_cd_to.acc_seg: 89.4460 2024/04/26 11:37:21 - mmengine - INFO - Iter(train) [ 3700/40000] base_lr: 9.2473e-04 lr: 9.2473e-04 eta: 6:27:35 time: 0.6298 data_time: 0.0184 memory: 6214 loss: 0.9818 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5307 decode.semantic_cd_from.acc_seg: 74.7535 decode.semantic_cd_to.loss_ce: 0.4511 decode.semantic_cd_to.acc_seg: 81.7130 2024/04/26 11:37:54 - mmengine - INFO - Iter(train) [ 3750/40000] base_lr: 9.2371e-04 lr: 9.2371e-04 eta: 6:27:08 time: 0.6488 data_time: 0.0257 memory: 6215 loss: 0.9911 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5244 decode.semantic_cd_from.acc_seg: 86.0803 decode.semantic_cd_to.loss_ce: 0.4667 decode.semantic_cd_to.acc_seg: 84.4518 2024/04/26 11:38:26 - mmengine - INFO - Iter(train) [ 3800/40000] base_lr: 9.2269e-04 lr: 9.2269e-04 eta: 6:26:40 time: 0.6507 data_time: 0.0210 memory: 6216 loss: 0.9312 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5119 decode.semantic_cd_from.acc_seg: 81.5348 decode.semantic_cd_to.loss_ce: 0.4193 decode.semantic_cd_to.acc_seg: 86.0709 2024/04/26 11:38:59 - mmengine - INFO - Iter(train) [ 3850/40000] base_lr: 9.2167e-04 lr: 9.2167e-04 eta: 6:26:12 time: 0.6481 data_time: 0.0264 memory: 6223 loss: 0.8778 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.4884 decode.semantic_cd_from.acc_seg: 81.8352 decode.semantic_cd_to.loss_ce: 0.3893 decode.semantic_cd_to.acc_seg: 87.2019 2024/04/26 11:39:31 - mmengine - INFO - Iter(train) [ 3900/40000] base_lr: 9.2064e-04 lr: 9.2064e-04 eta: 6:25:44 time: 0.6498 data_time: 0.0183 memory: 6213 loss: 0.9964 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5283 decode.semantic_cd_from.acc_seg: 75.5499 decode.semantic_cd_to.loss_ce: 0.4681 decode.semantic_cd_to.acc_seg: 85.7708 2024/04/26 11:40:03 - mmengine - INFO - Iter(train) [ 3950/40000] base_lr: 9.1962e-04 lr: 9.1962e-04 eta: 6:25:16 time: 0.6473 data_time: 0.0291 memory: 6215 loss: 0.9604 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.4902 decode.semantic_cd_from.acc_seg: 80.3928 decode.semantic_cd_to.loss_ce: 0.4702 decode.semantic_cd_to.acc_seg: 78.9787 2024/04/26 11:40:36 - mmengine - INFO - Exp name: scd_upernet_r18_256x512_40k_custom_second_reverse_20240426_105747 2024/04/26 11:40:36 - mmengine - INFO - Iter(train) [ 4000/40000] base_lr: 9.1860e-04 lr: 9.1860e-04 eta: 6:24:47 time: 0.6502 data_time: 0.0199 memory: 6218 loss: 0.9844 decode.binary_cd.loss_ce: 0.0000 decode.binary_cd.acc_seg: 100.0000 decode.semantic_cd_from.loss_ce: 0.5356 decode.semantic_cd_from.acc_seg: 80.5928 decode.semantic_cd_to.loss_ce: 0.4487 decode.semantic_cd_to.acc_seg: 83.6353 2024/04/26 11:40:36 - mmengine - INFO - Saving checkpoint at 4000 iterations 2024/04/26 11:40:39 - mmengine - INFO - Iter(val) [ 50/2968] eta: 0:02:41 time: 0.0447 data_time: 0.0022 memory: 600 2024/04/26 11:40:42 - mmengine - INFO - Iter(val) [ 100/2968] eta: 0:02:35 time: 0.0447 data_time: 0.0023 memory: 321 2024/04/26 11:40:44 - mmengine - INFO - Iter(val) [ 150/2968] eta: 0:02:18 time: 0.0397 data_time: 0.0021 memory: 321 2024/04/26 11:40:46 - mmengine - INFO - Iter(val) [ 200/2968] eta: 0:02:17 time: 0.0378 data_time: 0.0023 memory: 321 2024/04/26 11:40:48 - mmengine - INFO - Iter(val) [ 250/2968] eta: 0:02:10 time: 0.0432 data_time: 0.0020 memory: 321 2024/04/26 11:40:50 - mmengine - INFO - Iter(val) [ 300/2968] eta: 0:02:04 time: 0.0451 data_time: 0.0021 memory: 321 2024/04/26 11:40:53 - mmengine - INFO - Iter(val) [ 350/2968] eta: 0:02:04 time: 0.0450 data_time: 0.0023 memory: 321 2024/04/26 11:40:55 - mmengine - INFO - Iter(val) [ 400/2968] eta: 0:01:59 time: 0.0448 data_time: 0.0021 memory: 321 2024/04/26 11:40:57 - mmengine - INFO - Iter(val) [ 450/2968] eta: 0:01:57 time: 0.0502 data_time: 0.0021 memory: 321 2024/04/26 11:40:59 - mmengine - INFO - Iter(val) [ 500/2968] eta: 0:01:53 time: 0.0276 data_time: 0.0023 memory: 321 2024/04/26 11:41:01 - mmengine - INFO - Iter(val) [ 550/2968] eta: 0:01:49 time: 0.0150 data_time: 0.0022 memory: 321 2024/04/26 11:41:04 - mmengine - INFO - Iter(val) [ 600/2968] eta: 0:01:48 time: 0.0455 data_time: 0.0021 memory: 321 2024/04/26 11:41:06 - mmengine - INFO - Iter(val) [ 650/2968] eta: 0:01:45 time: 0.0448 data_time: 0.0021 memory: 321 2024/04/26 11:41:08 - mmengine - INFO - Iter(val) [ 700/2968] eta: 0:01:42 time: 0.0793 data_time: 0.0021 memory: 321 2024/04/26 11:41:10 - mmengine - INFO - Iter(val) [ 750/2968] eta: 0:01:40 time: 0.0379 data_time: 0.0020 memory: 321 2024/04/26 11:41:12 - mmengine - INFO - Iter(val) [ 800/2968] eta: 0:01:37 time: 0.0442 data_time: 0.0021 memory: 321 2024/04/26 11:41:15 - mmengine - INFO - Iter(val) [ 850/2968] eta: 0:01:36 time: 0.0425 data_time: 0.0023 memory: 321 2024/04/26 11:41:17 - mmengine - INFO - Iter(val) [ 900/2968] eta: 0:01:33 time: 0.0449 data_time: 0.0020 memory: 321 2024/04/26 11:41:19 - mmengine - INFO - Iter(val) [ 950/2968] eta: 0:01:31 time: 0.0796 data_time: 0.0022 memory: 321 2024/04/26 11:41:22 - mmengine - INFO - Iter(val) [1000/2968] eta: 0:01:29 time: 0.0425 data_time: 0.0020 memory: 321 2024/04/26 11:41:24 - mmengine - INFO - Iter(val) [1050/2968] eta: 0:01:26 time: 0.0447 data_time: 0.0020 memory: 321 2024/04/26 11:41:26 - mmengine - INFO - Iter(val) [1100/2968] eta: 0:01:24 time: 0.0398 data_time: 0.0020 memory: 321 2024/04/26 11:41:28 - mmengine - INFO - Iter(val) [1150/2968] eta: 0:01:21 time: 0.0403 data_time: 0.0022 memory: 321 2024/04/26 11:41:30 - mmengine - INFO - Iter(val) [1200/2968] eta: 0:01:19 time: 0.0727 data_time: 0.0020 memory: 321 2024/04/26 11:41:33 - mmengine - INFO - Iter(val) [1250/2968] eta: 0:01:17 time: 0.0346 data_time: 0.0021 memory: 321 2024/04/26 11:41:35 - mmengine - INFO - Iter(val) [1300/2968] eta: 0:01:14 time: 0.0433 data_time: 0.0022 memory: 321 2024/04/26 11:41:37 - mmengine - INFO - Iter(val) [1350/2968] eta: 0:01:13 time: 0.0445 data_time: 0.0022 memory: 321 2024/04/26 11:41:39 - mmengine - INFO - Iter(val) [1400/2968] eta: 0:01:10 time: 0.0448 data_time: 0.0020 memory: 321 2024/04/26 11:41:42 - mmengine - INFO - Iter(val) [1450/2968] eta: 0:01:08 time: 0.0750 data_time: 0.0020 memory: 321 2024/04/26 11:41:44 - mmengine - INFO - Iter(val) [1500/2968] eta: 0:01:06 time: 0.0450 data_time: 0.0020 memory: 321 2024/04/26 11:41:46 - mmengine - INFO - Iter(val) [1550/2968] eta: 0:01:03 time: 0.0448 data_time: 0.0022 memory: 321 2024/04/26 11:41:49 - mmengine - INFO - Iter(val) [1600/2968] eta: 0:01:01 time: 0.0407 data_time: 0.0023 memory: 321 2024/04/26 11:41:51 - mmengine - INFO - Iter(val) [1650/2968] eta: 0:00:59 time: 0.0429 data_time: 0.0019 memory: 321 2024/04/26 11:41:53 - mmengine - INFO - Iter(val) [1700/2968] eta: 0:00:57 time: 0.0799 data_time: 0.0018 memory: 321 2024/04/26 11:41:55 - mmengine - INFO - Iter(val) [1750/2968] eta: 0:00:54 time: 0.0388 data_time: 0.0021 memory: 321 2024/04/26 11:41:57 - mmengine - INFO - Iter(val) [1800/2968] eta: 0:00:52 time: 0.0450 data_time: 0.0021 memory: 321 2024/04/26 11:42:00 - mmengine - INFO - Iter(val) [1850/2968] eta: 0:00:50 time: 0.0377 data_time: 0.0019 memory: 321 2024/04/26 11:42:02 - mmengine - INFO - Iter(val) [1900/2968] eta: 0:00:47 time: 0.0447 data_time: 0.0019 memory: 321 2024/04/26 11:42:04 - mmengine - INFO - Iter(val) [1950/2968] eta: 0:00:45 time: 0.0448 data_time: 0.0018 memory: 321 2024/04/26 11:42:06 - mmengine - INFO - Iter(val) [2000/2968] eta: 0:00:43 time: 0.0450 data_time: 0.0021 memory: 321 2024/04/26 11:42:08 - mmengine - INFO - Iter(val) [2050/2968] eta: 0:00:41 time: 0.0449 data_time: 0.0020 memory: 321 2024/04/26 11:42:11 - mmengine - INFO - Iter(val) [2100/2968] eta: 0:00:38 time: 0.0612 data_time: 0.0020 memory: 321 2024/04/26 11:42:13 - mmengine - INFO - Iter(val) [2150/2968] eta: 0:00:36 time: 0.0337 data_time: 0.0020 memory: 321 2024/04/26 11:42:15 - mmengine - INFO - Iter(val) [2200/2968] eta: 0:00:34 time: 0.0385 data_time: 0.0021 memory: 321 2024/04/26 11:42:17 - mmengine - INFO - Iter(val) [2250/2968] eta: 0:00:32 time: 0.0403 data_time: 0.0021 memory: 321 2024/04/26 11:42:19 - mmengine - INFO - Iter(val) [2300/2968] eta: 0:00:29 time: 0.0431 data_time: 0.0023 memory: 321 2024/04/26 11:42:22 - mmengine - INFO - Iter(val) [2350/2968] eta: 0:00:27 time: 0.0562 data_time: 0.0021 memory: 321 2024/04/26 11:42:24 - mmengine - INFO - Iter(val) [2400/2968] eta: 0:00:25 time: 0.0261 data_time: 0.0020 memory: 321 2024/04/26 11:42:26 - mmengine - INFO - Iter(val) [2450/2968] eta: 0:00:23 time: 0.0328 data_time: 0.0022 memory: 321 2024/04/26 11:42:29 - mmengine - INFO - Iter(val) [2500/2968] eta: 0:00:20 time: 0.0415 data_time: 0.0018 memory: 321 2024/04/26 11:42:31 - mmengine - INFO - Iter(val) [2550/2968] eta: 0:00:18 time: 0.0402 data_time: 0.0021 memory: 321 2024/04/26 11:42:33 - mmengine - INFO - Iter(val) [2600/2968] eta: 0:00:16 time: 0.0602 data_time: 0.0022 memory: 321 2024/04/26 11:42:35 - mmengine - INFO - Iter(val) [2650/2968] eta: 0:00:14 time: 0.0276 data_time: 0.0021 memory: 321 2024/04/26 11:42:37 - mmengine - INFO - Iter(val) [2700/2968] eta: 0:00:11 time: 0.0228 data_time: 0.0021 memory: 321 2024/04/26 11:42:39 - mmengine - INFO - Iter(val) [2750/2968] eta: 0:00:09 time: 0.0402 data_time: 0.0020 memory: 321 2024/04/26 11:42:42 - mmengine - INFO - Iter(val) [2800/2968] eta: 0:00:07 time: 0.0412 data_time: 0.0019 memory: 321 2024/04/26 11:42:44 - mmengine - INFO - Iter(val) [2850/2968] eta: 0:00:05 time: 0.0789 data_time: 0.0020 memory: 321 2024/04/26 11:42:46 - mmengine - INFO - Iter(val) [2900/2968] eta: 0:00:03 time: 0.0351 data_time: 0.0019 memory: 321 2024/04/26 11:42:48 - mmengine - INFO - Iter(val) [2950/2968] eta: 0:00:00 time: 0.0460 data_time: 0.0031 memory: 321 2024/04/26 11:42:49 - mmengine - INFO - per binary class results: 2024/04/26 11:42:49 - mmengine - INFO - +-----------+--------+-----------+--------+-------+-------+ | Class | Fscore | Precision | Recall | IoU | Acc | +-----------+--------+-----------+--------+-------+-------+ | unchanged | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | | changed | nan | nan | nan | nan | nan | +-----------+--------+-----------+--------+-------+-------+ 2024/04/26 11:42:49 - mmengine - INFO - per semantic class results: 2024/04/26 11:42:49 - mmengine - INFO - +----------------+--------+-----------+--------+-------+-------+ | Class | Fscore | Precision | Recall | IoU | Acc | +----------------+--------+-----------+--------+-------+-------+ | unchanged | 88.93 | 80.06 | 100.0 | 80.06 | 100.0 | | water | nan | nan | 0.0 | 0.0 | 0.0 | | ground | nan | nan | 0.0 | 0.0 | 0.0 | | low vegetation | nan | nan | 0.0 | 0.0 | 0.0 | | tree | nan | nan | 0.0 | 0.0 | 0.0 | | building | nan | nan | 0.0 | 0.0 | 0.0 | | sports field | nan | nan | 0.0 | 0.0 | 0.0 | +----------------+--------+-----------+--------+-------+-------+The text was updated successfully, but these errors were encountered: