Question about multi-GPU training

[zyzhou11]

Hello, your work is outstanding. While reproducing the code, I encountered the following issue.
In a single-GPU environment, I successfully trained the model using the following command.

"python main.py --config cfgs/PCN_models/SymmCompletion.yaml --val_freq 10 --val_interval 50 --exp_name train_pcn"

However, in a multi-GPU environment, the training does not proceed as expected when using the following command.

"CUDA_VISIBLE_DEVICES=0,1 python main.py --launcher pytorch --config cfgs/PCN_models/SymmCompletion.yaml --val_freq 10 --val_interval 50 --exp_name train_pcn"

Could you provide the correct command for multi-GPU training?

[markhome1]

You can delete os.environ['CUDA_VISIBLE_DEVICES'] = "1" in main.py and try again

[zyzhou11]

I have already deleted the line "os.environ['CUDA_VISIBLE_DEVICES'] = "1"" from the code.
I'm wondering if the "--launcher pytorch" option is needed in the command.
If I don't use the '-- launcher pytorch' option, the code will be trained using DataParallel, but the speed will be very slow (multi-GPU). If I were to use this option, the following error would occur.

So I would like to know what command you use for multi-GPU training.

[markhome1]

我跑通了，时间节省了30%，
CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 main.py
--launcher pytorch
--config cfgs/PCN_models/SymmCompletion.yaml
--val_freq 10
--val_interval 50
--exp_name train_pcn
然后修改一下像图片中的代码，再用上面的命令运行

[zyzhou11]

非常感谢您的耐心解答，谢谢！

[markhome1]

@HongyuYann hope the author can give us a detailed explanation.

[shiboliu]

#4 (comment)

在 4090 X 2 的机器上，bs 设置为多少？
在 启动程序 和 save checkpoint 的时候很耗时？

[hanryyan]

I have already deleted the line "os.environ['CUDA_VISIBLE_DEVICES'] = "1"" from the code. I'm wondering if the "--launcher pytorch" option is needed in the command. If I don't use the '-- launcher pytorch' option, the code will be trained using DataParallel, but the speed will be very slow (multi-GPU). If I were to use this option, the following error would occur.

So I would like to know what command you use for multi-GPU training.

I'm sorry for not getting back to you sooner. I have solved the problem in terms of low training speed when using the DataParallel. You can use the following newest command "python main.py --config cfgs/PCN_models/SymmCompletion.yaml --val_freq 10 --val_interval 50 --exp_name train_pcn --deterministic" ("deterministic" is important for the mode of DataParallel.)