-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathnohup.out
35 lines (35 loc) · 2.57 KB
/
nohup.out
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
Let's use 2 GPUs!
Let's use 2 GPUs!
Tue Mar 8 10:02:17 2022 Rank: 0, Train Epoch: 0, Iter: 0/1563, Loss: 4.602168560028076
Tue Mar 8 10:02:35 2022 Rank: 0, Train Epoch: 0, Iter: 156/1563, Loss: 2.231948137283325
Tue Mar 8 10:02:53 2022 Rank: 0, Train Epoch: 0, Iter: 312/1563, Loss: 2.2177748680114746
Tue Mar 8 10:03:12 2022 Rank: 0, Train Epoch: 0, Iter: 468/1563, Loss: 2.5263755321502686
Tue Mar 8 10:03:30 2022 Rank: 0, Train Epoch: 0, Iter: 624/1563, Loss: 2.2935409545898438
Tue Mar 8 10:03:48 2022 Rank: 0, Train Epoch: 0, Iter: 780/1563, Loss: 2.060790538787842
Tue Mar 8 10:04:07 2022 Rank: 0, Train Epoch: 0, Iter: 936/1563, Loss: 2.24410080909729
Tue Mar 8 10:04:25 2022 Rank: 0, Train Epoch: 0, Iter: 1092/1563, Loss: 2.0432682037353516
Tue Mar 8 10:04:44 2022 Rank: 0, Train Epoch: 0, Iter: 1248/1563, Loss: 1.985356330871582
Tue Mar 8 10:05:02 2022 Rank: 0, Train Epoch: 0, Iter: 1404/1563, Loss: 1.8439347743988037
Tue Mar 8 10:05:20 2022 Rank: 0, Train Epoch: 0, Iter: 1560/1563, Loss: 1.6929256916046143
Tue Mar 8 10:05:21 2022 Rank: 0, Valid Epoch: 0, Iter: 0/625, Loss: 1.7742857933044434
Traceback (most recent call last):
File "/home/tracy/anaconda3/envs/open-mmlab/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/tracy/anaconda3/envs/open-mmlab/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/tracy/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in <module>
main()
File "/home/tracy/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/home/tracy/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/tracy/anaconda3/envs/open-mmlab/bin/python3', '-u', 'cifar10_ddp_syncBN.py', '--local_rank=1']' died with <Signals.SIGTERM: 15>.
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Killing subprocess 964893
Killing subprocess 964894