Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DL+DiReCT fails on CPU only workstation #3

Open
rcruces opened this issue Feb 28, 2023 · 1 comment
Open

DL+DiReCT fails on CPU only workstation #3

rcruces opened this issue Feb 28, 2023 · 1 comment

Comments

@rcruces
Copy link

rcruces commented Feb 28, 2023

Hello, I'm trying to run DL-DiReCT on a workstation with no CUDA, but 40 CPU Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz. I think I'm getting the next error because pytorch might be looking for CUDA but I have none.

Is there anyway to work thought this issue?

dl+direct --subject sub-01 --no-cth sub-01_T1w.nii.gz out/

If you are using DL+DiReCT in your research, please cite:
	Rebsamen, M, Rummel, C, Reyes, M, Wiest, R, McKinley, R.
	Direct cortical thickness estimation using deep learning�based anatomy segmentation and cortex parcellation.
	Human Brain Mapping. 2020; 41: 4804-4814. https://doi.org/10.1002/hbm.25159

Input: (0.5, 0.5, 0.5) [RAS]
Resampling
loading checkpoint /data/mica1/01_programs/DL-DiReCT/src/../model/v0_f1.pth
Traceback (most recent call last):
  File "/data/mica1/01_programs/DL-DiReCT/src/../src/DeepSCAN_Anatomy_Newnet_apply.py", line 439, in <module>
    checkpoint = load_checkpoint(model_file, device)
  File "/data/mica1/01_programs/DL-DiReCT/src/../src/DeepSCAN_Anatomy_Newnet_apply.py", line 402, in load_checkpoint
    return torch.load(checkpoint_file, map_location=device)
  File "/home/anaconda3/envs/DL_DiReCT/lib/python3.10/site-packages/torch/serialization.py", line 795, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/anaconda3/envs/DL_DiReCT/lib/python3.10/site-packages/torch/serialization.py", line 1012, in _legacy_load
    result = unpickler.load()
  File "/home/anaconda3/envs/DL_DiReCT/lib/python3.10/site-packages/torch/serialization.py", line 958, in persistent_load
    wrap_storage=restore_location(obj, location),
  File "/home/anaconda3/envs/DL_DiReCT/lib/python3.10/site-packages/torch/serialization.py", line 1055, in restore_location
    return default_restore_location(storage, str(map_location))
  File "/home/anaconda3/envs/DL_DiReCT/lib/python3.10/site-packages/torch/serialization.py", line 215, in default_restore_location
    result = fn(storage, location)
  File "/home/anaconda3/envs/DL_DiReCT/lib/python3.10/site-packages/torch/serialization.py", line 185, in _cuda_deserialize
    return torch.UntypedStorage(obj.nbytes(), device=torch.device(location))
RuntimeError: CUDA error: out of memory
ERROR: Segmentation failed
@mrunibe
Copy link
Member

mrunibe commented Feb 28, 2023

It should be possible to run DL+DiReCT on CPU(s) only. If no GPU is detected, the script should fallback to CPU only (takes more time).

Maybe you have a graphic card that is detected as GPU but with very limited amount of memory (check with nvidia-smi). In this case you may try to disable the GPU explicitly with:
export CUDA_VISIBLE_DEVICES=-1

Alternatively, the error above might also because there is not enough memory (RAM) available. You may try:

  1. Is your MRI already skull-stripped (brain without skull)? If not, add the --bet option to run brain extraction first
  2. If this still fails, you may additionally try the option --lowmem to further reduce memory consumption (see also FAQ)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants