ValueError: Conversion modality could not be found #1014

espentrydal · 2023-11-10T08:26:30Z

espentrydal
Nov 10, 2023

Hi

I got this error probably at the end of converting files from ADNI to BIDS (participants.tsv has been created). I am unsure what the error means, and if it means I need to rerun the conversion, and in that case if there is anything I need to fix first.

2023-11-10 08:56:02,755:INFO:Creating scans files...
Traceback (most recent call last):
  File "/home/espen/miniconda3/envs/clinica/bin/clinica", line 8, in <module>
    sys.exit(main())
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids_cli.py", line 74, in cli
    adni_to_bids.convert_clinical_data(
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids.py", line 154, in convert_clinical_data
    adni_utils.create_adni_scans_files(conversion_path, bids_subjs_paths)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_utils.py", line 1117, in create_adni_scans_files
    converted_mod = find_conversion_mod(file_name)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_utils.py", line 1158, in find_conversion_mod
    raise ValueError(f"Conversion modality could not be found for file {file_name}")
ValueError: Conversion modality could not be found for file sub-ADNI053S4813_ses-M000_FLAIR_real.json

Could it have something to do with this: #396 ?

Clinica version 0.7.6.
Converters: adni-2-bids
Ubuntu 22.04

Thanks!
Espen

NicolasGensollen · 2023-11-10T09:18:01Z

NicolasGensollen
Nov 10, 2023
Maintainer

Hi @espentrydal

You get this error because the function find_conversion_mod(file_name), which tries to guess the modality from a file path, fails to do so. The implementation looks like this:

https://github.com/aramis-lab/clinica/blob/a258c7252a31a7de59b30988b5fbfb760c07aabd/clinica/iotools/converters/adni_to_bids/adni_utils.py#L1142C5-L1161

As you can see, it is failing with your file because real gets interpreted as the modality (which is not a valid and supported BIDS entity) instead of FLAIR.

Usually, the files with real and imaginary suffixes are generated by dcm2niix when it is not able to correctly convert the full image.

It is difficult to tell you more without additional information. Did you notice any conversion error in the logs ? Could you provide them here ?

Best,
Nicolas

0 replies

espentrydal · 2023-11-10T09:36:27Z

espentrydal
Nov 10, 2023
Author

Hi @NicolasGensollen

Thank you for your very swift reply. I did not pipe stdout/stderr so unfortunately I only have the last 1400 lines which are related to fMRIs where there are some errors related to missing slices. I could not find a log file in the current directory or in the bids directory except for the modality_paths.tsv files. Are there any other logs available?

1 reply

espentrydal Nov 10, 2023
Author

I am redownloading the dataset (previously it was two downloads fused into one, apparently without any conflicts) to be sure. I will then retry the conversion and make sure I have stdout and stderr saved in files. It will take a few days downloading and rerunning the conversion.

espentrydal · 2023-11-15T11:01:43Z

espentrydal
Nov 15, 2023
Author

@NicolasGensollen

I have re-downloaded the dataset as one collection and rerun the Clinica conversion. I get a similar error message for another subject. And the conversions stop at this point as last time.

$ cat stderr.txt

/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids.py:206: DtypeWarning: Columns (19,20,21,50,51,104,10
5,106) have mixed types. Specify dtype option on import or set low_memory=False.
  adni_merge = pd.read_csv(adni_merge_path)
/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/bids_utils.py:114: DtypeWarning: Columns (19,20,21,50,51,104,105,106) have mixed types. S
pecify dtype option on import or set low_memory=False.
  file_to_read = pd.read_csv(file_to_read_path)
/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/bids_utils.py:114: DtypeWarning: Columns (19,20,21,50,51,104,105,106) have mixed types. S
pecify dtype option on import or set low_memory=False.
  file_to_read = pd.read_csv(file_to_read_path)
/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_utils.py:643: PerformanceWarning: indexing past lexsort dept
h may impact performance.
  dxsum_df.loc[(alternative_id, "sc"), "DIAGNOSIS"].values[0]
Traceback (most recent call last):
  File "/home/espen/miniconda3/envs/clinica/bin/clinica", line 8, in <module>
    sys.exit(main())
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids_cli.py", line 74, in cli
    adni_to_bids.convert_clinical_data(
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_to_bids.py", line 154, in convert_clinical_data
    adni_utils.create_adni_scans_files(conversion_path, bids_subjs_paths)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_utils.py", line 1117, in create_adni_scans_files
    converted_mod = find_conversion_mod(file_name)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/clinica/iotools/converters/adni_to_bids/adni_utils.py", line 1158, in find_conversion_mod
    raise ValueError(f"Conversion modality could not be found for file {file_name}")
ValueError: Conversion modality could not be found for file sub-ADNI002S4262_ses-M060_FLAIR_real.nii

1 reply

NicolasGensollen Nov 15, 2023
Maintainer

@espentrydal could you check whether you get files with real and imaginary suffixes for other subject / session ?

It could be that there is an issue with the conversion for this specific scan for some reason.

espentrydal · 2023-11-15T15:07:11Z

espentrydal
Nov 15, 2023
Author

@NicolasGensollen I have no files with imaginary suffix (find . -name '*imaginary*' -print). With real suffix there is a long list (find . -name '*real*' -print): https://gist.github.com/espentrydal/7151bd28a9cc0cb76d71d838c0d196d6

1 reply

espentrydal Nov 16, 2023
Author

In case it helps: For the example above, the sub-ADNI002S4262_ses-M060_FLAIR_real.nii is correctly stored in the anat/ folder. The accompanying json file lists (extract):

        "Modality": "MR",
        "MagneticFieldStrength": 3,
        "Manufacturer": "Philips",
        "ManufacturersModelName": "Intera",
        "InstitutionName": "OHSU MRI3",
        "MRAcquisitionType": "2D",
        "SeriesDescription": "Axial T2-FLAIR",
        "ProtocolName": "Axial T2-FLAIR SENSE",
        "ScanningSequence": "IR",
        "SequenceVariant": "SK",
        "ScanOptions": "OTHER",
        "ImageType": ["ORIGINAL", "PRIMARY", "M", "IR", "M", "IR", "REAL"],

(There is no json file accompanying the gzipped T1W scan in the same folder.)

In the func folder there is a functional MRI scan also with a _real suffix (sub-ADNI002S4262_ses-M060_task-rest_bold_real.nii). From the accompanying json file:

        "SeriesDescription": "Extended Resting State fMRI",
        "ProtocolName": "Extended Resting State fMRI",
        "ScanningSequence": "GR",
        "SequenceVariant": "SK",
        "ScanOptions": "FS",
        "ImageType": ["ORIGINAL", "PRIMARY", "M", "FFE", "M", "FFE", "REAL"],

espentrydal · 2023-11-16T11:30:25Z

espentrydal
Nov 16, 2023
Author

@NicolasGensollen After seeing this issue #1012 I redid the conversion of sub-ADNI002S4262 with Clinica 0.7.2. This works well without any errors. I receive the same errors as before with FLAIR or fMRI if I try this with 0.7.6. So it seems to be an error introduced between these versions.

5 replies

NicolasGensollen Nov 16, 2023
Maintainer

@espentrydal thanks for looking into this !
I'm a bit surprised that the conversion is successful with 0.7.2 and fails with 0.7.6. We will need to investigate further to understand why and fix the problem.
Can rely on Clinica 0.7.2 for your own work while we fix this ?

espentrydal Nov 17, 2023
Author

@NicolasGensollen Conversion is still ongoing, but it seems to be going well. Thanks!

I left the results from the previous conversion with hopes that this time it would go faster/skip previous successful conversions (would save me a lot of time). So there is a v0 conversion and the v1 conversion is ongoing. I can see that now I have this:

$ tree
├── ses-M060
│   ├── anat
│   │   ├── sub-ADNI002S4262_ses-M060_FLAIR_real.json
│   │   ├── sub-ADNI002S4262_ses-M060_FLAIR_real.nii
│   │   └── sub-ADNI002S4262_ses-M060_T1w.nii.gz
│   ├── func
│   │   ├── sub-ADNI002S4262_ses-M060_task-rest_bold_real.json
│   │   └── sub-ADNI002S4262_ses-M060_task-rest_bold_real.nii.gz
│   └── sub-ADNI002S4262_ses-M060_scans.tsv
├── ses-M60
│   └── anat
│       ├── sub-ADNI002S4262_ses-M60_FLAIR.json
│       ├── sub-ADNI002S4262_ses-M60_FLAIR.nii.gz
│       └── sub-ADNI002S4262_ses-M60_T1w.nii.gz

How does Clinica handle a second conversion into a folder with a previous one? It seems all images with the _real suffix have a three integer session id where the first is a 0, while sessions from the ongoing conversion only have two integers in the session id. Would it be safe to just delete all these folders?

ghisvail Nov 17, 2023
Collaborator

Between version 0.7.2 and 0.7.6, we have normalized session labels to be 3 digits.

ghisvail Nov 17, 2023
Collaborator

It's also not advisable to re-use the same BIDS folder for consecutive conversions. Consider 2 conversions as 2 separate BIDS datasets, which you may decide to merge into one eventually.

espentrydal Nov 17, 2023
Author

I see, thanks.

NicolasGensollen · 2023-12-04T11:21:01Z

NicolasGensollen
Dec 4, 2023
Maintainer

I was able to reproduce this problem using clinica 0.7.7 and dcm2niix v1.0.20230411.

When running the actual dcm2niix commands extracted from the converter:

subject ADNI002S5178 session M000 (no problem):

$ dcm2niix -w 0 -f sub-ADNI002S5178_ses-M000_FLAIR -o /Users/nicolas.gensollen/Downloads/bids/sub-ADNI002S5178/ses-M000/anat -z n -b y -ba y /Users/nicolas.gensollen/Downloads/ADNI/002_S_5178/Axial_T2-FLAIR/2013-05-16_19_31_16.0/I372847
Compression will be faster with 'pigz' installed http://macappstore.org/pigz/
Chris Rorden's dcm2niiX version v1.0.20230411  Clang14.0.0 ARM (64-bit MacOS)
Found 35 DICOM file(s)
Philips Scaling Values RS:RI:SS = 2.08132:0:0.00464127 (see PMC3998685)
Convert 35 DICOM as /Users/nicolas.gensollen/Downloads/bids/sub-ADNI002S5178/ses-M000/anat/sub-ADNI002S5178_ses-M000_FLAIR (256x256x35x1)
Conversion required 0.062921 seconds (0.028793 for core code).

subject ADNI002S4262 session M048 (problem):

$ dcm2niix -w 0 -f sub-ADNI002S4262_ses-M048_FLAIR -o /Users/nicolas.gensollen/Downloads/bids/sub-ADNI002S4262/ses-M048/anat -z n -b y -ba y /Users/nicolas.gensollen/Downloads/ADNI/002_S_4262/Axial_T2-FLAIR/2015-10-15_07_18_13.0/I573588
Compression will be faster with 'pigz' installed http://macappstore.org/pigz/
Chris Rorden's dcm2niiX version v1.0.20230411  Clang14.0.0 ARM (64-bit MacOS)
Found 35 DICOM file(s)
Philips Scaling Values RS:RI:SS = 2.11746:0:0.00437488 (see PMC3998685)
Convert 35 DICOM as /Users/nicolas.gensollen/Downloads/bids/sub-ADNI002S4262/ses-M048/anat/sub-ADNI002S4262_ses-M048_FLAIR_real (256x256x35x1)
Conversion required 0.062863 seconds (0.028903 for core code).

I'm not sure what is going wrong with these conversions but it seems like an issue with the data and/or with dcm2niix.

I think the best move for Clinica is to ignore those images instead of keeping them and crashing afterwards.

@MatthieuJoulot @ghisvail any idea ?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ValueError: Conversion modality could not be found #1014

{{title}}

Replies: 6 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

ValueError: Conversion modality could not be found #1014

espentrydal Nov 10, 2023

Replies: 6 comments · 8 replies

NicolasGensollen Nov 10, 2023 Maintainer

espentrydal Nov 10, 2023 Author

espentrydal Nov 10, 2023 Author

espentrydal Nov 15, 2023 Author

NicolasGensollen Nov 15, 2023 Maintainer

espentrydal Nov 15, 2023 Author

espentrydal Nov 16, 2023 Author

espentrydal Nov 16, 2023 Author

NicolasGensollen Nov 16, 2023 Maintainer

espentrydal Nov 17, 2023 Author

ghisvail Nov 17, 2023 Collaborator

ghisvail Nov 17, 2023 Collaborator

espentrydal Nov 17, 2023 Author

NicolasGensollen Dec 4, 2023 Maintainer

espentrydal
Nov 10, 2023

Replies: 6 comments 8 replies

NicolasGensollen
Nov 10, 2023
Maintainer

espentrydal
Nov 10, 2023
Author

espentrydal Nov 10, 2023
Author

espentrydal
Nov 15, 2023
Author

NicolasGensollen Nov 15, 2023
Maintainer

espentrydal
Nov 15, 2023
Author

espentrydal Nov 16, 2023
Author

espentrydal
Nov 16, 2023
Author

NicolasGensollen Nov 16, 2023
Maintainer

espentrydal Nov 17, 2023
Author

ghisvail Nov 17, 2023
Collaborator

ghisvail Nov 17, 2023
Collaborator

espentrydal Nov 17, 2023
Author

NicolasGensollen
Dec 4, 2023
Maintainer