Add initial support for the DICOM format

[kostrykin]

This PR builds on the work initiated by @hexylena in #18895 (the PR was abandoned).

DICOM is a popular file format in medical imaging.

DICOM files are identified using the pydicom package, which is also used to read metadata. The pydicom package is a pure Python package that has no further dependencies and is 2.4 MB in size.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[kostrykin]

Turns out, pydicom is not available on Python 3.9, only on ≥3.10. To cope with that, I have made the import in lib/galaxy/datatypes/images.py optional in c82854c. However, CI still tries to install the package on 3.9 and fails (e.g., here). How can I mark the dependency as optional?

[kostrykin]

Package tests are failing https://github.com/galaxyproject/galaxy/actions/runs/19894695277/job/57021998725?pr=21385#step:7:5453 because metadata is not set as expected.

I assume that this is because pydicom is not installed for the package tests (in this case, Dicom.set_meta will not set the metadata). Is it possible to add pydicom as a dependency for the package tests?

--
Edit: I hope that 2758853 does the trick…

[kostrykin]

Package tests still failing, not sure why:
https://github.com/galaxyproject/galaxy/actions/runs/19897139188/job/57030412010?pr=21385#step:7:8308

====================== 50 passed, 136 warnings in 19.03s =======================

• '[' 0 -eq 0 ']'
• mypy .
  galaxy/webapps/openapi/utils.py:66: error: Unexpected keyword argument
  "schema_generator" for "get_definitions" [call-arg]
  field_mapping, definitions = get_definitions(
  ^~~~~~~~~~~~~~~~
  /tmp/gxpkgtestenvdGWpmY/lib/python3.9/site-packages/fastapi/_compat/main.py:267: note: "get_definitions" defined here
  galaxy/webapps/openapi/utils.py:74: error: Unexpected keyword argument
  "schema_generator" for "get_openapi_path" [call-arg]
  result = get_openapi_path(
  ^~~~~~~~~~~~~~~~~
  /tmp/gxpkgtestenvdGWpmY/lib/python3.9/site-packages/fastapi/openapi/utils.py:250: note: "get_openapi_path" defined here
  galaxy/webapps/openapi/utils.py:92: error: Unexpected keyword argument
  "schema_generator" for "get_openapi_path" [call-arg]
  result = get_openapi_path(
  ^~~~~~~~~~~~~~~~~
  Found 3 errors in 1 file (checked 149 source files)
  test_galaxy_packages: exit 1 (1145.76 seconds) /home/runner/work/galaxy/galaxy/galaxy root> bash packages/test.sh pid=4850
  test_galaxy_packages: FAIL code 1 (1150.51=setup[0.70]+cmd[4.04,1145.76] seconds)
  evaluation failed :( (1150.58 seconds)
  Error: Process completed with exit code 1.

[nsoranzo]

Package tests still failing, not sure why: https://github.com/galaxyproject/galaxy/actions/runs/19897139188/job/57030412010?pr=21385#step:7:8308

====================== 50 passed, 136 warnings in 19.03s =======================

• '[' 0 -eq 0 ']'
• mypy .
  galaxy/webapps/openapi/utils.py:66: error: Unexpected keyword argument
  "schema_generator" for "get_definitions" [call-arg]
  field_mapping, definitions = get_definitions(
  ^~~~~~~~~~~~~~~~
  /tmp/gxpkgtestenvdGWpmY/lib/python3.9/site-packages/fastapi/_compat/main.py:267: note: "get_definitions" defined here
  galaxy/webapps/openapi/utils.py:74: error: Unexpected keyword argument
  "schema_generator" for "get_openapi_path" [call-arg]
  result = get_openapi_path(
  ^~~~~~~~~~~~~~~~~
  /tmp/gxpkgtestenvdGWpmY/lib/python3.9/site-packages/fastapi/openapi/utils.py:250: note: "get_openapi_path" defined here
  galaxy/webapps/openapi/utils.py:92: error: Unexpected keyword argument
  "schema_generator" for "get_openapi_path" [call-arg]
  result = get_openapi_path(
  ^~~~~~~~~~~~~~~~~
  Found 3 errors in 1 file (checked 149 source files)
  test_galaxy_packages: exit 1 (1145.76 seconds) /home/runner/work/galaxy/galaxy/galaxy root> bash packages/test.sh pid=4850
  test_galaxy_packages: FAIL code 1 (1150.51=setup[0.70]+cmd[4.04,1145.76] seconds)
  evaluation failed :( (1150.58 seconds)
  Error: Process completed with exit code 1.

You can ignore that, it's another issue that will be fixed by #21384 .

[kostrykin]

https://github.com/galaxyproject/galaxy/actions/runs/19934924298/job/57156962812?pr=21385#step:8:6244

534     >>> fname = get_test_fname('1.tiff')
535     >>> guess_ext(fname, sniff_order)
Expected:
    'tiff'
Got:
    'ome.tiff'

I don't understand this.

https://github.com/galaxyproject/galaxy/actions/runs/19934924298/job/57156962812?pr=21385#step:8:3697
The test/unit/data/datatypes/test_images.py::test_tiff_sniff test passes, so we know that for 1.tiff

✅ Tiff().sniff returns True and

✅ OMETiff().sniff returns False:

def test_tiff_sniff():
    for filename in (
        ...,
        "1.tiff",  # <--!!!
        ...,
    ):
        fname = get_test_fname(filename)
        ...
        assert not OMETiff().sniff(fname), f"filename: {filename}"  # <--!!!
        assert Tiff().sniff(fname), f"filename: {filename}"  # <--!!!

Why does ❌ guess_ext tell that 1.tiff is an ome.tiff then?

---

Note: I think this must be caused by something changed in https://github.com/galaxyproject/galaxy/pull/21385/files/df5fe20cf57b5084163645d4e63179bcc57e17a8..d8375ee948434b2600cefebc719301e0d4666c46, because the unit tests passed before that.

[nsoranzo]

I have a feeling this won't work for larger files where the image is larger than the contents_header_bytes .

Maybe you can use the @disable_parent_class_sniffing decorator from lib/galaxy/datatypes/sniff.py and restore the previous sniff method?

[kostrykin]

I tried it (see kostrykin#4), but when I put the @disable_parent_class_sniffing decorator on the OMETiff class, the sniff method does not get called. The decorator seems to disable sniffing.

The test/unit/data/datatypes/test_images.py::test_ome_tiff_sniff fails with

>       assert OMETiff().sniff(fname)
E       AssertionError: assert False
E        +  where False = <lambda>('lib/galaxy/datatypes/test/1.ome.tiff')
E        +    where <lambda> = <galaxy.datatypes.images.OMETiff object at 0x7f99946c3190>.sniff
E        +      where <galaxy.datatypes.images.OMETiff object at 0x7f99946c3190> = OMETiff()

Despite that OMETiff.sniff is implemented as

    def sniff(self, filename: str) -> bool:
        raise ValueError('OMETiff.sniff called')  # <-- !!! this error is *not* reported !!!
        with tifffile.TiffFile(filename) as tif:
            return tif.is_ome

thus, OMETiff.sniff is not called when the @disable_parent_class_sniffing decorator is in place.

Here is the error from running the test: https://github.com/kostrykin/galaxy/actions/runs/19942920690/job/57185121231?pr=4#step:8:11238

That said, I think the current solution should work, because tifffile.TiffFile only reads the headers/metadata of the file. The pixel content only is read when the corresponding methods are used (they are not used here).

On the other hand, since we do not know how large metadata can be, or how it is organized, if there is a possibility to get this working without sniff_prefix, I'd go for that.

[kostrykin]

I think all remaining CI failures are unrelated to this PR.