Can't use docling with rocm

[apalazzi]

Bug

I'm trying to use docling with AMD rocm, but I can't get it to work, I always get torch.AcceleratorError: HIP error: invalid device function

I install docling in its own venv with

pip install "docling[vlm,tesserocr,rapidocr,easyocr]"  --extra-index-url https://download.pytorch.org/whl/rocm6.4

~$ python3 
Python 3.12.3 (main, Jan  8 2026, 11:30:50) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(f"PyTorch version: {torch.__version__}")
PyTorch version: 2.9.0+rocm6.4
>>> print(f"HIP version: {torch.version.hip}")
HIP version: 6.4.43484-123eb5128
>>> print(f"GPU device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'None'}")
GPU device: AMD Radeon Graphics
>>> 

So far so good, but when I try to use it it doesn't workL

~$ docling https://arxiv.org/pdf/2206.01062[INFO] 2026-01-21 08:03:02,545 [RapidOCR] base.py:22: Using engine_name: onnxruntime
[INFO] 2026-01-21 08:03:02,549 [RapidOCR] download_file.py:60: File exists and is valid: /usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/rapidocr/models/ch_PP-OCRv4_det_infer.onnx
[INFO] 2026-01-21 08:03:02,549 [RapidOCR] main.py:53: Using /usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/rapidocr/models/ch_PP-OCRv4_det_infer.onnx
[INFO] 2026-01-21 08:03:02,593 [RapidOCR] base.py:22: Using engine_name: onnxruntime
[INFO] 2026-01-21 08:03:02,594 [RapidOCR] download_file.py:60: File exists and is valid: /usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/rapidocr/models/ch_ppocr_mobile_v2.0_cls_infer.onnx
[INFO] 2026-01-21 08:03:02,594 [RapidOCR] main.py:53: Using /usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/rapidocr/models/ch_ppocr_mobile_v2.0_cls_infer.onnx
[INFO] 2026-01-21 08:03:02,617 [RapidOCR] base.py:22: Using engine_name: onnxruntime
[INFO] 2026-01-21 08:03:02,623 [RapidOCR] download_file.py:60: File exists and is valid: /usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/rapidocr/models/ch_PP-OCRv4_rec_infer.onnx
[INFO] 2026-01-21 08:03:02,623 [RapidOCR] main.py:53: Using /usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/rapidocr/models/ch_PP-OCRv4_rec_infer.onnx
2026-01-21 08:03:07,117 ERROR   docling.pipeline.standard_pdf_pipeline: Stage layout failed for run 1: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
Traceback (most recent call last):
  File "/usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/docling/pipeline/standard_pdf_pipeline.py", line 280, in _process_batch
    processed_pages = list(self.model(good[0].conv_res, pages))  # type: ignore[arg-type]
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/docling/models/base_layout_model.py", line 35, in __call__
    predictions = self.predict_layout(conv_res, pages)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/docling/models/layout_model.py", line 180, in predict_layout
    batch_predictions = self.layout_predictor.predict_batch(  # type: ignore[attr-defined]
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/docling_ibm_models/layoutmodel/layout_predictor.py", line 226, in predict_batch
    self._image_processor.post_process_object_detection(
  File "/usr/share/llama.cpp/docling-venv/lib/python3.12/site-packages/transformers/models/rt_detr/image_processing_rt_detr.py", line 1094, in post_process_object_detection
    "scores": score[score > threshold],
              ~~~~~^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: HIP error: invalid device function
Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

Docling version

~$ docling --version
Docling version: 2.68.0
Docling Core version: 2.60.0
Docling IBM Models version: 3.10.3
Docling Parse version: 4.7.3
Python: cpython-312 (3.12.3)
Platform: Linux-6.8.0-90-generic-x86_64-with-glibc2.39

Python version

~$ python --version
Python 3.12.3

[dosubot]

Hi @apalazzi! I'm Dosu and I’m helping the docling team.

Docling's layout pipeline relies on HuggingFace transformers (specifically RT-DETR) and PyTorch for model execution. The "HIP error: invalid device function" is a runtime error from the AMD ROCm stack and usually means that either the model or one of its custom kernels (like MultiScaleDeformableAttention in transformers) is not compatible with your GPU or ROCm version. This is not a Docling-specific issue—there are known build and runtime failures with ROCm/HIP in Docling, especially in the layout stage, and similar errors have been reported by other users on Fedora and other platforms with ROCm and PyTorch [source].

Docling's device selection logic does not explicitly recognize ROCm/HIP devices; it only uses CUDA if PyTorch exposes it, so ROCm GPUs are only used if they present a CUDA-compatible API. There is no ROCm-specific adaptation in the code [source].

As a workaround, try running Docling in CPU-only mode (set the device to "cpu" in your pipeline options). This should avoid the HIP error, though it will be slower. If you need GPU acceleration, there is currently no documented workaround for ROCm—compatibility depends on upstream support in PyTorch and transformers. You can check for updates to PyTorch, transformers, and Docling in the future, as there is ongoing work to improve support for non-CUDA accelerators, but ROCm is not yet officially supported [source].

If CPU fallback is not acceptable, you may want to test alternative models or pipelines that do not rely on RT-DETR or custom CUDA/HIP kernels, but full ROCm compatibility for layout models is not confirmed at this time.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}