How to choose execution provider ?

[mykolamelnykml]

How to choose execution provider ?
How to force run only on CPU or GPU?

[felixdittrich92]

Hey @kolia1985 👋,

I have released a new version (https://github.com/felixdittrich92/OnnxTR/releases/tag/v0.3.0).
Now you have the full control to configure the underlying Engine (InferenceSession) to your needs :)

For example to force GPU usage:

from onnxtr.models import ocr_predictor, EngineConfig

providers = [("CUDAExecutionProvider", {"device_id": 0})]  # For available providers see: https://onnxruntime.ai/docs/execution-providers/

engine_config = EngineConfig(providers=providers)

predictor = ocr_predictor(
    det_engine_cfg=engine_config,
    reco_engine_cfg=engine_config,
    clf_engine_cfg=engine_config  # This will be only used with `assume_straight_pages=False`
)

So the single predictors are completly configurable for example if you want to run the detection model on CPU and the recognition model on GPU you can do this now 🤗

This gives also the option to test different execution providers like DirectML / OpenVINO / .. (but keep in mind it's not tested with these providers yet)

Best regards,
Felix

[mykolamelnykml]

@felixdittrich92 Thanks a lot!

[felixdittrich92]

FYI:

I deployed a patch release version which fixes:

• the super high VRAM usage
• ~5 times higher inference latency as described in the latency table