[Question]: How to run Flair SequenceTagger inference with ONNX?

[DaniSalva]

Question

Hi everyone!

I’m trying to optimize inference speed for a Flair Sequence tagger (flair/ner-lang-large) model based on XLM-R using ONNX to improve CPU performance.

I followed the guide here and managed to export the model to ONNX succesfully. However, I'm not sure how to actually run inference in an ONNX environment (using onnxruntime) and get the same prediction as with Flair.

Does anyone have an example or code snippet showing how to perform inference with a flair model in ONNX? I also tried JIT but I hadn't speed improvements.
Any reference or working example would be greatly appreciated🙏

[mdiermaier]

I have a similar issue in onnx. I guess you must also still run onnx with flair itself and load from flair.
e.g.

from flair.nn import Classifier
tagger = Classifier.load("ner-german-large-onnx.pt")

However this only worked for me with an not optimized model, which obviously has no real benefit...

Now when i try to optimize the model for cpu inference

tagger.embeddings = tagger.embeddings.export_onnx(
    "ner-german-large-embeddings.onnx",
    sentences,
    providers=["CPUExecutionProvider"],
    session_options={},
    opset_version=17,
)

tagger.embeddings.optimize_model(
    optimize_model_path = "optimized-ner-german-large-embeddings.onnx",
    opt_level=2,
    use_gpu=False,
    only_onnxruntime=False,
    use_external_data_format=True,
)

the optimize_model part uses all my memory and then my shell crashes (>60 GB RAM). I tried this for the model "flair/ner-german-large" as downloaded directly from huggingface.

Any Ideas what the problem is?