Support for Tensor Parallelism in openvino.genai.benchmark?

[Dazui-Wang]

Hi, I'm exploring performance scaling with large language models in openvino.genai. Does the benchmark(tools/llm_bench/benchmark.py) function support any argument similar to vLLM's --tensor-parallel-size to enable tensor parallelism across devices?

[as-suvorov]

Hi, there is no such option for llm_bench tool for the moment. However you can pass OpenVINO plugin configuration properties into LLMPipeline constructor.
Please find information about plugin configuration options in the documentation: https://docs.openvino.ai/2025/openvino-workflow/running-inference/inference-devices-and-modes.html
You could try to use ov::hint::model_distribution_policy({ov::hint::ModelDistributionPolicy::TENSOR_PARALLEL}) property, src: https://github.com/openvinotoolkit/openvino/blob/master/src/inference/include/openvino/runtime/properties.hpp#L409-L461