How to sepeifiy world_size for tensorrtllm backend?

[Saigut]

The ./scripts/launch_triton_server.py in tensorrtllm_backend repo has a parameter --world_size, but tritonserver don't.

So when I try to launch a "tp_size=4 pp_size=1" tensorrtllm model using tritonserver, it will report error like mpiSize != tp_size*pp_size, due to mpiNUm is always 1.

Is there any method for tritonserver to specify world_size for tensorrtllm model?

I think the most proper way maybe to set world_size in model repo of tensorrtllm model, like in config.pbtxt, but I can't find such a option.

[krishung5]

Hi @Saigut, it is required to launch Triton with the script launch_triton_server.py when using tensorrtllm backend in order to have the correct setup for MPI.

[Saigut]

It's a different way than other backends. And how about when there are many tensorrtllm models that have different tp_size*pp_size?

[nnshah1]

@Saigut We think its a good idea to make this simpler and more straight forward. Adding a feature request to the tensorrt-llm backend repository is probably the best place to track it.