❓ [Question] How do you ....?

[andrei-pokrovsky]

❓ Question

Hi, could you please explain how this is better than pytorch to Onnx to TensorRT export path?

[narendasan]

This path provides a few benefits vs. ONNX.

1. IR - One of the issues with ONNX is that you 1. need to have a valid mapping from all PyTorch operators to ONNX operators and 2. you need a valid mapping of all ONNX operators to TensorRT Layers via the ONNX parser. This is not always the case and has been a blocking problem for many users. Torch-TensorRT uses TorchScript which is a built in IR for PyTorch which has much better coverage of PyTorch Operators. Torch-TensorRT goes directly from TorchScript to TensorRT.

   a. Integration - Torch-TensorRT produces valid TorchScript programs. This means without any extra code you can fully integrate compiled graphs into larger PyTorch programs. The execution of these graphs will feel no different to standard PyTorch

   b. Partial Compilation - Because TorchScript comes with an interpreter and can be executed natively in PyTorch, this opens the possibility to compile subsets of the program to TensorRT while leaving components that are not currently supported in PyTorch. This lets users get some performance gains instead of being all or nothing

2. Ease of Use - We initially developed Torch-TensorRT in the context of automotive and embedded scenarios so we wanted to make common tasks like compilation and portability pretty simple. In contrast to the boilerplate needed to convert and then run TensorRT engines from ONNX, including a standalone inference application, Torch-TensorRT only requires a few lines of code to compile a model and produces a self contained artifact that is portable w.r.t. TRT portability rules. This means we need to only ship a single file which has the model code to our deployment platforms. The inference code for Torch-TensorRT programs is the same code you would have developed for your PyTorch code. There is other QoL components available like a CLI which is especially nice for compiling models for embedded systems that do not have python and a runtime which will automatically select the proper device for your model.

Those are probably the two major reasons but you can find more information here if you are interested: https://www.nvidia.com/en-us/on-demand/session/gtcfall21-a31107/

[dongb5]

@narendasan Hi，is there any comparison of inference efficiency between ONNX-converted TensorRT engine and TensorRT-optimized Torchscript model in C++?

[narendasan]

Moving this to a discussion thread so others can find it later