Converter Versioning and Capability Declaration

[narendasan]

TL;DR

There are many reasons to have multiple versions of a converter, as well as many reasons why a converter would not work for a specific use case. This feature adds the ability to inform the compiler of this information so that it may select the right converter or inform earlier phases that no valid converter exists.

Goal

With ongoing work such as #1128, #1009 etc. dealing with potentially supporting multiple versions of TensorRT, certain converters that have specific limitations and more it might be worth looking into including metadata about converters so that instead of running and failing converters can report incompatibilities prior to compilation.

Usecases

1. Versioned Converters
   • There are reasons to have more that one version of a converter. For instance if there is a breaking API in PyTorch or TensorRT change, this enables backwards compatibility (Torch-TensorRT Versioning & Stack Support #1009) or just more incremental improvement
2. Incompatibility detection
   • Converters currently have a number of checks for various failure cases which get caught at conversion time. At this point it is too late to recover and fallback to PyTorch. Therefore by declaring these limitations beforehand, during partitioning, it can be taken into account.
3. Specialization
   • Each converter needs to handle all of its special cases, data type limitations for instance. This means some converters become very lengthy. Breaking these converters up can help code clarity.

Proposed API:

Inspired by how evaluators have extra information associated with the definition such as valid schemas, unsupported output types, etc. (

TensorRT/core/conversion/evaluators/evaluators.h

Line 38 in 484fc90

struct EvalOptions {

) the proposal would be to have a ConverterCapabilities struct which would include the following information:

• Supported TensorRT versions
• Known failure cases:
  • e.g. Number of supported free dimensions for dynamic shape
• Supported DataTypes

There may be a common set of settings most converters would support, i.e. all TensorRT versions, no failure cases, all data types, so converters should default to these settings unless the developer explicitly sets more restrictions.

Implications on the rest of the Codebase

Conversion Phase

There should now be an additional step in the conversion phase to cross reference the user provided and environment settings before the actual converter is called

Converter Library

Guard Registration

The converter library could prevent (with warning/debug info) that a converter is not registered because it is incompatible with the environment

Converter Overloading

One of the goals here is to avoid having a bunch of #ifs around the codebase guarding features. As such it would be nice to be able to associate multiple converter variants to the same function schema as long as their capabilities do not collide.
(as mentioned in #306) This might mean that we might want to as well supporting more than one schema with a single converter implementation instead of doing a pure 1:1 association.

Implementation Phases

Prototype - L

• Implement the capabilities/versioning class, extend the registry to manage them (Implement the converter capabilities/versioning class, extend the converter registry to manage them #1428)
• Basic resolution of constraints, e.g. only register converters known working with the target TRT version
• Expose declarations via OpIsSupported

MVP 1.4.0 - M

• Use this information to modify supported operations for partitioning

Converter overloading Y23Q1+ - M-L

• Allow for multiple converters to coexist as long as constraints are disjoint

[peri044]

Can you please give examples of ops which doesn't conform to default ConverterCapabilities ?

[narendasan]

One case @andi4191 found was some layers in TensorRT only support 1 free dimension. Another would be for plugin based converters which have type restrictions (in the case a user enforces strict types)

[narendasan]

aten::div is a converter which fits here as well, it uses unary layers internally which do not accept Int32. (#660)

[narendasan]

Sister topic, support doing a dry run conversion to see if a layer can be converted without error

[narendasan]

#1426, #1439