Support dynamic fallback

[Njuapp]

Previously, when encountered ops/modules that have to fall back and meanwhile dynamic shape is used, shape analysis would fail when doing generateRandomInputs(), because tensors with -1 contained in shape cannot be created.

To support dynamic fallback, we have to firstly overcome this failure, by doing generateRandomInputs() with min/max/opt shape info. Then shape analysis can be executed three times, with min/max/opt shapes correspondingly.

Such a solution can cover most of user cases, only if the input shapes of subgraphs are determined by the input shape of whole graph. However, in some cases such as Object Detection where input shapes of subgraphs may be data-dependent and variable even if input shapes of whole graph is fixed, such a solution would fail.

For failure cases, we need to 1. detect such cases and make error messages easy to understand. 2. If undetectable, at least some warning messages should be given when dynamic fallback is encountered.

[Njuapp]

However, the above case of object detection would also cause error even if it is static shape analysis. As long as fallback is needed and some input shapes of certain subgraphs are data-dependent, shape analysis (whether it is static or dynamic) is intractable.
Should we focus more on cases where static shape analysis could succeed but dynamic ones would not? @narendasan @bowang007

[narendasan]

We should probably start with closing the gap between static analysis and dynamic analysis

[ncomly-nvidia]

Data-dependent shapes is a different issue and really falls on TRT itself. They are adding in the future (can confirm internally). We should assume data-independence to start

[ncomly-nvidia]

@dheerajperi please modify the original content with a precise description of the current approach (min-opt-max) as well as the known limitations of this stage.

Agreed updated 7/14.

[ncomly-nvidia]

@dheerajperi can you update by 7/21?

[narendasan]

IMO we should start with dynamic batch, this seems like a high impact good start

[ncomly-nvidia]

Agreed. dynamic batch --> dynamic input shape --> data dependent shape seems like the logical flow.

[ncomly-nvidia]

New RFC here