RIFE + TRT_RTX is this as intended?

[Selur]

I tried using RIFE with TRT_RTX:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
from fractions import Fraction
import logging
import site
import sys
import os
core = vs.core
# Limit frame cache to 48473MB
core.max_cache_size = 48473
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# Force logging to std:err
logging.StreamHandler(sys.stderr)
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vstrt_rtx.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vstrt.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vs-mlrt/vsort.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/akarin.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/BestSource/BestSource.dll")
# Import scripts
from importlib.machinery import SourceFileLoader
vsmlrt = SourceFileLoader('vsmlrt', 'F:/Hybrid/64bit/vs-mlrt/vsmlrt.py').load_module()
from vsmlrt import Backend
import validate
# Source: 'G:\TestClips&Co\files\test.avi'
# Current color space: YUV420P8, bit depth: 8, resolution: 640x352, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg, format: MPEG-4 Visual
# Loading G:\TestClips&Co\files\test.avi using BestSource)
clip = core.bs.VideoSource(source="G:/TestClips&Co/files/test.avi", cachepath="J:/tmp/test_bestSource", track=0, hwdevice="opencl")
frame = clip.get_frame(0)
# setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=vs.MATRIX_BT470_BG)
# setting color transfer (vs.TRANSFER_BT601), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=vs.TRANSFER_BT601)
# setting color primaries info (to vs.PRIMARIES_BT470_BG), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=vs.PRIMARIES_BT470_BG)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=vs.RANGE_LIMITED)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=vs.FIELD_PROGRESSIVE) # progressive
clip = core.misc.SCDetect(clip=clip,threshold=0.150)
# adjusting color space from YUV420P8 to RGBH for vsRIFEmlrt
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# adjusting frame count&rate with RIFE (mlrt)
clip = vsmlrt.RIFE(clip, multi=Fraction(2.4), model=44, backend=Backend.TRT_RTX(fp16=True,device_id=0,verbose=True,use_cuda_graph=True,num_streams=1,builder_optimization_level=3,engine_folder="J:/TRT"), ensemble=True)# new fps: 60
# adjusting output color from: RGBH to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_in_s="full", range_s="limited", dither_type="error_diffusion") # additional resize to allow target color sampling
# set output frame rate to 60fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=60, fpsden=1)
# output
clip.set_output()

I first got ModuleNotFoundError: No module named 'onnx', so in installed onnx (pip install onnx).
Then it complained about ModuleNotFoundError: No module named 'modelopt', so I installed modelopt (pip install nvidia-modelopt).
After that, it complained about ImportError: No module named 'onnx_graphsurgeon', so I installed graphsurgeon (pip install onnx-graphsurgeon).
Then ModuleNotFoundError: No module named 'onnxconverter_common' was missing, so I installed that (pip install onnxconverter-common).
=> It worked.

Is this as intended, or should these packages come with the downloads?
Or do they already come with the downloads and I missed them?

Cu Selur

[WolframRhodium]

Sorry, that's why the TRT_RTX backend is experimental.

[zelenooki87]

Tensorrt_rtx with the vs-mlrt plugin in Hybrid worked correctly in fp32 mode. However, for fp16, it reported the errors related to the libraries that @Selur mentioned.

Unfortunately, tensorrt_rtx does not natively support precision flags at the moment, so it's necessary to convert the model to fp16 using onnxconverter_common. NVIDIA support recommends first converting the model with the TensorRT-Model-Optimizer https://github.com/NVIDIA/TensorRT-Model-Optimizer , which also has support for bf16.

In the tensorrt_rtx GitHub issue I opened, they said that a version for Cuda 13.0 will be released in about a week. Hopefully, it will fix many of these issues and add native support for precision flags.