InputOperator, OutputOperator

[hhd-shuai]

Hi, I am working on deploying a model on the RK3588 and performing inference. I found that the InputOperator and OutputOperator run on the CPU, consuming a lot of resources and time. Could you tell me what processing is done inside these operators? Are they performing tensor dimension transformations? Is there a way to execute these operators on the NPU?


If so, would it be possible to provide some guidance.

[yuyun2000]

输入输出是把数据在cpu和npu之间切换的，不过你这个竟然要6秒，有点夸张了，实测也是这么慢吗？用的什么版本？

[hhd-shuai]

感谢解答！版本是
RKNN Model Information: version: 6, toolkit version: 1.5.2+b642f30c(compiler version: 1.5.2 (c6b7b351a@2023-08-23T07:30:34))
RKNN Driver Information: version: 0.9.2
速度还好，它这个是微秒，主要是cpu占用较高。我的模型head数量多，叠加起来的数据搬运cpu占用就比较高了。另外请教一下，如果是NPU不支持的算子，内部是否会将数据拷贝到CPU执行完了再拷贝回NPU呢？rknn_infer接口里面的CPU占用，主要来源于输入输出把数据在cpu和npu之间切换，还是这种不支持算子带来的

[yuyun2000]

是不支持的算子，导致数据在npu和cpu之间来回切换从而使推理时长变多

[hhd-shuai]

好的，感谢！