throughput fps vs inference fps

[shrinand1996]

When i run the benchmarking script, there are 2 throughput fps. One is shown as throughput(batch size =32)=2.06 fps and then there is one when bechmarking is complete (around 9 fps).

My question is which one is correct?
When i run inference, it takes around .5 s per image regardless of the model i choose.

How can i make the inference speed reach the speed that the benchmarking script returned?

I want to run it image by image (not batches) and get the desired speed.

[callmesora]

I have a similar question. Trying to infer directly on a webcam so any insights here would be helpful. Thank you for the amazing library btw

[ashwinvaidya17]

The throughput (batch size) ... is computed from the TimerCallback. It is calculated based on batch inference called by trainer.test. The one computed when benchmarking is complete uses the MockDataloader and calling predict for each image. Hope this is clear.

As for the difference in throughput from benchmarking, I am still trying to debug the issue. One major factor is that benchmarking uses MockDataloader which creates a dataset of random tensors with the same length as testing set. Performance numbers based on this are misleading as preprocessing time varies for each real test image. So the benchmarking script is showing a more optimistic value. Second, for some reason when the model is trained first (as in benchmarking) before creating TorchInferncer object, the prediction times are lower thus also leading to higher throughput.

[samet-akcay]

it is also possible to write a custom inferencer as mention in #1183.