How to improve detections per second

[Nekronik]

I am getting a not too high detection/execution rate in the terms of 8 detections per second on a mobile device with Qualcomm SDM632.

Considerations of my use case:

• I am only executing detection and mesh models
• the face area will be more or less 60% of the video input
• the face will not move too much
• I only care about the result of gestures

I tried lowering the detection resolution using the width and height properties or the filter. But how low can/should I go with it?
Any other setting I can tweak to improve detections per second?

[vladmandic]

• make sure that everything you don't need is disabled:
  body, hand, object, face.iris, face.description, face.emotion
• try using smaller detector model
  `face.detector.modelPath = 'blazeface-front.json'
• since there is only one face in video, you can be very aggressive with caching
  face.detector.skipFrames = 50
• increase score thresholds as that can reduces some math:
  face.detector.scoreThreshold = 0.6
• reduce input resolution:
  you could do it by setting filter.width and filter.height or even faster (by a tiny bit)
  you could completely disable filter and reduce size of input video to start with by setting constraints
  and then initializing video stream using navigator.mediaDevices.getUserMedia(constraints)

not much else comes to mind for the moment

i am planning to provide uint quantized models and supporting usage of @tensorflow/tfjs-tflite once it's stable (it's in alpha now) which is explicitly designed for mobile devices - but i guess it's going to be 2-3 months at the minimum

[Nekronik]

I already disabled everything but what I needed. Good for the record though. 👍
Thanks for pointing out to use blazeface-front.json since for whatever reason I was using blazeface-back.json.
About the skipFrames, are these frames of the source video or frames in terms of detections accomplished? It is not clear in the docs.

[vladmandic]

how it works is that mesh detection relies on a box determined from face detection to start with
but if videoOptimized = true, we cache that box since face doesn't move that much between each frame and just adjust it by little bit after running face mesh on each frame
only when face moves so much that face mesh becomes unreliable, we need to re-run actual face detection

skipFrames is the maximum value of frames before new face detection is forced.

[Nekronik]

skipFrames is the maximum value of frames before new face detection is forced.

where one frame is a "video frame" or a "human.detect execution"?

[vladmandic]

Detection execution

[Nekronik]

Then, if the device is running at 8 detections per second and I skip 50 frames, it would be ~6 seconds before the next box detection happens. I guess this should be set inverse proportional to performance. Could this feature be implemented in a time based approach where you would set the time to wait between box detections?

[vladmandic]

True.

And yes, I'll give that some thought.
I think I'll leave it as number of frames, but force detection based on time as number of elapsed seconds since last.

But in reality, no danger of skipping detection for a longer period of time on stable video input as initial detection is all thars needed - bounding box adjusts depending on results of each mesh detection run.

[vladmandic]

Btw, you can also set skipInitial to true which means detection will be forced whenever there is no detected face. That is a good way to avoid waiting for new detection even if skipFrames is high.