confidence threshold in preprocessing stage

[amotz1]

first of all i used openface for awhile and i see how much effort were put in on it so well done for all that work and thank you very much!

in our experiment there are times that i saw in the visualisations of the videos, especially sometimes if subjects move the head etc... that openface think it detects a face because it draws a cube, but there is no face inside the cube. i thought that maybe openface gives those kind of frames a low confidence value and i may be able to use some kind of confidence values threshold in my pre-processing stage to filter those bad frames out of the analysis. something like if the confidence is less then "x" i am not using this frame in my analysis.

i then thought what this "x" may be, and i am not sure so i thought to ask here. what kind of threshold would be advisable to use if at all? is there a confidence threshold that above it i can be quite confident that openface detected the lendmarks and recognized a face correctly?

thank you very much again
amotz

[brmarkus]

There is no good answer. If you (very) try different videos in (very) different environment of (very) different persons/heads (orientation, hairs, beard, pair of glasses, emotions; differenent lightning, number of persons/heads, distance between camera and person/head, used camera lense, etc etc) you woull notice different thresholds would be required.

Usually you would need additional processing, like see it as a consistency check, plausibility check - like does the returned bounding-box/cube make sense, could landmarks be detected, would the landmarks (number, spreading) make sense, do multiple consecutive frames show the same/still similiar-enough bounding box/cube?

[amotz1]

thank you for answering! as for the visualisation, i do see it as a consistency check and it helps and indeed most of the times openface seems to detect the faces correctly and landmarks looks good according to the visualisation which is great.

as for the confidence column, if i understand you correctly you think that if i try to choose a confidence threshold to my analysis under which i would not include frames to my analysis it would change according to many factors that changes during the video and i can never be sure what this threshold should be.

would you advise against using such a confidence threshold for pre-processing then?

[brmarkus]

Would your use-case mean that the persons/faces in your processed video(s)/stream(s) be under the same or very similar environment (they don't move freely, they don't overlap with each other or with other obstacles, lightning would be the same, the distance to the person/faces would stay the same, the persons were asked to look straight into the camera, remove pair of glasses, look neutral, etc)?
Then using a threshold could be a start. Try it.
However, I think, the persons/faces and conditions vary, we are just unique, in varying conditions all the time...
You might find an adaptive threshold.