Face mesh jumpiness and other topics

[Nekronik]

Hello,

I've recently updated the library to the latest version (2.3.5) from an old one (1.5.1) and I noticed a few things to discuss.

1. Starting from 1.9.3 (where you introduced better typing) my project stopped compiling due to some missing types.
   • I used the compilerOptions.skipLibCheck = true to make it not fail for typing.
   • I have read the documentation but I have not found a way to get around this.
   • Is there a way to avoid this skipLibCheck on my project?
2. At least on the latest version (2.3.5), the face mesh is jumping at the same pace the general face detection is performed (blazeface).
   • My input is a video stream and if I stand still without moving my face, the face mesh is changing position as described.
   • The jumpiness frequency is related to the skipFrames.
   • Should it be jumping if the face did not change a single pixel?
   • Is there a way to mitigate it for now apart from increasing the value of skipFrames?
3. Reading the documentation around skipFrames, you mentioned that you set it to X considering the detections are running at a speed of Y, in other words, calculating it based on the desired time interval.
   • The detection rate changed based on the device's capabilities, making it hard to come up with a one number fits all.
   • Additionally to skipFrames, does it make sense to introduce a skipTime?
4. When calling human.draw.face, if I provide the drawLabels: false option, it is still rendering the "Face: 100%, Real NaN%, Roll Yaw and Pitch".
   • Is it ok that I always get a 100% face score?
   • I am not using Real-of-Fake model, is it ok that Real: NaN% is rendering?
   • I would expect that if drawLabels is false, no labels would be drawn, am I correct?
5. I just discovered the cacheSensitivity config property which says "values 0..1 where 0.01 means reset cache if input changed more than 1%", and I find it hard to understand.
   • The caching of "what" is it referring to?
   • What does it mean that "it" changed by a percentage?

I can provide more information or open issues related to the topics I just described if you require.

Thank you very much for your time.

[vladmandic]

re: 1 - i'd like to fix that, can you give me more information?
re: 2 - i'll test it
re: 3 - i agree its difficult to come up with number that fits all, but i need to have some defaults as running all models all the time is too slow
re: 4 - thats a bug, i'll fix it today - can you open an issue for this so i can track it?
re: 5 - it refers to input pixels (image, data, whatever) - if input content has changed more than cacheSensitivity then any caching is disabled for next frame. and it overrides any values for skipFrames for next frame. this is to make sure that there is no caching if input images are different or video scene changed too much.

[Nekronik]

re: 1

If I remove the skipLibCheck rule from the tsconfig file that's what I get.
On my pacakge.json I only have this line "@vladmandic/human": "2.3.5", in relation to the library or TensorFlow.

And those are the two imports I have:

import type { Config, DrawOptions, Error, Result } from '@vladmandic/human';
import Human from '@vladmandic/human';

Could it be a peer dependencies thing?
Let me know if this helps.

[ ERROR ]  TypeScript: ./node_modules/@vladmandic/human/types/dist/tfjs.esm.d.ts:2100:18
           Cannot find namespace 'OptimizerConstructors'.

   L2099:  export namespace train {
   L2100:      import sgd = OptimizerConstructors.sgd;
   L2101:      export { sgd };

[ ERROR ]  TypeScript: ./node_modules/@vladmandic/human/types/src/tfjs/types.d.ts:6:24
           Cannot find module '@tensorflow/tfjs-core/dist/index' or its corresponding type
           declarations.

      L5:   */
      L6:  export { Tensor } from '@tensorflow/tfjs-core/dist/index';
      L7:  /**

[ ERROR ]  TypeScript: ./node_modules/@vladmandic/human/types/src/tfjs/types.d.ts:11:28
           Cannot find module '@tensorflow/tfjs-converter/dist/index' or its corresponding type      
           declarations.

     L10:   */
     L11:  export { GraphModel } from '@tensorflow/tfjs-converter/dist/index';
     L12:  //# sourceMappingURL=types.d.ts.map

[ ERROR ]  TypeScript: ./node_modules/@vladmandic/human/types/src/human.d.ts:164:31
           Cannot find module '@tensorflow/tfjs-core' or its corresponding type declarations.        

    L163:  image(input: Input, getTensor?: boolean): {
    L164:      tensor: Tensor<import("@tensorflow/tfjs-core").Rank> | null;
    L165:      canvas: HTMLCanvasElement | OffscreenCanvas | null;

[Nekronik]

re: 3.

I know, but the skipTime could be another way of throttling it and could work as an OR condition.
When one of the conditions is met the detection is performed again while resetting the condition check logic.
The defaults could still be as they are setup now and leave to the consumer the choice wether to use one or the other.

i.e. To only be time based

detector: {
      skipFrames: Number.MAX_SAFE_INTEGER,
      skipTime: 500,     // milliseconds
}

[vladmandic]

I like the idea of skipTime - can you open an enhancement request to track that?

[vladmandic]

regarding skipLibCheck, this is a bigger topic...

no, its not about peerDependencies or dependencies at all - i wish it was :(

default dist/human.esm.js bundles tfjs for convinience

how it works is that my build process creates dist/tfjs.esm.js depending on the build target,
e.g., on nodejs its just a re-xport of @tensorflow/tfjs-node while on browsers its just a re-export of @tensorflow/tfjs plus backends
(see tfjs/*.ts - those are re-export definitions)

but...

• packging entire typings for tfjs would be massive
• tfjs itself is built using ancient typescript 3.5.3 with veeery relaxes ts rules
• tfjs official build uses silly hacks with symlinks so for example, a tf-backend-webgl imports symlinked tfjs-core package
  thats why you see errors about */dist/index
• human is built using typescript 4.4.4 with very strict rules
  so even if i packaged its typings, id get even more errors in human

tfjs team is working on porting it to typescript 4.4,
but thats going to be a while and even then its questionable how strong of checks theyll enforce

one option is to use dist/human.esm-nobundle.js instead - its the same library,
but marks @tensorflow/* as external dependencies instead of bundling them
(so you have to import tfjs manually before importing human)
and then you can see if there are any lib issues with human itself - which id be happy to fix

what im currently working on is fully custom build of tfjs to be used by human with goals of:

• much cleaner bundle
• up-to-date since tfjs release cadence is major release every 2-3 months with no patch releases in the meantime
  (and i have quite a few fixes included in the tfjs main branch)
  btw, my custom tfjs is under https://github.com/vladmandic/tfjs

if youre interested, i just pushed first one: dist/human.custom.esm.js

[Nekronik]

Thanks for the detailed explanation.

This is what I have done.

import '@tensorflow/tfjs';
import type { Config, DrawOptions, Error, Result } from '@vladmandic/human';

import { Human } from '@vladmandic/human/dist/human.esm-nobundle.js';

removed skipLibCheck and I still get this error.

[ ERROR ]  TypeScript: ./node_modules/@vladmandic/human/types/dist/tfjs.esm.d.ts:2100:18
           Cannot find namespace 'OptimizerConstructors'.

   L2099:  export namespace train {
   L2100:      import sgd = OptimizerConstructors.sgd;
   L2101:      export { sgd };

Does this ring a bell? or am I doing something wrong?

Also, the IDE says that it could not find a declarations file for this import, making the Human have an any type.

[vladmandic]

Does this ring a bell? or am I doing something wrong?

I'll try to recreate, give me a day.

Also, the IDE says that it could not find a declarations file for this import, making the Human have an any type.

Ah, the magic of package.json mappings - as soon as you step out of using default import, it cannot make a connection (typings are the same, but it doesnt even try) :(

[Nekronik]

A quick workaround that I found was to just use the type from the default path (type import) and then rename the Human (class import) from the other path.
But I was not very comfortable with it since I did not know if they shared the same typing nor could guarantee that would be always the case.

import type { Human } from '@vladmandic/human';
import { Human as HumanNoBundle } from '@vladmandic/human/dist/human.esm-nobundle';

[vladmandic]

But I was not very comfortable with it since I did not know if they shared the same typing nor could guarantee that would be always the case.

They are guaranteed the same, you can see they reference same entry points in .build.json (that is the config file for my build process), only compiler target and link settings are different

[vladmandic]

regarding skipLibCheck - ive turned on every strictest tsc rule i could think of and went exploring :)
and fixed pretty much everything that is under my power (new version is on github)

for default import

ive fixed all tsc lib errors except for one where tfjs does things no longer looked upon nicely by tsc:

human/types/dist/tfjs.esm.d.ts:2105:18 - error TS2503: Cannot find namespace 'OptimizerConstructors'.

its a totally irrelevant piece of code (and not at all used by human)
where tfjs exports OptimizerConstructors as a class, but tsc typings insists on referencing a namespace instead
and thats not a problem for my compiler, but tsc creates invalid typings

for named import

issue is like simply that tsc doesnt know where to look for them
so if you're using

import { Human } from '@vladmandic/human/dist/human.custom.esm';

easiest thing is to create human.d.ts with:

declare module '@vladmandic/human/dist/human.custom.esm' {
  export * from '@vladmandic/human/types/src/human';
}

and reference it in tsconfig.json

    "typeRoots": ["src/"] // whereever human.d.ts resides

best thing, with custom build there are no tsc issues reported at all!

ive created enhancement issue #200 to track this

[vladmandic]

i've added generic typings that reference all individual human modules,
if using named imports and tsc cannot find type definitions now you can simply add line to your .ts file:

/// <reference path='./node_modules/@vladmandic/human/dist/human.d.ts' />

[Nekronik]

Could you elaborate a little bit more on how to consume @vladmandic/human/dist/human.custom.esm?
Does it require importing tfjs independently (like for human.no-bundle) or is it automatic?

I have tried it without any tensorflow import and the detections seem to be running but I get this error spammed in the console:

Human: error: cannot set backend: tensorflow Error: Backend name 'tensorflow' not found in registry

Isn't it strange? 😕

[vladmandic]

human.custom.esm is special bundle with custom-built tfjs for browsers (embedded, so no import needed), its not intended for node
node support is via human.node and human.node-gpu

[Eduard-Serra-Fourthline]

Hey, I am @Nekronik I will use this account interchangeably from now on. I will try to stick to this one.

I implemented a smoothed solution like it is explained in wiki/Caching#smoothing.
I had the impression that the next function would try to predict where the face would be, but instead, it only interpolates from the last results, am I correct?

The jumpiness issue is still present even after the smoothing. It mitigates it a little bit but it keeps jumping nonetheless.
I got a couple of videos recorded (a before and after the smoothing), is there another more private channel where I could show them to you?

Thanks for your time.

[vladmandic]

The jumpiness issue is still present even after the smoothing. It mitigates it a little bit but it keeps jumping nonetheless.

interpolation only works if number of results is identical - if number changes, then it assumes a major change in the frame has occurred and used latest results as-is without smoothing. so if face flickers in-and-out, smoothing will not help. but it should not flicker to start with.

I had the impression that the next function would try to predict where the face would be, but instead, it only interpolates from the last results, am I correct?

correct

I got a couple of videos recorded (a before and after the smoothing), is there another more private channel where I could show them to you?

create a free account on any of the cloud storage services or file sharing sites and send me the link once uploaded.

[Eduard-Serra-Fourthline]

To clarify, it is not skipping detections, just detecting it "bigger" than it is. It is easier to see than explain.
I've uploaded it to youtube, hope that works for you.
jumpiness without smoothing
jumpiness with smoothing

[vladmandic]

got it. definitely something wrong.

i just pushed an update on github, issue was that box scale factor for detected boxes vs calculated boxes is never the same and should not be treated as such. so when box re-detection got triggered, boxes were scaled incorrectly for the first frame.

which made me notice that each different module should have its own crop scale factor to optimize per-module - i've done preliminary rescaling for face emotion and description, so that should improve precision in those modules as well.

[Eduard-Serra-Fourthline]

Nice find.
I tried to go through the sources to guess what the root cause was, but I am not too familiar with it and I could have never guessed that. 😓
Will this fix be released in a 2.4.4 or are we looking to 2.5.X?

[vladmandic]

its in 2.5.x only
you can use it directly with npm if you want to try, npm install git+https://github.com/vladmandic/human

[Eduard-Serra-Fourthline]

@vladmandic/human/dist/human.custom.esm is not available directly from github repo. Did you remove human.custom.esm?

[vladmandic]

yes, thats the main difference so far between 2.4 and 2.5, custom has become a default :)