Using for a gallery app like Google Photos

[neckaros]

Hi, I'm using this package in my application "à la" google photos to identify friends and familly in photos.

What are the recommendations/best models/best settings to use for this non real time use?

Right now my settings are:

const myConfig: Partial<Config> = {
  cacheSensitivity: 0,
  filter: {
    equalization: true
  },
  face: {
    enabled: true,
    detector: { rotation: true, 
      maxDetected: 30, },
    mesh: { enabled: true },
    description: { enabled: true },
  },
};

I detect person in photos like this:

  const queryImage = tf.node.decodeJpeg(query);
  const resultsQuery = await human.detect(queryImage)

i load existing descriptors from database

labeledDescriptors: {personId: string, descriptor: number[]}[}

fore each faces from step 1 i get the best match from those existing descriptors:

const match = await matchFace(labeledDescriptors, face.embedding)

const best = human.match(face, embeddingArray); 

if best.similarity is above 0.5 i auto tag. if not the user has it as suggestion and if he validate i had the 'embedding' result i had from step 1 in database

  const resultsQuery = await human.detect(queryImage)

Not sur if i should use enhance manually before saving to DB?

[vladmandic]

your configuration sounds reasonable.

note that config.face.detector.rotation is auto-disabled for tfjs-node as required op is not supported in underlying tensorflow library.
regarding enhance, typically there is no harm by enabling it as it performs histogram equalization, but best if you try yourself.

btw, regarding app itself, check out https://github.com/vladmandic/pigallery

[neckaros]

Thanks, yes I had a look at your galley. That's where I took the configuration.
I didn't see you use enhance though in your app. I will try to add it on detect faces before trying the match

[neckaros]

Do you think not having the second rotate in node is very impactful ? Do you think it will ever be implemented in TSNode?

[vladmandic]

I didn't see you use enhance though in your app. I will try to add it on detect faces before trying the match

i haven't done updates to gallery app in a long time and in the meantime there is a lot of new stuff in human :)

Do you think not having the second rotate in node is very impactful?

its very impactful if head angle is more than 30 degrees, otherwise very little.

Do you think it will ever be implemented in TSNode?

i've opened a feature request a while ago, its not getting much traction.
op does exist as software-only and webgl implementation, so i'll experiment with tfjs-node -> cpu conditional forwarding.

[neckaros]

Ohh that's why I noticed some weird match when I have a head titled more than 30deg in my db or in the image analyzed!

That would be great! Thanks!

Ps: do you have a ref to your issue on Ts repo?

[vladmandic]

sure - its tensorflow/tfjs#5473

[vladmandic]

new version of human is on github (not yet released on npmjs) and there is extra config flag: softwareKernels
if enabled, it will allow human to use cpu based implementations for kernel ops that are not implemented in a given backend

that allows face rotation correction to work in nodejs

its disabled by default as it does have a performance impact

[neckaros]

WOW thanks so much! I will try this fallback asap!