Age & gender models

[nicoeiris11]

Hi Vlad!

I'm testing alternative age and gender models instead of using the default description config with faceres.json model.

While reading the wiki I found that ssrnet supports age.json, gender.json and gender-ssrnet-imdb.json.
When testing the last one using the demo I found an error regarding model input shape:

exception: The shape of dict['input_4'] provided in model.execute(dict) must be [-1,64,64,3], but was [1,64,64,1]

Am I missing something or is it actually an error in the pre-processing of the frame?

This is my current config:

ssrnet: {
      enabled: true,
      modelPathAge: 'age.json',
      modelPathGender: 'gender-ssrnet-imdb.json',
      skipFrames: 99,
      skipTime: 3000,
      minConfidence: 0.1,
    },

[vladmandic]

thats weird, in my code its clearly marked as it requires float grayscale input normalized to -1..1:

human/src/gear/ssrnet-gender.ts

Lines 45 to 53 in c8571ad

	t.enhance = tf.tidy(() => {
	const [red, green, blue] = tf.split(t.resize, 3, 3);
	const redNorm = tf.mul(red, rgb[0]);
	const greenNorm = tf.mul(green, rgb[1]);
	const blueNorm = tf.mul(blue, rgb[2]);
	const grayscale = tf.addN([redNorm, greenNorm, blueNorm]);
	const normalize = tf.mul(tf.sub(grayscale, constants.tf05), 2); // range grayscale:-1..1
	return normalize;
	});

but model signature is definitely RGB:

signature gender-ssrnet-imdb.json
2023-02-15 14:48:35 INFO:  graph model: /home/vlado/dev/human-models/models/gender-ssrnet-imdb.json
2023-02-15 14:48:35 INFO:  created on: 2022-02-09T16:25:35.010Z
2023-02-15 14:48:35 INFO:  metadata
2023-02-15 14:48:35 { generatedBy: '2.3.1', convertedBy: 'TensorFlow.js Converter v2.4.0', version: '440.undefined' }
2023-02-15 14:48:35 INFO:  model inputs based on signature
2023-02-15 14:48:35  { name: 'input_4:0', dtype: 'DT_FLOAT', shape: [ -1, 64, 64, 3 ] }
2023-02-15 14:48:35 INFO:  model inputs based on executor
2023-02-15 14:48:35  { name: 'input_4', dtype: 'float32', shape: [ -1, 64, 64, 3 ] }
2023-02-15 14:48:35 INFO:  model outputs based on signature
2023-02-15 14:48:35  { id: 0, name: 'Identity:0', dytpe: 'DT_FLOAT', shape: [ -1, 1 ] }
2023-02-15 14:48:35 INFO:  model outputs based on executor
2023-02-15 14:48:35  { id: 0, name: 'Identity', dtype: 'DT_FLOAT', shape: undefined }
2023-02-15 14:48:35 INFO:  tensors: 157
2023-02-15 14:48:35 DATA:  topology: { nodes: 294 }
2023-02-15 14:48:35 DATA:  weights: {
  files: [ 'gender-ssrnet-imdb.bin' ],
  size: { disk: 161236, memory: 161236 },
  count: { total: 157, int32: 72, float32: 85 },
  quantized: { none: 157 },
  values: { total: 40309, int32: 141, float32: 40168 }
}

i added that code for a reason, but shoot me if i recall why.
i'll need to double check.

created issue #339 to track

[vladmandic]

just closing the loop - issue is resolved.

[nicoeiris11]

Hi Vlad!

Is it possible to include crop configuration to age and gender preproc? (face.description, face.gear, and face.ssrnet)
Accuracy improved a lot for emotions detection using this setting, I'd really like to test it also for age and gender.

Thank you!

[vladmandic]

sure.
in addtional of previously added face.emotion.crop, i've added optional config params:

config.face.description.crop
config.face.gear.crop
config.face.ssrnet.crop

note that this has minimal testing (e.g. only automated tests, i don't have time right now for extensive manual testing).

when you do have a chance to try it out and find out what are the best values, please share and i'd like to set them as defaults.
(seems like you've already found a good value for emotion.crop?)

[nicoeiris11]

Thank you so much Vlad!

Of course, I'll let you know which was the optimal crop value after the testing phase.
For emotion.crop in my specific case, the best value was emotion.crop = 0.1, but this could definitely vary for other users.

[nicoeiris11]

Hi Vlad!

Is there any specific reason why face result does not contain ageScore but only genderScore?

[vladmandic]

simple - can't return what model does not have :)
more specifically, there are 3 supported models that report age: faceres, gear (with sub-variants) and ssrnet.
faceres and ssrnet return a single float value while gear returns score for each possible decade (so 0% chance that person is 0-10y old, 10% chance he is 10-20 years old, etc.), so even actual reported age value is calculated by human.
looking at faceres model architecture, it simply doesn't have activation layer, it simply does a softmax at the end and return whatever value has highest chance without actually calculating chance.
and ssrnet actually runs age calculation (somehow using tanh function, that model is really strange, but somehow elegant), not value/score as you'd normally expect.

[nicoeiris11]

Did you have the chance to assess which model has the best accuracy? From my experiments ssrnet and gear has a very bad accuracy while faceres in some cases has a decent accuracy.

Given we don't have ageScore, I was thinking about doing a kind of average between all the age predictions for the same person (at the moment of tracking people). In other words, for person X, average all age predictions in time and assign that average as the age for analytics purposes. Mi initial idea was to use ageScore to do a ponderated average given model confidence at the moment of assigning an age to a person.

Do you think there might be a better approach to have a reliable age prediction?

[nicoeiris11]

Given the accuracy of supported age/gender models are not too good, may I search for other alternatives (lightweight) models that could be converted to tfjs for your review and possible integration into human?

[vladmandic]

absolutely.
and yes, i found faceres to be the best of the existing ones, thats why its default.

[nicoeiris11]

Here are some preliminary findings for age and gender models.
Please let me know your thoughts.

• https://github.com/yu4u/age-gender-estimation/blob/master/demo.py (here is a webgl demo)
• https://github.com/dpressel/rude-carnie#pre-trained-checkpoints
• https://github.com/justadudewhohacks/face-api.js/tree/master/weights (see age_gender_model, seems to be very promising)

I'll continue looking for more alternatives, but I'd like to have your feedback about these ones while I continue my research.

Thank you so much Vlad!

[vladmandic]

interesting that you pointed out faceapi as that repo was abandoned and now i maintain active version: https://github.com/vladmandic/face-api, so yes, i'd say i'm quite familiar with it. but unfortunately, those weights cannot be used outside faceapi (long story).

yu4u is anything but lightweigth - its 126mb just for age/gender.
and rude-carnie (btw, that model is considered a standard from which a lot of other models started) is even bigger - 166mb just for age and additional 166mb for gender.

issue with both is they are trained using very heavy backbones (efficientnet for yu4u and inception for rude-carnie). maybe they would be usable if they are modified to use a more suitable backbone for edge such as mobilenet, but that requires far more work than just conversion (plus entire retraining).

[nicoeiris11]

Totally understand, I thought yu4u and rude-carnie models had the chance to be quantized, etc. to be lightweight, my apologies for referencing those. I'll continue looking for other alternatives.

Meanwhile, could you please give me your feedback regarding the following:

Given we don't have ageScore, I was thinking about doing a kind of average between all the age predictions for the same person (at the moment of tracking people). In other words, for person X, average all age predictions in time and assign that average as the age for analytics purposes. My initial idea was to use ageScore to do a ponderated average given model confidence at the moment of assigning an age to a person.

Do you think there might be a better approach to have a reliable age prediction?

[vladmandic]

averaging age over time is a good approach and you can do ponderated average using face.faceScore, but do a high normalization on it (so for example, 0.8 for face score which is ok-ish means only 0.1 for age) - idea is that if face landmark detection has high confidence, chances are age detection will also be more accurate given they work on the same snapshot of a face.

[vladmandic]

regarding quantization, its easy to go from fp32 to fp16 as that is just a small loss of resolution. but going to int8 would require a complete re-normalization. so even if you divide current sizes by 2, they are still massive.
plus i'm familiar with efficientnet and inception backbones and while they are great, they are not suitable for real-time.

[nicoeiris11]

(so for example, 0.8 for face score which is ok-ish means only 0.1 for age)

Do you mind elaborating a bit more on this? I don't get how an 80% of face confidence maps to 10% of age confidence.
Actually what you mention is a great idea, I could average ages ponderating face score, so if face detection is not confident, age detection should not be too much ponderated in the average.

[vladmandic]

idea is when doing pondering to assign more spread out age score than original face score. so something like ageScore = Math.max(0, 2 * face.faceScore - 1). (so 1.0 is 1.0, 0.8 becomes 0.6 and 0.5 or lower becomes 0)

[nicoeiris11]

Thank you so much for the explanation, now I totally understand what you mean. Great idea!!

[nicoeiris11]

What do you think about this possible implementation?

const ageConfidence = (faceScores: number[]): number[] => {
  return faceScores.map((faceScore: number) => Math.max(0, 2 * faceScore - 1));
};

const ageWeightedAverage = (ages: number[], ageConfidences: number[]): number => {
  const [sum, weightSum] = ageConfidences.reduce(
    (acc, w, i) => {
      acc[0] = acc[0] + ages[i] * w;
      acc[1] = acc[1] + w;
      return acc;
    },
    [0, 0]
  );
  return sum / weightSum;
};

[vladmandic]

looks ok to me. reduces can be much shorter, but this is easily readable.