Embedding

Face Feature Embedding and Simmilarity Compare

Usage

To use face simmilaity compare feature, you must first enable face.embedding module
and calculate embedding vectors for both first and second image you want to compare

To achieve quality results, it is also highly recommended to have face.mesh and face.detection.rotation
enabled as calculating feature vectors on non-quality inputs can lead to false results

For example,

const myConfig = {
  face: {
    enabled: true,
    detector: { rotation: true, return: true },
    mesh: { enabled: true },
    embedding: { enabled: true },
  },
};

const human = new Human(myConfig);

const firstResult = await human.detect(firstImage);
const secondResult = await human.detect(secondImage);

const simmilarity = human.simmilarity(firstResult.face[0].embedding, secondResult.face[0].embedding);

console.log(`faces are ${100 * simmilarity}% simmilar`);

If the image or video frame have multiple faces and you want to match all of them, simply loop through all results.face

for (let i = 0; i < secondResult.face.length; i++) {
  const secondEmbedding = secondResult.face[i].embedding;
  const simmilarity = human.simmilarity(firstEmbedding, secondEmbedding);
  console.log(`face ${i} is ${100 * simmilarity}% simmilar`);
}

Additional helper function is human.enhance(face) which returns an enhanced tensor
of a face image that can be further visualized with

  const enhanced = human.enhance(face);
  const canvas = document.getElementById('orig');
  human.tf.browser.toPixels(enhanced.squeeze(), canvas);

Embedding Vectors

Embedding vectors are calulated feature vector values uniquely identifying a given face and presented as array of 256 float values

They can be stored as normal arrays and reused as needed

Simmilarity function is based on Eucilidean distance between all points in vector
Eucliean distance is limited case of Minkowski distance with order of 2
Minkowski distance is a nth root of sum of nth powers of distances between each point (each value in 192-member array)

Changing order can make simmilarity matching more or less sensitive (default order is 2nd order)
For example, those will produce slighly different results:

  const simmilarity2ndOrder = human.simmilarity(firstEmbedding, secondEmbedding, 2);
  const simmilarity3rdOrder = human.simmilarity(firstEmbedding, secondEmbedding, 2);

How simmilarity is calculated:

  const distance = ((firstEmbedding.map((val, i) => (val - secondEmbedding[i])).reduce((dist, diff) => dist + (diff ** order), 0) ** (1 / order)));

Once embedding values are calculated and stored, if you want to use stored embedding values without requiring Human library you can use above formula to calculate simmilarity on the fly.

Face Image Pre-processing

To achieve optimal result, Human performs following operations on an image before calulcating feature vector (embedding):

• Crop to face
• Find rought face angle and straighten face
• Detect mesh
• Find precise face angle and again straighten face
• Crop again with more narrow margins
• Convert image to grayscale to avoid impact of different colorizations
• Normalize brightness to common range for all images

Demo

Human contains a demo that enumerates number of images,
extracts all faces from them, processed them and then allows
for a selection of any face which sorts faces by simmilarity

Demo is available in demo/embedding.html which uses demo/embedding.js as JavaSript module