Face contours

[shahkashani]

Hi friends!

Just wondering if there's a good way to essentially create an outline of the detected face?

I've tried using the silhouette annotation, but in many cases, it doesn't seem to cover the entire face. I've also tried using some of the contour points that I've found in the code, but as you can see, it suffers from the same problem as the silhouette:

(The numbers are mesh indexes, which should correspond to the contour, but visually don't seem to.)

Interestingly, the mesh does have the information to cover the entire face (at least more of the cheek), but I can't seem to find the right indexes that work consistently for all rotations:

Essentially I am trying to do something like this, but I don't want the open mouth to appear in the black overlay, so I figured I would draw a polygon around the entire detected face (hence the need for a silhouette / contour that covers the entire face):

Here is the raw image:

[vladmandic]

contours you saw in the code are disabled as i don't see a reason to enable them as they are 2d-only and better results can be found in already present results data.

best is to consider sets of connected points as triangles and fill them. so for each face mesh point, there are 2 other points that should be considered to form a triangle - that then covers entire face precisely and leaves areas such as open mouth as unmarked by default as its not part of any triangle.

and that is exactly how its already implemented - simply set human.draw.options.fillPolygons = true; and you'll see how draw method behaves (see example screenshot).

if you want to see how to map the triangles, see src/draw/face.ts:drawFacePolygons() (and note that triangulation point map is exported as human.faceTriangulation)

now, it can also be done using silhouette in results, but then you'd have to deal with unmasking open mouth, etc.
if you want to go down that path, take a look at src/face/mask.ts which extracts only face silouetthe from entire input.
(your use case is basically inverse of that, you need everything but silouetthe)

and one more approach (best if you're dealing with some 3d framework) is to use uv map projections.
but i don't recommend that unless you're already familiar with such concepts (using three.js or babylonjs) - but its absolutely best if what you're trying to do is project something else instead of face (e.g. perform a face-swap or something like that)
for those examples, you can take a look at https://github.com/vladmandic/human-motion, specifically how its done in src/mesh.ts:drawFace()

[shahkashani]

Thank you so much for your reply! This is incredibly useful.

So, based on what I'm reading, silhouette is actually perfect for me because I actually want the mouth to be filled. The ideal result would be this:

However, the issue is that silhouette doesn't seem to cover the entire face in some cases, for instance with the above example image. I get the following:

const f = result.face[0];
const opt = {
  fillPolygons: true,
  useDepth: false,
  color: 'black'
};
lines(ctx, f.annotations.silhouette, opt);

If I use the triangulation point map, the entire surface is covered, but the mouth is not filled:

const f = result.face[0];
const triangulation = human.faceTriangulation;
const opt = {
  fillPolygons: true,
  useDepth: false,
  color: 'black' 
};
for (let i = 0; i < triangulation.length / 3; i++) {
  const points = [
    triangulation[i * 3 + 0],
    triangulation[i * 3 + 1],     
    triangulation[i * 3 + 2]
  ].map((index) => f.mesh[index]);
  lines(ctx, points, opt);
}

I guess I could combine the two, but I feel like I'm missing something.

[vladmandic]

issue with silhouette is that edges are really close to ears so plane it defines goes through head (z-axis), not in front of face - so when you turn your head, cheekbones or nose can project more to the left/right than position of the edge of the face.

i'd stick with triangulation method and augment it with either:

• extend triangulation to define triangles that cover open mouth
  (you can see the points at https://github.com/vladmandic/human/blob/main/assets/facemesh.png)
• augment it with plane that covers mouth only
  use lipsUpperOuter and lipsLowerOuter landmarks instead of silhouette

[shahkashani]

Got it! Thank you kindly for the explanation, I'm glad I'm not missing anything obvious. Both of those strategies are straight-forward enough, so I'll do some experimentation and see 😄