Unable to find Calculator "mediapipe.tasks.vision.face_landmarker.FaceLandmarkerGraph"; Initialize was not ok

[SpookyCorgi]

Hi, I was trying to test out the newly committed face landmarker web api.

I build the package by running bazel build vision_pkg and then try to use it using the follow code.

   const vision = await FilesetResolver.forVisionTasks(
        "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
    );
    const faceLandmarker = await FaceLandmarker.createFromModelPath(vision,
        "https://storage.googleapis.com/mediapipe-tasks/face_landmarker/face_landmarker.task"
    );

However the browser is giving me the follow error.

Is there anything wrong within my process or is the task file still need to be updated?
Thank you.

[GeorgeS2019]

@kuaashish
@schmidt-sebastian

Instead of labeling this as type:others => it would be great for the community to be type:blendshape <= soon to be hot topic for mediapipe so the community can easily track the effort done by others

Create a "mediapipe blendshape" label

[baronha]

Hi, I was trying to test out the newly committed face landmarker web api.

I build the package by running bazel build vision_pkg and then try to use it using the follow code.

   const vision = await FilesetResolver.forVisionTasks(
        "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
    );
    const faceLandmarker = await FaceLandmarker.createFromModelPath(vision,
        "https://storage.googleapis.com/mediapipe-tasks/face_landmarker/face_landmarker.task"
    );

However the browser is giving me the follow error.

Is there anything wrong within my process or is the task file still need to be updated? Thank you.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);

const faceLandmarker = await FaceLandmarker.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/face_landmarker/face_landmarker.task"
);

Where have you seen the documentation for this part? I looked on @mediapipe/tasks-vision but didn't see any mention of it at all

[schmidt-sebastian]

FaceLandmarker is not yet released via NPM. While you can build from source, you will need to wait for the new Wasm binaries to land on NPM. I am working on a release today, so this should be resolved soon.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[baronha]

FaceLandmarker is not yet released via NPM. While you can build from source, you will need to wait for the new Wasm binaries to land on NPM. I am working on a release today, so this should be resolved soon.

@schmidt-sebastian
Can you tell me the specific time for the upcoming NPM release ?

[schmidt-sebastian]

Published: https://www.npmjs.com/package/@mediapipe/tasks-vision

[baronha]

Published: https://www.npmjs.com/package/@mediapipe/tasks-vision

Thanks, @schmidt-sebastian. You are my destiny !!!

[FishWoWater]

@schmidt-sebastian Hi! I have just tried the npm release with the script

   const vision = await FilesetResolver.forVisionTasks(
        "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
    );
    let options: FaceLandmarkerOptions = {
      baseOptions: {
        modelAssetPath:  "https://storage.googleapis.com/mediapipe-tasks/face_landmarker/face_landmarker.task"
      },
      outputFaceBlendshapes: true
    };

    const faceLandmarker = await FaceLandmarker.createFromOptions(
       vision, options
    );
    
    const image = document.getElementById("image") as HTMLImageElement;
    const landmarks = faceLandmarker.detect(image);

But the browser says [Error] Unhandled Promise Rejection: Error: StartGraph failed: $BLENDSHAPES Tag and blendshapes model must be both set. Get BLENDSHAPES is set: true, blendshapes model is set: false; Initialize was not ok
What's wrong and what should I do?

Thank you very much.

[baronha]

You can download and try in your project.
blendshapes.task.zip

Here are the results I got:

Untitled.mp4

[FishWoWater]

@baronha Thank you for your help, but it has some errors too :(
The one thing I have tried that worked for me is to compress tflite models in published task file and the blendshape tflite model in this issue. Quite strange, right?

Could you please share you code? I am a newbie in typescript :(

[baronha]

@FishWoWater

My code to initialize blendshap:

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
  )

const response = await FaceLandmarker.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath: "asset/blendshapes.task", // your blendshape model
  },
  numFaces: 1,
  outputFaceBlendshapes: true,
  runningMode: "VIDEO",
  outputFacialTransformationMatrixes: true,
});

[FishWoWater]

@baronha still not working for me ; the initialization was ok but error occurred during inference, I guess I have problems with the calling procedure. Appreciate it a lot if you could give me some advice

async init(){
      const vision = await FilesetResolver.forVisionTasks(
          "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
      );
      let options: FaceLandmarkerOptions = {
        baseOptions: {
          modelAssetPath: "./blendshapes.task"
        },
        numFaces: 1,
        outputFaceBlendshapes: true,
        runningMode: "VIDEO",
        outputFacialTransformationMatrixes: true,

      };

      this.faceModel = await FaceLandmarker.createFromOptions(
        vision, options
      );
      
  
    }, 
createCameraElement() {
          this.isLoading = true;
          
          const constraints = (window.constraints = {
                      audio: false,
                      video: true
                  });
          navigator.mediaDevices
              .getUserMedia(constraints)
              .then(stream => {
                  this.isLoading = false;
                  
                  this.myVideo = this.$refs.camera as HTMLVideoElement;
                  this.myCanvas = this.$refs.canvas as HTMLCanvasElement;
                  this.myVideo.srcObject = stream;

                  window.requestAnimationFrame(this.faceMocap);
              });
          },

    faceMocap(){
      {
        const delta = this.clock.getDelta();

        if(this.isCameraOpen && this.myCanvas && this.faceModel){
          const context = this.myCanvas.getContext('2d');
          context?.drawImage(this.myVideo, 0, 0, 200, 150);

          const landmarks = this.faceModel.detectForVideo(this.myVideo, this.videoTimestamp);
          if(landmarks.faceBlendshapes && landmarks.faceBlendshapes[0]){
            let cnt = 0; 
            const target: Category[] = landmarks.faceBlendshapes[0].categories;
            for(const res of target){
              if(typeof this.head.morphTargetDictionary !== 'undefined' && typeof this.head.morphTargetInfluences != 'undefined'){
                const prefix = "";
                // const prefix = "asFaceBS.";
                let index = this.head.morphTargetDictionary[prefix + res.categoryName];
                if(index){
                  this.head.morphTargetInfluences[index] = res.score;
                  cnt += 1;
                }
              }
            
            }
          }
          this.videoTimestamp += Math.round(delta) > 0 ? Math.round(delta) : 1;
          window.requestAnimationFrame(this.faceMocap);
        }
      }
    }, 

[FishWoWater]

It WORKS using models from this link, taken from this issue

[ArghyaDasIntel]

@FishWoWater , the latest npm package looks broken. Use the version 0.1.0-alpha-11 instead of "latest" and it should work well. All the best.

[FishWoWater]

@ArghyaDasIntel Thank you~ Yes, I have tried the 0.1.0-alpha-11 and it works with the model provided by baronda