-
-
Notifications
You must be signed in to change notification settings - Fork 471
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
face blendhapes support #935
base: master
Are you sure you want to change the base?
Conversation
3de96ca
to
2c1fd5e
Compare
@newnon Hello, I tried this submission code in Unity and Unity reported the following error. What is the reason? @homuler Can you integrate this contributed code for Blendshape functionality? Thank you so much!
|
I'm planning to port the FaceLandmarker Task, but since this PR is implemented in a different way, I cannot merge it as it is. |
You need to recompile mediapipe binary to use this. |
@newnon Sorry, I don't quite understand what you mean, can you explain in detail? Thank you so much. |
https://github.com/homuler/MediaPipeUnityPlugin/wiki/Installation-Guide#prerequisites |
Thanks @newnon! Its working perfectly after rebuild the package myself |
@rudrjain , I hope it helps:
|
Hi @newnon, I've checked your PR and it works perfectly. I'm quite new to MediaPipe, but the FaceLandmarker API also supports facial transformation matrixes (see: https://github.com/google/mediapipe/blob/085840388bbebd322889026fae235b3120a4bce7/mediapipe/tasks/web/vision/face_landmarker/face_landmarker_result.d.ts#L35). Is it possible to add this feature to your solution? I can try to implement it by myself but if you have any hints how to make that, then it would be easier for me to start. Thanks in advance! |
you can implement it by yourself
headPos - head position |
@homuler Any updates on when you'll get the the FaceLandmarker Task implemented or this PR merged? Having the landmarks/blendshapes is going to be soooo amazing to have in this plugin! |
+infinity! |
+1 |
@newnon in your PR you forgot to also include |
Interestingly I got this running on Android, and the blendshape for Update: Yeah I just checked, there's nothing especially unique about the implementation, and it's using the default mediapipe tflite model. newnon/mediapipe@997ef9c Sloth sure dun look too happy on Android. Update again: the values are apparently getting returned, they're just super small. So I multiplied jawOpen by 10,000 and and clamped it from 0-100 and it seems to move it. It's likely the same for cheekPuff, etc. Still don't know why its different on Mac and Android, or if the result will be different on other devices, also it runs quite slowly and rarely in sync, usually a couple seconds delayed on my Samsun Galaxy S20. Update yet again: I added a calibration step to my android app, and other then the extreme lag of a couple seconds behind which is another problem to solve, as long you calibrate where you set you min and max value and use a remap function to make that 0 to 100, you can fix the outliers. Perhaps just coming up with a gallery of a few "regularization" expressions the user needs to make, in order to calibrate, would be an intuitive way to handle this for the time being, instead of having them set all 52 shapes, or identifying and fix each problematic blend shapes. Actually thinking about it... since this is image based theres no reason I couldnt just find a way to the funnel 52 blendshape face images from somewhere like here https://arkit-face-blendshapes.com/ and use this to calibrate on device before it ever gets to the user. |
Any idea how to reduce the lag?? |
thanks @panpawel88 it helped me in my project |
Looks like homulers integrating things: https://github.com/homuler/MediaPipeUnityPlugin/tree/feature/tasks/face-landmarker !! YAY! IT'S ON IT WAAY HEY HEEEEY!! :-) So excited! So my guess is this PR will go poof soon, in lieu of a fully integrated face landmarker task with blendshapes! |
How can I call this function ? Please let me know Please !! Thank you . |
How can I use this ? Would you explain about this code for using for example ? Thanks |
@jefflim69 you should check out the latest on master. Blendshapes are in From
That gives you the head rotation and blendshapes. For eye rotation you can approximate it from the blendshapes the way SpookyCorgi did with Phiz: or you can approximate from eye/iris detection. Hope that helps. |
Hey, I have been looking to implement blendshapes in my project but didn't know it wasn't included in the latest release (I assumed it was since they were introduced in mediapipe 0.9.3 I guess). I am going with github actions too but I am not too familiar with how arguments to workflows work. Looking at build-package.yml file, I am assuming the workflow would take default values if I leave that field empty right? Also, how do I turn off building of certain plugins. Thanks. |
support for face blendshapes