Blendshape data and functionality with AR

[skango]

Plugin Version or Commit ID

V0.14.0

Unity Version

2022.3.11f1

Your Host OS

Windows 11

Target Platform

Android, iOS

Description

I would like to know if there is any way to get blend shape information. I want to use blendshape data to apply that to my 3D model in AR. Also can I run this without webtexture? Because I want to use this in AR and webtexture is not suitable. Maybe I can pass my own texture 2D? Can I do something like that? Please let me know, thanks!

Code to Reproduce the issue

No response

Additional Context

No response

[skango]

@homuler Please when you have time just tell me 1 little thing, I got blend shapes working, I just need to understand how can I pass mediapipe my own texture 2Ds instead of using webcamera? Please, this is the only thing I need to get everything working.
I am using facelandmarkrunner script to get blendshape data, I just don't know how to provide this plugin with my own textures and not webtextures. Thanks!

[skango]

If anybody needs to get blendshapes, access it from facelandmarkrunner script at this part:
private void OnFaceLandmarkDetectionOutput(FaceLandmarkerResult result, Image image, long timestamp)
{
File.WriteAllText(Application.persistentDataPath + "/mpblend.json",
result.faceBlendshapes[0].ToString());
_faceLandmarkerResultAnnotationController.DrawLater(result);
}

[homuler]

I just need to understand how can I pass mediapipe my own texture 2Ds instead of using webcamera?

You can build an Image from Texture2D.

MediaPipeUnityPlugin/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Formats/Image.cs

Lines 47 to 50 in 6eb8026

	// TODO: detect format from the texture
	public Image(ImageFormat.Types.Format format, Texture2D texture) :
	this(format, texture.width, texture.height, format.NumberOfChannels() * texture.width, texture.GetRawTextureData<byte>())
	{ }

See also https://github.com/homuler/MediaPipeUnityPlugin/wiki/API-Overview#imageframe and #343.

[skango]

I just need to understand how can I pass mediapipe my own texture 2Ds instead of using webcamera?

You can build an Image from Texture2D.

MediaPipeUnityPlugin/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Formats/Image.cs

Lines 47 to 50 in 6eb8026

	// TODO: detect format from the texture
	public Image(ImageFormat.Types.Format format, Texture2D texture) :
	this(format, texture.width, texture.height, format.NumberOfChannels() * texture.width, texture.GetRawTextureData<byte>())
	{ }

See also https://github.com/homuler/MediaPipeUnityPlugin/wiki/API-Overview#imageframe and #343.

Thank you for your response, it is very clear there how to create an image frame, however, I don't understand where to pass that frame variable. facelandmarkrunner is the script I want to use only, since that one has blend shapes, so how do I input there this image frame? Or it is global? If it's global where does that go? Everything else is clear and I understand, thank you! In the sample code you pass it to some graph drawer or something, and I don't use that, I need this facelandmarkrunner.

[homuler]

I mean you can build an Image from Texture2D as well as ImageFrame.

[skango]

Okay thanks I got it working on android, however, I am not able to make IOS build, is there anything that I am missing? The Xcode project with full of errors, that was not the case before I imported this plugin. I made a build from windows machine, should I make it from MacOS? Could that be an issue?

[homuler]

If necessary, please create a new issue.