How can i use Mixamo Animations in threejs and vroid model?

[wbq9224]

Hello guys, I do some animations using mixamo and unity. And how can i play these animations in threejs?
In gltf models, the way to play animations usually like this:
mixer = new THREE.AnimationMixer(gltf.scene)
clip = THREE.AnimationClip.findByName(gltf.animations, 'animation_0');
action = mixer.clipAction(clip)
action.play()
But in vroid models, i dont know how to find animations which i have already maked in mixamo or 3dmax.
Please help.

[0b5vr]

Does this help?: #891

[wbq9224]

Does this help?: #891

Thank you for your reply!
It seems that that is what i need. But unfortunately I can not open glitch for some reasons. So i cant watch working demo that you privide in glitch. Could you please privide some key point or code here? Or maybe just a little keypoint hint is also helpful!
Thank you!

[wbq9224]

Does this help?: #891

Finaly i use vpn, open glitch and watched the working demo. it's cool.
I noticed that you use fbx loader to load a fbx model with mixamo animations and built keyframe track by threejs. And then use mixer to apply the keyframe animation to vrm model. It worked. If i download animated mixamo model and then alreay make these animation in vrm by unity, can i find these animation in vrm model directly? In other words, Is this fbx model and fbx loader necessary?
Forermore, If my skelon animation is just maked by 3dmax or maye, not by mixamo, can i use this method to play?
Thank you again!

[0b5vr]

I'm not sure which format you are using to export your animations.
In the demo, the animation data is contained in the FBX which doesn't have its own meshes, and the FBX is directly retrieved from Mixamo.

The VRM file format itself doesn't have the ability to contain any animations.
I assume you have modified animations taken from Mixamo using Unity and trying to use the animation in Three.js, is this right?
I'm afraid I don't fully understand your intention.

[wbq9224]

I'm not sure which format you are using to export your animations. In the demo, the animation data is contained in the FBX which doesn't have its own meshes, and the FBX is directly retrieved from Mixamo.

The VRM file format itself doesn't have the ability to contain any animations. I assume you have modified animations taken from Mixamo using Unity and trying to use the animation in Three.js, is this right? I'm afraid I don't fully understand your intention.

Yes. I'm finding animation data in vrm file just like "gltf.animations" in glb model file and then play it by threejs. Now look like vrm file itself doesnt contain these animation data, and I need to get these animation data by other animated model format. Then mix it to vrm model, right?
By the way, is 1.0-beta version of three-vrm necessary? Now i'm using 0.6.0 and everything seem fine except animation. I try 1.0.0 beta and material seem have a little different. And when i add HemisphereLight to scene, the glsl shader will throw error.
Thank you for your reply again.

[0b5vr]

and I need to get these animation data by other animated model format. Then mix it to vrm model, right?

Yes, that's right. The demo uses Mixamo FBX but technically any animation format will do the job.

By the way, is 1.0-beta version of three-vrm necessary? Now i'm using 0.6.0 and everything seem fine except animation.

It's mainly for VRM 1.0, which is the next generation of the VRM specification.
Once VRM 1.0 gets released, we are going to migrate 1.0 soon.

And when i add HemisphereLight to scene, the glsl shader will throw error.

That is a separate issue, I will investigate later. Thanks for the report!
#972

[wbq9224]

and I need to get these animation data by other animated model format. Then mix it to vrm model, right?

Yes, that's right. The demo uses Mixamo FBX but technically any animation format will do the job.

By the way, is 1.0-beta version of three-vrm necessary? Now i'm using 0.6.0 and everything seem fine except animation.

It's mainly for VRM 1.0, which is the next generation of the VRM specification. Once VRM 1.0 gets released, we are going to migrate 1.0 soon.

And when i add HemisphereLight to scene, the glsl shader will throw error.

That is a separate issue, I will investigate later. Thanks for the report! #972

Thank you for you reply. It almost solved my question. Now the animation seem worked.
Since the whole animation system ready for vrm 1.0, I have to update my code which is baseed on 0.6.0 vrm to the latest. And I found some error such as "THREE.VRMSchema.HumanoidBoneName.Hips" and so on. So I want to ask if any document for the latest update change point or example for 1.0.0-vrm especially for blendshape and bone. And what time for releaseing 1.0.

[0b5vr]

I found some error such as "THREE.VRMSchema.HumanoidBoneName.Hips" and so on. So I want to ask if any document for the latest update change point or example for 1.0.0-vrm especially for blendshape and bone.

We have to prepare the migration guide, yes!

Addressing this ad-hoc, about bones and expressions (formerly called blend shape proxies), you can specify bone names and expression names by raw string e.g. 'hips' 'head' in three-vrm v1.0.

And what time for releaseing 1.0.

Since it depends on the release of VRM 1.0 spec, I'm still not sure about the date.

[wbq9224]

Thanks！

[wbq9224]

Hello, I have to reopen this issue. I use https://glitch.com/edit/#!/three-vrm-1-sandbox-mixamo demo code to load animation download from mixamo and play the animation onto vrm model, it's worked.
And when I use animation download from different web such as https://actorcore.reallusion.com/actor and do the same work, it's totally not work. I did many work to try to solve the problem and I failed.
I tryed build a new RigMap which can map new animated model rig to vrm rig. I tryed retarget the new animated model to mixamo rigify rig. All these work not useful. So I wonder how can I use diffierent animation(not only from mixamo) to bind vrm model and then play ? Have a general method ?

[0b5vr]

I tryed build a new RigMap which can map new animated model rig to vrm rig. I tryed retarget the new animated model to mixamo rigify rig. All these work not useful.

How did it failed to work? It appears to me that you have done proper step to make it work properly.

[wbq9224]

I tryed build a new RigMap which can map new animated model rig to vrm rig. I tryed retarget the new animated model to mixamo rigify rig. All these work not useful.

How did it failed to work? It appears to me that you have done proper step to make it work properly.

It's disapperaed. I try many diffierent motions model and they all disapperaed.
Such as https://actorcore.reallusion.com/motion/free?asset=b083-runtoblastb, I built a new rigmap like this:
'CC_Base_Hip': 'hips',
'CC_Base_Waist': 'spine',
'CC_Base_Spine01': 'chest',
'CC_Base_Spine02': 'upperChest',
'CC_Base_NeckTwist01': 'neck',
'CC_Base_Head': 'head',
'CC_Base_L_Clavicle': 'leftShoulder',
'CC_Base_L_Upperarm': 'leftUpperArm',
'CC_Base_L_Forearm': 'leftLowerArm',
'CC_Base_L_Hand': 'leftHand',
'CC_Base_L_Thumb1': 'leftThumbProximal',
'CC_Base_L_Thumb2': 'leftThumbIntermediate',
'CC_Base_L_Thumb3': 'leftThumbDistal',
'CC_Base_L_Index1': 'leftIndexProximal',
'CC_Base_L_Index2': 'leftIndexIntermediate',
'CC_Base_L_Index3': 'leftIndexDistal',
'CC_Base_L_Mid1': 'leftMiddleProximal',
'CC_Base_L_Mid2': 'leftMiddleIntermediate',
'CC_Base_L_Mid3': 'leftMiddleDistal',
'CC_Base_L_Ring1': 'leftRingProximal',
'CC_Base_L_Ring2': 'leftRingIntermediate',
'CC_Base_L_Ring3': 'leftRingDistal',
'CC_Base_L_Pinky1': 'leftLittleProximal',
'CC_Base_L_Pinky2': 'leftLittleIntermediate',
'CC_Base_L_Pinky3': 'leftLittleDistal',
'CC_Base_R_Clavicle': 'rightShoulder',
'CC_Base_R_Upperarm': 'rightUpperArm',
'CC_Base_R_Forearm': 'rightLowerArm',
'CC_Base_R_Hand': 'rightHand',
'CC_Base_R_Pinky1': 'rightLittleProximal',
'CC_Base_R_Pinky2': 'rightLittleIntermediate',
'CC_Base_R_Pinky3': 'rightLittleDistal',
'CC_Base_R_Ring1': 'rightRingProximal',
'CC_Base_R_Ring2': 'rightRingIntermediate',
'CC_Base_R_Ring3': 'rightRingDistal',
'CC_Base_R_Mid1': 'rightMiddleProximal',
'CC_Base_R_Mid2': 'rightMiddleIntermediate',
'CC_Base_R_Mid3': 'rightMiddleDistal',
'CC_Base_R_Index1': 'rightIndexProximal',
'CC_Base_R_Index2': 'rightIndexIntermediate',
'CC_Base_R_Index3': 'rightIndexDistal',
'CC_Base_R_Thumb1': 'rightThumbProximal',
'CC_Base_R_Thumb2': 'rightThumbIntermediate',
'CC_Base_R_Thumb3': 'rightThumbDistal',
'CC_Base_L_Thigh': 'leftUpperLeg',
'CC_Base_L_Calf': 'leftLowerLeg',
'CC_Base_L_Foot': 'leftFoot',
'CC_Base_L_ToeBaseShareBone': 'leftToes',
'CC_Base_R_Thigh': 'rightUpperLeg',
'CC_Base_R_Calf': 'rightLowerLeg',
'CC_Base_R_Foot': 'rightFoot',
'CC_Base_R_ToeBaseShareBone': 'rightToes',
And I use the new animation clip name(not "mixamo.com"). And finally I play this new animation track, my vrm model is disapperaed : (

[wbq9224]

does anybody can help, plz?

[0b5vr]

Could you provide a demo using services like Glitch or CodeSandbox?

[wbq9224]

Could you provide a demo using services like Glitch or CodeSandbox?

Thank you for your kind help!
Here is a live demo, basiclly same with your code #891.
I just change the animate model, clip name and rigmap. But everything goes to bad :(
https://codesandbox.io/s/optimistic-dan-qlqgsb?file=/main.js

[0b5vr]

It seems that the animation data needs to be retargeted for the VRM you use and that is unfortunately out-of-scope for three-vrm for now.

[jasondalycan]

I have encountered the same issue. I started creating a CC3 rig map (similar to that posted above by @wbq9224) and while the animation loads and moves the character the animation's motions are offset quite a bit. Looks like the same issue faced by the following Godot user: smix8/GodotAnimationRetargeting#19

It appears that the CC3 bone strucutre is farily different than that of Mixamo/VRM and thus requires a significant amount of retargeting.

[jasondalycan]

Similar issue trying to use DeepMotion (https://www.deepmotion.com/). Using the workflow 1) upload VRM to deep motion -> 2) create animation -> 3) Download as FBX -> 4) Import into three-vrm with appropriate rig map, the animation loads and plays some of the animation correctly although the character's torso jumps up and down sporadically. See the attached videos:

The skeletons between the exported DeepMotion FBX and the MixamoFBX are quite similar (rig map correlates well). Any thoughts on what may be causing this? See video clips for an example of what is happening:

DeepMotion Animation:

three-vrm Animation:

[0b5vr]

would you provide a demo (Glitch, CodeSandbox, etc) ?

[jasondalycan]

Sure here's the code in codesandbox: Here

[jasondalycan]

Here's another codesandbox that's using an FBX from mocaponline.com. This seems to work aside of the fact tha the arms appear offset by the t-pose: do you know how to resolve this? Codesandbox code is here.

[0b5vr]

both demo does not seem to working for me at all. seems it's failed to load fbx

[0b5vr]

Seems the animation rotates by 180deg / 360deg and it seems to be an error of quaternion/euler conversion for me. Not sure what causes this issue though.

For hips movement, the format seems to be different from mixamo's one. You may need to transform the hips translation track. Probably flipping axes and removing 0.01x scale will fix this?

[witnesstodark]

We was fighting problem with animations a lot. With old mixamo animation, then with new, and eventually we created GLB importer for animation so now it is much easier to do any stuff. Here is our viewer for testing glb and fbx anims - https://vrm-viewer.ownverse.world
Git is not public yet, but we will do it soon I guess.

[ButzYung]

What is the "new" Mixamo animation that you guys are talking about? All Mixamo FBX motions I have tried so far seem to work properly on VRM.

[aMidnightNova]

If i try Loading the Gangnam Style animation that is in this demo it works correctly for me.

https://glitch.com/edit/#!/three-vrm-1-sandbox-mixamo

however if I download Gangnam Style from Mixamo myself it results in

There is probably a change in basis needed that I haven't figured out yet.

[ButzYung]

The Gangnam style animation works properly in the exampler loader demo. Did you use the example loader or did you write your own loader which caused the problem?

[aMidnightNova]

It does this with the loader code found on glitch.

to be a bit more clear about the animations:
If I download the Gangnam Style animation from glitch via the ref link in the code, that one works correctly. If i go to Mixamo and get Gangnam Style there, that one does not work correctly, along with anything else I tried.

[aMidnightNova]

This is no longer an issue for me, I realized today there is an updated version in the examples https://github.com/pixiv/three-vrm/blob/dev/packages/three-vrm-core/examples/humanoidAnimation/loadMixamoAnimation.js

[ButzYung]

@wbq9224 I have been trying to build a FBX motion loader for VRM in my app so your case raised my interest. I have tried with your FBX sample motion, and after some testings and stuff, I can identify the problems and fix them (at least I can have your FBX motion displayed properly).

The FBX loading example in three-vrm is somewhat oversimplified. It works fine with FBX animations that are also based on normalized T-pose rig like VRM, such as Mixamo animations. However, things begin to become tricky when the rig is not in T-pose or is rotated at the root/hips. For example, the bone movement offset needs to consider the bone/rig rotation as well. Also if the rest pose of root/hips has been rotated, the axes of all rotations have to adjusted accordingly as well.

@jasondalycan If you can provide a sample FBX motion I would like to try as well.