Animating a model using the 'body' values returned

[AizenKS]

This may have been asked already so apologies if thats the case. I'm looking to use the data returned from video detection to animate a model (character) in babylonjs through bone rotations. I have looked at the 'human-motion' demo as well as the 'vrm' demo and tried applying what I saw but to not so great results, for example using the angle method from the vrm demo but this doesn't work correctly and often causes the limbs to clip through other parts.

Any tips or assistance would be greatly appreciated, thanks.

[vladmandic]

i never had much luck with vrm, but that was never my priority.
regarding bablyonjs, human-motion does not use skeleton object, it uses shapes and that works well.
but if you want to remap to actual skeleton, i suggest to take a look at https://github.com/motionIQ-ai/motion-intelligence/blob/main/src/person/Kinematics.ts

[AizenKS]

Unfortunately the link provided does not exist (404), do you have the source file? If so could you post it.

[vladmandic]

ah, it exists, its my repo, but its commercial so its private. sorry.
anyhow, the idea is to

• use babylonjs skeleton concept
• rely on babylon's lookController to set bone
• then adjust bone.rotation so its facing the right way
  also, skeleton and results must be scaled to match and they must have normalized origin (so normalize results with average).
  not easy, but results are really good.

something like

    this.person.lookControllers[name].target = this.anchor[name].absolutePosition;
    const position = new Vector3(kpt[0], kpt[1], kpt[2]);
    this.origin[name].setPositionWithLocalVector(position);
    this.bone[name].setPosition(position, Space.BONE);
    this.bone[name].computeAbsoluteTransforms();

[AizenKS]

Thanks for the answer, could you give a brief explanation of the code snippet provided (I'm rather new to babylonjs)
Also could you give some tips on the normalisation and scaling required, I looked into it and found I need to normalise by performing calcs using the bounding box, however that has also not worked as expected.

[vladmandic]

im traveling right now so its hard to provide much more detailed answer.
for normalization, i typically use mean coordinates of the torso, not the bounding box for entire body as its too unstable.