Correct way to manually deform models with skeletons in BabylonJS

[Mazax]

Hello everyone!

I'm working on a BabylonJS project and having problems with the movement of the 3D-model. I have used tens of hours to find a solution with poor results. The projects goal is to get 3D-model/avatar move according to the movement data from Kinect-sensor. I'm using a C#-server to send Kinect data through WebSocket to JS and BabylonJS for visualization.

The problem what I'm having is that I don't know how to correctly move the model. BJS documentation seems to focus only to ready-made animations and not to the manual deforming of meshes with skeletons. 

I've moved and deformed the skeleton in BJS, but cannot get the meshes to move with the skeleton. Everything works fine in Blender (moving the skeleton deforms the mesh), but not in BJS. I've also tried to make custom animations for the skeleton in BJS to animate the model frame by frame with the data, but it's so complex I'm having a hard time understanding it (with no proper tutorial).

What is the correct way to manually deform models with skeletons in BabylonJS? If the right way is to make custom animations can you help me understand how to build an animation for a skeleton in BJS? (I'm familiar with the BJS tutorials and documentations for animating meshes with no skeletons).

My previous post about the problem

Here is a bit creepy custom animation I'm working on. (How to delete the ready-made animation?)

Here is one debug-model I'm using. How to make skeleton animation from scratch if your model doesn't have one?

 

Thank you for your help.

Mazax

[Wingnut]

Hiya Mazax...

   Man, sorry you have gotten no replies to this.  hmm.  Not much recent activity in your previous thread, either.

I wish I knew about skeletons, but I am weak in that area (and many other areas)    A forum search for 'bone', 'skeleton', 'weighted', 'armature', etc... might bring results.

Have you had any success/advancement in this issue?  Any satisfaction?  I will try to "assemble" a decent response... and provide some playground links... soon.  Sorry for the delay.  Maybe others will respond to the "bump", too.

[GameMonetize]

Hello Mazax,

sorry for the late response. Did you check this demo: http://www.babylonjs-playground.com/#I6RJJ#44

[Mazax]

Hi again and thank for your answers!

@Wingnut  I'm glad to be able to inform you that I've found part of the answer I was looking for myself from the BabylonJS GitHub-site. 

I found out that the new BJS 2.5 features include bone rotation and scaling (Added to GitHub 5 days ago, Oct 29 2016 by abow. Link). There I found basically the same demo that @Deltakosh linked previously and used that information to find the solution to set transformations, rotations and scales to the bones in my project. 

What i found out is:

• You need to update the bone.getLocalMatrix to set the values to bone, however I cant get the bone.getWorldMatrix work the same way. (Why?)
• To update the mesh with it's skeleton data you need to call the bone.markAsDirty(). This was my main problem what I was having and it isn't told in the BJS documentation (I didn't find it at least).
• The new BabylonJS 2.5 (now in beta) version has at least some support of the manual bone transformation (Bone.rotate and Bone.scale). Hopefully the new version will cover more about the subject (Bone.translate maybe? And hopefully something to ease the work with the matrices like in Bone.scale).

The code I'm using now to deform the model is:

 

//get translation matrix
var matrix = bone.getLocalMatrix(); 
//set position values
matrix.m[12] = joint.x;
matrix.m[13] = joint.y;
matrix.m[14] = joint.z;
//set mesh to be updated
bone.markAsDirty();

The rotation and scale values can be set in BJS 2.5 by calling the Bone.rotate and Bone.scale.

This problem now solved, my project faces a new issue: I noticed that the skeleton I'm using doesn't have the right rotation coordinate system compatible with Kinect v2 data and it might be constructed wrong in some other way too (I think I know how to fix it and am currently working with it. I'm new to blender, but learning fast!). The next step in my project would be to fix that skeleton. If someone have one spare Kinect v2 compatible Blender armature I'm more than happy to have it  

I'll update my progress again when I get something major done.

 

Thank you for your help.

Mazax

[getzel]

Welcome, I wish you good luck in your project.