solved Morphing with armature animation

[ozRocker]

I'm struggling with getting this girl to walk and make facial expressions.

If I separate the head, the head moves when using the "pout" morph.

https://playground.babylonjs.com/#BDG7ME#11

 

If I apply to the whole body, the rest pose is applied to the walk.

https://playground.babylonjs.com/#BDG7ME#15

 

Both work fine in Blender though.  Is there something else I need to do to get morphing working with armature animation?

[JCPalmer]

I do not know.  When I morph with a skeleton in QI, the base vertices / normals are recomputed on the CPU, & resent to GPU.  In this scenario, the skeleton weights being applied in the shader are blissfully unaware of the change.

You are not running the default shader set with a custom material.  There I see morphing is done before bone influence in default.  You might want to just use a standard material, not one that changes.  This will show whether this a general problem or just with custom materials.

BTW, I never had the need to switch my texture for toons (change expression in the dropdown).  If you are going to show full characters without zoom, this might be just an expense without any real payoff that you can see.  Just sayin.

[ozRocker]

1 hour ago, JCPalmer said:

You are not running the default shader set with a custom material.  There I see morphing is done before bone influence in default.  You might want to just use a standard material, not one that changes.  This will show whether this a general problem or just with custom materials.

ok, here's the playground without any custom material. The morph problem can be seen here as well.

https://playground.babylonjs.com/#BDG7ME#16

[GameMonetize]

It should work in a whole body approach:

Can you make sure that the morph targets are all the same but the head?

Bones work at matrix level where morph targets work at geometry so they SHOULD be compatible

 

[GameMonetize]

Also another test: can you try with all morph targets identical to the source?

[ozRocker]

35 minutes ago, Deltakosh said:

Also another test: can you try with all morph targets identical to the source?

Ok, this playground has a morph target applied to the whole body, but the target is exactly the same as the source (no change)

https://playground.babylonjs.com/#BDG7ME#17

It looks like its applying the rest pose to the animation

[GameMonetize]

On it!

[GameMonetize]

Ok so it seems that there is a data issue.

Let me try to show you why.

If you run this PG: https://playground.babylonjs.com/#BDG7ME#18, you will see that skeletons are disabled but still the target and the source are not the same. Seems like the target is based on T pose while the source is on the first animation frame

[ozRocker]

3 hours ago, Deltakosh said:

Ok so it seems that there is a data issue.

Let me try to show you why.

If you run this PG: https://playground.babylonjs.com/#BDG7ME#18, you will see that skeletons are disabled but still the target and the source are not the same. Seems like the target is based on T pose while the source is on the first animation frame

Ah ok.  Do you know how I can fix that?  I thought morph targets were relative, so the values for everything would be zero, apart from the face that changes.

[JCPalmer]

Thinking out loud.  Shapekeys are based on the actual values of the vertices.  The actual positions are not read when the position values are exported.  They are a read from a copy made using a bmesh.  The bmesh copy has the advantage of having all modifiers already applied like Mirror or sub-surface smoothing.  An armature is also implemented as a modifier.

The process currently processes all armatures first & along with any actions (animations).  The frame number is then set back to what it was.  All meshes are then processed with the settings of the modifiers, in your case also the first frame.  It never made a difference before.  For my morphing, I never exported with an action, though Tower of Babel can.  I just set the armature to the rest pose before exporting, and got the poses from an armature pose library exported separately.

I wonder if just before creating the bmesh:

• set the currentAction to None
• went into pose mode
• selected all bones
• cleared all transforms

If this does not just destroy the export, all morph targets & the main vertices would be as if from the rest pose.  Afterward, maybe set currentAction back to what it was. 

[ozRocker]

11 hours ago, JCPalmer said:

Thinking out loud.  Shapekeys are based on the actual values of the vertices.  The actual positions are not read when the position values are exported.  They are a read from a copy made using a bmesh.  The bmesh copy has the advantage of having all modifiers already applied like Mirror or sub-surface smoothing.  An armature is also implemented as a modifier.

Ok, your thoughts gave me an idea.  I did this with my virtual world.  I had a mesh .babylon file and I had an animation .babylon file and I was able to apply the animation to the mesh.  I used this technique as a workaround for this.

What I did here was create a .babylon file with meshes and skeleton (but no animation) and morph target.  I was able to separate head from body so I only have to apply morph targets to the head, which is a smaller mesh.

I then created a .babylon file with mesh and animation.  I removed morph targets from that.  I only need the animation but you can't export Blender project that doesn't have a mesh, or has a mesh but its empty.  I basically cleared out the mesh data by hand (editing the text file).  This cut a 3.3MB file to 122KB so it only has the animation data.

I then load both files and copy the animation to the morph target meshes.

You can see it here https://playground.babylonjs.com/#BDG7ME#20

[GameMonetize]

Woot! this works very well!

Congrats

[JCPalmer]

Nice work around.  I did try to implement what I talked about on Saturday.  My first attempt only worked when you exported when the armature was selected when you did the export.  Then tried to select the armature in code & got errors, and gave up for the day.

One improvement on the 2 export solution might be to have the armature be the only thing in the currently selected layer.  Check the 'currently select layers' box in the custom exporter properties on the Scene tab.  Or adding a 'Only armatures / animations' check box on Scene exporter properties seems like it might be useful & easy to implement as well.

Not working on this right now, but will pick it up again during a break.  Having quite a difficult time finishing my Blender Kinect Mocap project.  The pipeline turned out to be the easy part.  Getting the resulting data transformed without destroying the mesh is difficult.  I foresee frequent break opportunities, unfortunately.