Skeletal Animation, Picking Info, and Morph Shapes

[WolfKodi]

Whoa! Okay, there are two parts to this.

 

Case/Issue 1: Scene picking info for animated rigs only obtains picking information from the first frame of the rig animation.

 

I've setup a test case, http://jsfiddle.net/7x0nkh0g/3/ , where there's this cube that is being animated and moved by this bone. You can click on the cube to change it's colour between green and red. Clicking only works if you click on the mesh where it was at the first frame of animation. Doesn't canvas picking info casts a ray into the scene and looks for active faces everytime? It appears that it's using the first frame of the rig as if it was not being animated at all.

This issue occurs from files exported from both Babylon and Tower Of Babel exporter from blender.

Case/Issue 2: Using the Morph Extension (ver 1.1) and Tower Of Babel to import and animate shape keys created in Blender, on a mesh that also has skeletal rig and animation, morph shape offsets are relative towards the Rest Position (edit mesh position in Blender before the mesh has been set to a pose by a rig).

Using the previous jsFiddle example: http://jsfiddle.net/7x0nkh0g/3/, clicking on the blue circle morph's the cube's shape into a pyramid. Only the top four keys are morphed. Notice how the top four keys try to move towards the rest position of the original mesh instead of following the translated skeletal rig's first frame.

After discovering this, I created another animation rig but this time, I set the first frame of the rig to match the original rest position of the mesh and then have my usual looping animation after that. In Babylon, I then set the rig animation to skip over that first frame. This is sort of a work-around but it actually works! See: http://jsfiddle.net/7x0nkh0g/4/ The cube is being morphed to a pyramid correctly now, while still being animated by a skeletal rig.

I've only tried this trick out on the test case. Can't wait to try it out on my facial shapes on top of a skeletal animation.   

I read in some old threads that JCPalmer might have a working copy of the Morph Extension version 1.2 hasn't been released? I'm getting depreciation messages from BJS using this old Morph Extension: "BJS - [22:36:45]: Mesh.updateVerticesDataDirectly deprecated since 2.3."

On a sidenote for those test cases I've created, if you click on the blue circle to morph the shape of the cube first, then you can't click on it to change it's colour later. JS gives an error "Uncaught TypeError: Cannot read property 'subtractToRef' of undefined". Click on the cube to change its colour first and then clicking the morph next works fine however.

If you need the blender file, I've attached it.

cubearmatureplusdeform.zip

[GameMonetize]

Hello!

 

1. This is because meshes are updated GPU side and picking is done CPU side. So CPU is only aware of the initial state

 

2. This one is for JCP I guess

[WolfKodi]

Hi!

1. Oh, how interesting. Is there a way to update the CPU before the a picking request? What about casting a ray or collision picking? Are they done on the CPU side and are only aware of the initial state? Surely there must be a way to update them.

2. I guess too 

[JCPalmer]

Your calls have been received on all your topics about morphing.  North America is in the midst of a 4 day holiday, so no one was able to come to the phone just now.  Too much to do, naps to be taken, etc.

 

Your call was important to us, and I will look at these again on Monday.

[WolfKodi]

Happy holidays and enjoy Black Friday!

These aren't all that important or critical but thanks for the heads up.

[GameMonetize]

You can process the bones CPU side (mostly for compatibility) but for obvious performance reasons we do not save the computed vertices. THere are just used for rendering.

 

PIcking is done against optimized and cache values. 

[WolfKodi]

I see, thanks! I know where to dig deeper if I want to go there. I think it is better to harness as much power from the GPU as possible while still being mostly compatible. Do you suppose the GPU can perform a picking task?

[GameMonetize]

I don't say it is impossible but this is a tough task (as it has to be done inside the same shader with everything else like rendering and we are already instructions limited)

[WolfKodi]

I figured it would be a tough task. Thank you kindly for answering!

[GameMonetize]

My pleasure

[JCPalmer]

Glad you are interested in Morphing.  I have a major new release in the works.  This version may still not work correctly against a mesh with a skeleton using the standard BJS animation system, but MORPH (renamed to QueuedInterpolation, QI) will have an integrated bone interpolator right inside the beforeRender.  BABYLON.Animation objects can loaded directly from Motion capture files (only Acclaim format right now) without running them through Blender.  I still use a BABYLON.Animation object for data storage, but it is designed to work best if you only include key frames.  In your example, I think you would only need 1 frame, the last one.  The final pose will be where it ends up, no matter what state is left after the earlier queued end state.
 
I looked at the behavior you are getting with skeletons & meshes with ShapeKeyGroups.  Part of the problem may be that you are using BJS 2.3.  The MORPH in the repository still calls updateVerticesDirectly().  I have done a lot of work in 2.3 to ensure things work together.  First was upgrading the standard updatingVertices() to be optimized to work with Float32Arrays.  updateVerticesDirectly() is no longer needed, so it was not updated & depreciated.
 
Next I updated the cpu skinning process, such that morphing & it use the same Float32Arrays to inter-operate as best as possible.
 
Have done a lot of testing using MakeHuman 1.1.  One can have 50 facial shapekeys.  I have converted my proof of concept VoiceSync tool to be for MakeHuman meshes.  I have not gotten all the face shapes coded for each sound yet, but you can interactively develop them using slider controls, then write the typescript snippet needed to put back into the program onto the console.  It is the first scene I know of where you can actually direct a mesh deforming using controls.
 
I also have an interactive tool to bring in Motion Capture files from Carnegie Mellon Graphics Lab http://mocap.cs.cmu.edu/

This is not likely to be released though, in a pre-built way at least.  You have to down load a 1 GB zip file of the entire library, then put the html, JS, and texture files in the top level directory.  You then launch the html via double click.  The individual online files are not CORS enabled.

 

Really close to getting this to turn over.  Not going to spend time to get the error you have fixed, sorry.

[WolfKodi]

In your example, I think you would only need 1 frame, the last one.

Assuming you were referring to shape key frames, so that means it would only require the one shape key frame data? instead of having the Base Shape as well? This would make it more efficient on data size.

 

 

Really close to getting this to turn over.  Not going to spend time to get the error you have fixed, sorry.

 

I'm glad to hear that it's coming along well and it sounds like the updated version would fix many issues with the current repository and to be in line with BJS 2.3. Would QueuedInterpolation still be an extension like Morph or would it be integrated directly into BJS?

That's quite already, I figured it would be better to debug from the new version and the old errors can be disregarded. I spent the weekend doing a lot more testing more shape keys and ended up getting more errors with incorrectly morphed shapes and how adjusting one shape key in blender messes up another when exported using TOB and using the Morph extension in BJS. I finally called it quits for the old version when I found out that you can animate individual bones on top of an existing running skeletal animation. That's what I will be using for now.

Thank you.

[Dal]

@JCPalmer

Did you release any of this in the end? I am going to need this kind of stuff soon

[NasimiAsl]

i work on GPU side picking system 

good news it is possible 100%

bad news need time for  do that and   optimize

[JCPalmer]

@Dal, the exporter, Tower of Babel, is in the extensions.  Shape key groups for morphing have been done for a while.  Vismes are getting close.  The skeleton interpolator is not operational.  I did use it in a test scene recently.  No skeleton or actual talking, but exercising the FACE & WINK shape key groups.  The Eyes are using a rotation POV animation.

I have abandoned motion capture in favor of inverse kinetics inside of Blender.  Not sure when this will be done, sorry.

[Dal]

Really cool stuff!

[NasimiAsl]

ok i give a result from GPU picking system

http://www.babylonjs-playground.com/#JAG8B#4

this is compilicated a lot

 

flow this topic for GPU Picking System :