Blender exporter problems with many animations

[ozRocker]

My animations don't seem to work when I export a lot of them together from Blender.  I recall struggling with 3 animations before and fixing it by changing the order in the NLA editor.  This time I can't get it working.  All the ranges appear in the .babylon file and I'm able to play the ranges, but the animations are jumbled up.

Here's what my NLA editor looks like.  All the animations run fine in there:

What happens is the "Idle" animation ends up copied into the other ranges.  If I move them around and put Idle last then I get a Walk animation that has a bunch of mixed anims in it.  This is happening in Blender exporter 4.5 and 4.4.

My ranges seem ok:

"ranges":[{"name":"Float","from":0,"to":107},{"name":"Fly","from":120,"to":198},{"name":"Idle","from":210,"to":459},{"name":"Jump","from":470,"to":502},{"name":"Land","from":510,"to":572},{"name":"LeftTurn","from":580,"to":607},{"name":"RightTurn","from":620,"to":647},{"name":"Run","from":660,"to":678},{"name":"Walk","from":690,"to":720}]

Here's a playground with the exported .babylon file:

http://www.babylonjs-playground.com/#1IGIDN#2

and here's a ZIP with the Blender file and texture:

http://www.punkoffice.com/downloads/brokenanims/animator.zip

I'm just wondering if there's something simple that I'm forgetting to do.

My anims are 30 fps and I've set Blender to 30 fps.  I've made sure to apply position, rotation and scale to the mesh and skeleton.  I've set the end of the timeline to the last frame of the NLA editor.  I make sure to select all animations before exporting.

 

 

[ozRocker]

Here's another example using the same Blender file.  I just removed all animations except the flying ones.  So my Blender screen looks like this:

Some of the animations seem to have bits of "Jump" and "Fly" in them.

Here's the playground:

http://www.babylonjs-playground.com/#1IGIDN#3

 

[JCPalmer]

First, I have no clue what an NLA editor is. 

• Loaded your blend.
• Switched to Animation layout. 
• Changed Dope sheet form to Action Editor
• Selected Fly

It showed frames 0 - 78 as having data for this action, but when I clicked in this frame range no flying.  Frame 426 way outside this range was flying.

Think your process pipeline prior to export is cause.

[ozRocker]

Thanks for the quick response @JCPalmer  The NLA editor is just a way to lay out animations on the timeline in the form of blocks.  I thought it was needed to select which animations were required for the exporter.  Turns out I've been doing it wrong the whole time!  Any kind of global timeline editing actually interferes with the exporter as it confuses that with the local timeline of an animation.  Anyway, I need to make sure I have F (fake) against the animations to give them a fake reference since nothing is referencing them and they would get discarded when I save the file.  It works now!

[JCPalmer]

Good.  Doing less work, fixes problem.  Not a common occurrence.  Did do some searches after I posted on NLA.  Basically, it strings animations together, just like the export does. The big difference is the actions are still editable without NLA, since the exporter changes nothing to the blend.

The reason you might want to edit skeletal mocaps is for frame reduction.  I finally got the bone interpolater I have been working on close to completion.  Until the new version of Matrix.Lerp came out, it was just useless.  You can now remove all the frames which are not at some major direction change.  Here is a step left - step right animation, reduced from 45 frames to 6.

The missing frames are smoothly determined on the fly in a Mesh before render.  The same before render can also morph multiple shape keys groups, for walking and talking at the same time.  Root bone translation is also converted to position, as well.  I have that turned off right now.  This test file does not have translation, anyway.

Notice that it is not a good idea to have frames at the beginning, but you should always populate the last frame.  As the name of the extension, Queued Interpolation, implies, you can string as many animations as you wish by just submitting them to the mesh's skeleton queue.  All at the same time, if you want / know in advance.

Going from animation to animation is always smooth by definition.  No easing / fixing required.  The first pose in the animation above is arrived at, after 6 interpolations with whatever the state of the skeleton was when starting.

This is also more than about file space.  If you standardize your skeleton across all your characters, you can make a library of every animation you have for that skeleton, when each under 10 key frames.  You can then export your character(s) without any animation, and assign the animation ranges in BJS from the same library. 

I just got the thing to work for the first time today.  Have not tested the rescaleAsRequired arg for Skeleton.copyAnimationRange. I do not even have a library skeleton where the skeleton you are copying to is taller or shorter.  Much more to do ...

[ozRocker]

At the moment I have standardized the skeleton across all my characters.  All my animations are in a separate file.  I don't know how to export to a .babylon file without a mesh so I export normally, then edit out the mesh data from the text file by hand.  This isn't a perfect solution though.  Animations change the shape of a person.  If I'm using animations that came from a male then all my avatars, including female, end up with wide shoulders.  If I'm using animations that came from a female then all avatars, including male, end up with a thin waist and wide hips.  I think the most realistic yet most efficient method is to have 2 separate sets of animations for male and female.  

[gryff]

Quote

Any kind of global timeline editing actually interferes with the exporter as it confuses that with the local timeline of an animation.

Never known why you were using the NLA Editor Oz. If you want to combine animations there are other ways - Merging Animations - go to about 19 minutes as most of the video is about creating two animations and the lady does wander about a bit ;-) Note she is using Blender 2.77.

AS Jeff does not take the NLA Editor into his exporter and once you add an action strip to the NLA Editor it is converted into a data block that the NLA Editor can use - but the action strip is lost to the BJS exporter - and it ignores the NLA Editor? Once you create the "Fake" user those action strips are still there for the exporter. The video above creates a new action strip with all the individual strips connected. You just then delete the individual strips - and export the one strip. However, I think JCP's exporter will export multiple action strips anyway without the need for the NLA Editor. And I think @Satguru suggested this to you before

Quote

At the moment I have standardized the skeleton across all my characters.

How can you Oz? The skeleton may have the same number of bones and linking - but look in Blender at the Armature(skeleton) of a male and female figure you are using. Select the Armature then go into Edit mode. In 3d Window hit N key now look at the Transform properties of a bone (shoulder) - the "head" and "tail" positions. Are they the same for your musclebound dude and your skinny fashion model lady?

cheers, gryff

 

[ozRocker]

I didn't realise how it worked @gryff  I thought the ranges were set according to where the animations were laid out on the timeline.  Turns out this is the worst way in the world to export animations 'cos it interferes with the local timelines.  So yeh, no more NLA for me!

With regards to the standardized skeleton, I meant that I'm using the same structure (same bone names) and one set of animations that I apply to each character.  The different characters do have their own skeletons, but the problem is the animations were rigged for one specific skeleton and it tends to make other characters bones move in a way to make their shoulders wider, or waists bigger.  Without an animation the character looks normal but the problem is the character is always animating.  If its not moving its still running the "idle" animation.  That animation itself is enough to warp the character slightly to form the shape of the original template.

[gryff]

Quote

At the moment I have standardized the skeleton across all my characters.  All my animations are in a separate file.  I don't know how to export to a .babylon file without a mesh so I export normally, then edit out the mesh data from the text file by hand.  This isn't a perfect solution though.  Animations change the shape of a person.  If I'm using animations that came from a male then all my avatars, including female, end up with wide shoulders.

As I said above Oz although the skeleton/armature/rig you are using may look the same in terms number of bones, bone names, and links etc, in two different figures they will not be the same.

Below is an image of two different figures A: a big bodybuilder and B: a slim petite model. The two figures were created in MakeHuman 1.1  and both were given the same CMU rig/armature. I have highlighted the "LeftShoulder" bone. The image also includes the various bits of Transform data for each bone - note the differences.

I exported each figure separately as .babylon files,  then looked for the "LeftShoulder" bone in the files.

Figure A (the male):

{"name":"LeftShoulder","index":9,"matrix":[-0.1382,-0.9007,0.4118,0,0.9873,-0.1583,-0.0148,0,0.0785,0.4045,0.9112,0,0.2743,1.7343,0.44,1],
"rest":[-0.1382,-0.9007,0.4118,0,0.9873,-0.1583,-0.0148,0,0.0785,0.4045,0.9112,0,0.2743,1.7343,0.44,1],"parentBoneIndex":8,"length":2.2224},

Figure B (the female):

{"name":"LeftShoulder","index":9,"matrix":[-0.112,-0.9914,-0.0676,0,0.9937,-0.1119,-0.0055,0,-0.0021,-0.0678,0.9977,0,0.1849,0.7277,0.1441,1],
"rest":[-0.112,-0.9914,-0.0676,0,0.9937,-0.1119,-0.0055,0,-0.0021,-0.0678,0.9977,0,0.1849,0.7277,0.1441,1],"parentBoneIndex":8,"length":1.3462},

They have the "same" rig/armature - but the data in the "matrix" array, "rest" array  and perhaps most obviously the bone "length" (at end of data illustrated above) are different for the two figures.

Attaching an action from, say, a .bvh file would involve a process called "retargeting" and would take skeletal differences into account, but the method you are trying does not seem to take the skeletal differences into account.

cheers, gryff

[ozRocker]

I suspect there might actually be a problem so you could be onto something @gryff The animations look fine in Blender but they seem to be changing the shape of the players slightly in my Babylon.js scene.

I have an animation.babylon file which has a mesh with no vertices or materials, but it has a skeleton and all the walking animations.  I'll be referring to that later as "Assets.animator".

I have character files (andrew.babylon, josh.babylon, etc..) which have a mesh and a skeleton, but no animations.  Those skeletons were rigged to their particular mesh so they do not have the same transformations, only the same bone names.  I have 4 characters which are real people I have scanned in.  I'll be referring to them as "Characters.arrMeshes[]".  When a user logs in they can choose one of those 4 as their avatar.  When they do that, an avatar mesh will be created which will be a clone of the matching character mesh and an avatar skeleton will be created which will be a clone of the matching character skeleton.  When that's done, I copy all the animations from the animator.

this.mesh = Characters.arrMeshes[character].clone("Avatar");
this.mesh.visibility = true; //the original characters are hidden from scene
this.mesh.skeleton = Characters.arrMeshes[character].skeleton.clone("Avatar");
this.mesh.skeleton.copyAnimationRange(Assets.animator.skeleton, "Idle");
this.mesh.skeleton.copyAnimationRange(Assets.animator.skeleton, "Walk");
this.mesh.skeleton.copyAnimationRange(Assets.animator.skeleton, "Run");
this.mesh.skeleton.copyAnimationRange(Assets.animator.skeleton, "LeftTurn");
this.mesh.skeleton.copyAnimationRange(Assets.animator.skeleton, "RightTurn");

I'll be delving into this deeper when I have time but I thought I'd put this up there in case you spot an obvious problem with my process.

[ozRocker]

Here's a better demonstration @gryff

In this playground I've imported Josh and I'm cloning the "idle" animation from Sandy.  When I play the "idle" animation you can see that it warps Josh's mesh quite noticeably.  Maybe there's something wrong with my code, but as you can see, I'm just copying the animation from Sandy, not her skeleton.

http://www.babylonjs-playground.com/#25M97N#3

However, in Blender the meshes don't warp.

This is Josh with Sandy's "idle" animation applied to him:

 

This is Sandy with her "idle" animation:

 

There's something definitely not right with the Babylon.js result.  Can someone check my code and see if there's bugs in it?

[ozRocker]

Here I'm starting from a t-pose just to see if Josh's original pose has anything to do with the warping:

http://www.babylonjs-playground.com/#25M97N#4

But as you can see the end result is the same.  It still warps 

[ozRocker]

This is Sandy and Josh side-by-side both running the same animation

http://www.babylonjs-playground.com/#25M97N#5

[adam]

I'm not sure if this will help, but here is what the skeleton looks like in BJS:

http://www.babylonjs-playground.com/#25M97N#6

http://www.babylonjs-playground.com/#25M97N#7

 

[ozRocker]

Thanks @adam , that does help.  I tried turning on the skeleton with the debug layer but that doesn't seem to be working.  But here now I can see that the skeleton breaks up a bit when I play that animation :/

[adam]

It looks like the copyAnimationRange code needs an option to copy relative positions.

[gryff]

@adam: I liked that skeleton viewer - you learn something every day

Here is the LeftShoulder bone data now from three of oz's sources:

From animations.babylon:

{"name":"mixamorig_LeftShoulder","index":7,"matrix":[0.1901,0.0012,-0.9818,0,0.846,-0.5075,0.1632,0,-0.4981,-0.8616,-0.0975,0,0.1578,0.3195,-0.0151,1],
"rest":[-0.0913,-0.0021,-0.9958,0,0.9567,-0.2777,-0.0872,0,-0.2764,-0.9607,0.0273,0,0.1578,0.3195,-0.0151,1],"parentBoneIndex":3,"length":0.3385

From josh.babylon

{"name":"mixamorig_LeftShoulder","index":7,"matrix":[0.2389,0.0253,-0.9707,0,0.8757,-0.4376,0.2041,0,-0.4196,-0.8988,-0.1268,0,0.1884,0.3521,-0.0079,1],
"rest":[-0.0402,-0.0009,-0.9992,0,0.9795,-0.1974,-0.0392,0,-0.1972,-0.9803,0.0088,0,0.1884,0.3521,-0.0079,1],"parentBoneIndex":3,"length":0.3915

From sandyidle.babylon:

{"name":"mixamorig_LeftShoulder","index":7,"matrix":[0.1946,0.0359,-0.9802,0,0.8545,-0.4969,0.1515,0,-0.4816,-0.8671,-0.1274,0,0.1754,0.355,-0.0168,1],
"rest":[-0.0913,-0.0021,-0.9958,0,0.9567,-0.2777,-0.0872,0,-0.2764,-0.9607,0.0273,0,0.1754,0.355,-0.0168,1],"parentBoneIndex":3,"length":0.3761

Easy to notice the difference in bone length and differences in "matrix" data

I was under the impression that the animations.babylon data had been manually extracted from from one of sandy's files - and was expecting them to be the same

cheers, gryff

[adam]

https://github.com/BabylonJS/Babylon.js/blob/9991f3a0f1c7607b29dbbab0ad9f72c0ecf2dcf5/src/Bones/babylon.bone.js#L83

It appears that the rescale option in the copy function was for situations like this.  Unfortunately it isn't solving the problem.

http://www.babylonjs-playground.com/#25M97N#8

[JCPalmer]

I was waiting for someone to try the rescale.  That function tried to change the translation (not scale or rotation).  It did it using a decompose - change - recompose.  That was a major problem when I tried use that method for my interpolator.  In 2.4 there are Matrix.getTranslation() & Matrix.setTranslation().

I have just upgraded to Unbuntu 16.04 LTS, and my Gulping ability is impaired at the moment.  If someone changed Bone.copyAnimationRange() section to this, better results might happen:

            // loop vars declaration / initialization
            var orig: { frame: number, value: Matrix };
//            var origScale = scalingReqd ? Vector3.Zero() : null;
//            var origRotation = scalingReqd ? new Quaternion() : null;
            var origTranslation = scalingReqd ? Vector3.Zero() : null;
            var mat: Matrix;

            for (var key = 0, nKeys = sourceKeys.length; key < nKeys; key++) {
                orig = sourceKeys[key];
                if (orig.frame >= from && orig.frame <= to) {
                    if (scalingReqd) {
                        mat = orig.value.clone();
                        origTranslation = mat.getTranslation();
                        mat.setTranslation(origTranslation.scaleInPlace(ratio));
//                        orig.value.decompose(origScale, origRotation, origTranslation);
//                        origTranslation.scaleInPlace(ratio);
//                        mat = Matrix.Compose(origScale, origRotation, origTranslation);
                    } else {
                        mat = orig.value;
                    }
                    destKeys.push({ frame: orig.frame + frameOffset, value: mat });
                }

 

[ozRocker]

25 minutes ago, gryff said:

 

[ozRocker]

26 minutes ago, gryff said:

@adam: I liked that skeleton viewer - you learn something every day

Here is the LeftShoulder bone data now from three of oz's sources:

(ignore my previous post. editor screws up sometimes when quoting text)

I must've exported at 2 different instances.  I've fixed that so animations.babylon is exactly the same as sandyidle.babylon but without the mesh.

[adam]

7 minutes ago, JCPalmer said:

If someone changed Bone.copyAnimationRange() section to this, better results might happen:

http://www.babylonjs-playground.com/#25M97N#9

[gryff]

@ozrocker: I just look at the data you are posting - and saw the three different sets.  And thought to myself - that adds a complication.

I'm just glad that @adam and @JCPalmer  are around to go deeper into the issue.

cheers, gryff

[ozRocker]

@gryff yeh, this is not a clean process at all.  So many things that need to be done to create these .babylon files for this purpose so there are times when I think I've done the same process but the resulting files end up different.  I'm probably the only person here that's using the Blender exporter and animations to this extent so I understand why there's no simple "1-click" solution.

[JCPalmer]

Yeh, being on bleeding can be messy.  I do not have time to both program & build extensive models.  This clearly makes it worse.  The bone visualizer definitely helps see the problem.  What you are doing is one of my primary design goals, which is could be good for you.

I was just starting to tackle root bone translation & rotation transfer to mesh position & rotation.  Noticed a flaw in re purposing the BABYLON.Animations property within each Bone.  The

I am working in this area right now (along with tackling root bone translation & rotation transfer to mesh position & rotation), but am not hopeful this is going to be solve in the next day or 2 though.