Translating an acm file to BABYLON.Animation

[JCPalmer]

Carnegie Mellon has a whole library of motion capture files for their 31 bone skeleton http://mocap.cs.cmu.edu.  The acm zip file is over 1gb.  It is output in a number of formats.  acm is the only non-binary format, so it should be easiest to work with.

 

I am thinking about a class which will process one of these files into the animation for each bone, after removing as many frames as possible.  What I have so far:

module MORPH{    export class Mocap{        private _boneNames = {}; // format  is "rclavicle": 1        private _animation : Array<BABYLON.Animation>;                constructor(acmFile : string, frameRate = 120, toleranceToStripFrame = 0){            // Todo: read the acm file and parse to members            // remove as many frames as possible, working backwards; final frame must be provided        }                /** Memory efficient reference copying of a Mocap animation to a skeleton.          */        public transferAnimations(destSkeleton: BABYLON.Skeleton, animationIdx = 0){               var destBones = destSkeleton.bones;            var nBones = destBones.length;            for (var i = 0; i < nBones; i++){                var boneIdx = this._boneNames[destBones[i].name];                if (typeof boneIdx === "undefined"){                    BABYLON.Tools.Error("Mocap: not same rig");                    return;                }                destBones[i].animations.splice(animationIdx, 0, this._animation[boneIdx]);            }        }    }}

Make Human can rig using this, so seems to be worth looking into.  One problem is a small # of vertices need more than 4 influencers.  This is not a problem as long as I use software skinning (probably going to have to for mobile anyway), and do not compress my mesh weights.  Tower of Babel does not do this, so still good.

 

Would have to put in a subclass, so applySkeleton could be over ridden (thinking about variable length influencers).  Still Good.

 

Not sure how to take the root element & combine with each bone to make a frame.  Any ideas? Here is the first frame of a file:

#!OML:ASF H:\Terrain\Patient Classification 1\Walking\liu\liu.ASF:FULLY-SPECIFIED:DEGREES1root 8.87208 15.7511 -31.7081 5.5217 4.9122 3.70117lowerback -6.36782 -4.70696 -6.683upperback -0.643012 -5.87124 2.06862thorax 3.78681 -3.04691 5.29674lowerneck -24.0759 -2.27082 -11.0665upperneck 24.0856 -2.48175 11.8592head 12.2401 -0.563692 6.00894rclavicle -2.43511e-015 -1.90833e-014rhumerus -44.9636 -1.75199 -74.7834rradius 18.5852rwrist -1.70884rhand -28.8562 -18.8005rfingers 7.12502rthumb -2.21279 -48.7688lclavicle -2.43511e-015 -1.90833e-014lhumerus -1.92727 -17.3217 82.8203lradius 47.7499lwrist 17.0419lhand -19.6688 -28.7993lfingers 7.12502lthumb 6.65951 1.07601rfemur -38.858 0.527314 17.3997rtibia 30.0014rfoot -4.01149 0.174858rtoes -20.7668lfemur 24.5842 9.53209 -22.1071ltibia -1.90833e-014lfoot -3.73452 -5.77693ltoes -22.05172...

[GameMonetize]

Woot..there are 6 values for root...position and rotation?

[JCPalmer]

Nice guess.  Seems there is also a .asf file that goes with the .amc files.  Here is s stripped down version describing only 2 of the bones :

# AST/ASF file generated using VICON BodyLanguage# -----------------------------------------------:version 1.10:name VICON:units  mass 1.0  length 0.45  angle deg:documentation   .ast/.asf automatically generated from VICON data using   VICON BodyBuilder and BodyLanguage model FoxedUp or BRILLIANT.MOD:root   order TX TY TZ RX RY RZ   axis XYZ   position 0 0 0     orientation 0 0 0 :bonedata  begin     id 1      name lhipjoint     direction 0.692024 -0.648617 0.316857      length 2.68184      axis 0 0 0  XYZ  end  begin     id 2      name lfemur     direction 0.34202 -0.939693 0       length 6.92462       axis 0 0 20  XYZ    dof rx ry rz    limits (-160.0 20.0)           (-70.0 70.0)           (-60.0 70.0)  end  ...:hierarchy  begin    root lhipjoint rhipjoint lowerback    lhipjoint lfemur    lfemur ltibia    ltibia lfoot    lfoot ltoes    rhipjoint rfemur    rfemur rtibia    rtibia rfoot    rfoot rtoes    lowerback upperback    upperback thorax    thorax lowerneck lclavicle rclavicle    lowerneck upperneck    upperneck head    lclavicle lhumerus    lhumerus lradius    lradius lwrist    lwrist lhand lthumb    lhand lfingers    rclavicle rhumerus    rhumerus rradius    rradius rwrist    rwrist rhand rthumb

I am hoping that at least 1 of the formats from this DB can be loaded into Blender.  Then I could generate an export.  This could be compared to my processing to know If I am doing it right.

 

Also, I now think compressed matricesIndices is not really a problem for variable influencers, so long as they are padded out to be an even multiple of 4.  TOB is still not doing to do it though, since I do not want replicate the code to undo it as in line source.

 

I never really like this format difference TOB & JSON variants being handled in pass 1 of  python script.  Going to make pass 1 store all influencers in variable length format (a 0 weight being the delimiter).  Provide 2 functions in both variants which get called in pass 2:

• toFixedInfluencers(nFixed)
• compressMatrixIndices()

No one probably knows what I am talking about.  Just talking to myself.

[JCPalmer]

Update:  I am not working on the Mocap class yet.  I have ran a Make Human mesh though the exporter with the CMU rig, writing of variable # of influencers. I captured some really interesting statistics in the log file:

Avg # of influencers per vertex:  2.361660439260002Highest # of influencers:  7

If this is representative at all, sounds like variable # of influencers would crush fixed length, when computeBonesUsingShaders = false.

 

DK, although TOB always subclasses Mesh to begin with, I would either have to in-line the override the applySkeleton which did variable, or inherit from MORPH.Mesh.  Inheriting when you actually had shapekeys is going to happen anyway, but kind of weird otherwise.  I do not like inlining at all.

 

How about allowing BABYLON.Mesh do either?

[JCPalmer]

Wait the fixed method checks that each weight is > 0 before doing a matrix operation with it, so there is no other advantage to variable than it will actually do things fixed cannot.

[JCPalmer]

have not run this yet, but this subset of applySkeleton was changed to this, then if skinUsingVariableNumInfluencers was read by FileLoader, then any exporter that supported writing variable could use this:

var maxInfluencers = this.skinUsingVariableNumInfluencers ? Number.MAX_VALUE : 4;var matWeightIdx  = 0;var totalInfluencers = 0;var inf : number;for (var index = 0; index < positionsData.length; index += 3) {    for (inf = 0; inf < maxInfluencers; inf++){        var weight = matricesWeightsData[matWeightIdx + inf];        if (weight > 0) {            totalInfluencers++;            Matrix.FromFloat32ArrayToRefScaled(skeletonMatrices, matricesIndicesData[matWeightIdx + inf] * 16, weight, tempMatrix);            finalMatrix.addToSelf(tempMatrix);                                }else break;               }    matWeightIdx += this.skinUsingVariableNumInfluencers ? inf + 1 : 4;    Vector3.TransformCoordinatesFromFloatsToRef(this._sourcePositions[index], this._sourcePositions[index + 1], this._sourcePositions[index + 2], finalMatrix, tempVector3);    tempVector3.toArray(positionsData, index);    Vector3.TransformNormalFromFloatsToRef(this._sourceNormals[index], this._sourceNormals[index + 1], this._sourceNormals[index + 2], finalMatrix, tempVector3);    tempVector3.toArray(normalsData, index);    finalMatrix.reset();}

[GameMonetize]

But with more than 4 influencers per vertex, you have to change the file format, and all shaders supporting bones as well. This is not a simple task.

 

or you just want to do it for CPU skinning?

[JCPalmer]

Thought the .babylon format could stay the same except that matriceIndices would have up to 3 0's at the end.  Changing shaders seems like a high hurtle. 

 

Think setting skinUsingVariableNumInfluencers = true could just force the CPU route.  In a perfect world, they should be separate choices.  Since many people do not build their own rigs though, at least they have some vehicle other than build your own rig.  Maybe do GPU route at a later date.

 

FYI, that code above not perfect. Starting testing.

[JCPalmer]

Note to self & anyone stumbling into this thread in the future:

 

The .acm & .asf file collection is a format called "Acclaim".  The format is documented here:  http://research.cs.wisc.edu/graphics/Courses/cs-838-1999/Jeff/ASF-AMC.html and here http://www.darwin3d.com/gamedev/acclaim.zip

[JCPalmer]

Anyone know what do you specify as an Animation object for a bone that does not participate in a particular animation? Null?

 

The first link says in the bonedata section of the ASF file, if a bone has no Degrees of Freedom, or DOF, that it will have no motion data in the ACM file.  I have observed this in the one animation I pulled out of the zip file.  Some of the bones are missing from the ACM file.  In the ASF file, they have no DOF.  See the 'lhipjoint' bone in the ASF above.

[JCPalmer]

Well, I got the variable # of influencers for CPU running all the way from Blender to the code above, edited slightly.  I now realize that I am varying the number by vertex.  Looking at the vertex shaders & my opengl books, this is simply not possible.

 

Varying by mesh should be possible though.  DK, how does this sound as a template to modifying the shaders to do vary by mesh?

#if N_BONE_INFLUENCERS > 0    // having bone influencers implies you have bones    uniform mat4 mBones[BonesPerMesh];    #if N_BONE_INFLUENCERS < 5        attribute vec4 matricesIndices;        attribute vec4 matricesWeights;    #else        attribute mat2 matricesIndices;        attribute mat2 matricesWeights;    #endif#endif...void main(void){#ifdef INSTANCES    mat4 finalWorld = mat4(world0, world1, world2, world3);#else    mat4 finalWorld = world;#endif#if N_BONE_INFLUENCERS > 0    mat4 influence = mBones[int(matricesIndices[0])] * matricesWeights[0];        #if N_BONE_INFLUENCERS > 1            influence += mBones[int(matricesIndices[1])] * matricesWeights[1];        #endif            #if N_BONE_INFLUENCERS > 2            influence += mBones[int(matricesIndices[2])] * matricesWeights[2];        #endif            #if N_BONE_INFLUENCERS > 3            influence += mBones[int(matricesIndices[3])] * matricesWeights[3];        #endif            #if N_BONE_INFLUENCERS > 4            influence += mBones[int(matricesIndices[4])] * matricesWeights[4];        #endif            #if N_BONE_INFLUENCERS > 5            influence += mBones[int(matricesIndices[5])] * matricesWeights[5];        #endif            #if N_BONE_INFLUENCERS > 6            influence += mBones[int(matricesIndices[6])] * matricesWeights[6];        #endif            #if N_BONE_INFLUENCERS > 7            influence += mBones[int(matricesIndices[7])] * matricesWeights[7];        #endif        finalWorld = finalWorld * (influence);#endif    gl_Position = viewProjection * finalWorld * vec4(position, 1.0);    ...

It would mean that numBoneInfluencers be added to mesh, default 4.  I did not want to start modifying all the files, without your input.  Will just move the CPU skinning to my subclass, if that was the case.

[GameMonetize]

I see no problem with this. Actually this will be even better when num influences < 4

[JCPalmer]

Ok, one question, what about this BONE4 = false fallback on a couple?  Seems like this is never going to work right, except in the case when 3 or less is in use.  This way handles that.

 

Couldn't cpu skinning be the fallback?  Sorry that's 2 questions.

[GameMonetize]

With your system we do not need BONE4 at all actually

[JCPalmer]

Comedy of errors,  (I should do standup). 

• First mat2 is not 2 rows of four.  I switched to mat3, which is a 3 by 3
• If you access a mat by only 1 value, you get a vector not a scalar.  Switched to 2d access, matricesWeights[1][0]
• Made all other changes, but hit a wall with gl.vertexAttribPointer().  Even though you can specify up to a mat4 for a vertex attribute, the max size you can specify is only 4.

I am done for today, but tomorrow think I will do this, unless stopped:

• Add 2 more vertex buffer types (do not think you would like to attributes per buffer)
• Re-do cpu skinning slightly.

This is what the shader would not be:

#if NUM_BONE_INFLUENCERS > 0	// having bone influencers implies you have bones	uniform mat4 mBones[BonesPerMesh];	attribute vec4 matricesIndices;	attribute vec4 matricesWeights;	#if NUM_BONE_INFLUENCERS > 4		attribute vec4 matricesIndicesExtra;		attribute vec4 matricesWeightsExtra;	#endif#endif...#if NUM_BONE_INFLUENCERS > 0	mat4 influence;	influence = mBones[int(matricesIndices[0])] * matricesWeights[0];	#if NUM_BONE_INFLUENCERS > 1		influence += mBones[int(matricesIndices[1])] * matricesWeights[1];	#endif		#if NUM_BONE_INFLUENCERS > 2		influence += mBones[int(matricesIndices[2])] * matricesWeights[2];	#endif		#if NUM_BONE_INFLUENCERS > 3		influence += mBones[int(matricesIndices[3])] * matricesWeights[3];	#endif			#if NUM_BONE_INFLUENCERS > 4		influence += mBones[int(matricesIndicesExtra[0])] * matricesWeightsExtra[0];	#endif		#if NUM_BONE_INFLUENCERS > 5		influence += mBones[int(matricesIndicesExtra[1])] * matricesWeightsExtra[1];	#endif		#if NUM_BONE_INFLUENCERS > 6		influence += mBones[int(matricesIndicesExtra[2])] * matricesWeightsExtra[2];	#endif		#if NUM_BONE_INFLUENCERS > 7		influence += mBones[int(matricesIndicesExtra[3])] * matricesWeightsExtra[3];	#endif		finalWorld = finalWorld * influence;#endif

Overall, things went pretty well.

[GameMonetize]

I rahter prefer this way actually:) Keep up the good work!

[JCPalmer]

Ok, works both CPU & GPU for StandardMaterials.  I have modified all 6 vertex shaders which use bones.  I have been forcing a 4 influencer mesh to say it was 6.  So in addition to testing which something that actually needs more than 4, for the next commit need to add the extra attributes for the other shaders when more than 4.

 

I also changed EffectFallbacks class as follows:

private _mesh : AbstractMesh;private _meshRank : number;...public addCPUSkinningFallback(rank: number, mesh : AbstractMesh){    this._meshRank = rank;    this._mesh = mesh;    if (rank > this._maxRank) {        this._maxRank = rank;    }}...public reduce(currentDefines: string): string {    ...    if (this._mesh && this._currentRank === this._meshRank){        this._mesh.computeBonesUsingShaders = false;        currentDefines = currentDefines.replace("#define NUM_BONE_INFLUENCERS " + this._mesh.numBoneInfluencers, "#define NUM_BONE_INFLUENCERS 0");    }    ...}

I was going to call it in the places BONES4 fallback was added.  Do you think this is going to work?

[GameMonetize]

It should!