Polpatch

Sorry for my absence, i really thank you for all the help that you give me. Unfortunately at this time (between holidays and family) I am not at home and I have not easy access to the Internet. I'm still working on the project, hoping to give you good news soon. Merry Christmas (even if late) and Happy New Year folks!!!

Sorry Guys, I did a mistake...When I've create the list of sourceBox i didn't consider the hierarchy between the boxes. Now i have issues in the initialization of the rotation of sourceBox(i'm not so good ahahaha)Unfortunately all the changes that I apply to an object becomes local when I assign a parent.So, here the list of source box//I tried an avatar kinect similar to the skeleton, but I failedvar listMesh = [ BABYLON.Mesh.CreateBox("source_spineBase", scale, scene), //0 BABYLON.Mesh.CreateBox("source_spineShoulder", scale, scene), //1 BABYLON.Mesh.CreateBox("source_neck", scale, scene), //2 BABYLON.Mesh.CreateBox("source_head", scale, scene), //3 BABYLON.Mesh.CreateBox("source_elbowLeft", scale, scene), //4 BABYLON.Mesh.CreateBox("source_wristLeft", scale, scene), //5 BABYLON.Mesh.CreateBox("source_handLeft", scale, scene), //6 BABYLON.Mesh.CreateBox("source_elbowRight", scale, scene), //7 BABYLON.Mesh.CreateBox("source_wristRight", scale, scene), //8 BABYLON.Mesh.CreateBox("source_handRight", scale, scene), //9 BABYLON.Mesh.CreateBox("source_kneeLeft", scale, scene), //10 BABYLON.Mesh.CreateBox("source_ankleLeft", scale, scene), //11 BABYLON.Mesh.CreateBox("source_footLeft", scale, scene), //12 BABYLON.Mesh.CreateBox("source_kneeRight", scale, scene), //13 BABYLON.Mesh.CreateBox("source_ankleRight", scale, scene), //14 BABYLON.Mesh.CreateBox("source_footRight", scale, scene) //15 ];I created the hierarchy:var InitializeHierachy = function(ListBox){ ListBox[1].parent = ListBox[0]; ListBox[2].parent = ListBox[1]; ListBox[3].parent = ListBox[2]; ListBox[4].parent = ListBox[1]; ListBox[5].parent = ListBox[4]; ListBox[6].parent = ListBox[5]; ListBox[7].parent = ListBox[1]; ListBox[8].parent = ListBox[7]; ListBox[9].parent = ListBox[8]; ListBox[10].parent = ListBox[0]; ListBox[11].parent = ListBox[10]; ListBox[12].parent = ListBox[11]; ListBox[13].parent = ListBox[0]; ListBox[14].parent = ListBox[13]; ListBox[15].parent = ListBox[14]; } I also created some functions to simplify switching between sourceBox / targetBone / sourceJoint etc. This is the function for the positioning of the list of sourceBoxes (with hierarchy), it works:var SetPositionFromTarget = function(boxList){ for(var i = 0; i < boxList.length; i++){ //compute the global position of related targetBone var bone = targetSkeleton.bones[HelperSourceBoxTargetSkeleton(i)]; var boneMatrix = bone.getWorldMatrix(); x = boneMatrix.m[12]; y = boneMatrix.m[13]; z = boneMatrix.m[14]; var positionBone = new BABYLON.Vector3(x, y, z); //in case of rootBox use only global position if(boxList[i].parent == null){ boxList[i].position = positionBone; } //In case of non rootBox compute local position from parent else{ //global position of parentBone var parentBone = targetSkeleton.bones[HelperSourceBoxTargetSkeletonL(boxList[i].parent.id)]; var parentMatrix = parentBone.getWorldMatrix(); x = parentMatrix.m[12]; y = parentMatrix.m[13]; z = parentMatrix.m[14]; var parentPosition = new BABYLON.Vector3(x, y, z); //compute local position var position = positionBone.subtract(parentPosition); boxList[i].position = position; } } } But when I try to insert the (local) rotations through the following algorithm, this does not coincide: var SetRotationFromSource = function(boxList){ for(var i = 1; i < boxList.length; i++){ //global orientation of related sourceJoint var sourceJoint = HelperSourceBoxSourceSkeleton(i); var jointOrientation = sourceSkeleton.jointOrientation[sourceJoint].normalize(); //global orientation of the related sourceJoint of the box's parent (it's so hard to write ahahaha) var parentJoint = HelperSourceBoxSourceSkeletonL(boxList[i].parent.id); var parentOrientation = sourceSkeleton.jointOrientation[parentJoint].normalize(); //compute the local rotation var localOrientation = BABYLON.Quaternion.Inverse(parentOrientation).multiply(jointOrientation).normalize(); boxList[i].rotationQuaternion = localOrientation; } } I colored the various zones to have a greater distinction (the parts are mirrored for kinect):right arm - Greenleft arm - redright leg - magentaleft leg - yellow The rotations aren't random, I think I have to work on the axes but I do not know how. Maybe, is my function that calculates the local rotation incorrectly? for the skype call... my english is very very very very bad

ok, I'm almost done... just one thing, I can not assign the rotations of targetBox (reference coordinate global) in target bone (with its own local reference system). Tips on how to perform the transformation of the reference system from global to local?

Hi Griff!You're right, I said "native of blender" meaning it was a .blend file. I'm in wrong However, I believe that the avatar in the scene is already in T-pose, or at least satisfies the description.

Hi @dbawel and thank you for all the time you are devoting to me!Regarding your question: 1.Yes, the server Kinect (github: https://github.com/wouterverweirder/kinect2) uses exactly WebSocketIO. The server does not have all the features of the Kinect SDK native (eg missing the status of joint), but I can easily make changes. The server sends the following types of frame:Kinect2.FrameType = { none : 0, infrared : 0x2, //Not Implemented Yet longExposureInfrared : 0x4, //Not Implemented Yet depth : 0x8, bodyIndex : 0x10, //Not Implemented Yet body : 0x20, audio : 0x40, //Not Implemented Yet bodyIndexColor : 0x80, bodyIndexDepth : 0x10, //Same as BodyIndex bodyIndexInfrared : 0x100, //Not Implemented Yet bodyIndexLongExposureInfrared : 0x200, //Not Implemented Yet rawDepth : 0x400, depthColor : 0x800}; In particular the structure of the frame-type body (and all its components)://bodyFramebodies: Array[6];floorClipPlane: Quaternion;//body objectbodyIndex: int;joints: Array[25];leftHandState: int;rightHandState: int;tracked: bool;trackingId: int;//joint objectcameraX: double;cameraY: double;cameraZ: double;coloX: double;colorY: double;depthX: double;depthY: double;orientationX: double;orientationY: double;orientationZ: double;orientationW: double;Currently I just memorize (orientationX, orientationY, orientationZ, orientationW) of each joint. 2. Avatar that I use is a native of blender (http://blog.machinimatrix.org/avatar-workbench/) and the initial pose coincides (luckily) with the one you described. I think I understand in outline the process that you have explained, now I have to study it thoroughly.Just one thing, Taking as an example the forearms, saying "at the center of each bone" you mean that the cube should be placed between the elbow and wrist, or exactly in the corresponding joint (in the case of the Kinect in the wrist)? In my ignorance I thought was enough to read the orientation of the joint kinect and add it (with brute force hahaha) to the bone corresponding with the necessary corrections to the rotation.Undoubtledly follow your advice. Thank you so much, I will keep you updated in the coming days!! I owe you a lot of coffee

@JCPalmer no problem ahahah Hi Dbawel, first of all I want to thank you for the great help that you give me.i have almost no experience, just in these months I'm dipping (or drowning hahahah) in this area. I use a local server to send frames kinect to my project via socket.For each frame, I store the original information of each joint in an appropriate object. This object provides a match between joint kinect and skeletal bones of Babylon, the hierarchical map of the skeleton kinect and rotation/position of the joints (use only the position of the joint spineBase for the global position). The "pseudo-code" you mentioned has the task to get the local rotation to be applied to the bone of my skeleton, since I can not apply global changes in the bones Babylonjs (or at least I have not found anything). I tried to study the behavior of a joint orientation to change the reference coordinates and then use the data directly into the bone (with the appropriate corrections to correct the offset between the two skeletons. But for every bone I am obliged to further modify the axes, since the rotations did not correspond to the rotations of the joint kinect.I have now found the right combination to match the right forearm of the two skeletons, but over the weekend I was not able work to find the combination of other bones, I will try this afternoon

Thank you all for the answers!! But I'm working on a specific project and I can not use programs in background for animation in real time, the kinect server for the acquisition of the frame is already an important compromise.. sigh I've got to move my avatar only with the information contained in the kinect frame

Kinect documentation is very poor, but looking around I found that the Kinect uses a right-handed coordinate system.I tried to use your combination on a single bone (right forearm) of the avatar but I have not had a correct reply. I retried once again with this new combination and the forearm moves correctly ....var orient = new BABYLON.Quaternion(kinectOrientation.x, -kinectOrientation.y, -kinectOrientation.z, kinectOrientation.w);Perhaps this could be given by the different local reference axes of each local bone respect the scene. However, your answer made me realize that my reasoning was correct.

Hello everyone!I plan to use the Kinect v2 (that for xboxOne) to move an avatar in my scene.My idea was to gain the quaternions of jointOrientations and then change the appropriate bone matrix. Nevertheless, the jointOrientations are expressed in global rotations (each jointOrientation indicates the direction of the joint in absolute coordinates) while, if I understand it, I can only modify the local matrix of skeleton bone. So I am trying to convert local jointOrientation in local rotation:var joint; //the joint of kinectvar parent = joint.Parent();var localOrientation = BABYLON.Quaternion.Inverse(parent.Orientation).Multiply(joint.Orientation);But I'm having trouble in the transformation of the reference coordinate between kinect joints and avatar bones int the babylon scene... I tried to change the axes by swapping values (x, y, z), but I'm probably wrong var kinectOrientation; //orientation of the joint expressed in quaternionreturn new BABYLON.Quaternion(kinectOrientation.y, kinectOrientation.x, kinectOrientation.z, kinectOrientation.w); //this is just one example, I have tried in different casesDo you have any advice? Thanks in advance

mmmmh... ok, I will study what to do,, perhaps by creating a small box to be inserted in the middle of the button ... sigh Thanks for your help

http://www.babylonjs-playground.com/#TLXTN#24 Indeed, adding a split line between posButton and posButton[i-1] the text remain distorted. The same fate adding other intermediate points in the lines, hahaha http://www.babylonjs-playground.com/#TLXTN#25 Dynamic Texture is the only way to write the text in a Babylon's scene, right?

yes, adding a middle point the corner is ok, but text is choppy.... I can say "It's not a bug, it's a feature" ahahaha http://www.babylonjs-playground.com/#TLXTN#23 P.s. Thanks

Hi guys, I have this type of menu split into section (buttons). I want insert (line 111) a text in each button using a DynamicTexture but, as you can see, the text have a irregularity in the corner (i guess it's due by the ribbon's triangulation).There are other ways to enter text in a Mesh, perhaps with the possibility of indicating the inclination? Here my code (it's just a draft )http://www.babylonjs-playground.com/#TLXTN#21Thanks to all.

k, I did it Thanks to you i understand the potentialof registerBeforeRender() method ahaha I have also changed the way of give the rotation to the bone, his rotation is clean as you can see:) http://www.babylonjs-playground.com/#WBIXK#5 I believed that the method .updateMatrix() would serve to update the value of the matrix, but clearly I was wrong. Thanks again guys... you're saving my life ahahha

maybe i can find here the answer... i'll try it later