blender Kinect V2 skeleton

[JCPalmer]

This topic is a follow up to one of the "what's next?" comments I made.  I have added facility into an addon for Blender, which allows post import operations on MakeHuman meshes / skeletons.  The button highlighted, converts the 160 some odd bone, default skeleton to one made for Kinect2.

The bones are named for the bone tail which lines up to the joints the sensor's output.  There was an old topic about this, but:

• The people are gone.
• They shared nothing but pictures.
• This includes their skeleton, which could have been the reason for all their problems.

Here is a .blend of my converted skeleton.  Any comments would be appreciated, @gryff, @dbawel maybe?  The skeleton units are in decimeters, which are easy to convert the sensor's meters to, but any solution should probably have a units multiplier.  The .blend is already in weight paint mode, seen below.  Just RIGHT click any bone to see its weighting on vertices.

For the rest of this topic, I am just sounding out how I think this might be implemented in 3.2.  DB, for your electronics company work, I think you are going to have time requirements which are too tight for waiting for 3.2.

First, I wish to use this for pose capture, but making changes to the Skeleton class (or QI Skeleton sub-class) to do live animating may actually help debug things & therefore be a twofer.  To this skeleton class, if the bones array cannot be changed to a { [bone: string] : BABYLON.Bone } dictionary for compatibility reasons, I think there needs to be some additional data elements & a method to switch on Kinetics, because calling getBoneIndexByName() 25 times every update is probably slower.

Next thing is, yes bones are basically a local matrix to the bone parent, & kinetics returns data in it's world space,  BUT that is no reason to go all the trouble of converting sensor output to local space (unless doing mocap - more later), WHEN skeleton.prepare() is just going to have to convert it back.  Performance could suck.  Having an alternate  _computeTransformMatrics() seems like the level to branch off to convert from kinect space to world space.  Here is a non-running mock up of all these changes:

public onAfterComputeObservable = new Observable<Skeleton>();
protected _kinectsDictionary : { [bone: string] : BABYLON.Bone };
protected _useKinetics : boolean;
protected _kineticsBody : any;
protected _kinecticsBodyIdx : number;
protected _kinecticsUnits : number;
protected _kineticsFloorClipPlane : BABYLON.Quaternion;

public switchToKinetics(val : boolean, units = 1, bodyIdx = 0) : void {
    this._useKinetics = val;
    if (val) {
        for (var i = 0, len = this.bones.length; i < len; i++) {
            this._kinectsDictionary[this.bones[i].name] = this.bones[i];
        }
        this._kinecticsUnits = units;
        this._kinecticsBodyIdx = bodyIdx;
        
    } else this._kinectsDictionary = null;
}

public incomingKineticsDataCallback(eventData : string) : void {
    var parsed = JSON.stringify(eventData);
    this._kineticsBody = parsed.bodies[this._kinecticsBodyIdx];
    
    // account for right handed to left handed for this here
    this._kineticsFloorClipPlane = new BABYLON.Quaternion(
        parsed.floorClipPlane.x,
        parsed.floorClipPlane.z,
        parsed.floorClipPlane.y,
        parsed.floorClipPlane.w
    );
    this._markAsDirty();
}

/**
 * @override
 */
public _computeTransformMatrices(targetMatrix: Float32Array, initialSkinMatrix: BABYLON.Matrix) : void {
    if (this._useKinetics) {
        this._kineticsTransformMatrices(targetMatrix, initialSkinMatrix);
    }else{
        super._computeTransformMatrices(targetMatrix, initialSkinMatrix);
    }
    this.onAfterComputeObservable.notifyObservers(this);
}

protected _kineticsTransformMatrices(targetMatrix: Float32Array, initialSkinMatrix: BABYLON.Matrix) : void {
    // ...
}

For using this for capture, maybe a method in the bone class say, worldToLocalMatrix(), which could be called by code monitoring the onAfterComputeObservable.

This is my current straw man.  I do not even have the hardware, right now.  Thoughts?

[dbawel]

@JCPalmer-

Do you have the scene available for upload into Blender? Either FBX, STL, OBJ, etc? FBX is preferable for me. Although I'd love it if you have time to define your general process in connecting the Kinect V2 to Blender - if indeed you are doing this. I still use the Brekel python plugin from my good friend Oshri's lab in Switzerland. Man that takes me way back when Oshri and I were in Thailand at a Bhuddist temple capturing Motaro for the Mortal Kombat films.

DB

[JCPalmer]

@dbawel, do you have this, or is there a Blender Python plugin somewhere?

There is a link to a .blend file in first post.  Think Blender can export an fbx from it.   If you could see how well the converted skeleton performs, that would be great!  I want to make the most perfect skeleton possible from MakeHuman for use with Kinect2.  If the skeleton is poor, it sort of puts an upper limit on what can be done.

I have no interest in making a Kinect2 interface for Blender myself, but might use one since it would fit right into my workflow.  My final output will be using my pose interpolation, which is improving (float free).  There is one important set of poses where IK might take a long time / or I might have to have to settle for low quality.  I am quite able to perform it myself using mocap then keyframe reduce.  Finding some mocap, probably from a different skeleton, from the web is both unlikely & never going to translate right.

I found eBay has new Kinect2 sensors at $80, so it definitely is within my budget.  Need to check around first, though.

I have other news on this proposal front, but still need to check some things, first.

[dbawel]

@JCPalmer-

I just looked and can't believe that the Kinect V2 sells for as little as 50.00 US. I spent $200.00 a short while back, so $50.00 is quite a good deal as the SDK can be downloaded for free. I should have waited, although I couldn't have worked on the device when it was released 2 years ago now.

Let me know if you believe it isn't too difficult to create a real-time open port plugin for the Kinect V2. I know I could certainly use such a tool.

Cheers,

DB

[dbawel]

@JCPalmer -

Yes, I've been using the Brekel Pro Body V2 for a couple of years now, as a my good friend Oshri owns the lab which this was developed out of. And the plugin works extremely well and has saved me countless hours of animation. However, I'd like to take the next step and create this as an extension directly into babylon.js since the skeleton is a set and known bone configuration. So if you have any thoughts which might help create this as a BJS extension, then i'd love to hear any ideas you might have. My new job requires me to delve into these areas, and I'd much prefer to work on such tools with others thn entirely on my own.

Cheers,

David

[JCPalmer]

Ok, I ordered my used Kinect V2 sensor, $39.99⁹⁵ & free shipping, Saturday afternoon.  Today, Monday its on my front porch before Noon.  First order of business is to prepare the room.  It really needs it with stacks of stuff, but also I have a motherboard with 3.0 USB, but the case it is in only has USB 2.0 ports on the top / front.  Need to plug-in from the back.  Might also need some sort of adapter, since the end of the sensor cable looks female.  Some picture I saw looks shows a piece I never got.  Investigating, there is an adapter needed to be bought when connecting using USB.  Probably is going to cost me more than the sensor.

I have been combing over the large collection of examples / github projects, as well as the SDK.  There was a javascript interface supplied by the SDK 1.8, but not included for 2.0.  Even that seemed slightly clunky.  If you go down to the comments of this, you have go to a sample program, compile it, and make changes to a config file to make the server part.

The websocket client / server approach, which the "thing" in 1.8 may or may not have been implemented in, seems like the best way to do it.  It might even allow python clients, depending on if a "websocket" is really any different from a socket.  A free standing server program you might launch does not exist in all the things I have seen, so far.  Having to install node.js, even though I have, restricts the usefulness to people who either cannot or do not want to have to install all this stuff.  Seems like too much can go wrong.

One of the repos, seemed to have a reasonable JSON output, but when I asked a question about a JS file referenced in the readme which was missing, I was told the project was proprietary.  I saw one blog, that even mentioned babylonjs (about says he works for MS), that talked about using supersocket in a 22 line console app which wrapped everything in JSON.  There is a REPO based on this, no license, & no actual binary that I can see.  I do not see everyone building their own .net app from source.  I do not know how to do it.

The saga continues..

[dbawel]

@JCPalmer-

If the pic you posted above is your actual device, then it appears you have everything you need to plug in a get up and running immediately. The only item I don't see in the pic is the power supply. And I don't entirely follow your issue with USB 3.0, as you mentioned that your laptop is USB 3.0, and you're aware the Kinect V2 won't work with USB 2.0. However, with the adapter shown in the picture, there is a cable to plug into your standard USB 3.0 port - unless you're missing the unique cable adapter.

But if you are up for developing a plugin / extension to babylon.js from the Kinect V2, then I know allot of developers which will be using such an extension daily. And I'm glad you can see the advantages in using the Kinect V2 over the Kinect V1 - as there is no comparison over the quality which the Kinect V2 provides - in all aspects especially the quality of the texture it captures which can easily be applied to the depth map, as well as so many other uses for the image data.

I'm ready to move forward if you are.

And on a completely separate note, I just purchased the Microsoft Surface 2 laptop, and this is the very best laptop I've ever owned - by far. It's a bit pricy at almost $3,000, however as I need to use it for work, it is entirely worth the price. It's faster than any laptop or desktop I own, and when removed from the bluetooth keyboard, weighs 1.7 lbs. In addition, I ordered a device I'd never heard of before which is the "surface dial" and this device is awesome. I attach anywhere on the screen, desktop, or anywhere in proximity, and can assign it as a controller for practically anything. Working in Photoshop is a real pleasure as I connect it to a variety of tools and it makes my workflow incredibly fast. For $99 (or less), I've found it to be invaluable for me personally. But take a look at the Surface 2, as it is well worth the sticker price - and they were just released a couple of weeks ago. You're looking at a TB of solid state hard drive and 8 GB of video RAM on the NVidea 1060 graphics card. Not to mention the best looking highest quality screen I've ever had on any device. And playing games is unparalleled even compared to my latest game consoles. Unbelievable.

DB

[JCPalmer]

That was not my picture.  There is a "Xbox Kinect Adapter for Xbox One S and Windows 10 PC". I ordered it.

I cloned the https://github.com/peted70/kinectv2-webserver, but I do not think I can build it with Visual Studio Code.  After getting no where, decided to just start with a new / empty .Net core 2.0 app & start adding stuff from the  REPO.  Did not get very far.  As soon as I added some using statements, I got error adding 'using System.Windows.Media'.  After searching web, found it needed a reference to 'PresentationCore', which I found in the repo.  I added the section to my .csproj, but that got rejected.

I don't think this is going to work for .Net Core, which is command line.  Why would you need graphics for a command line program was probably the thinking? .Net Core is the only thing that runs in Visual Studio Code, not ".Net regular", so unless I can find & remove the section of the program that needs this, I am going to have to go the Node.js route. 

[JCPalmer]

First @dbawel, I hope you are OK.  Yesterday on the evening news, the small place you mentioned you live, was indicated as the location of one of the Ventura county fires.  I remembered I had no idea how pronounce it, which triggered my memory when I heard an odd name.

My Kinect adapter unit arrived yesterday, but I resisted doing anything till today.  My back room was a victim of being the last room to clean for a couple of visits this year (never got beyond just dumping stuff there).  That combined with being the entrance point for a large dog made the amount of dirt even larger than I thought.  After cleaning all morning, was ready to play.

After hook up, Kinect Studio did a successful capture.  I have more, but will wait till things firm up.  Having an actual system means there are things I can actually try.

[dbawel]

Hey @JCPalmer-

I actually lost my house in the "Thomas" fire this past Thursday night. Not much left, and just sorting through what Is potentially recognizable. However, I have to be grateful as my family is all OK, and my pets as well. Stuff can always be replaced. After something like this, it really puts my life into perspective. I just happy for what I do have. Thanks for asking and thinking about me and my family.

As for the Kinect V2, my current task is much simpler to begin with, and perhaps @Deltakosh might be able to help. I just need to connect 8 - 10 Kinect V2 units together in an array and use these in an array to detect where an object such as a ball touches the wall with a projector projecting the render canvas onto the wall - and detect where the ball hits the wall to return a relative X,Y position on the canvas to use as a control point. The best example would be that I have objects flying around in the render canvas, and when a ball hits the wall and intersects an object on the render canvas at the corresponding X,Y position, then the object is destroyed. Any thoughts from anyone on how I might link more than one Kinect V2 together to cover a 36ft. X 10ft. projected canvas; and is there currently a limitation as to how many Kinect V2s can be currently linked together in a single array which can register a relative reasonable X,Y position to then affect the objects on the render canvas? I've not done this yet, so I would appreciate any help at all with this.

If I can get this to work, then I will be able to get a large company I'm working for to switch from Three.js to Babylon.js for the future and commit to Babylon.js for all of their online game content in the future; and this is one of the 3 largest game companies in the world which I now haw considerable influence. But only if I am able to deliver content in the coming months to meet their needs. I'm confident I can do this myself, however, I'm not so confident I can get it done in the short few weeks I have to deliver. So please help if you have any ideas or experience in how I might approach this or have any code examples to point me in the right direction.

Again, than you for asking about my family and personal safety, as I barely got out with my family and pets before the house literally exploded. But I just started a great job, and have my family and all I really need to be happy - as stuff is only stuff. And I just bought the brand new Microsoft Surface 2 laptop with the NVidea 1060 graphics card with 8 GB ram and 1 TB of Solid State Drive for memory. It cost almost $3500.00, but so well worth it! This machine is a monster! I've yet to work on a new desktop that can out-perform this beast - and it only weighs 1.7 lbs if you remove the keyboard - and the screen is absolutely beautiful. Even if you can't afford it, buy it - as you'll work that much faster and that much easier that it will easily pay for itself.

Anyway, please let me know any thoughts all you devs might have. I could really use the great brains on this forum right now, as mine are all but spent.

Cheers,

DB

 

[JCPalmer]

That's brutal.

As far as your "Kinect array", think you should start at both ends, and try to see if you can make them meet "on paper", first.  From the hardware side, you clearly need more than one computer.  Each Kinect is going to need a USB 3.  I recall that you can have Multiple kinect systems can be on a single machine. I do not recall the limit, but as you indicate, 8 seems way high.  As production of Kinect stopped in Oct 2017, your use of the word "currently" in relation to the limitation, seems more "final" to me, at the SDK level.  Also, the amount of data transferred for each sensor, could put a big load on each system, limiting that way as well.

From the other side, you clearly want one scene (on one machine ).  Since web sockets are the only way into browser based Javascript anyway, you could listen on multiple ports on a small lan of all the machines. 

Just start filling in what the final scene wants, & what "Kinect offers".  I have looked at the source of the node based solution from the old thread, and see that it is using a single "default" kinect.  I am not so sure trying to get more than one sensor per computer is not a trap.  That's all I got.  I am not doing anything like this.

[JCPalmer]

Alright, I am now ready to say what I am going to do after much exploration.  I am going to have a small body tracking command line program / DLL program:

• Run as a command line program, as a socket server when going to WebGL (implemented later)
• Run as Dynamic Linked Library when animating in Blender.

The output either way is going to be JSON.  Javascript & Python have very fast parsers.

I looked at many things from directly referencing Microsoft.Kinect.dll inside of Blender, to C# & node implementations.  They all required tons of extra stuff:  like nest libraries to handle com objects in python.

I ended up starting with this gist I found, which was not filled up with tons of garbage.  Right now it is has a main, and I am successfully compiling, running, and outputting to console.  Want to break into 2 source files, one with a main & future socket server (console writer for now), and one which does body tracking.  Also need to start using a thread & callbacks.  Here is how the MakeHuman Blender add-in looks below.  (The recording controls are just mockups at present). 

My effort will be to get tracking working for Blender.  Blender is more important to me as I can then use its facilities to key frame reduce. 

[JCPalmer]

Well, I have the preliminary program, generating JSON out to console.  It is a simple 3 source file visual studio c++ project, as below:

• Common ConsoleKinect.h file, 42 lines
• ConsoleKinect.cpp to connect / disconnect to sensor.  It also has a main() which writes JSON to console, 79 lines.  Could maybe be changed to web sockets.
• BodyTracking.cpp which sends JSON of up to 6 bodies to a callback function, 276 lines. One frame below (in decimeters):

{                                                                                             
"floorClipPlane": {"x":-0.069041,"y":0.977668,"z":0.198492,"w":0.000000},                     
"bodies": [                                                                                   
        {                                                                                     
        "id": 72057594037966008,                                                              
        "bones": {                                                                            
                "SpineBase": {                                                                
                        "state": "Tracked",                                                   
                        "location": {"x":-7.602520,"y":-2.450854,"z":12.575587},              
                        "rotation": {"x":-0.038790,"y":0.984983,"z":-0.007158,"w":-0.168086}  
                },                                                                            
                "SpineMid": {                                                                 
                        "state": "Tracked",                                                   
                        "location": {"x":-7.874529,"y":0.989391,"z":12.571926},               
                        "rotation": {"x":-0.038751,"y":0.984069,"z":-0.007366,"w":-0.173358}  
                },                                                                            
                "Neck": {                                                                     
                        "state": "Tracked",                                                   
                        "location": {"x":-8.050920,"y":4.244757,"z":12.405807},               
                        "rotation": {"x":-0.011741,"y":0.983150,"z":-0.042320,"w":-0.177447}  
                },                                                                            
                "Head": {                                                                     
                        "state": "Tracked",                                                   
                        "location": {"x":-8.169784,"y":5.621467,"z":12.831615},               
                        "rotation": {"x":0.000000,"y":0.000000,"z":0.000000,"w":0.000000}     
                },                                                                            
                "ShoulderLeft": {                                                             
                        "state": "Inferred",                                                  
                        "location": {"x":-9.356917,"y":3.143413,"z":12.689640},               
                        "rotation": {"x":0.780255,"y":-0.620334,"z":-0.053141,"w":0.059698}   
                },                                                                            
                "ElbowLeft": {                                                                
                        "state": "Inferred",                                                  
                        "location": {"x":-10.125637,"y":0.933223,"z":12.700338},              
                        "rotation": {"x":-0.504695,"y":0.087254,"z":0.847071,"w":0.141917}    
                },                                                                            
                "WristLeft": {                                                                
                        "state": "Inferred",                                                  
                        "location": {"x":-9.458263,"y":3.967202,"z":12.008181},               
                        "rotation": {"x":-0.085888,"y":0.203032,"z":-0.126433,"w":0.967169}   
                },                                                                            
                "HandLeft": {                                                                 
                        "state": "Inferred",                                                  
                        "location": {"x":-8.712178,"y":4.470363,"z":11.882681},               
                        "rotation": {"x":0.033128,"y":0.206811,"z":-0.471186,"w":0.856806}    
                },                                                                            
                "ShoulderRight": {                                                            
                        "state": "Tracked",                                                   
                        "location": {"x":-6.881767,"y":1.855229,"z":11.495045},               
                        "rotation": {"x":0.905937,"y":0.326226,"z":-0.209925,"w":-0.169664}   
                },                                                                            
                "ElbowRight": {                                                               
                        "state": "Tracked",                                                   
                        "location": {"x":-6.203947,"y":-0.852729,"z":11.262550},              
                        "rotation": {"x":0.950260,"y":0.128880,"z":-0.283492,"w":-0.005223}   
                },                                                                            
                "WristRight": {                                                               
                        "state": "Tracked",                                                   
                        "location": {"x":-6.054850,"y":-3.174261,"z":11.341275},              
                        "rotation": {"x":0.951029,"y":0.025301,"z":-0.306971,"w":0.025948}    
                },                                                                            
                "HandRight": {                                                                
                        "state": "Tracked",                                                   
                        "location": {"x":-6.148480,"y":-3.594989,"z":11.472821},              
                        "rotation": {"x":0.939234,"y":-0.144842,"z":-0.290934,"w":0.110530}   
                },                                                                            
                "HipLeft": {                                                                  
                        "state": "Tracked",                                                   
                        "location": {"x":-8.223284,"y":-2.363104,"z":12.518145},              
                        "rotation": {"x":-0.655357,"y":0.753883,"z":-0.028560,"w":0.036759}   
                },                                                                            
                "KneeLeft": {                                                                 
                        "state": "Tracked",                                                   
                        "location": {"x":-8.740862,"y":-5.240495,"z":13.740103},              
                        "rotation": {"x":0.687393,"y":0.081403,"z":0.693923,"w":0.198332}     
                },                                                                            
                "AnkleLeft": {                                                                
                        "state": "Inferred",                                                  
                        "location": {"x":-8.586180,"y":-5.091630,"z":10.653872},              
                        "rotation": {"x":0.485320,"y":0.539348,"z":-0.490311,"w":-0.482870}   
                },                                                                            
                "FootLeft": {                                                                 
                        "state": "Inferred",                                                  
                        "location": {"x":-8.778518,"y":-4.952463,"z":9.319104},               
                        "rotation": {"x":0.000000,"y":0.000000,"z":0.000000,"w":0.000000}     
                },                                                                            
                "HipRight": {                                                                 
                        "state": "Tracked",                                                   
                        "location": {"x":-6.598404,"y":-2.416044,"z":11.948594},              
                        "rotation": {"x":0.665414,"y":0.693741,"z":-0.206221,"w":-0.182811}   
                },                                                                            
                "KneeRight": {                                                                
                        "state": "Inferred",                                                  
                        "location": {"x":-7.947070,"y":-5.174057,"z":11.059258},              
                        "rotation": {"x":0.815460,"y":-0.261802,"z":0.516198,"w":-0.004877}   
                },                                                                            
                "AnkleRight": {                                                               
                        "state": "Inferred",                                                  
                        "location": {"x":-9.306696,"y":-7.740426,"z":10.477659},              
                        "rotation": {"x":0.751671,"y":-0.248703,"z":0.606808,"w":0.070159}    
                },                                                                            
                "FootRight": {                                                                
                        "state": "Inferred",                                                  
                        "location": {"x":-9.703592,"y":-7.874325,"z":9.320196},               
                        "rotation": {"x":0.000000,"y":0.000000,"z":0.000000,"w":0.000000}     
                },                                                                            
                "SpineShoulder": {                                                            
                        "state": "Tracked",                                                   
                        "location": {"x":-8.020656,"y":3.453608,"z":12.468588},               
                        "rotation": {"x":-0.025380,"y":0.983288,"z":-0.025873,"w":-0.178415}  
                },                                                                            
                "HandTipLeft": {                                                              
                        "state": "Inferred",                                                  
                        "location": {"x":-8.550290,"y":4.696713,"z":11.654633},               
                        "rotation": {"x":0.000000,"y":0.000000,"z":0.000000,"w":0.000000}     
                },                                                                            
                "ThumbLeft": {                                                                
                        "state": "Inferred",                                                  
                        "location": {"x":-8.642012,"y":4.133314,"z":11.844402},               
                        "rotation": {"x":0.000000,"y":0.000000,"z":0.000000,"w":0.000000}     
                },                                                                            
                "HandTipRight": {                                                             
                        "state": "Tracked",                                                   
                        "location": {"x":-6.172046,"y":-4.336384,"z":11.558290},              
                        "rotation": {"x":0.000000,"y":0.000000,"z":0.000000,"w":0.000000}     
                },                                                                            
                "ThumbRight": {                                                               
                        "state": "Tracked",                                                   
                        "location": {"x":-6.375901,"y":-3.567194,"z":10.899231},              
                        "rotation": {"x":0.000000,"y":0.000000,"z":0.000000,"w":0.000000}     
                }                                                                             
        }                                                                                     
]                                                                                             
}                                                                                                                                                         

There are only 4 entry points, which are as simple as possible for Python:

// entry points of ConsoleKinect.cpp
DllExport HRESULT openSensor(char Left_or_Righthanded, char Forward_or_Mirror, char Decimeters_Meters_or_Inches);
DllExport void closeSensor();

// entry points BodyTracking.cpp
DllExport HRESULT beginBodyTracking( void(*cb)(char *) );
DllExport void endBodyTracking();

Will start tomorrow trying to get Blender to first call then use the information. @dbawel, I do not know which reader you need, possible color if you have a special colored ball, you might use BodyTracker.cpp as a template.  Everything that be borrowed from on the web jams all readers into one completely unreadable file, like this.  That is only helpful when you need to use a multi-source reader.  Thank god, I do not need anything but one.

I do not yet have a place where the source lives on the net.  Mixing a c++ project with anything else seems like a bad idea.

[dbawel]

Hi @JCPalmer -

I agree completely - this could become an extremely difficult task due to the source we're working with on so many different levels. And I've only seen successful applications for the Kinect V2 written using Python, as it's the most compatible wrap around language available which has many of the functions necessary already written into the API. Regardless, we're most likely looking at something quite messy if I rquire anything beyond "simple."

And you nailed the features I initially need for my current project. I need to track different colored "bean bags" or small bags with some weight behind them which will be thrown at a projected image on a wall. So the info you've provided gives me a starting place, and I can use all the help I can get in the coming weeks. So far, it appears d-able; however, I have little experience with more than 2 Kinect units, and have used primarily the basic functions clearly written and easily accessible through the SDK. So if you have time to assist, I might really need the help. After this, then I'd like to try and implement a real-time capture volume using more than one Kinect V2 - however, I do know what this will require, and it's quite a bit above my pay grade. But I'll cross that bridge soon - perhaps real soon - but I absolutely need to track 10 - 12 different colored bean filled bags or spheres in order to get the current project finished - and finished on time which is before March 1st.

 I've been looking at all resources online, as there has been a good bit of work done using three.js, but there's not any projects which the developers provide all of the code - which I certainly understand as the wok that's been done is certainly no small task, and definitely worth revenue as many developers using three.js have been asking to use the code. But first, I'm looking for info such as the limits for opening and maintaining a Web Socket connection between multiple Kinect V2s -which I've yet to find out how many Kinects can currently be connected at any given time. The limit used to be 8, however, I'm not certain if this limit still exists, or what work has been accomplished since the release of the V2 and other peripherals which might help maintaining the connections over time. So if you know of any resources concerning these and other areas of new development, I'd love to share whatever I've discovered, and possibly embark on a project to make the most use of depth maps for imaging devices such as the Kinect. One advantage I do have is that I'm part of a group of many of the best engineers and developers anywhere in the world, as we're writing most of the applications for Sony devices; and I personally work on the R&D side of Sony where we get to push the envelope as far as we possibly can. Needless to say, I love my job - possibly more than any job in my recent past. But I can make use of many of these resources as needed, which provides me with advantages few others have in any industry - as Sony is definately leading in the areas of smart phones and most all other entertainment devices.

So let me know what you might think about all of the above, and I'd love to consider what we might be able to pull together in search of new applications few have ever considered designing - yet building. This is why I often work with @Pryme8 - as he has never been afraid to push the barriers of practically anything - and usually if not always able to deliver on his ideas. He just needs someone like me to push him through to the end, as his attention span is extremely low, since he knows he is able to accomplish most everything he sets out to do - and that's a dangerous place in which to allow your mind to live. Anyway, I mentioned him as I'm looking for others to join our small team of "thinkers" who aren't afraid to push way outside of what's currently considered "do-able." And you seem to be living in that world as well; while there are very few of us who are willing to go there. Just let me know what you think, and I'm sure I'll hear from @Pryme8 the moment he reads what I've written above. Not to mention I'll definitely give him a call this weekend, as he might be able to assist with the various barriers I have in front of me which I must solve and put into production within weeks. But that's where I find the most "fun" in any aspect of technology, and where I've been able to accomplish the most. It's places like this where I brought the Matrix to life as well as Gollum - and people often marvel about Avatar - however, most everything we did to accomplish Avatar was already a decade old and developed for films such as the first Mortal Kombat movie among other projects overlooked due to the times or often due to their audience. But it's been exciting for me personally, although it's often reported that we broke technical barriers on Avatar; whereas the real truth is that we had already developed and used the technologies years before Jim ever conceived of the script - but that's another story.

Again, let me know your thoughts as you appear to be interested in looking into this further. It's now after 7pm here in San Diego, and I have a presentation tomorrow which I'm just beginning to build now. I'm just praying that I remember enough to push through the night and "sell" this WebGL framework by tomorrow morning. No more syntax errors, and no more Chrome. Only FireFox tonight as I can't afford any browser errors which aren't purely due to poor development skills. If there's an error, then I need to know it's me...

Cheers,

DB

[JCPalmer]

Well, about to break for xmas.  I have data spilling into blender, though I have to call the functions in the DLL right now by ordinal #.  Think it may have something to do with the namespace, but there is only 4 entry points, so good enough for now.

The data I am getting back is inside a b'mystuff'.  If that was the only problem, I could just chop 2 from the front, and 1 off the back.  The other issue is my escape characters \n \t's are literals.  I hope the Json parser can ignore them, or maybe they just show up that way in the Blender console.  These are not major problems.  This is as far as I got.  Kinda of making notes for myself, as no one is going to read this.  @dbawel, I really did not get a good look at your long note yet. After xmas.

Here is what my print statements to Blender console show (did a couple of line breaks & shortened):

C:\Users\Jeff Palmer\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\MH_Community\kinect_sensor\ConsoleKinect_x64.dll
HRESULT from openSensor: 0
HRESULT from beginBodyTracking: 0
length of data: 4468
b'{\n"floorClipPlane": {"x":-0.066457,"y":0.977163,"z":0.201832,"w":0.000000},\n
"bodies": [\n\t{\n\t"id": 72057594037991404,\n
\t"bones": {\n\t\t"SpineBase": {\n\t\t\t"state": "Tracked",\n\t\t\t"location": {"x":0.227578,"y":-0.296393,"z":1.874896},\n\t\t\t"rotation": {"x":-0.082223,"y":0.993997,"z":-0.003135,"w":-0.072109}\n\t\t},\n\t\t
"SpineMid": {\n\t\t\t"state": "Tracked",\n\t\t\t"location": {"x":0.177004,"y":0.007977,"z":1.876632},\n\t\t\t"rotation": {"x":-0.082180,"y":0.993055,"z":-0.004126,"w":-0.084086}\n\t\t},
...
\n\t\t"ThumbRight": {\n\t\t\t"state": "Tracked",\n\t\t\t"location": {"x":-0.398474,"y":0.266390,"z":1.821667},\n\t\t\t"rotation": {"x":0.000000,"y":0.000000,"z":0.000000,"w":0.000000}\n\t\t}\n\t}\n]\n}\n'

 

[dbawel]

Hi @JCPalmer-

I appreciate the insight, and that you're even looking at the issue. I made it through my presentation at Sony, and am still Senior Engineer in the Electronics division. So I must be doing something right. Let's catch up when you have time. My Wife is in Thailand, so I'm working through the holidays. Thus, if you get bored or feel like you want to touch base over the holidays, just shoot me message - as I want to move forward on integrating the Kinect V2 with babylon.js on a level that's useful. There is so much we could use the Kinect to do, that it's certainly worth beginning an integration. Just capturing the skeleton as a bones controller in real-time would send babylon into a whole new arena for most all users. And this doesn't appear to be that hard to do. So I'll be working on a project for Sony which doesn't utilize the Kinect, however, I'll take a look at the connections over the break when I find time.

Have a great holiday!

DB

[JCPalmer]

We are on two different tracks here. 

 - - - - - - - -

I changed my mind on going directly into BJS for bones for now.   It would take more dev work, since I would have to implement a socket client / server as well, then after I got it do work, I would need to capture the results & make a way to reduce frames in BJS.  Editing is built in for Blender, so I only have to get it there.

If the data for bones proves useful though, it could be a decent starting point for BJS.  If a websocket Server was put in BJS, on say port 519, then program(s) controlling kinect(s) could post 30 times a second on the BJS server.  Having client / server (poster / listener) this way around seems much better.  My notes / comments in the first post of this topic would change to get out of changing BABYLON.Skeleton itself.  If the conversion from global space to local could be made in C++, then you could just assign the values passed to the bones.

- - - - - - - - -

For your bean bag wall thing, I would try very hard to avoid having to "Track" the bags.  I the Kinects were mounted at the top of the wall pointed down ( the clipping plane would be the same as the walls if there is no accelerometer in them), then you might be able to just isolate in the reader program when & where a wall strike occurred.  Only then would you need to bother sending up data to the BJS server.  BJS just listens, and if something comes in, sort of simulate what happens when a mouse click comes in.

[dbawel]

Hi @JCPalmer -

Your insights are quite apparent, and you have obviously thought this through further than I have at this stage. Although I have no excuse, but am still sorting through my burned out house here in Ojai. Thank God I have this week off, or I'd never get to this before the rains come. And it's unbelievably messy now even before any rains.

Your thoughts on not tracking the bags is precisely where I'm currently heading. I don't need to know anything except when the bags come in contact with the wall. Prior to this, there is no need to track where, when, or any other attribute of the bags. I definitely have a clear position in pixel depth when the bags hit the wall, and can track this with simple acceleration/deceleration if necessary. However, I'm certain that the depth map will provide a position in space independent of any physics which will show when each bag meets the position of the wall as the projector is projecting the image. So thanks for your insight into this, as it simply re-enforces where I am currently looking at reliably returning a position for each bag thrown.

And I'm not certain how much time you've spent with the Kinect V2, but We've only got approx. an area 2 X 2 meters where we will be able to reliably track each bags position without artifact problems. So again, your insight re-enforces the approach I'm currently taking to determine if and where a bag makes contact on the wall.

As for future work in tracking objects and more specifically skeletons, 30 fps is definitely enough data to deliver a relatively smooth IK animation. When we began using mocap back in the early 90s (yes, I know many of you were still crawling around on the floor - if even born yet), 30fps was all we had to work with. Only when we were able to increase our fps to 60 fps, it was only due to the increase in processing power. However, it wasn't long after where we achieved 24o fps for the Matrix films in the late 90s, otherwise, there would be no Matrix films. So @JCPalmer - I hope you continue to offer up any thoughts you have in moving this forward, as with the new AR devices coming to market, the Kinect and other devices to capture human bipedal motion will play a much larger role than developers realize now.  I'll keep moving towards the integration of such devices (the Kinect is a first and obvious choice due to its extensive SDK) into WebGL and specifically the babylon.js framework. Thant is, providing Sony allows me to keep moving forward with BJS now that it's supported my Microsoft. However, they've been incredibly supportive thus far - I just need them to realize why we need to continue with babylon.js and avoid the legacy traps of three.js and other frameworks.

So I hope we can keep the discussion and progress moving forward, as I now have some resources, and can make far greater progress than before on my own. So I'll keep it moving forward as long as possible, and hope you stay involved as well. But for now, you've been a huge help, and if I can assist you in any way with this or any other forward thinking innovations, please message me, and I'll let you know what I have available as resources in moving forward.

Cheers,

DB

[JCPalmer]

Well making progress slowly.  I assume the AR devices coming to market you meant was Magic Leap.  FYI, did you see the Rolling Stone recent piece? Kind of a long PR article, but the googles look interesting unless you need corrective glasses.

Am not sure how AR affects this Blender integration project specifically, unless Magic Leap has a Blender exporter though.  They are fairly likely to be relying on gltf, but let them know I'll write them an inline source code generating exporter for one for free .  The exporter is multi-pass, so the thing is already half written!

I have found that I need local rotations to write Blender actions, so I have re-coded some of the BJS Matrix functions into C++.  If this ever gets ported for BJS, then that part is already being done and in it's own thread.

The animation is still all screwed up once in blender.  Just hopefully need to divide and conquer: the Y-Z swap, and establish a baseline location of the root bone with the first frame.  Then subtract the baseline out for all the frames.  Will first do things in python, and move them to the C++ side as they make the picture better.

Speaking of first frames, most of my captures so far are me just running into the scene after I hit the record button.  I need a clean first frame.  Not that I cannot delete frames in front in Blender, but the baseline location is not zeroed when I want it to be.  I have come up with an "Action" pose, where frames get discarded until the finger tips of both hands are wider than the shoulders of the opposite side.  This is done in C++, and I even call MessageBeep() when it is triggered, which is oddly very satisfying.

For multi-body recordings, I have a UI problem.  I want to do the recording, but not assign Blender actions to the individual skeletons until later.  Bringing up a list in Blender is not that easy to understand, at least.  See.

[JCPalmer]

Hey @dbawel, this weekend I put my computer to sleep.  Saturaday, I noticed by the drive light & kinect sensor that it was running.  It was so cold in my back room under electric heat, that I just decided to power down.

This morning sure enough this update needed to completed upon power up.  Things with Kinect changed.  First, before the white light on right side on the Kinect was always on when the computer was on.  Now it is only on when sensor was open & acquiring data.  That does not sound bad, but it is different.  Since I have very little experience with Kinect, I do not know which it is supposed to be.

A less benign issue is when calling AcquireLatestFrame() for the body reader.  Before it would never return an error code.  Now it returns a 0x8000000A (E_PENDING, " The data necessary to complete this operation is not yet available. ") when there is no body detected.  I was checking my return code and wrote a problem to console then ignored the frame, so nothing bad happened except a whole bunch of errors I had never seen before.

I & other examples I have seen, am checking further down in code to make sure there was at least 1 body.  The net effect is the body-less frame just gets ignored at a different point in the code.

Just letting you know.  The color reader also has an AcquireLatestFrame().  It could also just be as a result of rebooting.  I rarely re-boot.  I am using sleep as a poor man's SSD drive, even though I have an M.2 socket on my motherboard.  The grief of trying reload on a DDS is not worth it.

[JCPalmer]

I have not done anything with this in a couple of weeks.  Once the C++ & corresponding python DLL loading was done, then it fell to actually using the data.  It got ugly fast.

Just an FYI, I posted a topic to Blender Stack Exchange asking there.  I doubt many here could deal with Blender at the Python level.  I reused the .blend link of the first post to add a script (in a Text editor window) with one frame of data & one of the failed ways I tried to assign it.