Multi-threading BABYLON.SceneLoader.ImportMesh

[Hagop]

Let me start by stating that I have thousands of meshes to import into a scene.

Since the following  loop will not work because i is incremented before the callback function of ImportMesh is complete

for (var i:=0; i< max;i++)

{

BABYLON.SceneLoader.ImportMesh("", "products/", productBabylonName, scene, function (newMeshes) {});

newMeshes.position = new BABYLON.Vector3(x,y,z)

}

I solved the problem by creating a recursive loop. Here  is my code

productLocationData (JSON format) & productCount  are retrieved from database using a AJAX-PHP-MySQL call

var Pindex=0;

populateRack ( productLocationData , productCount,Pindex );

var populateRack = function(productLocationData,productCount,Pindex)
            {
                var popRackProdPositionX = parseFloat(productLocationData.products[Pindex].positionX);
                var popRackProdPositionY = parseFloat(productLocationData.products[Pindex].positionY);
                var popRackProdPositionZ = parseFloat(productLocationData.products[Pindex].positionZ);
                var productBabylonName   = productLocationData.products[Pindex].productBabylonName;
                                                                                
                                        BABYLON.SceneLoader.ImportMesh("", "productshr/", productBabylonName, scene, function (newMeshes) {
                                                     newMeshes[0].position = new BABYLON.Vector3(popRackProdPositionX,popRackProdPositionY,popRackProdPositionZ);
                                                                     
                                        });//end import
                         
                Pindex +=1;
                if (Pindex < productCount)
                {
                populateRack(productLocationData,productCount,Pindex);    
                }
            }

Everything works fine. However, if there is a simpler way please advise.

HOWEVER my MAIN question is the following. Can I run many instances of the above code in parallel ? Reason being of course to have everything load faster. I know that javascript is a sinle-threaded language. Could HTML5 web workers be of help here? Or any other plug-in such as parallel.js ? Has anybody applied parallel processing to BABYLON.SceneLoader.ImportMesh. I am aware that Web Workers has been applied to collisions. Does anybody have an example?

 

 

[fenomas]

JS is indeed single-threaded, so the only way to parse multiple things in parallel would be workers. Workers don't share memory with the main thread though, so this would only be beneficial if parsing took significantly longer than copying the memory for the results back into the main thread. Have you measured where the time is being spent currently? If most of the time is being spent waiting for the files to load, then I'd expect the browser already does those in parallel, so code changes might not help much.

Incidentally, you can fix your initial loop with closures:

for (var i=0; i<max; i++) {
    var callback = makeHandler(i)
    BABYLON.SceneLoader.ImportMesh("", somePath, someName, scene, callback)
}

function makeHandler(i) {
    return function() {
        console.log('Value of i: ', i)
    }
}

 

[Hagop]

Thanks Fenomas. Indeed most of the "time" is spent, waiting for one mesh to load before the other. So if i could run multiple  Import Meshes in parallel, then I would save precious time. I would segment my database so that each ImportMesh command accesses  a different chunk of data. I will test the loop closure trick.

[Pryme8]

webworkers can handle this as well.

[Hagop]

Pryme8 Will the worker file be able to access the BABYLON engine and the scene which is a DOM element ?

[GameMonetize]

Unfortunately no and this is a shame

[Hagop]

Bad news. Delkatosh do you have any suggestions on workaround?

[adam]

This could be helpful:

https://github.com/optimalisatie/webworker-preload

[GameMonetize]

this is a constraint from WebWorkers...Not sure how to work around it in a performant way

[Hagop]

Delkatosh. I meant workaround solution on parallel import.

Adam. Interesting link. I will look into it

[GameMonetize]

It is impossible to do parallel things with JS as it is mono thread by essence

[adam]

I've always understood that a webworker runs in its own thread.

It looks like that webworker-preload can be used to load images, js, css to cache in a separate thread.

Here is the example that webworker-preload was built from:

https://gist.github.com/mseeley/9321422

[X3MC2]

On 9/22/2016 at 1:11 PM, Hagop said:

Let me start by stating that I have thousands of meshes to import into a scene

Are you really importing thousands of models in a single scene ? Because that is beyond imaginable, I can't imagine a web browser handling such a tedious task.

If I were you, I would have included all the models in a single .babylon scene and loaded it using the SceneLoader.

[Hagop]

X3MC2, yes indeed, thousands. I am thinking as a workaround as you mentioned to group them is babylon files, store in a database and load them with SceneLoader using the data: command. I will check if Web Worker could be helpful in this scenario as XMLHTTP is supported in Worker.

Adam, how would the webworker-preload example be applied BABYLON.SceneLoader.ImportMesh callback function, which is the main reason for delay?

[adam]

I'm not sure how it would work with .babylon files.

TowerOfBabel exports js files:

https://github.com/BabylonJS/Babylon.js/tree/master/Exporters/Blender

[adam]

I linked to the wrong repo:

https://github.com/BabylonJS/Extensions/tree/master/QueuedInterpolation/Blender

 

[Pryme8]

You would just use the webworker to load the file and then serlialize it to pass to the main thread. If your not familiar with webworkers then it might be tough. Also you could do a recursive function to load.  JavaScript is single thread so you would have to fake it or use webworkers.

[Pryme8]

Also why so many models? Why not package them into groups and load a smaller batch of files then just separate then on runtime.

[adam]

7 hours ago, Pryme8 said:

You would just use the webworker to load the file and then serlialize it to pass to the main thread.

It doesn't appear that you have to do that.  Once the webworker loads the file, it is available from cache.  So as long as you go to access it after the webworker has loaded it, you are good.

 

[adam]

[adam]

I just did some tests with some large images and I didn't see any performance gains using webworkers.

[JCPalmer]

Ok, I agree with Adam that textures & audio are just as important as geometry.  If coming from Blender, you can in-line (base64) textures.  Assuming you have gzip on your server, only a slight increase in total transfer.

As far as using JIT (Tower of Babel in-line source code) as opposed to JSON (babylon file), I am not sure if dynamically added script files in a worker are going to be visible in main thread.  Might even vary by browser. 

I have a tool to generate in-line (base64) audio, but have not committed it.  It is Javascript source code.

 

[Pryme8]

The way I was handeling passing objects with webworkers is always Serialize them @adam, I have a recursive loop function that checks the the objects keys and then that's keys keys and so on an so forth until it's completely a string then you can pass what ever.   A lot of the confusion I had initally with web workers was what data actually needs to be passed back and forth, if you just have the web worker grab the file and stringify it then pass that over to the main thread to an array for a file loading stack you should be good to go and get async loading of objects.

 

and I guess you won't see any performance increase really using webworkers to load a file because that's not really considered a intensive operation.

[fenomas]

I'm lost. Generally the point of webworker is to do things without blocking the UI thread. Loading assets doesn't block the UI thread, so what's the benefit of doing it in a worker?

I mean - the browser has a network stack running underneath the JS VM that knows how to async load multiple assets in parallel. When you load assets from a webworker, the requests go to the same network stack, don't they? All it does is add overhead for copying memory back and forth. Am I missing something?

(Of course if you parsed the results in a worker, that might have some benefit, but that's not what's under discussion here, right?)

 

[Pryme8]

That's what I was discussing I'm not sure what they are trying to do though.