Help with Dynamic Textures

[dbawel]

Hello Babylonians,

 

I hope everyone is having a great weekend, and not working like me.  I have a scene where I'm drawing on objects with textures using decal.  I was drawing on objects using multiple canvas' onto a dynamic texture - however, since this is a real-time multiple user app, I'm not able to create additional canvas' for each user who is on the session to allow unique colors (textures) to be drawn from remote events (the example below is single user and just a basic drawing function and minimal elements.)

 

The issue is of course an expected problem with this new draw function: as I add texture color to my scene objects, it is using memory and decreasing my framerate to the pont where it becomes not a useful tool.  Is there a way for me to use a dynamic texture in this script using the same function (decal), or to instance the textures in a way to avoid memory usage for each pick point (draw)?  I would really like to use a dynamic texture so that I could save out each object's texture image.  However, I was doing this initially, and I have many, many colors (textures), as well as many users drawing together in real time on any object.  This doesn't allow for me to use the canvas to pass color information (textures) to a dynamic texture on each object, as I require multiple canvas' for each remote user.

 

The following is a scene to be copied into the playground.  Unfortunately, it is too many lines to save as a playground scene.  But just copy and paste, and it will run.  

 

var createScene = function () {

var scene = new BABYLON.Scene(engine);

 

var camera = new BABYLON.ArcRotateCamera("Camera", -Math.PI/2, 1.1, 10, new BABYLON.Vector3(2, 0, 0), scene);

//camera.attachControl(canvas, true);

 

var hemilight = new BABYLON.HemisphericLight("hemi", new BABYLON.Vector3(0, 1, 0), scene);

hemilight.groundColor = BABYLON.Color3.Gray();

hemilight.specular = BABYLON.Color3.Black();

 

 

var face = BABYLON.Mesh.CreateSphere("sphere", 3.0, 3.0, scene);

face.scaling = new BABYLON.Vector3(1.0,1.3,1.0);

face.rotation = new BABYLON.Vector3(Math.PI / 2, Math.PI / 5, Math.PI / 2);

face.material = new BABYLON.StandardMaterial("sMat", scene);

face.material.diffuseTexture = new BABYLON.Texture("http://i75.photobucket.com/albums/i309/Athox/Scared_face.jpg", scene);

face.material.diffuseTexture.hasAlpha = false;

face.position = new BABYLON.Vector3(2, 0, 0);

face.material.diffuseTexture.uOffset = 1.0;

face.material.diffuseTexture.vOffset = 1.85;

face.material.diffuseTexture.uScale = 1.1;

face.material.diffuseTexture.vScale = -1.0;

 

var hat = new BABYLON.Mesh.CreateCylinder("cylinder", 4, 3, 3, 5, scene);

hat.scaling = new BABYLON.Vector3(0.4, 0.5, 0.4);

hat.rotation = new BABYLON.Vector3(Math.PI/4,Math.PI/9,-Math.PI/27);

hat.material = new BABYLON.StandardMaterial("Mat", scene);

hat.material.diffuseTexture = new BABYLON.Texture("http://st.depositphotos.com/1000635/3769/i/950/depositphotos_37695355-Seamless-yellow-sponge-texture-background..jpg", scene);

hat.position = new BABYLON.Vector3(2.7, 1.5, 0.9);

hat.setEnabled(true);

 

var draw_mat = new BABYLON.StandardMaterial("draw_mat", scene);

draw_mat.diffuseTexture = new BABYLON.Texture("http://i.imgur.com/f1ARb.png", scene);

draw_mat.diffuseTexture.hasAlpha = true;

draw_mat.zOffset = -.5;

 

var isDown = false;

 

var onPointerDown = function (evt) {

if (evt.button !== 0) {

return;

}

 

isDown = true;

var pickInfo = scene.pick(scene.pointerX, scene.pointerY);

if (pickInfo.hit) {

console.log(pickInfo.pickedMesh.name);

var decalSize = new BABYLON.Vector3(.1, .1, .1);

var newDecal = BABYLON.Mesh.CreateDecal("decal", pickInfo.pickedMesh, pickInfo.pickedPoint, pickInfo.getNormal(true), decalSize);

newDecal.material = dmat;

}

}

 

var onPointerUp = function() { isDown = false; }

 

var onPointerMove = function (evt) {

if (evt.button !== 0) {

return;

}

 

var pickInfo = scene.pick(scene.pointerX, scene.pointerY);

if(isDown) {

console.log(pickInfo.pickedMesh.name);

var brush = new BABYLON.Vector3(.1, .1, .1);

var idraw = BABYLON.Mesh.CreateDecal("decal", pickInfo.pickedMesh, pickInfo.pickedPoint, pickInfo.getNormal(true), brush);

idraw.material = draw_mat;

}

}

 

        canvas.addEventListener("pointerdown", onPointerDown, false);

        canvas.addEventListener("pointerup", onPointerUp, false);

        canvas.addEventListener("pointermove", onPointerMove, false);

 

        scene.onDispose = function () {

        canvas.removeEventListener("pointerdown", onPointerDown);

        canvas.removeEventListener("pointerup", onPointerUp);

        canvas.removeEventListener("pointermove", onPointerMove);

}

 

    return scene;

};

 

Hey DK - is there any way we might increase the # of characters to save a playground scene?  Who might have control over this?

 

Thanks,

 

DB

[iiceman]

I don't really have a solution at the moment, but I tried to get your playground working.

 

The saving shouldn't be a problem, at least it worked for me: http://www.babylonjs-playground.com/#X1VDE

I think we already had a lot bigger playground scenes that worked well, too. Maybe something wrong locally with your PC?

 

One textures is missing since it got blocked by photobucket because of the same origin policy:

 

Image from origin 'http://s75.photobucket.com' has been blocked from loading by Cross-Origin Resource Sharing policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://www.babylonjs-playground.com' is therefore not allowed access.

Edit: fixed in this version by using another url for that image: http://www.babylonjs-playground.com/#X1VDE#1

 

And I got an error on Line 50:

 

Uncaught ReferenceError: dmat is not defined

Not sure where that is supposed to be defined... maybe some piece of code missing here?

 

 

Otherwise it seems to work well

[dbawel]

@iceman - THanks for providing the feedback, as this scene was a bit tempermental at first - for no apparent reason.  This was very odd... If I copied and pasted into the playground using Chrome, the scene won't load and I received all sorts of errors.  However, If I place the post into edit mode, and copy the script while in edit mode, the scene loads without errors - except for a couple of reference errors that I might have expected (perhaps), but the scene does load all assets.  

 

OK - I think I figured it out - when copying, only lines with code are to be copied and pasted.  Any empty lines with space and no code before or following the script body appear to consistantly cause errors.

 

This time, I was also able to save a good playground scene - but cannot explain why, as I received the message that the script was too large to save previously with no changes to the script whatsoever.

 

Here is a playground scene that works now:

 

http://www.babylonjs-playground.com/#1UHBBD

 

So please use this link to test the scene as it creates no errors.  I hope someone can help find a solution to my overhead problem - as well as the other issues as explained in the initial post requesting help.

 

Thanks.

[iiceman]

I started thinking about it and played around in the playground. But seems like I don't really know what I am doing: http://www.babylonjs-playground.com/#9U086

 

I have no clue how to convert the coordinates of the clicked point to the proper coordinates on the plane (or the sphere - even harder). But I think with a dynamic texture like that you should be able to get good performance even with alot of draw calls. Did you already try it out that way? Where did you get stuck and why did you switch to decals?

[dbawel]

Hi iiceman,

 

I was using multiple canvas' for each draw function, and this is very specific to draw on a dynamic texture and doesn't paing on non uniform texture mapped objects well.  Also, since the app is real time multiuser, I require 2 additional canvas' for each multi user that logs on in addition to the ones used locally.  So the overhead becomes unmanageable and breaks and/or slows sown to a snails crawl and is also incompatible with any Gui extension with so many canvsas.  I hope that someone can show be how to use decals with a dynamic texture - as this woud be the best of all worlds.  So that is what I'm asking for help with.  if you read my initial post, it goes into all of the issues to solve, and what would be idea.  I hope you can help - it would be a huge help to me if someone can make a dynamic texture work with my decal draw function.  Otherwise, perhaps I can instance the textures I'm drawing with to lower memory, but I don't know if this is possible.  But either possible solution would allow me to deliver on Monday which is my FINAL deadline.  I've missed everyone for almost a month, as each method I use has a serious problem associated with it - a game breaker.  So I know there is a brilliant mind out there who can save my job on this one.  Otherwise, I'll have to conceed to the BJS Gods that this can't simply be done for real time multiple users.

 

Cheers.

[Wingnut]

Hi DB, good to see that you adopted my "plenty of whitespace" suggestion. heh. Anyway, I have some questions:

 

Have you seen ANY other webGL demos anywhere on the net... where multiple users are editing a texture on a webGL shape?

 

If not, what would make you think that setting a Monday deadline on something never before done with webGL/JS... was a wise or attainable goal?

 

Do you honestly think that BJS is the problem?  Could it possibly be webGL, browser, or general JS that is the problem?

 

Did you have a contingency plan in place for IF the multiple context2D idea... ran into some showstoppers?

 

Has the person who has put you in fear of job loss... been briefed about how bleeding-edge and mid-evolution... webGL is?

 

It is my opinion that if you can get free of the handcuffs that you have somehow gotten yourself locked-into, then some benchmark and proof of concept testing can be done, and more knowledge/insight gained.  Plus, you can also find WHICH area of the project is causing the logjam... webGL spec, browser, JS, or, least-likely, BJS.

 

To repeat something said earlier, don't expect forum users to climb aboard your panic.  I would tend to think that most forum folk are not here to find stress. 

 

Sometimes situations are unavoidable, but, goodness, DB... could you have placed yourself any more perfectly... under a ready-to-fall anvil?  (Like you have time to answer ANY of THESE questions, eh?) 

 

I'm not helping the situation, am I? 

 

Ok, I'm off to do web searches for reasons why dynamic textures (context2d canvi) might be webGL/JS memory/performance hogs.  I'll let you know if/when I find some goods.  I'm on it, but try to get yourself out of that dangerous and fear-infested deadline cage, eh?  It's just plain cruel that you have been placed/forced into that position.... when trying something SO new and untested (imho).

 

http://code.tutsplus.com/tutorials/html5-canvas-optimization-a-practical-example--active-11893  A little old, but even I can almost understand it.  Not sure if anything there will help you, DB... but I'm trying. Here's another interesting one.  http://html5.litten.com/understanding-save-and-restore-for-the-canvas-context/  Wrangling context2d's... sucks, eh? 

[iiceman]

Sorry, have been offline pretty much the whole day... I fear I have no idea how to solve that problem of your, sorry man Drawing decal on a dynamic texture might be the same problem as drawing directly in the dynamic texture since you can't really map the point in 3d space to the proper coordinates on that object, I guess. I don't see why the canvas should be the bottle neck here and why you would need multiple canvas. I still think it should be possible that everybody draws to the same canvas that is then  used as a dynamic texture on the object.

 

Maybe you can manage to make things a bit easier for a first presentable result. Maybe let the users draw on a 2d canvas and then use that instead of drawing directly onto the object? There are a bunch of tutorials for that out on the internet. And using that canvas as a texture should be doable, right?

[JCPalmer]

Rough week to have a deadline. I am operating out of ignorance, but that is not always bad.

First, are you sure that it is not a network number of connections issue? Probably not, but if it has not been eliminated you could be trying to solve the wrong problem.

One thing that might work is to have 1 canvas per connected player, but that canvas is only on their machine? The implication being that at each Decal that is created is then serialized and shipped to others. Might also speed up due to all the code to make a decal. Probably have to fork repository though. I have never really liked the fact that basic meshes and ribbons are statics, and not subclasses. if they were you could just subclass Decals & extend. I looked up decals on a tablet, with small screen. looks like they are meshes. If not never mind. Must be the booze.

[JCPalmer]

Or fork lite. Just steal the createDecal function, and change it, so the decal can be transmitted to another machine

[JCPalmer]

Scrap that. Just make a class of the arts to createDecal (use an id for sourcemesh). Add toString() method & a static fromString(). Also add an apply() to actually carry out createDecal().

Create an instance of arg class in mouse events, toString() & send, then apply locally.

Receivers instance using fromString() then apply as well

[dbawel]

Hello Babylonians,

 

@ Wingnut - I appriciate the concern, but I already have an application (yes, the first one) that allows an unlimited # of users to draw remorely on a 3D object in real time.  

 

@ iiceman - the multiple canvas' are a problem as I need to display all remote user's drawing in their own color - so I have to add a canvas (or more) per remote user to pass thier own unique textues to the dynamic texture.

 

I posted the 2D canvas dynamic drawing function (partially constructed of a function that joshcamas posted earlier) in an earlier post, but changed to the code in this post as decals are so much simpler to use.  So the multiuser function is working perfectly - however, I'm not able to draw on the screen from remote users with their selection of color.  So using decals is a good solution, except the memory overhead is too much and the framerate drops with every stroke.

 

So the server and multiuser is working perfectly.  Please gnore any aspect of multiuser real time as I can implement this onto my node server in a few hours - regardless of the darw function.  If I don't find a solution today, I will simply use the decal function as it's fine for demos - and will solve the memory overhead later.  I appriciate all of the advice and everyone looking at this, and as I promised on this forum 2 weeks (or more) ago, I'll post the finished app.  I'll go head and use as well as post the decal function multiuser real time app this week if I can't find a better solution.  Either way, the app is allot of fun to use.

 

If anyone has any thoughts in the meantime on how I might instance textures, I'd really love the help.  Otherwise, I also have a 2D drawing app on a dynamic texture working in real time without any frame rate loss - but this is limited in it's use across many users remotely.

 

@DK - if you can implement the instance of textures, this would be a huge help to me - providing I can limit my memory usage.

 

Cheers.

[GameMonetize]

Hello, can you explain with the less words possible() what do you mean by instances of textures?

[dbawel]

Hi DK,

 

Perhaps I am asking for something that is already in Babylon.js.  If you look at my playground scene, it places decals textures onto 3D objects.  However, the beginning frame rate is 60fps, and drops dramatically as decals are added.  If I could find a way to use decals as I use dynamic textures to where the memory is not used as color information is added to the dynamic texture, this would be ideal.  So the idea was as objects are instanced, I would like to instance textures - however, I'm aware that the memory usage is not from cache of the texture repeatedly, but in additive resolution to the object's textures.  So perhaps a dynamic texture is the solution, however, I would need this to work with decals - or some function to add the new texture info (color info) to an object's dynamic texture without passing color info through a canvas.

 

I hope this makes sense.

 

Thanks DK.

[JCPalmer]

This has a positive effect for me in PotentialFPS.  Basically just merge all the decals, which can be done cause they are all the same material.  You are still adding vertices every decal, but the number of draws is at least under control.

 

http://www.babylonjs-playground.com/#1UHBBD#1

[dbawel]

Hey DK,

 

I noticed why I'm drawing the frame rate drops dramatically as I draw.  However, when I release the mouse to stop drawing, the frame rate quickly returns to 60fps.  But for me, this doesn't help as with multiple users, someone is always drawing - which keeps the framerate constantly low.  So I hope you can see my dilemma.

 

Any ideas?

 

Thanks.

[jerome]

I didn't read your code so far, but I guess you are updating or building a new texture each frame under the hood. 

This process is really expensive if you manage many textures (or canvas) at the same time.

Take a look at this old post : http://www.html5gamedevs.com/topic/11690-dynamic-texture-feedback/?hl=%2Bdynamic+%2Btexture where I faced the same problem.

 

The workaround I found so far was :

- to use the least canvas/texture as possible => example : draw everything on a single bigger canvas and display only the right needed part

- to update the texture less than each frame (say each 3, 5 or 10 frames depending on how it behaves... for displaying text values, 60 different values per second isn't required : the human eye can't understand anyway)

 

Maybe I will take some time some day to check in the Dynamic Texture Class code whether things could be optimized for the render loop. Some of the BJS functions weren't designed/thought initially to be used within the render loop, so there are still optimization possibilities here and there.

[dbawel]

Hi Jerome,

 

I had reviewed this post recently to look for anything that might help, but am trying to avoid passing textures from a 2D canvas - as I need at least one new canvas per remote user to pass their unique texture to the local user's object.  This requires many canvas', which I haven't found a way to implement or manage without a whole lot of limitations.

 

If you can look at the Babyln.js file to see what I might be able to achieve using the Dynamic Texture Class without passing textures through a canvas, that would be really valuable.

 

Much thanks for considering this.

[GameMonetize]

@DB: so when you click your mouse, does the FPS drop due to drawing to your canvas for the dynamic texture or because of multiple picking?

 

If this is the former, you can choose to update your canvas like just twice per second instead on every frame for instance.

 

What could be useful would be to have a playground with your current performance issue because right now there is no canvas on it

[dbawel]

I'm not using a dynamic texture anymore, as I can't manage a canvas for every remote user.  I'm using the decal function instead, and it appears to be the processing time of the function which draws the decal on the 3D object's texture.

[GameMonetize]

Actually decals are not textures but meshes WITH textures

[dbawel]

I overlooked linking the playground scene using decals to draw on a 3D object.

 

http://www.babylonjs...und.com/#1UHBBD

 

Any changes one might make to keep the frame rare from tumbling?

[jerome]

I'm thinking about this approach ... keeping the mouse button pressed will create hundred of decals, it to say meshes !

Please open the debug mode and keep the button pressed : you can see the meshes number increasing so fast.

 

I don't know if it's the best way to do if you want the users can draw things.

 

Anyway, I noticed that (using FF), if you draw a line and then release the mouse button for seconds, the frame rate re-increases slowly to 60 fps while you don't have too many meshes.

So the decrease seems correlated to both the number of meshes created and the delay between new pointerDown events (something to do with the events themselves or the memory allocated... this needs some investigation)

[jahow]

Hey DB,

 

This topic is intriguing, in a good way.

 

I think we all have some trouble understanding why exactly you switched to decals. As Jerome just said, using decals means you will have hundreds, then thousands of meshes pile up very quickly. Even if you merge those meshes along the way, vertex count will go up in a frightening way. All in all, I really don't think this is a reasonable approach. Decals are supposed to be deleted along the way, not stick around indefinitely.

 

On the other hand, using a Dynamic Texture seems to be the perfect approach. You will need uniformly mapped meshes, which is easy to obtain with a modeling software. The pickResult.getTextureCoordinates had a bug in it but it should be solved soon. This is what I'd call efficient mesh painting (thanks iiceman for the base PG): http://www.babylonjs-playground.com/#9U086#3

 

Now, I understand your problem stems from the multi-user aspect. If I understood correctly what you wrote, your approach was to create one DynamicTexture for each mesh and for each user, and basically display them on top of each other. How about:

- the scene only uses one DynamicTexture per mesh (even better: one big DynamicTexture for the whole scene, though this may be tricky for UV mapping meshes)

- when a client starts drawing on a mesh, a draw order is sent to the node server

- your node server receives all the draw orders and dispatch them to all the clients (except the one from whom the order originated)

- then all the clients' scenes are modified according to the draw order they received from the node server, effectively synchronising all the drawings

 

This is actually more a problem of server-client programming, and my solution is very simplistic, but you get the point. Wouldn't this allow you to use very few DynamicTexture in your scene, and still allow many simultaneous users on your app?

 

Hope that helped.

[iiceman]

That's exactly how I imagined it!!! ... awesome! Really, exactly what I wanted to suggest: one dynamic texture per mesh, everybody paints on that, only the data for the draw call gets submitted, and the coordinates get mapped from the mesh to the texture.

 

I just lagged the skills to do the a proper playground that actually works the way i intended (really good job jahow, I am gonna check that out in detail later to understand what you did there, that is very useful stuff I think!). And I obviously lagged the basic communication skills needed to explain the idea in an understandable way

[GameMonetize]

Can't agree more