Sprites and GUI discussion

[Temechon]

Hello guys, 

 

I'm trying to work on Babylon integration with Cocoonjs. It works great, performance are really good (with Canvas+, Webview+ seems slower) as far as I can tell (and THANK YOU JC for your great help ).

 

As you surely know, a game without a proper GUI/HUD is not really a game, but more a game prototype.

However, with canvas+, there is no DOM support = that means all the GUI/HUD should be in the main canvas. I tried to stack two canvas (one for a 2D library like pixi.js, and the other one for Babylon), but it does not work in canvas+ (a support ticket at Ludeai has been sent).

Another solution is to work with Babylon.Sprite, with an orthographic camera, but there are few features missing : 

 

First, only squared sprites are supported today. You can find a repro case here : http://www.babylonjs-playground.com/#KTGW2, with the original sprite here : https://dl.dropboxusercontent.com/u/17799537/pixi/test.png

With a non-squared sprite, results are very weird.

 

Second, layerMask are not supported by sprites or spriteManager. A solution would be to set the orthographic camera very very far away from the scene, with a fixed maxZ (otherwise the orthographic camera will see all the scene), but it does not seems to work --> repro case here : http://www.babylonjs-playground.com/#YF6KR (or maybe I don't use maxZ correctly, which is very possible).

 

Last, Bitmap font support. A game GUI have a lot of values changing all the times (text and number), and it's not a solution to create a sprite for each value. Bitmap font support would be awesome (see http://kvazars.com/littera/ to create custom bitmap font).

 

To me, as Babylon is more a game framework than a 3D engine, it should be able to do all of this. What do you guys think about all this ? Any other ideas to create awesome game HUD ?

 

Thanks !

[jahow]

Hey,

 

How about a full quad with a DynamicTexture covering the whole view? The DynamicTexture class is awesome in that you have all the flexibility of the canvas drawing functions! And of course you can keep parts of it transparent. I'm not sure how that would work with your particular setup though (no DOM support means no canvas drawing maybe...).

[Temechon]

It is indeed a solution, put I would like to use all babylon features with sprites, like animations for example. Using a DynamicTexture would need to re-code it all. Moreover, all scene.pick features would not work if the dynamic texture covers the whole view.

 

Dynamic texture on several planes could be a solution, but why not using sprites ? They are made for this

 

No DOM support means no HTML element other than <canvas> : that means no <div>, no <img>, ... and of course no css (very little support from Cocoonjs)

[jerome]

Well

I made many tests with dynamicTextures.

They don't suit for high refresh rate of text values : if you need to change many texts each frame, meaning generate and set a new texture for each sprite, performances fall down dramatically.

I used them first in my (pending) project here : http://logiciels.iut-rodez.fr/proto/weathermap/

Initially on each link, there were sprites with dynamicTextures displaying live metrology values : 40 sprites with evolving values.

This simple scene went then so slow, I had to refresh only one dynamicTexture per frame in turn to get back to 60 fps.

 

Finally I kept the dynamicTextures only for fixed texts (switch names) and opted for simple CSS to display metrology values on mouse pointing the mesh.

 

Webgl has no support for naitve text.

Canvas2D has this support but has not that good perfs for text.

DynamicTextures, which is a really good stuff (absent from threejs, needs an external plugin... inspired by the BJS one !) mixes the canvas2D and texture generation : a good way to inject text in 3D, but limited by canvas2D text perfs and texture handling (generation, passing to GPU, GC, etc). Remember that DynamicTexture creates a canvas2D element.

 

So, imho, for now, the place the browser is still the best at managing texts and menus keeps DOM+CSS... for now.

However we can use CSS3D to improve user experience, HUD/GUI behavior and communicate events, objects and coordinate between the Webgl world and the DOM environment.

 

Moreover many mobile frameworks don't want the developer to stack many canvases and allow only one unique canvas: do they allow dynamicTexture underlying canvas2d creation ? don't know.

If they don't support DOM either... well, we are dead

[jerome]

@Temechon

Let us know if you achieve good perfs with many live updated dynamicTexture please

 

http://www.html5gamedevs.com/topic/11690-dynamic-texture-feedback/?p=67519

 

Maybe, should we have a look at Phaser 2D game framework (same web site) : they are used, well, to create games with HUD / GUI and use a WeglRenderer (but in 2D) also and have a support for cocoonjs. Although I don't know if, when in cocoonjs mode, they fall back or not to Canvas2DRenderer.

[jahow]

Ok Temechon, I get your point. And I agree, expanding the sprite system would be a great thing for BJS. DynamicTexture does not have enough features for a complete GUI.

 

Wouldn't it be worth it to create a derived SpriteManager class that would be specific to GUI elements? This way we could offer a simplified API, like this:

var manager = new GuiManager(scene);manager.addElement( "button1", {  x: 100,  y: 100,  width: 300,  height: 50,  textureAtlas: "buttons",  textureIndex: 12,  hoveredTextureIndex: 13,  onClick: function() {    // do stuff  }} );

This class would handle all the rest, including batching the sprites, doing the ortho projection, scaling along the canvas size in different ways (stretching or not), allowing disabling/enabling elements...

 

Personally, I think this would be a great "game-oriented" addition to BJS.

[Temechon]

@jahow, it could be awesome. But the first and second problem (in my first post) should be resolved first

[jerome]

for #1 : http://www.html5gamedevs.com/topic/13070-sprite-widthheight/?p=75049

[jahow]

Yep, and I see DK already made a commit adressing issue #1: https://github.com/BabylonJS/Babylon.js/commit/8ff3d5533218cd5da5351c779cc510444da672f2

 

Temechon, have you tried the latest 2.1 file?

 

About issue #2, is layer mask an absolute necessity for GUI sprites? I'd be tempted to say it is more practical to enable/disable elements manually, for example hide an element if the 'close' button is clicked.

[Temechon]

I didn't see this commit, but it didn't seem to work correctly : http://www.babylonjs-playground.com/#KTGW2#1

My sprite is 200x100px and the final result is really weird :/

 

The layer mask is necessary, because otherwise all GUI sprites will be shown in the main scene  What is intended is to have two cameras, a main camera for the game, and an orthographic one for the GUI. The orthographic one will see only GUI elements, and the main camera will only see game elements.

[jahow]

Oh OK, now I understand. Good point! Now I guess one of us will have to get his hands into the Sprite code...

[RaananW]

Wouldn't using Billboard-Mehes be a solution as well? http://www.html5gamedevs.com/topic/13203-questions-re-sprites/ 

[fenomas]

Hey, I've been thinking about this too. But, stupid question: is just putting DOM elements on top of the game canvas not an option?

[jerome]

I guess it is the best option... if your environment allows DOM elements, other than <canvas>.

Seems not to be the case in cocoonjs  as Temechon said.

[jahow]

@RaananW: an advantage I see of using the sprite system is that it batches all the quads used, and allows color tuning of each. Also it only uses one material. It seems like the right way to go for a GUI. Adding separate meshes & several materials (for example if you want to change the color of one element but not the other) may not hinder performance if it's 10 or 20 quads, but it kinda feels 'cumbersome' in a way.

 

On the other hand, I don't see how to draw a dynamic text on a quad using the sprite system. Maybe a hypotetical GuiManager class could use one 'static' texture atlas (from a file) and a 'dynamic' one, the latter being used exclusively for dynamic text? cf. jerome's experiments with multiple DynamicTexture updates per frame.

[JCPalmer]

@Temechon, I may be a little ahead you in some respects, but behind you in the sprite area.  It is not an accident that I built a 100% webgl dialog extension.  My target platform right now is CocoonJS.

 

The dialog extension handles text as meshes of letters & dynamic numbers as LCD digits.  Both limit GPU / CPU memory using cloning.  Using 2D letters is reasonably efficient, and great quality.  There is a 2nd camera capability just starting in dev.  I got some seeming arcane changes to layermasks to help pull this off. 

 

There is also a mesh wrapper capability so you can integrate any mesh into the Panel layout (That's is how LCD digits are done).  I also have some very specific requirements, and that is how I am going to integrate them. 

 

Also, if you know all of your text at development time, you can trade more memory for higher frame rate.  Just use the fontgen.blend to make signs, which are the letters all strung together as a single mesh.  You can display directly, or put them in panel system, instancing a wrapper.

 

I have thought about how to make small panels visible all the time feasible (I only need dialogs at a time the game isn't being played).  I am thinking about that as I work on the dedicated camera option.  Need to be able to control the size & location.  Things like LCD scoreboards displayed all the time will then possible.

[Wingnut]

To expand on Fenomas' comment, we can also do absolute-positioning.  The html "panel" need not happen above or below the canvas.  It can also happen ATOP the canvas.

http://playground.babylonjs.com/#SVJ9R#4

Pretend that the single html button... is actually a container div... full-of RELATIVE-positioned gui widgets and ani-gifs.  They are positioned relative to the container div... which is invisible, and likely located at screen-center.

So, use percentages for relative placement of screen gems, and let the browser do the scaling for different platforms. I THINK a person could GUI-up every edge and corner of the screen with assorted eye candy, all contained "on" a single absolute-positioned div.   yum.  Never tried it.  Too scared. 

But yeah, if a person doesn't have dom support, then it has to be canvas all the way.  Interesting.  Sorry for the topic drift. 

[fenomas]

I guess it is the best option... if your environment allows DOM elements, other than <canvas>.

Seems not to be the case in cocoonjs  as Temechon said.

 

Ah, gotcha. In that case it might be useful to look at Feathers, a solid UI framework for Starling, a 2D framework originally made for stage3D (Flash's version of webGL) but which apparently now runs in webGL. I don't know if it can easily be used alongside BJS, but if nothing else it would make a good code reference if you're building something similar, since it's pretty mature.

 

 

To expand on Fenomas' comment, we can also do absolute-positioning.  The html "panel" need not happen above or below the canvas.  It can also happen ATOP the canvas.

 

Naturally, I meant "on top of" in z-index.

[GameMonetize]

@Temechon: can you help me understand why you think that sprite.width and height is not working? (And I'm so sad that you did not send me a mail about this bug )

[fenomas]

@Temechon: can you help me understand why you think that sprite.width and height is not working? (And I'm so sad that you did not send me a mail about this bug )

 

Compare the sprites to the same texture on a mesh. 

http://www.babylonjs-playground.com/#KTGW2#2

[Dad72]

I struggle to understand why the graphical interface is not just a layer on top of the canvas with absolute positions as Wingnut suggests.

 

Why use 3D objects, sprites, plan. it is the fixed 2D interface, why put an interface in the 3D world, I do not understand. I've never seen that.

[JCPalmer]

I put mine in 3D, cause I want it to work.  Canvas+ gets a lot of its speed from not having to drag around CSS / DOM.  As I was getting CocoonJS to work with BJS,  I began to be concerned having both a 2D & 3D canvas was going to be trouble.  From my earlier Blender work, I had stumbled onto its font potential.

 

This is my first adventure in computer games, not knowing any better, "I mixed peanut butter with chocolate".  I only need my UI to setup / control my "game" while paused.  This means I can splurge with the vertices & just zap them when done.  Cloning keeps my memory requirements low.

 

My "game" will not run on a desktop, but I can reuse most of it to do a browser based teaser scene, complete with 3D Letters flying around.

 

An alternative to CocoonJS might be BabylonHx.  It sounds like a porting tool though.  If you want do something with accelerometers, camera front or rear, touch screens, or cordova, you can actually develop with CocoonJS.  That stuff sounds like a nightmare to dev on a desktop, then port & pray.

[Temechon]

@Deltakosh - I didn't want to bother you because I didn't know if it was a bug or if it was me  

 

@Ranaan - Billboard could be a solution, but i think it is a painful one. Create a plane, add a texture, scale the place, ...A game GUI will be made of images in 80/90% cases, so why not using sprites ?

 

@JC - Pretty impressive work ! (I liked your "boobies" a lot if you don't mind xD) Indeed, you are far ahead ! But in my mind, using 3D meshes in order to create a HUD is not the right way. I'm sure it fits your needs though

 

@ Dad - It's all about Canvas+ and Cocoonjs without DOM support. Otherwise, i'll pick HTML/img without a doubt (like I did with Shinobomb).

[JCPalmer]

Well no one has ever said they liked my boobies before! (Not sure how I should feel about that. )

 

I was looking at the site on the couch this morning on iPad, and the dog wanted some attention.  He touched the big anchor in the top left before I could stop him, & I went to main forum page.  Dbawel's avatar was on last post of the "Kiwi" forum.  I had never heard of Kiwi.  We were at commercial, so I clicked on it.

 

They are CocoonJS based, so I clicked further into their API docs.  Saw nothing about 2D canvas, so maybe they are using Canvas+.  Went further into their HUDDisplay, and saw methods that looked very similar to what I have.

 

In my system, the base Class that handles all the layout is "BasePanel", 750 lines that took me far longer to get right than I would like to admit.  Everything sub-classes it, so layout can easily be recursive.

There also 2 content sub-classes: Letters & MeshWrapperPanel.  If layermask can be put into sprite, you could maybe write a third content sub-class, like SpriteWrapperPanel.  I do not use sprites to tell if this is a good idea or not.

 

This could reuse all the layout code, allow mix & match, and leverage the dialog camera.

[FlashyGoblin]

Did you check out bGUI? https://github.com/Temechon/bGUI