What are my options for UI in Babylon.js?

[stormwarestudios]

Hello all,

I apologize in advance if this question has been asked and answered N times already, but I have searched through the forums, and slogged through the vastness that is Google, and haven't quite come up with an opinionated-enough answer that influences my desires.

My (simple) question is this: What is my best option for implementing UI elements in a Babylon.js-based web game?

My (more complex) question is: Given that I want to provide (1) performant (2) support for responsiveness (handheld, tablet, computer) in a user interface which (3) allows for things like multiply-layered (e.g. controls over render-texture over background) UI widgets, (4) text rendering similar to DOM capabilities, that (5) integrates well with Babylon.js, and (6) is currently supported/actively developed, what are my actual options?

(1) my options are DOM (slow), canvas (better) and WebGL (best)

(2) can be influenced by media query-alikes in order to support different devices

(3) not sure, this sounds like custom witchcraft

(4) why not just use DOM??

(5) bGUI :-)

(6) Not bGUI :-(

I've built a UI library in the past, so the concepts are not foreign to me. But since I'm in the research phase for my own project, I figured I would reach out to the BJS community, and perhaps spark an A vs. B flame war. (Kidding!)

Any thoughts are welcome, so please fire at will

 

[JohnK]

Please do not fire at Will he's a good friend?. Sorry not able to answer question but castorGui and canvas2D (the gui ) should also be considered.

[Pryme8]

Just overlay Dom elements if its basic UI stuff if dynamic then it's best to go with canvas2d

[stormwarestudios]

Thanks guys for responding so quickly. CastorGUI seems to be a good starting place.

 

[Temechon]

You can also use the awesome Canvas2D if you want a full WebGL accelerated GUI. http://doc.babylonjs.com/overviews/Canvas2D_Home

[Pryme8]

Canvas2d is really cool and powerful for dynamic 2d gl stuff.

[royibernthal]

Both bjs and Canvas2D are using WebGL, it's much more ideal to not split your solution between different technologies, it can result in headaches in the future, especially when trying to make your project work on multiple platforms.

I'm not even getting into the performance cost of manipulating DOM, especially noticeable on mobile (added to the already very performance expensive WebGL).

From both development and performance point of views, Canvas2D would be the better choice in my opinion.

(2) It supports responsiveness, and even if it didn't, it would've been fairly easy to implement on your own.

(3) Canvas2D allows such complex UI, as far as I know. In addition, it can also be integrated as a 2D object within the 3D scene.

(4) I'm not completely sure about that one, but I believe text rendering capabilities probably are similar to DOM capabilities. Perhaps someone else can answer that question better than me.

(5) A no-brainer, Canvas2D is practically a part of bjs and therefore integrates with it very well.

(6) It is currently supported and actively developed.

 

As you can tell my opinion is completely biased towards Canvas2D and I didn't even bother comparing it to the other available solutions, as I truly believe it is the best choice when developing in bjs.

[GameMonetize]

No brainer: Use Canvas2D

[Nockawa]

20 hours ago, royibernthal said:

Both bjs and Canvas2D are using WebGL, it's much more ideal to not split your solution between different technologies, it can result in headaches in the future, especially when trying to make your project work on multiple platforms.

I'm not even getting into the performance cost of manipulating DOM, especially noticeable on mobile (added to the already very performance expensive WebGL).

From both development and performance point of views, Canvas2D would be the better choice in my opinion.

(2) It supports responsiveness, and even if it didn't, it would've been fairly easy to implement on your own.

(3) Canvas2D allows such complex UI, as far as I know. In addition, it can also be integrated as a 2D object within the 3D scene.

(4) I'm not completely sure about that one, but I believe text rendering capabilities probably are similar to DOM capabilities. Perhaps someone else can answer that question better than me.

(5) A no-brainer, Canvas2D is practically a part of bjs and therefore integrates with it very well.

(6) It is currently supported and actively developed.

 

As you can tell my opinion is completely biased towards Canvas2D and I didn't even bother comparing it to the other available solutions, as I truly believe it is the best choice when developing in bjs.

For the (4) let's say I try the best I can to reach the best rendering quality for text because I know it's crucial for everyone.

@stormwarestudios Canvas2D is a 2D Engine built 100% in WebGL using the babylon.js stack. It was designed/built to lay the foundation of a GUI library, but the foundation itself already required a lot of time to achieve the many goals set from the start. I'm starting since the beginning of the week to work on the GUI library, as you can expect, it won't be out soon, I intend to do something as rich as Canvas2D itself and with the best performance possible (nobody would want the GUI to take half of the CPU/GPU time, so I have to be cautious on this).

So for you, I guess the best is to start like the others did: learn the foundation feature, try to built some custom UI on the top of it. If you can live with it through time: that's great. Otherwise in a couple of months (I hope) you'll have a higher level library coming out with all the features (well, most of them) one could expect to write GUI for gaming/pro apps.

To conclude I'd like to say that both Canvas2D and the GUI lib are ending up as considerable amount of code, even if I tried to make it as tight as possible, there are so many possibilities, so many code paths which makes it inevitable to have bugs here and there. I try to be as reactive as possible to fix them and above all, I try not to make major changes in the engine itself since many weeks now. The major missing feature is to support property the Device Pixel Ratio for you.

If you have questions or want some design guidance, please tell about your project I'd be glad to think about it and how it should be done, it's a good exercise for me!

[fenomas]

Not to be the lone dissenter, but I'd say hack up what you need in the DOM. Assuming you're already familiar with html/css, it'll be faster and more powerful than any of the alternatives. If you later find it's not performing well, rendering components in webgl will still be there as an option.

I mean, if you want to learn a new library, I'm sure the above options are great. But if you want the shortest route to having a UI I expect that'd be the DOM.

[Pryme8]

Ill upload a cool example of a dynamic DOM menu, that works really well...  give me a little bit here.

[stormwarestudios]

It's great to hear the positive feedback, and thank you all for chiming in. Glad to see such a supportive community :-)

@Nockawa I'd like to know more about this GUI library you're working on. I'm in the process of building designs and mockups for what I want to accomplish, but I can share with you what I've got so far.

Some of the key elements I'm looking to build are:

1. morphable dialog/modal whose content is dependent on dimensions
2. using flexbox/grid-inspired layouts for dialogs (e.g. static corners, repeating edges, tileable background)
3. subscribe simple UI elements (text, slider, etc, anything numeric/string based) to a datastore, which update on datastore change
4. rendering non-primary scenes overlaid on UI widgets (e.g. alternate camera views, radar view)
5. heavy usage of image, text, navigable menus, objects (a la #4) renderable in a "wiki" style interface

I'd be really interesting in collaborating as a way to contribute, if this is interesting to you. Please let me know how I can get involved!

[Pryme8]

if you look at the menu on http://localhost/NeatFlax/Tutorial_2/

I made that in like one hour... and I have tons of other ones that feature stuff like you said, perhaps I should put it all together and make something for you guys.
Ive done dragable elements, all standard UI elements like sliders and color pickers, different menus being able to be active and hidden ect.

The flexable elements would be easy, it would just be a little bit of my flexText library and modifying it to do all elements.  If you would like to see more examples of UI stuff Ive done let me know.

[Nockawa]

5 hours ago, stormwarestudios said:

It's great to hear the positive feedback, and thank you all for chiming in. Glad to see such a supportive community :-)

@Nockawa I'd like to know more about this GUI library you're working on. I'm in the process of building designs and mockups for what I want to accomplish, but I can share with you what I've got so far.

Some of the key elements I'm looking to build are:

1. morphable dialog/modal whose content is dependent on dimensions
2. using flexbox/grid-inspired layouts for dialogs (e.g. static corners, repeating edges, tileable background)
3. subscribe simple UI elements (text, slider, etc, anything numeric/string based) to a datastore, which update on datastore change
4. rendering non-primary scenes overlaid on UI widgets (e.g. alternate camera views, radar view)
5. heavy usage of image, text, navigable menus, objects (a la #4) renderable in a "wiki" style interface

I'd be really interesting in collaborating as a way to contribute, if this is interesting to you. Please let me know how I can get involved!

1. and 2. there's possibility to plug a Layout Engine to a Primitive, so far there's the Canvas Layout which lets you position with absolute coordinates your prims or to rely on alignment/margin. I've also made a StackPanel Layout, started a GridLayout and planned to make a Responsive Layout like Bootstrap's one.

3. This is the part I'm working on right now: data binding, you'll have enough to create a relation between your models, optionally an intermediate object (ViewModel, Controller, whatsoever) and the View itself (the Control). I intend to provide an abstract class with the core feature implemented (property notification when changed, data binding capability, etc.) for you to code a GUI architecture quickly. The Data Binding feature will be pretty powerful with complex path to property (like Address.City.PostalCode), supporting collections, type conversion, string interpolation, different data source (dataSource of the Control, custom data source, UIElement as Data Source), two ways, one way, one way to source, explicit update of data, etc.

4. I'm not sure I understand this point, but if you want to render a 3D Scene inside a UI Element, you can simply create the scene, render it into a texture, then use this texture in a Sprite2d to render it on screen as a part of the UI. Concerning a Radar View: Canvas2D gives basic shape based primitives to realize such kind of things. I think that you'll miss the Polygon Primitive that I still not have written yet, but this one could be coded pretty quickly considering its closeness compared to Lines2D and Rectangle2D which are already available.

5. I plan to be able to support such kind of usage. Canvas2D relies heavily on caching strategies to cache static content (part of with composition, totally, etc.) to allow complex composition of primitives.

And at last: contributions of any kind are more than welcomed! People already help on this forum, which is cool, and if some can help on the code, it's even better!

I think a first contribution for you that would be ideal would be to code the Polygon2d class by cloning and existing one (say Rectangle2D or Lines2D), there's a documentation about writing your own Primitive Type sadly it's not up to date, but if you're interested I can make it up to date. 

You could also work on a new Layout Engine if you're interesting.

All in all you're two months early compared to my roadmap, if you can wait such time or start contributing I'll make sure you reach your goal, it's pretty close to mine anyway and I have two months to realize it.

[Nockawa]

19 minutes ago, Pryme8 said:

if you look at the menu on http://localhost/NeatFlax/Tutorial_2/

I made that in like one hour... and I have tons of other ones that feature stuff like you said, perhaps I should put it all together and make something for you guys.
Ive done dragable elements, all standard UI elements like sliders and color pickers, different menus being able to be active and hidden ect.

The flexable elements would be easy, it would just be a little bit of my flexText library and modifying it to do all elements.  If you would like to see more examples of UI stuff Ive done let me know.

it's going to be difficult for us to reach your "http://localhost"

[Pryme8]

Woo haha that was supposed to be

https://pryme8.github.io/NeatFlax/Tutorial_2/

[Nockawa]

1 minute ago, Pryme8 said:

Woo haha that was supposed to be

 

http://pryme8.girhubio/NeatFlax/Tutorial_2

still not working

[Pryme8]

Fixed it my phone was being a pain, also while I have you why would all the scene2d stuff come up as undefined?

 

its for the canvas at the bottom of the page I known it says vector2 instead of size but this was me troubleshooting it trying to get it to work because it would sometimes say BABYLON.size is not defined

[TheTrope]

Hi !

Does canvas2D work with VR ?

Edit: Yes it seems

[Nockawa]

5 hours ago, TheTrope said:

Hi !

Does canvas2D work with VR ?

Edit: Yes it seems

I couldn't have time to test, but I know I did changes for that, did you successfully tested it on VR? (just for my information)

If it doesn't work I'll make sure it is as soon as someone needs it. I've got a Vive just for that!  (well, officially)

[krum110487]

I am a bit inexperienced, but from my limited research, it seems using canvs2d on top of a 3d canvas inherently requires 2 canvases (I may be misunderstanding this process, I apologies if I am).

would it not be better to simply use a layer mask to represent an orthogonal camera on top of a 3d camera? I suppose the benefits of 2 canvases is they will run in different threads? or what?

Mostly just new and curious as I am playing with layers and rendergroups now to learn all about layering.

[Nockawa]

4 hours ago, krum110487 said:

I am a bit inexperienced, but from my limited research, it seems using canvs2d on top of a 3d canvas inherently requires 2 canvases (I may be misunderstanding this process, I apologies if I am).

would it not be better to simply use a layer mask to represent an orthogonal camera on top of a 3d camera? I suppose the benefits of 2 canvases is they will run in different threads? or what?

Mostly just new and curious as I am playing with layers and rendergroups now to learn all about layering.

Hello @krum110487 ,

I think you missed something or there's a misunderstanding. You can create a Canvas2D using the same HTMLCanvas than the 3D Engine. Examples such as this one: http://babylonjs-playground.com/#2AVSFH#35 demonstrate it.

Maybe you can explain to me a bit more what you're trying to achieve and then I'll be able to help and clarify things.

[krum110487]

Yes it seems I did misunderstand it, I looked up the documentation on "ScreenSpaceCanvas2D" and it makes more sense now, I was googling the wrong things.

I assume the advantage to this is you can use the 2D functions on that layer, instead of having to use 3d or sprites to make it work as a GUI? I was playing with this currently:

http://www.babylonjs-playground.com/#CSAVV#6 (Test code, please don't judge me :-))

notice how the sprites on the right in the foreground aren't moving at all.

What are the limitations to ScreenSpaceCanvas2D?
Vs the limitations of what I was doing in the trivial example above?

Thanks,

[Nockawa]

2 minutes ago, krum110487 said:

Yes it seems I did misunderstand it, I looked up the documentation on "ScreenSpaceCanvas2D" and it makes more sense now, I was googling the wrong things.

I assume the advantage to this is you can use the 2D functions on that layer, instead of having to use 3d or sprites to make it work as a GUI? I was playing with this currently:

http://www.babylonjs-playground.com/#CSAVV#6 (Test code, please don't judge me :-))

notice how the sprites on the right in the foreground aren't moving at all.

What are the limitations to ScreenSpaceCanvas2D?
Vs the limitations of what I was doing in the trivial example above?

Thanks,

The SpriteManager is an older feature of the 3D Engine, designed to render what we call billboard sprites (a 2D Sprite always facing the camera, but rendered in a 3D Scene). This is a technique which is typically used to render things like trees or grass really simply/quickly. So the intent of this feature is really to display a 2D bitmap in a 3D world.

Canvas2D is a whole brand new feature that is a 2D Engine, 100% powered by WebGL and using the babylon.js core stack. You can use this feature with a ScreenSpace Canvas, typically to draw UI content of the 3D Scene or any kind of 2D stuff that has to be parallel aligned with the screen. Or you can create WorldSpace Canvas which is the same thing, but rendered in a rectangle which is part of the 3D Scene (like this http://babylonjs-playground.com/#1BKDEO#22)

With a Screen Space Canvas, you can do a lots of things, it's a whole 2D Engine by itself, I suggest you to take a look at the documentation overview home page to get more information.

If you want to achieve the same thing as In your example but using a ScreenSpace Canvas, take a look at this pg: http://babylonjs-playground.com/#20MSFF#44

The Canvas2D feature is far more advanced than the Sprite Manager one, but they don't target the exact same purpose (even if you can achieve what the Sprite Manager does with a ScreenSpace Canvas and the Tracked Node feature, like this: http://babylonjs-playground.com/#1N9RJY#5 )

So all in all, it depends on what you're trying to achieve.

[JCPalmer]

If you wish your text to be formatted into hierarchical Dialogs / Forms yet still be Mesh(es), you can also use the Dialog extension.  I do not have documentation, but do have a test scene crammed with all features.  It is a bit slow with everything in one dialog, but most do not require such an amount.  Here is html format, and here is repository format of test scene.  Dialog has been updated for BJS 2.4.

If you just want like a sign you can animate in 3D space, there is a .blend file which can make signs, then export.  It is also the .blend I use to make the font letters.

I use DIALOG when I want the UI "in" the scene, and DOM when it just sits beside.