use Canvas2D as UI, can other 3D mesh display above on this UI??

[MasterK]

not find the method...

[chicagobob123]

Let me understand, do you want transparent 3D layer above the 2D layer?

The problem is when you click the mouse you wont be able to have the proper ZOrder. It will hit the 3D layer first. 

That is why you always see interface elements on top of a 3D scene. 

NOW, if you turned your 2D Canvas into a 3D mesh and replace the text with 2D text and buttons with 3D Meshes then you could

hit test that and do what you want. 

There is Text2D that is out there to make the text. And buttons are rounded corner squares. 

You know that is one odd thing. Has anyone come up with a library of 3D interface components that mimic the 2D interface world?

Things like buttons combo boxes and such. I guess since you can overlay them on 3D its not a big deal? 

Just some random thoughts. 

 

[Wingnut]

Good answer, and cool thoughts, cb.  I've DEFINITELY thought about what 3D gui widgets would look like.  But why copy 2D sliders, closers, buttons, etc, when you can click on a beer can, slide a snail, drag and drop a mushroom?  A guy could accidentally limit his creativity, here, by not being able to shake-off old ways.    Anyway... check this out, guys.

http://babylonjs-playground.com/#1BKDEO#31

Fun!!!  Lines 6/7 let you position and orient that worldSpaceCanvas... like a mad man.    Notice particles never go in front of Canvas2D things?  Not a bug.  Magic, gone wrong interesting.    It LOOKS like particles go in front, but we are seeing-through the 3/4 transparent world-space rectangle fill.  We are viewing particles that are rendered behind it.  Particles are unique in this way.  No issues with rendering-depth for normal mesh like ground and fountain.

Let's increase the alpha level in line 9... to FF...

http://babylonjs-playground.com/#1BKDEO#32

Yep, the particles are behind.  Again, not a bug... don't anyone report it as so.  It is a "phenomena". 

[chicagobob123]

Wingnut yeah but restricting a slider to move left to right and give proper feedback to the position etc Thats not just make a new thumb image. 

Combos have drop downs and sliders on the side. Particles could be used to be able to show two different UV's so the button can look like its clicked. One texture, On Down you move the UV .5 to expose the other half and you get that traditional feel. 3D artists could have a field day. Imagine a Dragon that had several images so it looks like its moving as a scroll bar thumb. Wild thoughts for fun but I wont have much time to ever do them. 

[chicagobob123]

You could even make a spinning elongated cube show different buttons on each side. The thoughts are endless. 

[GameMonetize]

You can use multiple scenes: one scene for the canvas2d and the main scene and a 2nd scene on top of the first one

[Nabroski]

i think he/she just not discovered  WorldSpaceCanvas2D jet.
http://www.babylonjs-playground.com/#4LXMG#0

[MasterK]

thank u for different options. I should describe more details.

I use 2d things as ui. And on the 2d ui,like a hero upgrade things. There is a 3D hero display in the center. So 3D hero covers the background ui.

And when I rotate the hero the ui is always face to camera.

that's what I mean.

maybe 2 scenes fit for this?

2 scenes means 2canvas and 2 engine?

 

worldspaceCanvas will move with camera, and can't use layermask.

[Nabroski]

Hey man, i dont want to upset you, but this is the best thing i read on this forum.

Thank you

[Wingnut]

Ohhh... I think I finally understand.

How about this, @MasterK...

http://babylonjs-playground.com/#1KRMMG#5

I was thinking "above" meant "in-front-of", but nooo.  You said it correctly the first time.  Duh, Wingnut!  I think I got it, now.

See line 91?  We return an object, with two properties...  canvas and node.  Now look way down at line 221.  I parent the node to the camera.  That's ONE way to keep a world space canvas ... docked to screen bottom, eh? 

I built a cool new button generator func at line 17.  It requires quite a mouthful of salad parameters to make it start running.  It's called repeatedly at lines 67-74.  Purina button chow! 

How we doing now, MasterK?  I bet THIS is what you were wanting.   Yep, yep, yep.  If you swing the camera around a bit, you may see the ground interfere with the gui.  That is because the ground is BIG, and the gui has a z-position of 300 (in line 4).  Even though the World Space Canvas is parented to the camera, it is still 300 units away from the camera, I believe.  Some experimenting and tweaking might be needed. 

We also might need to look-into scaling-upon-screen-resize, too.  And defaultColor in the docs (text2D), really means defaultFontColor.  The system is still evolving, so I guess we'll let that slide, for now.  I think I'll ping  @Nockawa to come visit, if only for his good company.

Hope this helps.  Party on!

[MasterK]

7 hours ago, Wingnut said:

How we doing now, MasterK?  I bet THIS is what you were wanting. 

It seems great! Thanks.

I'll study.

[MasterK]

@Wingnut

It's a little hard to set exact width and height by using z value of canvas2d...

Now i know i should use worldSpaceCanvas.  And my screen is 1136*640, i use a arcCamera, worldCanvasNode's parent is the camera, like you did. How can i fit a 1136*100 UI on the top of screen?

if z is too big, the ui will always behind the other 3d meshes.

if z is little, scale worldspacenode, content will blur...

if use 114*10 as node size, position will not very simple...

[Wingnut]

Yes, I agree with all.  Canvas 2D is new, still evolving, and we need more time and experiments to see when/where it is useful.  Let's look at a "viewports" method.

http://playground.babylonjs.com/#1VXHZA

That is ONLY a picture in the lower viewport.  But what if that lower panel was an HTML DIV?  (placed inside the canvasZone div, above/below the HTML canvas element).  Could your project use HTML "form widgets" to create your UI?  By using an HTML "form" in that lower viewport area, you get the clicking and dragging features of HTML, plus you get HTML "flow" or absolute positions, plus CSS positions, paddings, margins, borders, etc.  There are lots of advantages to using HTML for your control panel.

But I heard rumor that Android devices using CocoonJS... can't access the DOM, so HTML UI doesn't work there.  That is one of the reasons that our friends coded CastorGUI, bGUI, and Dialog.  If you use HTML for the UI, you probably say goodbye to these devices.  darn, eh?

There's another way.  Look at this.  http://playground.babylonjs.com/#13TVWJ

This time, a camera view is in the bottom panel.  In fact, the 2 thin blue lines are also viewports and camera views - of a blue plane placed at -500, 0, -500 (see line 74).

This type of system allows you to build a UI panel out of mesh.  You would manually "model" buttons, labels, readouts, sliders, listboxes, etc... out of mesh.  When done, you shoot a camera at them and put the camera view in the control panel.  Not an easy task, whatsoever, and the UI is certain to look "different" and have far less functionality than other UI.  Yet, it CAN be done, and it is super-cross-browser because... the UI is webGL.

There is no "perfect solution" in UI for scene controls.  At least I can't think of one.  If I had to build a control panel for a scene, I would use HTML (so far).  Canvas2D certainly has a great future to do such things, but maybe, not yet.  Still needs "evolve time", perhaps.

Others have suggested multiple BJS scenes and/or multiple BJS engines/canvas.  I don't know how many of our issues... that would solve.  Thoughts?

I think... the best UI we have "near" our webGL canvas, so far, is HTML.  I don't know if we can incorporate XUL or XAML.  Generally speaking, Firefox uses XUL to make it's UI widgets.  IE might use XAML as its UI widget maker.  SO, the chances to get XUL or XAML working cross-browser... is slim.  And using XUL or XAML for mobile devices?  Probably not easy or perhaps impossible.

I wish I could give you more options and hope.  Thoughts?  Maybe others will visit and give opinions.  I hope I didn't tell you anything incorrect.  I'll keep thinking.

 

[MasterK]

@Wingnut

I use The html things in our last project, I made a library of UI but not CastorGUI or bGUI(because they are not satisfy our need).  And i use 2 canvas/engine/scene to handle front/behind relationship. That's ugly but finally solve our problem.

Now we have a new project so i dont want to use HTML in core game, (maybe some not important place can also use html). And as i said html and one canvas cant handle position-z questions. If html can do, ScreenSpaceCanvas also can do.

I have a thought, use two camera, one is orth**** camera,  to set worldSpaceCanvas, so that any z-postion have same look. And use a free or arc camera to display 3d meshes. Use layerMask to separate them. Then ui and 3d mesh can mixed.

I dont know if it's ok. Maybe not.

[MasterK]

PLUS, html image has cross-origin problem, very trouble...

[MasterK]

I try two camera.

They have draw order, so ui and 3d mesh in different camera can't mix zOrder.

[MasterK]

and this is the final solution:

bottom: 3d scene

then UI:{

bottom: ORTHOGRAPHIC_CAMERA and world space canvas

middle: mesh camera

top: ORTHOGRAPHIC_CAMERA and world space canvas(screen space is ok, but position value is not same with bottom ui)

}

[Wingnut]

Cool!  I would love to see screen grab.

What is mesh cam?  Single mesh (the selected mesh) ?

Maybe I will try a playground the same, tomorrow.  Ortho camera...  good idea!  Well done.

[MasterK]

another problem.

world space canvas can't handle pointerEvent... Don't know is that a bug..

So i should use screen Canvas  in UI's top.

Mesh cam can be any... just show some 3d content on the bottom ui...like first i said in this topic.

 

[Wingnut]

oh, sorry, @MasterK, I didn't notice your follow-on question.  Um, hmm. 

I'm not sure what you mean.  http://babylonjs-playground.com/#1KRMMG#5

In that demo, we do picks/clicks (on the primitives)... using worldspacecanvas.  Clicks are pointer events.  Sort-of.  *shrug* 

Let me think.  What other pointer events exist?  OnPointerOver, onPointerOut, umm... are there more?

Ok, remember that there is a mesh used for worldSpaceCanvas?

http://babylonjs-playground.com/#1KRMMG#8

Look at lines 221-223.  A mesh (I named it canvMesh)... is returned from the createC2D call, and we parented it to camera.

Now look at lines 225 - 234.  I attached an actionManager to the mesh, and then registered a pointerOver and pointerOut action.

So now, you can mouse-over the red part of the button panel, and it will turn green (color2).  Mouse-out, and it goes back to red (color1).

I don't know if this is the "proper way" to do mouse over/out events on a world space canvas 2d, but it sure seems to work.

Did I understand your issue?  I hope so.  If I misunderstood, please explain again and use a playground to show me the issue.  thx!

I think pointerOver and pointerOut already works for primitives (like text2D, rectangle2D, etc).  I think there are some playground demos that use it. 

Be well, talk soon.  Sorry for the slow reply.

[Wingnut]

I see the discussion in the other thread, now.  I'm off-topic, I think.  Oh well. 

The pictures helped much.  A 3Dcontext canvas... right in the middle of a worldspacecanvas.  Cool.

Maybe we need a Canvas3d2d custom primitive.    I suppose it could be a renderTarget texture (a camera view as a texture) ...used for a Sprite2d.  hmm.  Canvas2D on all 4 sides (Rect2D's?), and a Sprite2D of the 3d rendering... in the center.  hmm.  Nice challenge.  Not sure if anyone has tried a renderTarget texture on a Sprite2D, yet.

But yeah, clicking mesh, on a renderTarget textured Sprite2D... won't work.  hmm.  Perhaps it needs FOUR worldSpaceCanvas parented to the camera... one bottom, one top, one left, one right.  Wow.  I think I'm going to try it. 

update:  http://babylonjs-playground.com/#1KRMMG#11  - Gruesome!    And fun!

[MasterK]

@Wingnut Thank you for spend lots of time.

I think the most important question between your solution and my need is Orthographic camera can use exact position but your  "add parent" method is hard to set position...

And maybe nockawa is fixing orthographic camera worldspacecanvas problem.. I'm just waiting for him... haha...

I find out all canvas2d bug that stop our project process. If they're solved, we can go ahead freely~