Pick a point in the space with mouse.

[Tomek]

Hi, this is my first post on this forum so I would like to say that BabylonJs is awesome and it has a great community!

I'm working on a project and I would like to allow the user to select a point in the space (so I could later pin a GUI button or a mesh there).

My first idea is that if I could detect that user is trying to click on the model then I would assume that the point is on the model, else if it's the space I could probably use some default distance from the camera.

I have read about collisions and end up with this example: https://www.babylonjs-playground.com/#RQ2T0A#6. The problem is that "click" handler always returns "hit: false" I assume that's because of the Yeti isn't something like mesh created with MeshBuilder. How could I fix it?

By the end, my final goal is to create some kind of indicator that would help the user visually pick that point (maybe some crossing lines or whatever). Any suggestions would be much helpful!

[Guest]

Hello and welcome!

you are on the right track: You can use scene.pick() but you just need something to pick

https://www.babylonjs-playground.com/#NU4F6Y#12

(I added comments in the code to explain it)

 

 

[Wingnut]

Hi guys!  Welcome, Tomek.  Cool invisible-plane click-in-space solution, Deltakosh!

I took it a few steps further.

https://www.babylonjs-playground.com/#NU4F6Y#13

Watch the JS console to try to learn what is happening.

I made Deltakosh's invisible plane... a LITTLE BIT visible... and textured some bricks on it.

I also parented the wall... to the camera.  Line 24.

I put actionManagers on the red, green, and blue boxes... each with pick-triggers and executeCode-actions.

I am testing IF/NOT we can pick the colored boxes...  THROUGH the (almost-) invisible brick wall. 

So far, no.  I think it might need "multi-pick" to do that... which IS an available feature.  More about that, below.

In current condition, picking red, blue, or green boxes... reports a WALL HIT at the console.  The "pick" cannot travel thru the wall... to hit the boxes behind the wall.

You can change '30' to '60' in line 25, and that will move the invisible wall BEHIND the colored boxes.

THEN... picking red, green, or blue (using their actionManagers)... works fine.

----------------

https://www.babylonjs-playground.com/#NU4F6Y#14

In this demo, wall is larger and farther-back (line 25), yet still parented to camera.  Picks work on invisible wall (distances over 100), yet ALSO work on red, blue, and green actionManagers.

Anyway, these are just some things to think-about and experiment-with.  Selecting things, and positioning 3d cross-hairs/reticles... can be challenging.

---------------

With 'multi-pick'... things change a bit:

https://www.babylonjs-playground.com/#NU4F6Y#16

See the 'multipick' in line 84.  Line 85 reports ALL the mesh that were hit.  So, a click in the center... reports 4 mesh (in an array) at the console.  (The little white cursor box is set isPickable = false, so it is not included.)

In versions #13/#14, the pickResult object had many cool properties, like .pickedMesh and .pickedPoint and .distance. 

Conversely, the pickResult returned from a multi-pick... is only a simple array of ALL the mesh that were hit.  No cool properties and no automatic distance, but SOME values can be calculated/measured.  Getting a .pickedPoint for each returned mesh... will be difficult, though.  The multipick's pickResult only contains the "hit mesh"... and no information about WHERE they were hit.

TMI, eh?  *nod*.  In general, there are many ways to get the screen X/Y position of a click/pick.  It is not so easy to determine the Z-depth when clicking in "empty space". 

Deltakosh installed the invisible plane... to show that "clicking in space" needs to hit SOMETHING, or else it won't be "seen" as a pick/click.  But using invisible planes for "space clicks"... has some "block clicks upon other mesh" -issues that need to be understood.  In many cases, the invisible "space-plane"... needs to be behind all other scene mesh.

Ok, we have plenty of playgrounds to experiment-with.  Good luck!  Tell us of your progress and/or problems.  We're here to help.  Be patient with yourself.  Soon, you will be an expert.  Sorry for all the talking... I hope I have been helpful.

PS:  'Collisions' are not really a mouse-pointer thing.  They are mesh-to-mesh and freeCamera-to-mesh intersection testing/systems.   But, that was some very intuitive thinking.  'Collision' is indeed a term that COULD apply to pointer hits, eh?  Not in BabylonJS, though.   

[dbawel]

I don't know if this might apply, however for a recent app, I used the (X,Y) canvas pick point to pass through, and set a separate control (mouse wheel, keypad) to set a dynamic Z distance from the camera in world space. This simple function allowed me to draw vertices and create meshes in three dimensions quickly and easily. I also drew a line from my camera to the (X,Y,Z) world space coordinate as a visual reference with a simple icon attached to the end point on the selected position to allow easier navigation. And for my app's specific needs, I provided a numeric Z depth reading in the UI with limits so that the user could set restrictions on Z depth (as well as X,Y world space, to provide a useful, selectable, and accurate determination of where to select to place mesh attributes.

DB

[Wingnut]

Hi DB.  Yeah, I was thinking something similar.  Perhaps hold-down the click, and then roll mousewheel... for depth-adjusting (mostly for positioning new mesh or "space markers").

I think many new users of BJS... want to "assemble".  Although the BJS Editor is nice, sometimes it seems... that mesh could use a "snap-on GUI"... perhaps .guiManager.

Add one, activate it, and very nice menu with arrow-buttons and readouts... happens somewhere on-screen.  You never really drag a mesh to new positions or rotations, but you use the GUI panel buttons to move/rotate the mesh.  In this way, the mesh sort-of carries-around its own editor.  We're still positioning via pointer/clicks, just... not dragging.  Instead, MeshMenu v1.0.    *shrug*  Later, with add "snap" 

Likely, somebody has already built it. 

I think I once made a right-click-on-mesh pop-open menu (context menu)... but I did it using Canvas2d.  Newer GUI system can handle it fine, too.  RIght-click to pop-open a mesh's "personal editor"?  Sounds ok, eh?  

Really, the GUI/funcs in my goofy space-flyer playground... is a pretty good start.  It has 6 buttons for rotate, 6 buttons for position, but needs 6 more buttons for scaling.  Use symbols on the buttons, instead of text... and you should be able to shrink the 18+ button menu... to 15% x 15% screen space (3 inches by 3 inches).  Have that pop-open under a mesh... when the mesh is right-clicked.  Nice.  Maybe allow the readouts to also be inputs... let users type-in numbers.  Same things that dbawel speaks-of... only 1500% overkill added, and never used by anyone. 

[Tomek]

Hi guys,

Thanks a lot! This gives me some ideas for further research. The general impression I get is that I should apply this kind of logic:

1. create a Plane far, far away to be sure it's behind the loaded model
2. then I can distinguish if the click happened on the space (when hit the plane) or else it must have been somewhere on the model

Right?

Using multi-pick option is also interesting, maybe I could do something with that information ? .

 @dbawel Your solution seems to be close to what I'm looking for, would you mind sharing some example?  

I will be sharing my progress.

[Wingnut]

Remember that ALL your mesh could have actionManagers listening for picks.  So, think about... ANY click that didn't pickResult.hit, or trigger a mesh.actionManager... MUST have been a click in space.  We don't need an invisible plane.... IF we can eliminate all other possibilities.  If all mesh have actionManagers, and none of them triggered... it HAD TO BE a "miss". 

So, you can run a function such as...  var onMiss = function() { var whereMiss = new BABYLON.Vector2(scene.pointerX, scene.pointerY); console.log(whereMiss); }

When you have this "where was the pointer on the screen at time of click-miss" value... things can be done with it. 

I BELIEVE one operation is called "unproject()"... which sort-of backwards-projects an imaginary line (a direction vector)... from the center of the camera, THROUGH that chosen screen X/Y... and you can use that to position mesh in "space" (along that projected line).  Or maybe that is called "project".  (You'll need to read about it). 

Still, you need SOME way of determining HOW FAR along Z-depth to place it (how far from camera).  Not only that, but camera can be turned... so that could cause trouble.  Maybe not.

And don't forget that an actionManager is allowed on the scene object.  I think we saw one used in one of the above playgrounds.  I'm not sure if THAT can test for "click happened without hitting anything", but it might be worth study.

Anyway, I figured I'd better tell you about the handy, constantly-updated... scene.pointerX and scene.pointerY properties.  If you put an actionManager with click-test on EACH mesh... then you can use these two properties to determine screen X/Y location of clicks that missed all mesh.  You don't need to use pickResults or hit checks... at all.  I am not experienced in 'unproject', but a forum search about it... looks good.  I dunno if any of these ideas are wise or useful... just passing along what little I know.

Heck, let's try a playground search for 'unproject', too.  That returns a few hits.  Others are far more qualified to talk about projections... than I.  Perhaps they will tell us what they know, soon.  Stay tuned, party on.

[Tomek]

@Wingnut thanks a lot, I really appreciate your feedback.

I'm missing something fundamental here. How is Mesh created with Babylon different from meshes loaded with SceneLoader (I mean `newScene.meshes` property)? Back to yeti example - I tried to create multi-pick but it always returns an empty array - https://www.babylonjs-playground.com/#RQ2T0A#7

[dbawel]

Hi @Tomek

My apologies - I haven't been on the forum much these past few weeks. If you would still like an example of this, then I expect to be able to set up a simple scene which accomplishes this.I have an insurmountable amount of work backlogged, but need to move back into the community. I'll try and let you know if I'm not able to get to this tomorrow; as it won't take long to build.

DB

[bghgary]

@Tomek I think there is a bug here perhaps related to the root transform for glTF files. There were some issues with playground you created, but even after I fixed the issues, the picking code still doesn't work. I've filed an issue for this: https://github.com/BabylonJS/Babylon.js/issues/5386

[Tomek]

UPDATE:

I think I'm getting somewhere. I mean I found a way with Gizmos how to position an element after it has been misplaced in space - https://www.babylonjs-playground.com/#RQ2T0A#12

Primarily the function of these pinpoints/anchors is to be something else, but I can imagine that this Gizmo function can be triggered by something like a double-click maybe (though I didn't found a way to receive double clicks) to change anchor position. 

Still, I'm don't know really how to pick a good initial Z distance from the camera (depth of anchor). Maybe something like a bounding box could help? Maybe then I could use this logic: if it's a bounding-box hit then I'll place an anchor on that specific place on the bounding box. If it's a space I will pin an anchor to the middle of the world. I both cases the user can correct the anchor with Gizmo.

[Tomek]

UPDATE:

* I have a way to move once created anchor with a Gizmo

* I have a way to intercept clicks on the scene and add a new anchor

The problem is I cannot properly set initial position for a new anchor after the user click.

I gave up on counting a distance between a model and camera for now. Having a gizmo set, it would be good enough to just click on the canvas and then create an anchor right under the mouse pointer and with some default distance from the camera.

But this is not easy, as I don't know how to take the camera position into account and set depth properly so the anchor stays under the mouse pointer. Is it possible even without some hard math?

This is where I stuck - https://www.babylonjs-playground.com/#RQ2T0A#14 .

Any help would be appreciated.

 

[JohnK]

Here is an experiment https://www.babylonjs-playground.com/#RQ2T0A#15

Still a lot wrong with it, as in

• not very accurate
• the scaling on line 85 is made up with no logical basis though it does seem to work (?????)
• rotating the camera means the x position of gizmo is reflected

Maybe it could lead somewhere! ?

[Tomek]

@JohnK thanks! I don't know all of this math so I cannot improve it ?

I thought about that wall from the first examples... https://www.babylonjs-playground.com/#RQ2T0A#16.

Here I have added a fake wall and after the click, the anchor is placed always under the pointer.

Though there are some problems with this approach:

1. it's a fixed depth.
2. if someone moves away enough from camera he will be limited to the wall size when placing anchor which also...
3. ...requires some computations probably so the wall is always bigger than the model
4. anchors are initially placed inside the model,
5. It's an extra mesh that someone has to be aware of (?)

As of point 4 - it works better when there are no bones https://www.babylonjs-playground.com/#RQ2T0A#17

Maybe most of these problems could be fixed with some kind of a bounding box ?

[JohnK]

How many of your problems are solved by having a really big wall? https://www.babylonjs-playground.com/#RQ2T0A#18

 

[Tomek]

I tried that before and I think it will be a good enough approach to what I'm looking for.

My initial aim was to be able to somehow pick the exact place in the space or a model.

Let's say that these anchors are gonna open up some comment, it would be a cool feature if a user could just rotate a model then click a place on this model and anchor would be then pinned exactly to that place. But this gets complicated because of the "Bones" (which I don't get yet) and basically handling clicks outside of the model.

Still curious about @dbawel solution though ?

I really appreciate all of the feedback you gave me.