Render Dynamic Line Connector Linking Label To Shape

[Rolento]

Hi All

I am trying to create an algorithm that dynamically draws a line (connector) that links a label to a shape.  At the moment I have a function for rendering a label at the location a shape resides.  Prior to rendeirng the connectors I wanted to resolve the issue of overlapping labels - i.e. dependent on the orientation of the camera the labels will overlap because the 2D X/Y screen coordinates projection occupies the same space.

In my attempt to fix this problem I wanted to obtain the X/Y screen coordinate for the shapes in my scene - I am using the following code:

        var p = BABYLON.Vector3.Project(mesh.position,
            BABYLON.Matrix.Identity(),
            scene.getTransformMatrix(),
            camera.viewport.toGlobal(engine.getRenderWidth(), engine.getRenderHeight()));

The problem I am finding is that the position returned has an X and Y coordinate of NaN.  I dont understand why this is happening and was hoping someone could kindly take a quick peak at my code to determine whats up:

http://www.babylonjs-playground.com/#1B1LNW#2

* See lines 154 ~ 158 for the above code snipplet

Beyond fixing the bug as reported above, if anyone can provide pointers as to how best resolve the overlapping labels issue that would be great.  Ideally I want all connectors to follow a uniform pattern / pathlinking labels to their corresponding shape.  Anyway, any pointers would be appreciated.

Thanks

[Wingnut]

What a cool idea!  I'm glad you are using Canvas2D for your labels.  Canvas2D is like having Phaser as a BabylonJS feature. 

And yes, the P object is not looking very healthy, is it? 

I don't know if this helps, but someone was playing with some funcs...  worldToScreen() and screenToWorld() in THIS playground...

http://www.babylonjs-playground.com/#1EJNKR#3

Might help... dunno.  Good luck! (and good to hear from you again. I hope things are goin' well for ya.)

[GameMonetize]

Hello this is because scene.getTransformMatrix is not set yet (no render was done). but you can use camera.getViewMatrix() * camera.getProjectionMatrix()

[Rolento]

Hi Wingnut

Thanks for the pointer, I tool a look at the code sample and it looks interesting - I may be able to utilise some of these concepts.  Once again, much appreciated for your help!

[Rolento]

Hi Deltakosh

I tried as you suggested:

        var p = BABYLON.Vector3.Project(mesh.position,
            BABYLON.Matrix.Identity(),
            camera.getViewMatrix() * camera.getProjectionMatrix(), // scene.getTransformMatrix(),
            camera.viewport.toGlobal(engine.getRenderWidth(), engine.getRenderHeight()));

However, upon making the above changes I receive an assert (e.m is undefined):

http://www.babylonjs-playground.com/#1B1LNW#4

Im not sure what the issue is...

As a workaround I rearranged the code and now request the connector function (not fully implemented yet) to be invoked via the onPointerMove() function - this seems to work nicely. 

http://www.babylonjs-playground.com/#1B1LNW#6

But... there is one last issue to resolve prior to moving on - when the application starts I can see in the bottom left corner of the viewport the labels quickly appear (flicker) and then get repositioned to their respective location.  I cant see an obvious reason why this happens - if you can help with this issue that would be great.

 

 

 

[GameMonetize]

This is because * is an operator not supported

In javascript you cannot overload operators so you have to use:  camera.getViewMatrix().multiply(...)

 

http://www.babylonjs-playground.com/debug.html#1B1LNW#7

[Rolento]

Hi Deltakosh

Ha, yes that makes perfect sense. 

Thanks again!

[Rolento]

Hi Guys

Right, im a step further now - I have been able to render the labels, detect intersections and move labels to resolve intersection issues.  However, there is a problem which is when the label is moved the bounding box that is calculated to determine collisions is nolonger correct:

http://www.babylonjs-playground.com/#1B1LNW#9

Lines: 122 ~ 140 detect label collisions and reposition labels if required.

Im hoping someone can advise as to why the label bounding box coordinates are invalid after the first collision as this is driving me crazy ... 

[Wingnut]

Hi Rolento!   Are you getting a few browser lockups with this beast?  I sure am. 

http://www.babylonjs-playground.com/#1B1LNW#11

I changed a couple things.  I moved the creation of the "canvas" (now called ssc2d) OUTSIDE-OF the renderLabel func.  I think you were making LOTS of screenSpaceCanvas2D's (ssc2d) and I think you only need one.  It can have lots of children.

I changed the renderLoop a bit, too.  No big deal.  Symptom is still the same.

I think SOME of the problem... is line 142:

g_labelArray[loop].label.children[0].y += 5;

That doesn't look right.  You are moving the children of the label, but not the group2d (not the labelObj itself).

But check this out.  Remark-out line 142, re-RUN... and mouse-drag into label intersection somewhere.  POOM, my FireFox locks-up like a bank in a ghetto.  What the hell?    Anyone else? 

I'm not sure, but this playground/project MIGHT be haunted. 

Hmm.  IF the browser DID allow me to move the label (group2d) itself...

g_labelArray[loop].label.y += 5;

...then that would be strange, because Group2D has no "y" property.    It DOES have a trackedNode... which we are fighting-against when we try to move the group2D.  (Wouldn't a Group2d.offsetFromTrackedNode = aVector2 ... be handy for us?)  To be frank, I don't understand 2D bounding boxes very well at all.  And I think Group2D's have two types, logical and renderable.  And did you notice the compactRect thing?  Apparently we can access some data from the "rectangle" that the Group2D itself uses.  Know about quads, Rolento? 

Topic-wandering:  Our non-SPS particleSystems use quads.  Think of them as a single "square" sub-division... of a sub-divided BJS Ground.  Each "quad" can be moved and rotated via moving its vertices... but they are each part of that ground mesh... so there is no .position, .rotation, and .scaling on the quad.  All particles... all quads... are "members" of the same mesh.  For example, a box... contains 6 quads... and each quad is a particle panel which can be positioned anywhere in world space.  Even though they are each part of the same master mesh, they are placed anywhere in space... apart from one another, making them appear to be separate individual mesh.  But they are not so.  Quads are used... due to their great performance speeds.  To wrap-up, if you ARE able to "query" the compactRect data, you might not find any classic positioning properties, but you MIGHT find 4-12 vector3 values (vertex positions).  They are moveable, and we might use THOSE to do the up-shift.  Re-position the quad.  Yeah! 

With YOUR line (moving children[0]), you are moving the label's rectangle2d... which seems like a pretty good idea, actually.  But, is the group2d actually "located" directly atop tracked-mesh center, no matter WHERE we set its child rect2d location?  hmm.  SO much to learn!

See the link to Prim2DPropInfo on that linked page?  Let's go there.  See the property named dirtyBoundingInfo?  Just possibly, you need to set that to true, somewhere, somehow.  That might be a flag that tells the Canvas2d system... that a boundingInfo needs refreshing... and do it.  Perhaps:

g_labelArray[loop].label.children[0].y += 5;  // your rect2d up-shifter
// then... 
g_labelArray[loop].label.sizeProperty.dirtyBoundingInfo = true;
// no promises... just an idea.  :)

On another subject... let's look at Group2D source.  Not too many properties, but lots of methods, and lots of available parameters in the constructor.  This possibly indicates that it might be best to dispose the intersected label (the label needing up-shift), and make a new one... whose rect2d Y position is further up.  I don't know WHEN or HOW you would bring the label back-to y: 50, once the intersection clears.  hmm.  You would probably need to dispose the up-shifted label, and make a new y: 50 label... AGAIN (get back to original position).

I don't think anyone has EVER tried what you are trying, trailblazer Rolento.    You are digging deeper into Canvas2D primitives than I have ever dug.  Speaking of primitives, group2D has a superclass... called Prim2DBase .  Let's look at it.

It has a crapload of properties.  One is named actualY.  I wonder what THAT puppy does?  Perhaps:

g_labelArray[loop].label.actualY += 5;  // try an up-shifter on group2D's superclass.
// then possibly STILL do... 
g_labelArray[loop].label.sizeProperty.dirtyBoundingInfo = true;
// no promises... just feeling-around in the dark, now.  :)

So where does this leave us?  I dunno. I wish I could get my browser to quit crashing so much.    It is driving ME crazy, too.  heh.  I'll keep fiddling with it... maybe... slowly.  I hope I didn't tell you too many wrong things.  I'm no expert... not even close.  Folks, feel free to correct me when I say wrong things... please. 

Rolento... let us know if you learn some things.  I am going to ping King @Nockawa and see if he has time to give us some advice. We'll talk again soon... but this code and this challenge... is scaring my dog.  heh

[Nockawa]

Hi @Rolento can you check this thread and tell me if you're looking to do something alike?

 

@Wingnut Group2D has a "y" property, unfortunately the documentation for a given class shows only the member for this class but not for the class it was extended from, in the case of Group2D, it extends Prim2DBase which contain the "y" property.

I'm sorry, I can't spend more time on this right now. Ping me next monday if you still need some help.

[Rolento]

Hi Wingnut

> I changed a couple things.  I moved the creation of the "canvas" (now called ssc2d) OUTSIDE-OF the renderLabel func.  I think you were making LOTS of screenSpaceCanvas2D's (ssc2d) and I think you only need one.  It can have lots of children.

Ahh, I see - I did'nt realise that - thanks.

g_labelArray[loop].label.children[0].y += 5;

> That doesn't look right.  You are moving the children of the label, but not the group2d (not the labelObj itself).

Exactly - when I was experimenting I did try and move the "group2d" but I could'nt find an "x" or "y" property for me to manipulate - hence I reverted back to moving the child label object.

> It DOES have a trackedNode... which we are fighting-against when we try to move the group2D.

Yes this is kind of frustrating, I did experiment with trying to offset the label against the position of the mouse when moved.  The result was not ideal - i.e. I was kind of making progress but the code was getting really mangled and turning into a mess so I have abandoned that approach at the moment hoping there is a more obvious cleaner solution.  

> And did you notice the compactRect thing?  Apparently we can access some data from the "rectangle" that the Group2D itself uses.  Know about quads, Rolento?  

Unfortunately I have not encountered quads at this time - thanks for the summary, I will have do some more research on this topic to better understand how they are used and/or hw they can be manipulated.  If you can point me in the direction of what you deem to be useful BJS tutorials and/or PG examples that would be great.

> I don't think anyone has EVER tried what you are trying, trailblazer Rolento.  You are digging deeper into Canvas2D primitives than I have ever dug.

Ha, lets hope I/we can find a solution to this mystery!

> Rolento... let us know if you learn some things.

I certainly will - I will be experimenting later today.  Your comments have provided me with some areas to explore and ideas!  Also, thanks for inviting @Nockawa

Once again, I really appreciate your help on this topic.

[Rolento]

Hi Nockawa

Quote

can you check this thread and tell me if you're looking to do something alike?

The exmaple provided in the thread is "close" to what I am wanting to achieve.  See below for the objectives I am trying to accomplish:

1) link labels to objects in 3D scene using connectors

2) ensure linking connectors (as defined in 1) do not overlap/intrsect

3) draw labels at X/Y coordinates (2D space) and "prevent" labels overlapping/intersecting

The example you provided accomplishes (1) bit not (2) and (3).  If you can provide any guidance I would really appreciate it.

 

 

[Nockawa]

2) and 3) are mostly algorithm problems that I can't cover in the Canvas2D lib. I honestly don't know how you could solve this, some serious thinking must be done, and some googling too...

I think you have the basics with the sample I gave, now you have to improve things to reach your goals: that's programmer's life for you!

[Wingnut]

@Nockawa, the original problem was with bounding area.  (and browser crashing, for me) 

That thread does not address label-overlap, overlap-avoidance, and after-avoidance... label bounding-area updates.

Not that it is YOUR problem, but you could still feel free to help, if you wish. 

http://www.babylonjs-playground.com/#1B1LNW#11

#1 - If I may speak freely, the loop in renderConnectors() [line 130] is a less-than-optimal way.  I think we should activate a label-intersection "observe".  The observe (observer/observable) constantly watches for two labels intersecting, and if observed, calls a func to de-overlap them (clear the intersect condition).  This is likely a programmer-created thing, and not the responsibility of Canvas2D system.  (but tools like this are hot items for future Canvas2D features).  All in all, ideally, we should let the observer system do the looping... so we don't have-to. 

#2 - Line 142... the label mover.  This was talked about earlier.  Now we know that Group2d has a Y value (inherited from Prim2d).  This functionality (this line) would leave this soon-gone function, and be moved to the onIntersection() function called by the intersection observer.

#3 - Here is where we could use Nockawa brains... in the bounding-area intersect detector.  This is currently a big pile of ugly code (no offense meant) in lines 150-220.  I think we need this label intersect-detector... clean-coded.  Again, let's try to avoid the looping.  Building an intersect observer/observable might be the answer.  I'm quite sure that observers already loop/iterate through all the observables... so we should tie-into that existing looper.  Still, we need a way to know when a label intersect has occurred.  If we derive an angle-relationship between the 2 intersected labels, would be sweet!  It would help our offsetter know the best method/direction for de-overlap.

Then... do the label separation, and then make sure our recently-moved Group2d bounding area is freshly updated... so the observer will clear (indicating no more intersect).

#4 - addConnector - Not needed, in my opinion.  (unless there is future plans to draw lines between label and trackedNode).  In this case, the programmer does not add connection.  Group2d's trackNode does that for us.  The only action we would do... is separate overlapped (intersected) labels.  This would be done with an "offsetting" of the label... from its trackedNode screenspace position (that's a type of connector, but not really).  Again, it's very important... that AFTER the separation (de-intersect) move, the observer is cleared-of intersection condition, and the bounding area of the newly-positioned Group2d is also updated (ready for more observing by the observer).    (enuf words for ya'll?)

Ok, that's all I got.  Plenty, huh?  Nockawa... if you feel like helping... we could use a really clean overlap detector with observer, and also a really nice moveGroup2dAndRefreshBoundingAreaAndResetObserver.  (gruesome func) 

IF Group2d Y-position live-adjusting... works nicely, then all is well. 

For me, so far... it crashes my Firefox. If I just LOOK at line 142, my browser crashes.  heh.  

Supposedly, g_labelArray[loop].label.y += 5; should move the Group2d Y-position... but... I'm scared.    So is my dog.  Tests ahead.  Help welcomed.  Thanks!

PS:  There is one more needed function/observer.  It is the "can I go home yet" observer.  AFTER a label has been offset (to avoid an intersect)... it will want to eventually... go home.  It wants to return to its original origin.  So, we need a "am I clear to return home" de-offsetter.  erf.    In a way, we are building a unique kind of "snap" system, eh?  *nod*  Cool project!  I hope Rolento doesn't mind us building a campfire at his project site, and throwing a coding party.    PARTY AT ROLENTO'S CAMPSITE!  YAY!

[Wingnut]

Wow, even if I change the line 142 area to...

141	// g_labelArray[loop].label.children[0].y += 5; // works, but weird bounding area ops
142	// g_labelArray[loop].label.y += 5;  // crasher
143	console.log(g_labelArray[loop].label);  // crasher
				

... I get browser crash upon intersect, IE or Firefox.  Nice.  Magical.  Crash a browser with a console.log... too good!    Just remove it all, still crashes.  LOVE IT!  What power we have!  heh  (Not really a crash.  It is a continuously running script... probably the intersect detector.  ReCurse of the Mummy's Tomb.  Still fun!)

[Nockawa]

There's currently a bug in BoudingArea computing, I'm aware of it since 2 weeks, but couldn't find the time to start working on it. I will have time this week.

[Rolento]

Hi Guys

Sorry for the radio silience - work has been crazy the past few days!  The good news is I was able to play around a little today and see if I can find out why the label intersection algorithm was not working.  Looking at the solution (thread) that Nockawa posted I could see that the function that calcs the screen coordinates differed from mine so I thought I would refactor my code to see what happens, boom... as soon as I made the change the bounding box collision detection of labels started working!  FYI here is a code comparison:

Original code:

p = BABYLON.Vector3.Project(mesh.position, 
	            BABYLON.Matrix.Identity(), 
	            scene.getTransformMatrix(), 
	            camera.viewport.toGlobal(engine.getRenderWidth(), engine.getRenderHeight()));

Updated Code:

var rh = engine.getRenderHeight();
var v = camera.viewport.toGlobal(engine.getRenderWidth(), rh);
var worldMtx = mesh.getWorldMatrix();
var proj = BABYLON.Vector3.Project((BABYLON.Canvas2D)._v, worldMtx, (BABYLON.Canvas2D)._m, v);
p = new BABYLON.Vector2(proj.x, rh - proj.y);

So we are a step closer now...  The update code with connectors added can be found below:

http://www.babylonjs-playground.com/#1B1LNW#16

The above code sample is far from complete - i.e. connectors still intersect and when dragging a shape for some reason the labels are moved appropriately but the connectors are not udpated/refreshed - I looked at the code and I cant see any obvious issues... :/  I must admit I have had a couple of whisky's this evening so dont shout if you see an obvious mistake!

With respect to Wingnut's comments, yes I 100% agree the code is messy (mangled) and far from optimised.  My strategy was to resolve the fundamental issues I was encountering and get a rudementary system working - once this is done then I was intending to refactor the code to make it more efficient.  So no offence taken when you provided your comments  

At the moment the current challenges I need to address are:

a ) determine why the connectors "sometimes" are not being redrawn

b ) update the line connector algorithm to prevent intersections

c ) update the line connector algorithm making it more intelligent such that labels can appear left / right of associated shape

d ) update the line connector algorithm making connectors follow a wave path (as below):

I think if the above challenges can be addressed then the label system would be quite an attractive and useful tool for annotating objects in a 3D scene.  If anyone has ideas on how the above can be achieved and/or wants to dive in and cut some code that would be great.

 

[Wingnut]

You're doing GREAT!  This is advancing nicely!  WELL DONE!  And no browser crashes for me!  YAY!

Way to go! 

[Rolento]

Hi All

Its been a bit of time since my last update (a new addition to the family - so lots of fun and games!).  Anyway, I have made a few tweaks to the code to resolve a series of glitches.  The code is still far from optimal but I think it is an acceptable foundation for anyone wanting to create dynamic labels/connectors for marking up objects added to a 3D scene.  Here is the latest code:

http://www.babylonjs-playground.com/#1B1LNW#26

The above is perfect for my needs - however, one possible enhancement would be to add additional logic to prevent the connectors from overlapping.

Once again a big thanks to [Wingnut], [Nockawa] who helped me to get this working...

 

[Flomotion]

I tried to create these kind of labels. But this is much better. Thank you!

[Wingnut]

Nice nob, Rolento!  I like it!  Congrats on the new family member, too.

[Nockawa]

Nice work @Rolento I'm glad you finally managed to get things done!

[Wingnut]

Hey Rolento... have you opened the browser dev tools window, and seen what happens to label lengths?   Just curious.

I bookmarked your playground... twice.  I have some plans for it, like perhaps using it as a demo in a tutorial, if that would be ok.  Advanced labeling section. 

[adam]

7 minutes ago, Wingnut said:

have you opened the browser dev tools window, and seen what happens to label lengths? 

This issue doesn't appear when testing the PG locally with the latest version of C2D.

[Rolento]

Hi Wingnut / all

I just checked the code and running locally (as per Adam's reply) I see no issue.  Do you see issues when running locally using latest version of C2D?

Cheers