BABYLON.Augmented Reality ...

[JackFalcon]

Hello, BabylonJS is fantasic. Thank you mr @Deltakosh and mr @davrous. 

Research: Augmented Reality making rapid advancements this year.

GIS AR, Multi-Marker AR, TableTop AR - 3D on flat surfaces... Happening fast.

Babylon is my 3D home. So I must ask:

          Is AR porting a priority for upcoming versions of BabylonJS. And if not, can it be?

I looked at wishlist and roadmap. Found milestone/5 with no mention of AR: https://github.com/BabylonJS/Babylon.js/milestone/5

Here is link with history I'd follow:

https://uploadvr.com/ar-js-efficient-augmented-reality-for-the-web/

Seems to be lineage of successful AR examples:

https://github.com/opencv/opencv_contrib/tree/master/modules/aruco

https://github.com/artoolkit/jsartoolkit5

https://github.com/jeromeetienne/ar.js

For AR -> Babylon is most promising to me.

How would one go about such a feature?

Thanks,

[HeadClot]

Just thought I would chime in on this.

ARtoolkit 6 is in open beta. They said they would be continuing support ARToolkit,js in a video I saw a while back. I cannot find said video at the moment.

That said I would like to see AR support for Babylon.js

[davrous]

Hello,

We've discussed about it this week to think how to integrate that in a near future. We don't know yet how it will be achieved technically but we're definitely interested in exploring AR.

David 

[JackFalcon]

Thank you Mr. @davrous, I'm happy to make a technical attempt ... : )

[GameMonetize]

Please do..We did not start anything yet

[JackFalcon]

Ok, scheduled deep-dive AR ...

[JackFalcon]

Ping, @jerome ... hello, : )

Apologies if this is bad timing, or if I misunderstand, or if I am completely off-course, not uncommon, etc. : )

 Considering Babylon AR integration study (attempt). Do you have tips?

Perhaps we chatted on twitter about level-of-difficulty - so curious about it. I'd like to learn AR better and show findings here.

 

Considering the complexities: cross-browser camera access, marker-tracking, ... ok that makes sense.

Thinking perhaps a good place to start might  be -> worldspace / local space / 3D translation over background video component? Is that it, would like to try...

For a basic hello world knot, are there other complications? (scaling, I wonder.) Any tips from the trenches? : )

Hoping to (try to) get down in there this next month.

 

All the best, Thanks. : )

[jerome]

I guess you are confusing me with another Jerome ( https://twitter.com/jerome_etienne ) because I don't know anything about VR ;-)

[JackFalcon]

Ah, yep! Thought that might be the case. Thanks...

[JackFalcon]

AR-DEEP-DIVE-DAY-1:

 

Learning comes from @jeromeetienne and @AndraConnect (follow twitter) +1. Hello && welcome. : )

For everyone who already knows, JE and AC are doing phenomenal work in AR.

Of the 3D rendering options for AR, BabylonJS would be an exceptional renderer - I think the best. However, I am biased by great results. 

 

Turns out -> there are two phenomenal "Jeromes".

Two phenomenal "Davids" we can handle! But now two phenomenal "Jeromes"!?! Does this happen? I only wish...

Jk. : ) Straight to the point.

 

Here-Is-What-I-Am-Learning about AR:

AR.js - is here -  https://github.com/jeromeetienne/AR.js thanks to @jerome_etienne

Reading through master +1. Great components under THREEx = {}.  the code is vanilla and beautiful..

The ReadMe (local) is the best starting point (https://github.com/jeromeetienne/AR.js/blob/master/README.md).

Too many links!!! A  good sign. : )

 

#1 KEY-INFO:

         From ReadMe: AR.js uses genius solution:

           -  "**emscripten and asm.js**! ... compile artoolkit c into javascript"

                      Thank you Brendan Eich. : ) Jerome is on it...

             - "Only thanks to all of them, i could do my part : Optimizing performance from 5fps on high-end phone, to 60fps on 2years old phone." 

            - ARToolkit, is also impressive: https://artoolkit.org/  wow - crashes Chrome Dev Tools over "asm". Oh Text search. Watch out for that!

                 mmm, look at those shaders!?!.

           - Innovations in; shadows, reflection, portals, multi-marker, table-top, & "Principles of AR" Look at that...  with principles. even. Like+1.

            - In addition to ARToolkit, it looks like two other engines may be used: aruco, tango. 

 

2. After a big  BLOB of minified ASM awaits this helpful hint:

Quote

                   " The ARController is the main object for doing AR marker detection with JSARToolKit. To use an ARController..."

 

Nicely done, and then...

var ARController = {};

... as a prototype : )

In a nutshell...

Quote

"ARController detects markers in  given image. The process() method dispatches marker detection events during run."

a good place to put a breakpoint? : ) Yes... ARController talks to artoolkit... : ) Basically accesses AR-API. Awesome. As expected...

obj.onSuccess = function() { } 

It calls back ... with a video? yep <video> saw that... new world. : )

 

PLAN:

Look: <a-frame><a-marker>export AND Threex base object full of tools. (interesting hoisting pattern) -> AND (using .position [gotcha] camers.matrix) maybe slip in BABYLON.Mesh.CreateKnot() ... at the last moment? Rendered to a second <canvas> <video> at a higher z-index... would it work? Could it work? -> Idk.

UPDATE -> It worked

UPDATE -> more of a "cannonball" than a "belly-flop".

UPDATE -> lucky is good.

Looks like: PresentationMatrix.... Transform Matrix.... Camera.matrix.

UPDATE -> It was BABYLON.Camera._computedViewMatrix and TransformMatrix calculation in an array to BABYLON.Matrix.FromArray().

 

Of course this is only a learning exercise in AR...

Because 1) The "Davids" will surely do something more phenomenal, and if not -> the"Jeromes" will. : )

And 2) I am only pedaling furiously on a tricycle, far, far... behind. 

 

If you are NOT a "Jerome" or "David",  like me, this thread sprinkles out a few breadcrumbs from a BABYLONJS + AR.js learning collision.

Hope it helps. 

 

 

 

[JackFalcon]

More AR Notes:

3) In UI Renderloop it looks like scene.visibility = true, onMarkerFound(); And position aligned by putting Matrix into camera.matrix.

Confirm: 3D is rendered in <canvas> above <video>

And scene.visibility flickers  on and off. 

 

Position Alignment and scaling, seems to be the big trick... Presentation Matrix, Tranformation Matrix, ... 

 

This is helpful: https://github.com/artoolkit/jsartoolkit5/blob/master/doc/introduction.html

And this: https://github.com/artoolkit/jsartoolkit5

And: https://threejs.org/docs/#api/math/Matrix4

... Yep, that describes problem .... 

[JackFalcon]

UPDATE: Found out why the freeze on "use asm"... in devtools.

It is because of word-find!  

Same thing happens in Sublime.

WARNING: Searching through the huge blob of ASM... can be slow!

SOLUTION: Search "around" it. Lol. Or separate file... : )

Also found out some other stuff.

   -> Into the ProjectionMatrix, init, float32Array(16) and the Matrix4 compatibility review.

[ozRocker]

is webRTC supported by iOS Safari and Chrome now?  Last I checked, there was no support but I heard Apple was planning on it.  Bowser from Ericsson supports it so you can always test there until Apple catches up

[JackFalcon]

AR Deep Dive Day 3 -> success.

"Hello World" - BabylonJS knotBox rendering in AR.js (with three.js knotbox):

TransformMatrix placed into camera._computedViewMatrix. 

Code posted: http://www.3dwebscapes.com/demos/arbjs5/ 

Insert this second line: look at with Hiro marker.

            markerObject3D.matrix.getInverse( modelViewMatrix )  
            cameraB._computedViewMatrix = new BABYLON.Matrix.FromArray(markerObject3D.matrix.toArray()); //SPLICE-FINAL!  

There is still some offset bug. Perhaps default world rotation thing in artoolkit (I'll look).

 

@Deltakosh & @davrous & @jeromeetienne I hope this helps.

Key-Takeaways: The type float23Array(16) looks compatible at the Transformation Matrix.

Building from Array, can re-render in a single line of code. : )

There was some difference in Matrix Math -> around multiply (so yeah, consideration)

BABYLON.Math seem sufficient coverage with (https://doc.babylonjs.com/classes/2.5/matrix) so no need for gl-matrix.js? I'd think, at first. But then...

       Ah wait, dependancy-consideration... AR API, so ok, understand fully. That might get tricky. : )

       Babylon.Matrix lives here (https://github.com/BabylonJS/Babylon.js/blob/master/src/Math/babylon.math.ts)

Thx,

[GameMonetize]

Hey this is UBBER cool.

Do you recommend we change stuff in bjs to make it easy for you to integrate?

[JackFalcon]

Thanks @Deltakosh ... thought about it. 

 

That was a really good question. So I rewrote the answers twice (quick)(thoughtful). : )

QUICK: I do not see changes for Babylon as a result of this test.

THOUGHTFUL: Here is (possible) minimum and maximum of the problem space.

 

At minimum: it is a single-line-code-splice (above) 

               - Theoretically all BJS could redirect through that line of code. Into the projection matrix...

               - Subject to robustness testing of more complex sets will show if anything breaks. I'll do that.

               - and fix the Projection bug (above) [possibly default axis mis-alignment from artoolkit].

 

At maximum: AR Integration is a fascinating landscape of vast latitude (it seems to me).

So maybe - thinking of AR in steps, stages, or phases might be helpful. For example; hello-world, multi-marker, flat-surface, gps, reflective, etc... 

1) #1 Big question of integration. Seems to be - who does the Matrix Math? This is the tricky delta I mentioned. I don't know the solution. It needs good ideas.

              Honestly, this was shown to me by @jeromeetienne. He mentions a big refactor for an API that reduces dependencies. Sensible. For gl-matrix.

              The question I have is: if BABYLON.Math is fully suitable (and can be extended quickly to match) - what then?

               Like I said a big question. Many options.  Probably it needs ideas from the group.

 

               My question, for that is: how could AR be added in as a package? And then pass Transformation Math to Babylon?

               AR - does come with a big house-guest - who eats a lot! JK, mostly.

                          The Artoolkit, (or any toolkit) is a big chunk of asm - which would be good in a separate file (package).

               Or, maybe it should be rewritten eventually? For example, there is barcode scanning code, so maybe it needs streamline, fruther optimization. Idk.

               A pure AR solution from ground up might be inevitable. And why tango integration may be cool. But that is only a guess.

              Hello Hololens ar-engine compiled to JS with enscripten? Idk.

 

               Strangely running both WebGL 3D renderers - at the same time - has not (yet) melted my laptop!

               This is the simple path through the woods or the mirage I mentioned.  I will look at robustness. : )

                As mentioned, emulating matrix math stymied at one spot around multiply and invert. Can still isolate - but not sure what that was, and it was late. 2am. : )

 

2) A number of important improvements likely remain for full AR implementation.

By that I mean all this: A) there is much latitude in continued improvements in AR for a while (...). B ) They do not appear to need BABYLON framework change. Probably package addition. And finally C) notion of fresh perspective of context, methodology, and optimization (below). Like building a flying vehicle - we have a ways to go yet. : )

Apologies for excessive detail (this is what I saw while looking):

              EXAMPLE 1: "Matrix Methodology" ->Placing the computed Transform Matrix into the camera._computedViewMatrix is ok. And possibly limited programmatically.

                          QUESTION: Are there ways to manipulate MatrixTransform with absolutePosition instead (making rotation scale independent)? Using Compose(), etc?

                                         In otherwords, leave camera matrix as is - and just compute mesh as a result of local space(ish). I'll run tests.. 

                                         No doubt, a deeper understanding of emulating by multi-marker may show ... something different. I'll try that.

             EXAMPLE 2: "UI stuff" User camera getUserMedia() requires https. Also it would be nice to know what symbol it is looking for, and a btn for camera shut off.

                                         All could be handled gracefully in standard way in the future. //TODO try  button, img, <span>... etc.

             EXAMPLE 3: There is barcode scanner code in the ar process loop, it seems, probably old example. Which I think is cool (barcode as anscestor - really?), but a performance optimization available? That is the notion of possibly a streamlined pure-ar context approach, from top down. But. Probably already happening. : )

 

 I will try those things and put results here. Thank you. 

[jeromeetienne]

You guys are crazy! i love the double rendering of the demo ?
 

[jeromeetienne]

I love your energy! so i just started working on it at 4am. sleeping in another life.

 

Code will happen here until it goes math and object3d ... enougth to be pushed into ar.js itself

https://github.com/jeromeetienne/AR.js/tree/dev/three.js/examples/object3d-glmatrix

[adam]

18 hours ago, aFalcon said:

1) Big question of integration. Seems to be - who does the Matrix Math? This is the tricky delta I mentioned. I don't know the solution. It needs good ideas.

This is an option:

https://github.com/toji/gl-matrix

Brandon Jones is well known for his work on WebVR.

https://twitter.com/Tojiro

[JackFalcon]

@jeromeetienne cheers. Thank you for object3D-glmatrix.  I have some availability to run tests with BABYLON.

[jeromeetienne]

2 hours ago, adam said:

https://github.com/toji/gl-matrix

I definitly agree, i plan to use this one

[max123]

That's some crazy shit!!

[JackFalcon]

Lol, AR is great, setting up next test now ... 10xcrazier!  

Maybe it should have warning label... 

Warning: may influence perception.

 

 

[JackFalcon]

BJS in ARjs: small planetary system test -> success

Orbiting spheres.... background (skybox) plane. 

Everything seems ok. Needs fine-tuning ...  z and y axis seem inverted.

-> Twisting and turning the marker makes a big difference.

Single line of code:

cameraB._computedViewMatrix = new BABYLON.Matrix.FromArray(markerObject3D.matrix.toArray());

 

[JackFalcon]

The world is off axis...  

...and maybe that is ok. Lol...