JackFalcon Posted July 14, 2017 Share Posted July 14, 2017 Hello, BabylonJS is fantasic. Thank you mr @Deltakosh and mr @davrous. Research: Augmented Reality making rapid advancements this year. GIS AR, Multi-Marker AR, TableTop AR - 3D on flat surfaces... Happening fast. Babylon is my 3D home. So I must ask: Is AR porting a priority for upcoming versions of BabylonJS. And if not, can it be? I looked at wishlist and roadmap. Found milestone/5 with no mention of AR: https://github.com/BabylonJS/Babylon.js/milestone/5 Here is link with history I'd follow: https://uploadvr.com/ar-js-efficient-augmented-reality-for-the-web/ Seems to be lineage of successful AR examples: https://github.com/opencv/opencv_contrib/tree/master/modules/aruco https://github.com/artoolkit/jsartoolkit5 https://github.com/jeromeetienne/ar.js For AR -> Babylon is most promising to me. How would one go about such a feature? Thanks, max123 and NasimiAsl 2 Quote Link to comment Share on other sites More sharing options...
HeadClot Posted July 14, 2017 Share Posted July 14, 2017 Just thought I would chime in on this. ARtoolkit 6 is in open beta. They said they would be continuing support ARToolkit,js in a video I saw a while back. I cannot find said video at the moment. That said I would like to see AR support for Babylon.js JackFalcon 1 Quote Link to comment Share on other sites More sharing options...
davrous Posted July 15, 2017 Share Posted July 15, 2017 Hello, We've discussed about it this week to think how to integrate that in a near future. We don't know yet how it will be achieved technically but we're definitely interested in exploring AR. David JackFalcon 1 Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 17, 2017 Author Share Posted July 17, 2017 Thank you Mr. @davrous, I'm happy to make a technical attempt ... : ) Quote Link to comment Share on other sites More sharing options...
GameMonetize Posted July 17, 2017 Share Posted July 17, 2017 Please do..We did not start anything yet JackFalcon and NasimiAsl 2 Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 17, 2017 Author Share Posted July 17, 2017 Ok, scheduled deep-dive AR ... Temechon 1 Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 24, 2017 Author Share Posted July 24, 2017 Ping, @jerome ... hello, : ) Apologies if this is bad timing, or if I misunderstand, or if I am completely off-course, not uncommon, etc. : ) Considering Babylon AR integration study (attempt). Do you have tips? Perhaps we chatted on twitter about level-of-difficulty - so curious about it. I'd like to learn AR better and show findings here. Considering the complexities: cross-browser camera access, marker-tracking, ... ok that makes sense. Thinking perhaps a good place to start might be -> worldspace / local space / 3D translation over background video component? Is that it, would like to try... For a basic hello world knot, are there other complications? (scaling, I wonder.) Any tips from the trenches? : ) Hoping to (try to) get down in there this next month. All the best, Thanks. : ) Quote Link to comment Share on other sites More sharing options...
jerome Posted July 24, 2017 Share Posted July 24, 2017 I guess you are confusing me with another Jerome ( https://twitter.com/jerome_etienne ) because I don't know anything about VR ;-) JackFalcon 1 Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 24, 2017 Author Share Posted July 24, 2017 Ah, yep! Thought that might be the case. Thanks... Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 28, 2017 Author Share Posted July 28, 2017 AR-DEEP-DIVE-DAY-1: Learning comes from @jeromeetienne and @AndraConnect (follow twitter) +1. Hello && welcome. : ) For everyone who already knows, JE and AC are doing phenomenal work in AR. Of the 3D rendering options for AR, BabylonJS would be an exceptional renderer - I think the best. However, I am biased by great results. Turns out -> there are two phenomenal "Jeromes". Two phenomenal "Davids" we can handle! But now two phenomenal "Jeromes"!?! Does this happen? I only wish... Jk. : ) Straight to the point. Here-Is-What-I-Am-Learning about AR: AR.js - is here - https://github.com/jeromeetienne/AR.js thanks to @jerome_etienne Reading through master +1. Great components under THREEx = {}. the code is vanilla and beautiful.. The ReadMe (local) is the best starting point (https://github.com/jeromeetienne/AR.js/blob/master/README.md). Too many links!!! A good sign. : ) #1 KEY-INFO: From ReadMe: AR.js uses genius solution: - "**emscripten and asm.js**! ... compile artoolkit c into javascript" Thank you Brendan Eich. : ) Jerome is on it... - "Only thanks to all of them, i could do my part : Optimizing performance from 5fps on high-end phone, to 60fps on 2years old phone." - ARToolkit, is also impressive: https://artoolkit.org/ wow - crashes Chrome Dev Tools over "asm". Oh Text search. Watch out for that! mmm, look at those shaders!?!. - Innovations in; shadows, reflection, portals, multi-marker, table-top, & "Principles of AR" Look at that... with principles. even. Like+1. - In addition to ARToolkit, it looks like two other engines may be used: aruco, tango. 2. After a big BLOB of minified ASM awaits this helpful hint: Quote " The ARController is the main object for doing AR marker detection with JSARToolKit. To use an ARController..." Nicely done, and then... var ARController = {}; ... as a prototype : ) In a nutshell... Quote "ARController detects markers in given image. The process() method dispatches marker detection events during run." a good place to put a breakpoint? : ) Yes... ARController talks to artoolkit... : ) Basically accesses AR-API. Awesome. As expected... obj.onSuccess = function() { } It calls back ... with a video? yep <video> saw that... new world. : ) PLAN: Look: <a-frame><a-marker>export AND Threex base object full of tools. (interesting hoisting pattern) -> AND (using .position [gotcha] camers.matrix) maybe slip in BABYLON.Mesh.CreateKnot() ... at the last moment? Rendered to a second <canvas> <video> at a higher z-index... would it work? Could it work? -> Idk. UPDATE -> It worked UPDATE -> more of a "cannonball" than a "belly-flop". UPDATE -> lucky is good. Looks like: PresentationMatrix.... Transform Matrix.... Camera.matrix. UPDATE -> It was BABYLON.Camera._computedViewMatrix and TransformMatrix calculation in an array to BABYLON.Matrix.FromArray(). Of course this is only a learning exercise in AR... Because 1) The "Davids" will surely do something more phenomenal, and if not -> the"Jeromes" will. : ) And 2) I am only pedaling furiously on a tricycle, far, far... behind. If you are NOT a "Jerome" or "David", like me, this thread sprinkles out a few breadcrumbs from a BABYLONJS + AR.js learning collision. Hope it helps. GameMonetize and jerome 2 Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 28, 2017 Author Share Posted July 28, 2017 More AR Notes: 3) In UI Renderloop it looks like scene.visibility = true, onMarkerFound(); And position aligned by putting Matrix into camera.matrix. Confirm: 3D is rendered in <canvas> above <video> And scene.visibility flickers on and off. Position Alignment and scaling, seems to be the big trick... Presentation Matrix, Tranformation Matrix, ... This is helpful: https://github.com/artoolkit/jsartoolkit5/blob/master/doc/introduction.html And this: https://github.com/artoolkit/jsartoolkit5 And: https://threejs.org/docs/#api/math/Matrix4 ... Yep, that describes problem .... brianzinn 1 Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 29, 2017 Author Share Posted July 29, 2017 UPDATE: Found out why the freeze on "use asm"... in devtools. It is because of word-find! Same thing happens in Sublime. WARNING: Searching through the huge blob of ASM... can be slow! SOLUTION: Search "around" it. Lol. Or separate file... : ) Also found out some other stuff. -> Into the ProjectionMatrix, init, float32Array(16) and the Matrix4 compatibility review. Quote Link to comment Share on other sites More sharing options...
ozRocker Posted July 30, 2017 Share Posted July 30, 2017 is webRTC supported by iOS Safari and Chrome now? Last I checked, there was no support but I heard Apple was planning on it. Bowser from Ericsson supports it so you can always test there until Apple catches up Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 31, 2017 Author Share Posted July 31, 2017 AR Deep Dive Day 3 -> success. "Hello World" - BabylonJS knotBox rendering in AR.js (with three.js knotbox): TransformMatrix placed into camera._computedViewMatrix. Code posted: http://www.3dwebscapes.com/demos/arbjs5/ Insert this second line: look at with Hiro marker. markerObject3D.matrix.getInverse( modelViewMatrix ) cameraB._computedViewMatrix = new BABYLON.Matrix.FromArray(markerObject3D.matrix.toArray()); //SPLICE-FINAL! There is still some offset bug. Perhaps default world rotation thing in artoolkit (I'll look). @Deltakosh & @davrous & @jeromeetienne I hope this helps. Key-Takeaways: The type float23Array(16) looks compatible at the Transformation Matrix. Building from Array, can re-render in a single line of code. : ) There was some difference in Matrix Math -> around multiply (so yeah, consideration) BABYLON.Math seem sufficient coverage with (https://doc.babylonjs.com/classes/2.5/matrix) so no need for gl-matrix.js? I'd think, at first. But then... Ah wait, dependancy-consideration... AR API, so ok, understand fully. That might get tricky. : ) Babylon.Matrix lives here (https://github.com/BabylonJS/Babylon.js/blob/master/src/Math/babylon.math.ts) Thx, Dad72, jerome and brianzinn 3 Quote Link to comment Share on other sites More sharing options...
GameMonetize Posted July 31, 2017 Share Posted July 31, 2017 Hey this is UBBER cool. Do you recommend we change stuff in bjs to make it easy for you to integrate? JackFalcon 1 Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted July 31, 2017 Author Share Posted July 31, 2017 Thanks @Deltakosh ... thought about it. That was a really good question. So I rewrote the answers twice (quick)(thoughtful). : ) QUICK: I do not see changes for Babylon as a result of this test. THOUGHTFUL: Here is (possible) minimum and maximum of the problem space. At minimum: it is a single-line-code-splice (above) - Theoretically all BJS could redirect through that line of code. Into the projection matrix... - Subject to robustness testing of more complex sets will show if anything breaks. I'll do that. - and fix the Projection bug (above) [possibly default axis mis-alignment from artoolkit]. At maximum: AR Integration is a fascinating landscape of vast latitude (it seems to me). So maybe - thinking of AR in steps, stages, or phases might be helpful. For example; hello-world, multi-marker, flat-surface, gps, reflective, etc... 1) #1 Big question of integration. Seems to be - who does the Matrix Math? This is the tricky delta I mentioned. I don't know the solution. It needs good ideas. Honestly, this was shown to me by @jeromeetienne. He mentions a big refactor for an API that reduces dependencies. Sensible. For gl-matrix. The question I have is: if BABYLON.Math is fully suitable (and can be extended quickly to match) - what then? Like I said a big question. Many options. Probably it needs ideas from the group. My question, for that is: how could AR be added in as a package? And then pass Transformation Math to Babylon? AR - does come with a big house-guest - who eats a lot! JK, mostly. The Artoolkit, (or any toolkit) is a big chunk of asm - which would be good in a separate file (package). Or, maybe it should be rewritten eventually? For example, there is barcode scanning code, so maybe it needs streamline, fruther optimization. Idk. A pure AR solution from ground up might be inevitable. And why tango integration may be cool. But that is only a guess. Hello Hololens ar-engine compiled to JS with enscripten? Idk. Strangely running both WebGL 3D renderers - at the same time - has not (yet) melted my laptop! This is the simple path through the woods or the mirage I mentioned. I will look at robustness. : ) As mentioned, emulating matrix math stymied at one spot around multiply and invert. Can still isolate - but not sure what that was, and it was late. 2am. : ) 2) A number of important improvements likely remain for full AR implementation. By that I mean all this: A) there is much latitude in continued improvements in AR for a while (...). B ) They do not appear to need BABYLON framework change. Probably package addition. And finally C) notion of fresh perspective of context, methodology, and optimization (below). Like building a flying vehicle - we have a ways to go yet. : ) Apologies for excessive detail (this is what I saw while looking): EXAMPLE 1: "Matrix Methodology" ->Placing the computed Transform Matrix into the camera._computedViewMatrix is ok. And possibly limited programmatically. QUESTION: Are there ways to manipulate MatrixTransform with absolutePosition instead (making rotation scale independent)? Using Compose(), etc? In otherwords, leave camera matrix as is - and just compute mesh as a result of local space(ish). I'll run tests.. No doubt, a deeper understanding of emulating by multi-marker may show ... something different. I'll try that. EXAMPLE 2: "UI stuff" User camera getUserMedia() requires https. Also it would be nice to know what symbol it is looking for, and a btn for camera shut off. All could be handled gracefully in standard way in the future. //TODO try button, img, <span>... etc. EXAMPLE 3: There is barcode scanner code in the ar process loop, it seems, probably old example. Which I think is cool (barcode as anscestor - really?), but a performance optimization available? That is the notion of possibly a streamlined pure-ar context approach, from top down. But. Probably already happening. : ) I will try those things and put results here. Thank you. GameMonetize 1 Quote Link to comment Share on other sites More sharing options...
jeromeetienne Posted August 1, 2017 Share Posted August 1, 2017 You guys are crazy! i love the double rendering of the demo ? JackFalcon 1 Quote Link to comment Share on other sites More sharing options...
jeromeetienne Posted August 1, 2017 Share Posted August 1, 2017 I love your energy! so i just started working on it at 4am. sleeping in another life. Code will happen here until it goes math and object3d ... enougth to be pushed into ar.js itself https://github.com/jeromeetienne/AR.js/tree/dev/three.js/examples/object3d-glmatrix jerome, JackFalcon, brianzinn and 1 other 4 Quote Link to comment Share on other sites More sharing options...
adam Posted August 1, 2017 Share Posted August 1, 2017 18 hours ago, aFalcon said: 1) Big question of integration. Seems to be - who does the Matrix Math? This is the tricky delta I mentioned. I don't know the solution. It needs good ideas. This is an option: https://github.com/toji/gl-matrix Brandon Jones is well known for his work on WebVR. https://twitter.com/Tojiro JackFalcon and jeromeetienne 2 Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted August 1, 2017 Author Share Posted August 1, 2017 @jeromeetienne cheers. Thank you for object3D-glmatrix. I have some availability to run tests with BABYLON. jeromeetienne 1 Quote Link to comment Share on other sites More sharing options...
jeromeetienne Posted August 1, 2017 Share Posted August 1, 2017 2 hours ago, adam said: https://github.com/toji/gl-matrix I definitly agree, i plan to use this one adam and JackFalcon 2 Quote Link to comment Share on other sites More sharing options...
max123 Posted August 1, 2017 Share Posted August 1, 2017 That's some crazy shit!! Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted August 2, 2017 Author Share Posted August 2, 2017 Lol, AR is great, setting up next test now ... 10xcrazier! Maybe it should have warning label... Warning: may influence perception. Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted August 2, 2017 Author Share Posted August 2, 2017 BJS in ARjs: small planetary system test -> success Orbiting spheres.... background (skybox) plane. Everything seems ok. Needs fine-tuning ... z and y axis seem inverted. -> Twisting and turning the marker makes a big difference. Single line of code: cameraB._computedViewMatrix = new BABYLON.Matrix.FromArray(markerObject3D.matrix.toArray()); Quote Link to comment Share on other sites More sharing options...
JackFalcon Posted August 2, 2017 Author Share Posted August 2, 2017 The world is off axis... ...and maybe that is ok. Lol... Dad72 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.