BABYLON.Augmented Reality ...

[GameMonetize]

Huge!!!! Need a video

[HeadClot]

15 hours ago, Deltakosh said:

Huge!!!! Need a video

Agreed! 

[JackFalcon]

Nice, yep. I'll figure out video...

[JackFalcon]

UPDATE: bought camtasia. @jeromeetienne, what do you use for AR Video?

Hopefully I didn't waste money....

 

[kopacabana73]

Hello,

I'm new on this forum. Thanks for BabylonJS, it's really great!

Actually, I'm working on a solution like AR.js but I detect images, not markers. I use a lot of algorithms for tracking. After few days, it's possible to track an image. 

I use SolvePnP to determine 3D environment . This algorithm returns 2 vector3, one for the position, one for the rotation. But i've got nothing for scaling. I have also a camera matrix. I'm not sure all values are good, especially for Camera Matrix.

Now, I want to use BabylonJS to map a simple plane on top of detected image but I'm completely lost.

I've got a first question : which component must be manipulate ? Camera or plane ? If it's plane, how to define camera properties (fov, etc) ? If it's camera, I think plane must be at 0, 0, 0, but  which size (size of trigger or just 1 unit) ?

How proceed with matrix ? Only on camera or on plane or on plane and camera ?

I inspect AR.js and Babylon-ar.js but I don't understand how to apply my results matrix on it. I've read many articles but nothing help...

 

I NEED HELP to solve this (little) problem !!! Could anyone help me ??? Thanks

 

[JackFalcon]

I can help...  : )

Here is a plane, with an image on it:

            var imageMaterial1 = new BABYLON.StandardMaterial("texture", scene);
            imageMaterial1.diffuseTexture = new BABYLON.Texture("../img/babylon_image1.png", scene);
            imageMaterial1.diffuseTexture.hasAlpha = true;


            var imagePlane1 = BABYLON.Mesh.CreatePlane("imagePlane1", 3.0, scene, false, BABYLON.Mesh.DEFAULTSIDE);  //CENTER
            imagePlane1.position = new BABYLON.Vector3(0,3,7);    
            imagePlane1.material = imageMaterial1;

 

2. possibly scale the image plane down  with a parent...

var arParentMatrix = new BABYLON.Mesh("parentMatrix", scene);
//then
imagePlane1.parent = arParentMatrix;
//and
arParentMatrix.scaling.x = 0.15;
arParentMatrix.scaling.y = 0.15;
arParentMatrix.scaling.z = 0.15;

 

Another little bit of code which helps figure out which way is up! helpful in AR...

This gives a nice x,y,z colored axis code adapted from a great playground:

    function displayMeshAxis () {

        //Center sphere
        var centerSphere = BABYLON.Mesh.CreateSphere("centerSphere", 6.0, 2.0, scene);
        centerSphere.position = new BABYLON.Vector3(0,0,0);

        //Change rendering group - always on top.
        centerSphere.renderingGroupId = 1;
        //centerSphere.parent = arParentMatrix; //scaling down
        centerSphere.visibility = 1;

        //render the x,y,z axis
        centerSphere.computeWorldMatrix();
        var matrix = centerSphere.getWorldMatrix();
        var origin = centerSphere.position;
        // find existing axis for this box and dispose
        var xAxis = scene.getMeshByName("xAxis"+centerSphere.name);
        var yAxis = scene.getMeshByName("yAxis"+centerSphere.name);
        var zAxis = scene.getMeshByName("zAxis"+centerSphere.name);
        if (xAxis!=null){ xAxis.dispose();}
        if (yAxis!=null){ yAxis.dispose();}
        if (zAxis!=null){ zAxis.dispose();}
        // calculate new normals for this mesh in world coordinate system
        var xNormal=BABYLON.Vector3.TransformCoordinates(new BABYLON.Vector3(100,0,0),matrix);
        var yNormal=BABYLON.Vector3.TransformCoordinates(new BABYLON.Vector3(0,100,0),matrix);
        var zNormal=BABYLON.Vector3.TransformCoordinates(new BABYLON.Vector3(0,0,100),matrix);
        // create axis lines
        xAxis = BABYLON.Mesh.CreateDashedLines("xAxis"+centerSphere.name, [origin, xNormal],3,10,200, scene, false);
        xAxis.color = BABYLON.Color3.Red();
        yAxis = BABYLON.Mesh.CreateDashedLines("yAxis"+centerSphere.name, [origin, yNormal],3,10,200, scene, false);
        yAxis.color = BABYLON.Color3.Green();
        zAxis = BABYLON.Mesh.CreateDashedLines("zAxis"+centerSphere.name, [origin, zNormal],3,10,200, scene, false);
        zAxis.color = BABYLON.Color3.Blue();
    }

That code might help you determine the orientation of your matrix, if it is unexpected etc...

It is also possible to pass a mesh in as a parameter, giving x,y,z normals to any mesh, above stubbed as:  ar-world-matrix-centroid. : )

 

You may need to rotate the worldMatrix, depending on your detection-orientation, for example:

arParentMatrix.rotation.x = 1*Math.PI / 2;

 

 

[JackFalcon]

https://twitter.com/3dwebscapes/status/911668425423863808

Video, and still trying to get it to embed here...

[brianzinn]

The thumb up at the end of the video is a nice touch!!  Great work  

[JackFalcon]

Lol, @brianzinn, that was the hardest part ... +1 thx. : )

[kopacabana73]

Thanks aFalcon

If I understand, the matrix returns by SolvePnp must be applied to arParentMatrix ! But how do I initialize the camera (fov, near plane, far plane, position, rotation, etc) ? I can retrieve the following camera properties : width, height, fov. Which orientation should I apply to the camera ? Is it the 3D object or the camera that needs to be handled?

 

[JackFalcon]

I see. Good questions. How to initialize the camera...

 I'd start with a playground search for "camera", in this situation, thanks to gr8 advice from sir @Wingnut.

LINK: https://doc.babylonjs.com/playground?code=camera

That link is good for 1 hour of entertainment. 

 

There are a few different cameras. So, I wrote a (stub) factoryMethod to init a bunch of them...

       
    asc.createCameraSets = function (){

        //CreateFollowCamera
        //CameraParams: name, alpha, beta, radius, target, scene.
        asc.sceneFollowCamera = new BABYLON.ArcRotateCamera("ArcRotateCamera1", -Math.PI/2, (Math.PI/2.9), 10, new BABYLON.Vector3(0, 5, 0), scene); 
        asc.sceneFollowCamera.wheelPrecision = 15;   
        asc.sceneFollowCamera.lowerRadiusLimit = 2;      //zoominandout
        asc.sceneFollowCamera.upperRadiusLimit = 300; 
        asc.sceneFollowCamera.upperBetaLimit = 1.5;//lock camera in place vertically.
        asc.sceneFollowCamera.lowerBetaLimit = 1;//lock camera in place vertically.
        asc.sceneFollowCamera.beta = 1.5;
        asc.sceneFollowCamera.radius = 5; //distance camera is from target.  
        asc.sceneFollowCamera.position = new BABYLON.Vector3(0, 20, -120);
        asc.sceneFollowCamera.setTarget( new BABYLON.Vector3(0,20,-119));

        //CreateFreeCamera
        asc.sceneFreeCamera= new BABYLON.FreeCamera("FreeCamera1", new BABYLON.Vector3(0, 10, 0), scene);
        asc.sceneFreeCamera.position = new BABYLON.Vector3(0, 20, -120);

        //CreateArcRotativeCamera
        asc.globalRotativeCamera = new BABYLON.ArcRotateCamera("GlobalRotativeCamera1",  0, 0, 0, new BABYLON.Vector3(0, 0, -0), scene);
        asc.globalRotativeCamera.setPosition(new BABYLON.Vector3(0, 20, -120));
 
        //Multi-Camera-Capability
        cameraSets = [asc.sceneFreeCamera, asc.sceneFollowCamera, asc.globalRotativeCamera ]; 

        if(false){ //TODO
            scene.activeCamera = asc.sceneFollowCamera;  //TODO
            asc.sceneFollowCamera.attachControl(canvas);
        }else if (true){
            scene.activeCamera = cameraSets[0];  
            cameraSets[0].attachControl(canvas);  
        }else if (false){
            scene.activeCamera = cameraSets[2];  
            cameraSets[2].attachControl(canvas,true);            
        }
    }

    /**********************END-CAMERA-CODE**************************************/

It refactors into a nice factory, via parameter. And, doesn't include  the VR camera (above) but it drops in very well.

So, that is how to initialize... a bunch of camera capabilities.

 

See how it switches between arcrotatecamera and freecamera, and followcamera, vrcamera...?

Of these multiple camera options - it depends entirely on your requirements.

 

That is why the playground search is probably best to get the camera behaviors.

 

Next step: BABYLON docs are really good... 

https://www.google.com/search?q=babylonjs+camera&rlz=1C1CHZL_enUS698US698&oq=babylonjs+camera&aqs=chrome..69i57j0j69i65l2j69i60l2.6314j0j4&sourceid=chrome&ie=UTF-8

Tutorials:

https://doc.babylonjs.com/tutorials

 

 

[kopacabana73]

Thanks aFalcon !

I'm gonna experiment and test your suggestions. I send you my results soon.

[gui17]

On 01/08/2017 at 1:27 PM, jeromeetienne said:

I definitly agree, i plan to use this one

HI i’m interested in WebAr i'm doing a search project and I would like to know if there is a way to use .obj and .mtl files on AR.js if could help me? i’ll be very thankful. I'm new in this AR stuff.

[Dad72]

Is there a tutorial and documentation to use AR? How does it work in practice? Do you need a tablet, a Iphone, PC ? how to view the results? I have a project using this technology, but I do not understand how does it work.
Thank you

[JackFalcon]

Hi @Dad72  : )

Great idea on a tutorial! 

Backs up dump-truck... Beep! : )

 

UPDATE: thoughts collected & terrible jokes removed, mostly...

 

The trickiest part of AR, at first -> was navigating the support across different devices. You need to get a 'proven-stack'.

Because... AR is asking for camera context access to OS (...) -> and not all browsers play the same way (sound familiar).

So the biggest win was to find a confirmed 1st Prototype, getting that first AR working... .

After that is a "now what" moment.

 

But first, you should anticipate a few -> "peculiar-AR-issues" encountered in 1st Prototype .

The Laptop was easy, Chrome worked. Then tablet I needed Explorer, and phone (I got a new one, ... pixel) : )

There is a  list of supported devices, on AR.js readme. good read. .... 

It is a wild-west of options. Hard to tell how all (core, kit, webkit, tango, aruco) options exist across browsers.

History repeats itself... 

 

Getting that first AR prototype has a few tiny hurdles with "marker-targeting":

 - distance, lighting, bending the paper,.... printing out the target, and which target to use? Hiro.

For example, one test, do NOT put my thumb over target! Lol.

Simple but important peculiAR-issues..... to  be anticipated.

 

After a successful render, then... how close to hold the device to the marker to device.

Far away is sometimes good. Too close stops render detection. Just like... too far stops render detection!

That surprised me - too close fails...

 

Another one ... the camera will show quickly, but if the framework is still loading in the background...

the experience is a delayed non-render. Which may need a background loading indicator. : )

Probably easy....

 

Last, confirming which way was UP turned out to be a big deal.... because, as usual - it was I who was confused (imagining AR which sticks to walls [lol]).

  Seems simple, not always. The Z or the scale can also be off pretty easily. Then - all things hidden behind camera (joy, pure joy... : ).

 

Ok getting down to INSTRUCTIONS: How does it work.

1. Check out AR.JS, download it.

2. Find "HIRO" marker (somewhere down in ARJS, or google). Print that out..

3. Inside AR.JS there is support for BabylonJS down in a folder: babylon.js/example

4. Inside that folder is a basic example! Run that in a server. localhost. Click Yes, Access Camera. Show the camera the Target = render on target "marker". Wait! It might work... : )

      - For example...make sure light is good. Not too close. Not bent. And thumb isn't in the way.  And in dev - that RENDER Z IS NOT BEHIND CAMERA - st(ar)angely common.

      - Look for stuff like that.

5. There are three new demos almost ready (solar-system, PBR, & forest-fire), for BABYLONJS/ARJS.

6. Once you get  1st Prototype running then - what now?

          -  challenges like "hiding the skybox". Water reflecting hidden sky?  Scale parentMatrix down... (ah,ha!)

7. Still you need to check  "cross-device" compatibility... like run a local server to get a demo on tablet... 

8. FYI: Two Concurrent Experiments:

9. BABYLON Extension - awesome, although remember those "peculiar-ar-issues" - they are back! AR is weird for packing into BABYLON extension. In progress...

           - Suggestions on how to build BABYLON extensions welcome! Whiteboard time...

10. Another  experiment: Loading ARJS/BABYLON into Ionic App. What I am working on now mostly...

          - I wonder if Camera Context is improved from within Native App Context - it should be. Right?

Derp, Jokes removed...: )

Which engine to choose...? That is a thing!?!

Do I have to choose an engine, or how does that work?

... question.

 

 default arwebkit - go!

But also configure out how to render on a table (tango?, arkit?, core?).

And there are others, just saw a new one the other day...

 

IMPROVED:  previously edited late night writing... :]

UPDATE: more jokes removed... :]

[JackFalcon]

@gui17  .obj -> https://doc.babylonjs.com/extensions/obj

something brewing with .gltf -> https://doc.babylonjs.com/extensions/gltf 

I've seen imports "mangled" not sure why that is yet...

That's all I know.... : )

 

[Dad72]

Thank you for all the information. I will try to experiment soon. I'll wait for a tutorial on how to proceed. I'm not sure to understand arwebkit? What is it used for?  is this an additional bookstore. Babylon.js and AR.js are not enough? I did not understand the marker to print, is that necessary. I have seen videos where there is no use of marker print. I think these are questions that many can ask. Thanks aFalcon

[JackFalcon]

Super cool@Dad72, yep will try for tutorial (better than above) [slightly improved], collecting thoughts...thx.

Follow-up questions, anyone can ask, nice approach -> here are thoughts on those:

- arwebkit, library extended from aruco, compiled from asm (if I am not mistaken). wish I knew asm. What is goin on in asm land?

- and that is also the primary reason for Extension (interoperability) consideration, how to package a confluence? White board time...

- Bookstore.... yeah! Pretty much! There you go, that helps ... thanks! That is an answer... kudos.

 

- "Marker" to print: there are many kinds!. : )

          - printed marker which is derived from QR-Code logic. OR

          - table-top which is derived from -> color-gradient plane analysis.... nice, right? OR

          - GIS, where something pops up at a coordinate.. AND

          - others..... face-recognition, and object recognition, and still more...

 

So the "HIRO" marker is a stand-in for other camera image analysis later.

One cool thing dug up, was .... how "Hiro" image works.

Understanding that it is like a simple bar scanner.

top-down, left-right pixel-tracking

<are you what I am looking for?> 

<You are! ok render. : >

 

It finds the QR (basically) and then instead of forwarding to some website - we forward to babylon - for canvas matrix overlay.

ding. (clear backcolor (!!!) ) . : )  And don't shut it off. Don't flicker canvas visibility.....

EXAMPLE: marker rendering is cutting in and out - just leave the matrix world on. Tada. Works great!

For testing, and also UX..... : )

 

 

TIP: great way to get big picture of AR:

- if the camera on your phone is the "reality", then the screen is the "Augmented Reality".

Then go cross-device just like in the old days: google-goggles, tablet, phones, laptops, etc.

A bunch of ar design principle pop up... with many options....

 

Sorry -> occasional blathering.. :]

 

 

 

[Dad72]

5 hours ago, aFalcon said:

GIS, where something pops up at a coordinate

and face-recognition, and object recognition, and still more

I think that's what I'm interested in.

exemple : 

 

 

[JackFalcon]

Hi @Dad72 that looks like "plane-detection". Notice when chair "snaps to floor" or how the rectangle matches the table...

It could be GIS theoretically - or big projection matrix. Which explains the problem with things hiding in z axis often.

Another thing happening in that demo, looks like "shadow emulation" under the chair, at one point the shadow disappears then reappears...shadows have a strong effect for realism.

I have not dug down into shadows, plane-detection, or big projection matrix yet. Ha! Eventually.. (idea) it would be cool to visualize the matrix ... for this use case.

Steps would be 1) get arjs running on Hiro, then 2) configure the init object to use arkit engine instead of arwebkit. 3) configure arkit to use "plane-detection", etc...

: )

[Dad72]

What is Hiro? what is the difference between arkit and arwebkit? what should be configured in arkit? Sorry for questions, I really like this technology, but I start completely with AR.
I look forward to finding a tutorial detailing with a documentation to use this.

PS: My phone and my tablet are on android, is this compatible

Thanks again aFaclon for your time.

[JackFalcon]

All good. : )

"Hiro" is an Image Marker. It is  a first step to working with AR....

When the camera sees this "marker" it renders something, in this case the Y-axis.

Other versions of "markers" can come from "plane-detection" like "plane color gradients" or GIS points...

Example above, works by scanning the image of the video (left to right, top to bottom) and if it sees the target, an event is sent ...to render the canvas overlay.

The same would be true for plane detection or GIS, just the "targeting" is different... color or GIS point... (targeting can be anything-btw).

That is where BABYLONJS comes in. Turns out it is a piece of cake to put a <canvas> overlay over the camera image. : )

The tricky part was aligning the ProjectionMatrix. And inverting it (apparently)... still don't understand that part.

We have an example working where Babylon includes AR.js and uses arwebkit to create AR overlay ProjectionMatrix [scale down parent : ].

 

To your question, AR.js supports arkit, and it can be configured...

For that answer and for compatibility the following link is best:

https://github.com/jeromeetienne/AR.js/

For compatibility I've found a number of work-arounds, different browsers, localhosting, etc. So iterating different options is good. : )

 

[Dad72]

Ok, I'm starting to understand better now. Thank you.

[JackFalcon]

Cool, thank you for help generating docs....

 

[Dad72]

I sent you a PM. I do not know if you saw it ?