Contemplating a digital ruler using BabylonJS

[Veksi]

I was recently thinking to educate myself a bit more regarding BabylonJS and mixing it with reality, so to speak. I had an idea to create a digital measuring too, but quickly realized it's a good deal more difficult to pull of than one initial thinks. So hence a question if it's really impossible without access to native APIs currently and if so, what might be the deal breaker?

I saw news such as https://www.theverge.com/2018/6/22/17492846/google-augmented-reality-measure-app-arcore-tango-update where both Apple and Google are showcasing their new framework capabilities. From those examples I gather plane detection is at least a very desirable usability thing, but I'm not sure if it's necessarily needed. I don't know if BabylonJS can do plane detection since it looks like it requires access to native capabilities and it could be achieve by opening a WebRTC camera (using BabylonJS as usually) and sending them to backend or perhaps hosting BabylonJS inside a web frame in a native app and calling it. What I think is more important is some sort of depth perception (which, I suppose, would mean access to the lens properties such as focal length or more than one camera or some other sensor such as laser?) so that when a beginning and end of some line have been marked, the length of the line could be (accurately enough) calculated.

I wonder if the above is about the right kind of reasoning? Or put otherwise, does someone have an idea how to implement a digital measuring ruler in BabylonJS and how to go about it? I see there's some new WebXR stuff going on in the GH repo and this might actually pave a way towards. Naturally it'd be nice to work towards this already now.  

By the way, I've looked into examples like https://www.babylonjs-playground.com/#YGKM4#1 that combine some virtual object with a real world scene. Instead of a ball it could be a ruler fixed other end being fixed into some specific point.

<Edit: Found also https://newatlas.com/arrim-one-laser-measurer/55001/ 

Edited October 7, 2018 by Veksi
Found material that supports some conclusions.

[JCPalmer]

Though a webgl framework might help you display your findings or take actions with those findings, getting the findings through photo analysis is the really hard part.  This has little, if not nothing, to with webgl, though once you know webgl might benefit.  BabylonJS does zero photo analysis, I think.

Getting data from WebRTC and then passing it around is going to be slow.  If you are going to use native capabilities anyway, you should probably follow their examples of retrieving the camera data.

I will say using Cordova is probably not going to help you without a lot of work.  When you access the camera in Cordova using the common plug-ins,  those just call either the videocam or photo app for a given OS.  When you close those, control returns to your javascript.  If you write your own plug-in to access the hardware directly, which I am doing, then you have 2 new problems:

• It is platform dependent, so you will have to code for each OS.
• You are probably going to have to do your analysis in the plug-in itself.  Reason is that Cordova plug-ins can only return strings.  Turning one to base64, passing it back, converting back to an image will slow you down to about 5 fps.  This is even before you start trying to work with the data.

You might write a Cordova plug-in which accesses each platforms native AR offerings, or scourer the net for a plug-in which already does.  The same problem of passing the camera data back in string format is still going to be the bottle neck.  Using WebRTC on browser, and a second native app which also accesses the camera is probably the only realistic way today.

 

[Guest]

As you mentioned, we will soon be able to use WebXR to add AR capabilities to Babylon.js.

In the meantime your idea of merging BJS with native app to get ArKit or ARCore features is a great idea (done by the folks at Jig.space for instance)

 

[Veksi]

Thanks, @JCPalmer and @Deltakosh. It appears a hybrid one is the option to do that sort of stuff or a mobile device without a backend API. Until WebXR thingies make the roll a bit more.

[trevordev]

WebXR should make it into popular browsers early next year ?. but you can already use it via chrome canary and support in babylon is already in progress: https://github.com/BabylonJS/Babylon.js/issues/3899 .

[Guest]

?