Creating new camera for room scale VR

[brochington]

Hey Yall,

I've started to work on a new camera in Babylon called "VRRoomScaleCamera", which with allow for walking around in a virual environment. There are a couple reasons I'm working on this:

1) I want get my Vive working nicely with Babylon.

2) I've noticed that the VR implementations within Babylon use the "old" webVR API, and want to see how the newer API works. 

3) To learn! 

I've really only started this weekend. So far I've been able to get the positioning and orientation information through to Babylon, and project a scene into the HMD. At the moment I'm working on getting the projectionMatrix updated correctly so that a "ground" mesh actually is on the ground. 

There is a learning curve for me in all this (first time using typescript, doing dev work on a PC, much lower level graphics code), so I'm going to be taking things super slow. A lot of this is discovery for me.

I'll leave a link to my fork at the end of this, but please understand that this in NOT functional yet. Constructive comments are very much welcome on the commits! This is especially true if you want to point to a class/method that I should be using, or gotchas that I should account for.

Will keep this thread updated, but for now:

https://github.com/brochington/Babylon.js 

[Mike Pelton]

Hi @brochington- I think this will be a very popular addition! Coupla tangential questions - does the Vive give you some sense of absolute scale? Is there depth data so you can tell if you're in a small or large room? Will look into WebVR and WebRTC to see if there's a depth feed available - with Tango and many more rumoured depth-sensing phones we'll be able to get the data, if we can use it in VR for occlusion and obstacle detection many new opportunities open up.

It sounds like you're no stranger to 3D so apologies if this is a dumb observation - one gotcha to watch out for is "up" and what happens if you look straight along the "up" axis - I'd imagine the Vive will deal with all that for you but it's always a potential pitfall.

Good luck!

[brochington]

Hello @Mike Pelton! 

The WebVR API does report the "Stage Parameters" of the playzone, which consists of a sizeX, sizeY of the floor in meters. It also has a transform for determining the correct hight of the user. The Vive software detects when you are close to the boudaries of your play area, and will display a nice virtual wall if you get too close. This wall is independent of what you show through the HMD, so it works in my trials. If you want more info on the WebVR API, check out https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API, with the caveat that some of the docs aren't current. 

The depth perception stuff in Tango looks very interesting. I think that a rich mobil VR experience will be very important if more people are to try out VR.

I am very much a stranger to 3D, and your observation is a good one! I appreciate it. So far the Vive seems to handle "up" pretty well, but their aren't many times I've need to look directly up yet. 

[davrous]

Hey! Super cool! It was in my plan to do something similar as I got the Vive also recentely but don't have much time to work on Babylon.js lately :-(

Feel free to contact either me or @Deltakosh if you need any kind of help. I'll be able to test your implementation on my Vive too ;-)

[brochington]

@davrous,

Hi! Thank you very much for the kind offering of help. A couple questions will really help me understand the internal workings of the Babylon camera system. I'll hit you up soon with a PM. Thank you, and looking forward to making headway!

[brochington]

Update time!

So I've been able to Orientation working, as well as positioning. This means that I've been able to "walk around" spheres and boxes so far. The biggest issue as of now is the correct "room space" or basically making it feel like you are actually in a room. The room scale demo at webvr.info has been very helpful, especially with some of the matrix math. It's frustrating to be "close" but also far off. Just part of the game, I guess =). 

My (extremely messy) code is on github: https://github.com/brochington/Babylon.js/tree/vr_room_scale_camera, with most of what I've been working on in: https://github.com/brochington/Babylon.js/blob/vr_room_scale_camera/src/Cameras/VR/babylon.vrRoomScaleCamera.ts. If you ANY tips/tricks/suggestions/advice, I am very much willing to listen. 

Also on the list of things to tackle is the projection matrix for each of the Vive's eyes, as well as some checks to make sure users can actually use the camera.

That's it for now. Hopefully much progress will be made soon. 

[GameMonetize]

Hello some comments on your great camera:

- We have BABYLON.Matrix.PerspectiveFovLHToRef : https://github.com/BabylonJS/Babylon.js/blob/master/src/Math/babylon.math.ts#L2874

- We also have a way to create a translation matrix and a rotation matrix and multiply then directly

- this line https://github.com/brochington/Babylon.js/blob/vr_room_scale_camera/src/Cameras/VR/babylon.vrRoomScaleCamera.ts#L159 could be replaced by an override of CheckInputs

 

[brochington]

@Deltakosh

Thanks for taking a look a my work so far, really appreciate it =)

Yeah, the fromRotationTranslation and perspective method are in there just to try and get things going correctly. Would the method you are referring to for rotation/translation be `Compose`?

Forgive my lack of knowledge, but would BABYLON.Matrix.PerspectiveFovLHToRef  be used to update the Projection Matrix?

Right now I'm setting the position and rotation directly in my _updatePosition2 method (also will be changed =)), but is there a way to set the view Matrix directly here, or are there other factors in play here?

One of the more interesting parts of the WebVR API (at least to me) is the addition of a new `requestAnimationFrame` on the actual VRDisplay object. This version of this method runs at the native speed of whatever platform you are running. In this case (with a Vive) it calls it at 90 FPS. This is great, but i don't know how well it would play with `Engine.runRenderLoop()` and `Scene.Render()`. My initial thought is to create a new runRenderLoop function specifically for VR Displays. I assume this would let me run `checkInputs` at 90 FPS. very curious to your thoughts on that. 

Again, appreciate the guidance. I have to admit that even though objects aren't well placed yet, and the positiong and view are all wrong, it's pretty fun to be able to "walk" around a scene =)

[GameMonetize]

1.  Would the method you are referring to for rotation/translation be `Compose`? Yep

2.  BABYLON.Matrix.PerspectiveFovLHToRef  be used to update the Projection Matrix? correct

3. is there a way to set the view Matrix directly here? You can override getViewMatrix 

 

For requestAnimationFrame we can think about replacing the one used by the engine to run the render loop

[brochington]

I've had a little bit of time to work on the room scale camera tonight, and It's coming along. 

I've been able to get the projection matrix outputs correctly enough to not make me cross-eyed. I've also gotten positioning working fairly well, though it isn't the smoothest. Still plenty to do.

@Deltakosh Thank you for the feedback! I've gotten rid of the extra functions I using instead of  `Compose` and `PerspectiveFovLHToRef`.

I am running into an in issue where the light source doesn't seem to be correct. I've attached two short videos, One which is what the camera looks like currently, and another of the same scene, just with the FreeCamera. Not sure why I'm getting the black bottoms of the boxes, and a `shadow` of the box in the center. Thoughts?

Latest code is in https://github.com/brochington/Babylon.js/tree/vr_room_scale_camera.

Getting there!

roomscale_demo_1.mp4

what_room_scale_should_be.mp4

[GameMonetize]

Hello this is because your cubes are inside-out

Go to their material and set mat.backFaceCulling= false;

[brochington]

@Deltakosh Okay, awesome, that fixed it right up =)

[brochington]

Update time:

The VRRoomScaleCamera works! Projection and View Matrix seem to be updating correctly. I'm having a couple of Typescript errors, and I'm not sure the best way to approach them. here is my current code: 

https://github.com/brochington/Babylon.js/tree/master_with_vr

And the errors I'm seeing: 

..\..\src\Cameras\babylon.camera.ts(551,68): error TS2339: Property '_viewMatrix' does not exist on type 'Camera'.
..\..\src\Cameras\babylon.camera.ts(553,41): error TS2339: Property 'getEyeFOV' does not exist on type 'Camera'.
..\..\src\Cameras\babylon.camera.ts(559,68): error TS2339: Property '_viewMatrix' does not exist on type 'Camera'.
..\..\src\Cameras\babylon.camera.ts(561,41): error TS2339: Property 'getEyeFOV' does not exist on type 'Camera'.
..\..\src\Cameras\babylon.camera.ts(578,43): error TS2339: Property 'getEyeFOV' does not exist on type 'Camera'.
..\..\src\Cameras\babylon.camera.ts(676,34): error TS2346: Supplied parameters do not match any signature of call target.

Their seems to be a bug in Chrome WebVR dev that is breaking the 90FPS for me, FYI. 

If anyone would like to try it out ( @davrous ) I've created a very basic dev project. You can find it at https://github.com/brochington/webvr-babylon-test-1. This does require the dev build of Chrome that can be found at webvr.info.

Please let me know if you see issues, or have any questions. Thanks!

[GameMonetize]

Hey..which version of babylon.js are you using? (2.5?)

[GameMonetize]

Where do you use _viewMatrix and getEyeFov?

[Steffen]

Hi @brochington, @Deltakosh

are there any news about the room scale vr cam? In the latest nightly I've seen a "trackPosition"-Flag as a WebVR option.

btw, thank you for sharing

edit: Tested my project with the latest nightly and the WebVRFreeCamera with the flags "compensateDistortion" and "trackPosition" set to true. Works! But I think I need some more tweaking - now you get really fast some motion sickness

 

[GameMonetize]

@davrous and @RaananW worked on it and it should work (I don't have a VR device to test  )

[RaananW]

Just in case this wasn't answered yet -

I am not sure what room-scale camera is. The WebVR camera takes all parameters from WebVR (hence from the headset itself), so it should actually fit your scene perfectly. You can adjust the position changes (scale the position) in case your world units are not the same as the units used in the camera itself.