Using the GearVR controllers

[HoloLite]

I guess the questions are for @brianzinn or anyone else who knows :)

Now that 3.2 is officially out and does have the basic support for gearVR, I am really eager to try it on.

1. So tried my app on the GearVR, but the controller does not show up. It works on winmr and vive btw.
Do I need to do anything special ? 

2. Which of the observables are currently supported ?
In particular I am using the following four. Please let me know which one(s) are supported.
I believe the GearVR does not have menu button, so that one is probably not available, but the touchpad values and button are needed.

this._vrHelper.onControllerMeshLoadedObservable.add((controller: BABYLON.WebVRController, eventState: BABYLON.EventState) => {
	controller.onMenuButtonStateChangedObservable.add((eventData: BABYLON.ExtendedGamepadButton, eventState: BABYLON.EventState) => {
		// do something...
	});

	controller.onTouchpadValuesChangedObservable.add((stickValue: BABYLON.StickValues, eventState: BABYLON.EventState) => {
		// do something...
	});

	controller.onTouchpadButtonStateChangedObservable.add((gamepadButton: BABYLON.ExtendedGamepadButton, eventState: BABYLON.EventState) => {
		// do something...
	});
}

Btw, I really look forward to testing babylonjs webvr app on the new OculusGo. The oculus and Samsung browsers seem to have pretty good support for webvr.
I am hoping the bjs controller support for gearvr would work for Go as well.

[brianzinn]

hi @HoloLite - sorry for the late reply. been on vacation.

#1.  No, you won't see the controller mesh.  Reason for that is that the GearVR and Daydream controllers are 3DoF (rotation only), so does not have any position.  The position needs to be inferred from rotation and where your elbow would be (using 1.7 default height last I checked).  The basic support only adds that the button will work, so was really only a minor improvement to make it usable.

#2. If you use the GearVR top button, you will notice that you can rotate yourself, so TouchPad observables ought to work, if I remember correctly.

I have to dust off my controller and try 3.2 - have been away and not on my computer in over 3 months...  otherwise would have been at Build 2018!

[HoloLite]

@brianzinn welcome back :)

Well, someone has fixed it (I believe deltakosh), now the controller is showing up - although the mesh is not the precise same one used by Go.

I am still having troubles with the Go event handler. Seems like I am not getting the callback.

 

[brianzinn]

You are right - it was added (https://github.com/BabylonJS/Babylon.js/commit/85decdfccc5d3aa99ea294fa4e8228c59b95b1fc) in 3.3.
You can swap out the controller mesh itself quite easily with MODEL_BASE_URL and MODEL_FILENAME public static properties.  You can grab the controllers in a-Frame - see link from davrous.  Also, there are controllers in the unity toolkit, but can't find link right now - they were also low-poly.  Double check licensing on those projects.

edit: OculusGo is set to use the GearVR controller and I don't have one to check that button indexes are the same. (https://github.com/BabylonJS/Babylon.js/blob/1c3c25a72ec2785089d2b64c08d0e2d404a53b1a/src/Gamepad/Controllers/babylon.poseEnabledController.ts#L92)

[MackeyK24]

Where are guys getting this to make gear V games in Babylon js.

all I see is webvr camera here and there and a gear controller. But see nothing how to actually build and package and deploy... and most of all how to run it in a gear vr or occulus go device...

 

or or is it just me not finding it ?

[HoloLite]

@MackeyK24 Hey! I thought this was the very idea of webvr itself! No plugin, no install, no store!

You only need to run the oculus browser that comes with gear vr and oculus go and point to your website where you host your webvr app.

Check this sample app: https://hololite.github.io/CityExplorer/?speed=10

It runs at ~30 fps on the Go or Gear. Decent but can be improved imo.

 

[MackeyK24]

That’s it ... Kool

But I still would love to see some starting point. Something explaining the basics. Something to explain control, something to explain movement.

I see tons of different WebVR Cameras and controllers but nothing real specific on how to use them.

I have never done any kind of VR app or game so I don’t know the ins and outs of WebVR in BabylonJS.

i guess I am looking for some kind of walkthrough for VR and BabylonJS 

 

[HoloLite]

I am also still learning this stuff.

Here are  a few pointers to get you started.

https://doc.babylonjs.com/how_to/webvr_helper

Excellent articles by DavRous

https://www.davrous.com/2017/12/22/babylon-js-vrexperiencehelper-create-fully-interactive-webvr-apps-in-2-lines-of-code/

https://www.davrous.com/2017/07/07/from-zero-to-hero-creating-webvr-experiences-with-babylon-js-on-all-platforms/

[MackeyK24]

Thanks @HoloLite ... I say the Zero To Here. But thought was for the whole HTC WebVR which i thought used a differnet API than trying to do GearVR

I couldn't tell what is for what... WebVR, GearVR, VRDeviceOriented... I just make out what is what. And no one has made any training outlining what you have do to make each of these type of games... Start to finish.

I spent SOOOOO much time making the toolkit, i never get to play with actually making mobile or vr type games.

I tell you... When i finally figure out how to actually do all things. i will build Mechanics for these type games right into the toolkit and i make courses showing how to actually do all this stuff using toolkit... From Start To Finish

 

[HoloLite]

It is great to have an accomplished toolkit dev like you to start focusing on webvr! AFAIK most folks on this forum are html5 games devs, the webvr devs are not many. I want to see more getting into babylonjs vr. This will be awesome. 

Yeah the beauty of webvr is you develop once for all different vr headsets. WebVR api and BabylonJS layer take care of the platform details for you. Usually the only platform specific stuff that your webvr apps need to worry is the controller event callbacks. 

I think bjs will move into webxr in the near future as well, by then we can develop apps for both vr and ar devices. 

[brianzinn]

If you dig a bit into the VR Experience Helper, you will notice that it does require a pointer down event (click on button) to enable WebVR.  It's not enough to be using a VR device (with the Experience Helper) that you will automatically get the VR immersive experience, so that is maybe one difference for the toolkit that you need to add.  I may be wrong, but I think that is required for the fallback mechanism on device orientation for full screen (ie: Google Cardboard) and perhaps just good netiquette for connected WebVR devices?

The identifier of the WebVR device itself is used by BabylonJS to create device specific experiences.  So, once you add the VR Experience Helper then it works for all supported devices and browsers ie: for Gear VR you can use Oculus or Samsung browsers (https://webvr.info/).

[MackeyK24]

I read the articles, I a have sooo many questions. Is there anybody out there besides @davrous that would like to help me with the Camera Rig in the toolkit. I need more clarification on what cameras are still be used and for what device.

Still unclear about what makes WebVR device. For example I thought the whole 3D with glasses or headset or even cardboard was WebVR. But from what I can tell everything that is not a FULL headset device like a Vive ... does not actually use WebVR (or at least not the WebVR free camera) ... but a fallback to VRdevice oriented camera which sucks for VR scenes... that distortion post process slows thing down and has a jitter when moving. So how can that be WebVR

i just don’t get it, I don’t want to switch to another engine just to try a learn the main concepts for WebVR.

i see a lot of just run it browser. But don’t see anything about creating VR scenes... what makes it VR... how do you control what’s going on... I see stuff like if your device has controller they will show up... but I don’t see anything explaining the rhyme or reason as to actually use them. . How to interact with objects... like see hands to manipulate...

I guess your already supposed to know how to do this stuff... but I don’t ?

[brianzinn]

Just my interpretation, but I would say that Cardboard is not webVR.  To me WebVR (and WebXR) are standards.  When you connect a VR Device supporting the standard that extra information is available in supported browsers.  When you have a full headset the browser + graphics card redirect the scene to the headset, but otherwise BabylonJS does the best it can, which is to split the screen and post process the distortion for a realistic effect.

Once you are in a VR scene, you have a start position and facing direction.  The direction you point the controller is used for navigation and interaction.  The scene itself is the same as a regular scene without VR.

Somebody else will need to chime in on all the cameras.  All good questions - it is overwhelming to get started!

[davrous]

Hi @MackeyK24,

I'm interested in better understanding why you're lost with our approach. I've tried to make it as simple as possible so if you're lost, I'm missing my goal.

Have you watched the BUILD session we've done with @Deltakosh : https://channel9.msdn.com/events/Build/2018/BRK2436

Regarding complexity, all Unity VR developers I've met who have seen our Babylon.js WebVR approach were amazed by the simplicity and productivity. But you're coming from another background so we maybe need to write better doc.

Let's try to answer some of your questions:

- the VRHelper is your best friend to build WebVR scenes : http://doc.babylonjs.com/how_to/webvr_helper & https://www.davrous.com/2017/12/22/babylon-js-vrexperiencehelper-create-fully-interactive-webvr-apps-in-2-lines-of-code/

- building a VR scene in Babylon is not different from building a normal scene with some interesting particular details to note:
 - the unit 1 is the Babylon.js world should be considered as 1 meter in VR
 - you need to pay much more attention to performance as you need to target 90 fps on high end HMD and 60 fps on mobile while doing a double rendering of the scene (left/right eyes). So check the stats/performance tab of the Debug Layer to optimize your scene. 

- Once you've build your scene switching to WebVR should only be a matter of 1 to 2 lines of code thanks to the helper which covers:
 - download and display of the current VR headset controllers used (HTC Vive, Oculus Touch, Windows MR, etc.)
 - provide a gaze doing the mesh/hit testing for your
 - provide a teleportation option which has been optimized for performance and to reduce motion sickness

 - WebVR is a standard living on top of WebGL. If you're watching our BUILD session, you'll see that Microsoft has implemented it inside Edge and Mozilla in Firefox on Windows. On Android, you've got support thanks to Chrome, Samsung Internet or Oculus Browser. Devices that has a WebVR support are: HTC Vive, Oculus, Oculus Go, Gear VR, DayDream and Windows Mixed Reality devices (including HoloLens in last build). 

- if the browser or device doesn't support WebVR, we're fall-backing automatically to the VRDeviceOrientation camera using the sensor to simulate a HMD and doing a double rendering of the scene with lens deformation (which can be disabled). But, the VRHelper has been made to hide this complexity.

To enter WebVR, you are forced, by the specification and for security reasons, to have a user's interaction. That's why the VRHelper is creating a VR button for you to let you enter VR. This can be customized as described in the documentation. 

David

[MackeyK24]

Thanks Dave... I am a total newbie to VR ... it seems to be there would different kind of mechanics... but after reading your post I get it more... I still don’t have a picture in my head to how to write code to handle movement and what the common practice to handle touching or grabbing etc... nice to know that the VRhelper takes care of all or most of this for you... I will watch the build video and read the articles to get a better picture of what I need to do to make WebVR games...

Btw, it was not you... and you do a great job... it is my lack of knowledge on the whole VR thing and thinking there would different game mechanics especially moving and grabbing ... but again, that’s me not you ?

[MackeyK24]

Yo @davrous ... Hey man I really need your help in anchoring the virtual joystick

check this out... I have been working for months on fine tuning the toolkit and the various art tools to optimize your scene for Xbox console still running Webgl 1. 

My buddy was using the toolkit to make mobile scene after being unhappy with unreal engine mobile. He said the performance was killing Ue4 mobile. 

So I broke out my iPhone 7 and tried my space shooter stress test scene where I purposely use a lot of physics and instantiate prefabs up the wazzoo ... to gage what the device can handle... was really meant for pc... but my iPhone 7 played it almost as smooth as my Mac with no modifications... I went and bought a iPhone X last week and that same scene run so smooth at 60 FPS ... again not even optimized for mobile... but runs the scene Just as good if not a little better than my Mac.

that fueled me to wanna redo a mobile optimized game and I am getting killer frame rate on my iPhone ... but the joysticks are to hard to control... we gotta be able to anchor thos guys to the bottom corners ... or where ever we need to.

Do you think you can help with that... please ? 

[MackeyK24]

I got it working on my mac... thank god... and thank you you @davrous ...

Great freakin work... I just added support for the vr helper right into the Camera Rig on the toolkit...

So now using the toolkit you can go from 2 lines code to ZERO lines of code... Just a few options on the camera rig at design time but if you need runtime access... the scene manager exposes its internal vr helper used on the camera rig.. You can get a reference from scene components using the scene manager:

this.manager.webvr is the actual VRExperienceHelper used to active the scene.

 

 

Still playing the controller and mesh options... but at least you got me going bro... Thanks

 

[MackeyK24]

Yo @davrous or any other VR Helper Guys...

Uh Ohh... I had things going great in playground. But now for some reason when using the scene.createDefaultVRExperience() the VR button display for about 1 second then during splash loading, but when it renders the scene... the button goes away.

It does not not show any error, the VR button at bottom of screen just disappears...

UPDATE

Found it

 

[MackeyK24]

Thank goodness... I found my problem with disappearing VR Helper Button

Yo @davrous Can we pleas get a higher zindex style for the VR Button... I think its at ZERO now thats my problem... My canvas is default to zindex 1.... the button was showing behind canvas... something 10001... make sure its always on top

 

 

[HoloLite]

Well you are referring to the bottom-right small VR button right? When you click the button, you will enter the VR mode where the scene starts rendering inside the VR headset, by then the button will disappear. This is expected behavior. You can programmatically call bjs api to exit the VR to go back to the browser mode again. 

[HoloLite]

Btw, just curious, which VR headset are you using to run this? 

[MackeyK24]

Thanks @HoloLite ... found the issue... The z-index on the VR Button is 0... My canas was set to 1... So the button was BEHIND the canvas... so i could never select it to go into VR Mode... I just put my canvas on ZERO index for now... But would love to see the VR button with a default z-index higher than 0...

 

[MackeyK24]

Oh Yeah... Finally got things going smoothy WITHOUT any code, the Camera Rig component will handle all that plus expose the instance of the VRExperienceHelper as manager.webvr for any runtime access you will to the vr helper.

Now all the WebVR function is now easily enabled on your scene like so:

 

I also use built in tag system for floor meshes... So you can just set the primary Unity Tag or use the Babylon Tags Component to add the floor mesh tag you want to use and the scene manager will getMeshesByTag and use those as the floor meshes... Also supports tagging Terrains as floor meshes.

Thanks @davrous for the complete breakdown... That BUILD 2018 session and the 2 lines of code article cleared up alot. Again, i had NO CONCEPTS of what WebVR games should be, how you move, how you interact... But now i do (thanks to the VRExperienceHelper) and Looking at the VRExperienceHelper.ts class

 

[MackeyK24]

7 hours ago, HoloLite said:

Btw, just curious, which VR headset are you using to run this? 

I am using a GearVR and Occulus Go... I dont have a full headset yet.

[HoloLite]

It seems like there are problems with recent bjs build with regard to Go support. 

Qs for you:

Which version of bjs are you using? 

Does the controller mesh actually show up on the scene?