Panoramic View in BJS: 360° Equirectangular Photos and Videos?

[Fabri]

Hello Everyone, I want to use Babylon.js (great library!!) to show 360 panoramic view for equirectangular photos like http://www.babylonjs-playground.com/textures/equirectangular.jpg

The example code http://www.babylonjs-playground.com/#11GAIH is perfect for my needs, but result scene looks like mirrored (look at the child in the original photo and in the rendered scene)... I can suppose the main reason (skyboxMaterial.backFaceCulling = false and camera inside the box)...

However, does exist a way to show in a 360 panoramic view an equirectangular photo avoiding a mirrored result? 

Thanks for your help 

[Wingnut]

Hiya @Fabri, welcome to the forum!  Sorry it is taking so long to get replies.  Although I have found no solutions, I've certainly tried a few (hundred) things.  I'm no expert, though.

This is an interesting challenge/issue!  No matter what I tried (so far), I couldn't get that dark blue speedboat to move to the OTHER side of the pier.    hmph.

BUT... I ONLY tested material and texture things (which I suck-at).  This MIGHT BE a job for the view matrix. In other words, do something to the camera->world relationship... instead-of the skybox/texture (I suck-at matrices even more). 

I dunno.  I think there's a matrix transformation that could do something magical, for this.  hmm.

All in all, good find, good challenge, not a bug, but a missing feature or yet-to-be-discovered adjustment.  Let's hope we get some more replies/help, soon. 

Welcome again, good to have you with us.  We'll talk again soon.  I'm going to keep goofing-around with your nice playground (thx for that, too), and I'll report discoveries, here.  Be well, party on!

[Fabri]

Hi Wingnut, I am happy to have received a response from you today...(I was almost losing every hope to get an answer)

I agree with you that what I found is not a bug but may be just a missing feature...unfortunately I need such feature...

I am completely new to 3D world and its terminology (...view matrix? camera->world relationship?)...so please be patient if I don't understand deeply your suggestions...

As far as I could verify with Three.js https://threejs.org/examples/webgl_panorama_equirectangular (just drag our equirectangular texture http://www.babylonjs-playground.com/textures/equirectangular.jpg into the page)

it works like a charm but with a little trick: geometry.scale( -1, 1, 1 )...in practice Three.js allows to invert the mesh "inside-out" by setting geometry.scale.x = -1;

Anyway I don't like Three.js's spherical view and I prefer much more Babylon.js's skybox view...

Do you know how to invert a mesh "inside-out" in Babylon.js?

I think that Babylon.js authors or any other expert, could solve my little problem in a few minutes, by checking how Three.js is working...

As a I said, I'm just a beginner and for me everything is something magic 

Hope to hear from you soon!

Regards,
 Fabrizio. 

[Wingnut]

Ooh, negative scaling, what a good idea.  I tried some experiments with it... no change.  Darn.    http://www.babylonjs-playground.com/#11GAIH#1 (lines 31-33)

In webGL, there is a "matrix" (a data structure used to store orientation data... used by webGL shaders... to render a scene)... called worldViewProjection.  It is "derived" by multiplying the worldMatrix * viewMatrix * projectionMatrix.  You can see a diagram of it... here... https://www.eternalcoding.com/?p=113 in the section called The Voodoo Behind Matrices.

It is MY theory... that reversing (180-degree "transform") of the camera viewMatrix... might fix our issue.  I'm not sure how to do that, but I will continue investigating it.

Many of the BJS "big dogs" will be returning from holiday vacations, soon... so maybe we will get responses from smarter people than I, soon.  Thanks for your patience and kind attitude.  We'll find the answer... or die trying.  heh.

To recap, we wish to reverse (rotate 180 degrees around Y axis) the image... as viewed from inside the skybox (current cam position).  All help highly welcomed!  Thanks!

[Fabri]

Hi Wingnut, I did my homework and read the excellent document that you suggested!

Waiting for BJS "big dogs" (as you call them), I found a temporary workaround (just a copy and paste of parts of babylons.js code and examples about shaders...), thanks to your very valuable suggestions!!!

You can take a look at http://www.babylonjs-playground.com/#11GAIH#2 , now the dark blue speedboat is on the correct side of the pier

A trick to flip the texture is "coords.x = 1.0 - coords.x;" [line 30] 

Anyway I keep on waiting for a better approach and final solution from our BJS Experts...hope they will find time for us.

Thanks again!!!

[Wingnut]

WOW!  Excellent!  You're a genius!  Talk about taking the bull by the horns... holy crap!   Script-based shader code... nice!  (I think Babylon.ShadersStore would work, too.)  (There's also an on-the-fly shader code assembler... called shaderBuilder - external JS extension, but that's for another day).

Amazing solution, Fabs!  What a dive-into hacking!  Have you been lurking for 2-3 years?    Hacking shader code is certainly not a "noob" thing.  I predict that you are going to become a local BJS superstar. 

Thanks for sharing that... and congrats.  Yeah, the big dogs will be looking into this, including a new big dog named Fabri. 

[jschwuch]

Hey!

You can easily fix this by setting the coordinate system to right handed.

http://www.babylonjs-playground.com/#11GAIH#3

[Fabri]

Great!! Simple and effective! Thank you very much for your support.

Are there any side-effects or limitations of changing coordinate systems? 

All further considerations or suggestions are welcome

[Wingnut]

Excellent solve, @jschwuch!  3rd post on the forum!  Amazing.  I'm hanging out with geniuses!  I'm honored!  Welcome jschwuch!  And thanks for the help!

[jschwuch]

The feature is quite new so there might be bugs. Apparently in one of my projects a texture on a ground disappeared but I didn't look into why that happened, because I don't need the right handed coordinate system there, just tried it.

There "shouldn't" be any limitations caused by changing the coordinate system as far as I know, but I didn't add the feature so I don't really know if it is fully supported yet.

Alternatively, if the change of coordinate system proofs problematic. You could also load the image into a canvas first, mirror it horizontally, generate a data url and give this data url to babylon. But you'll have to wait for the image to load of course. I played with it, but it's tricky to do in the playground. I'll look into it later or tomorrow and give you an update

[jschwuch]

Here is a working version with flipping of the image via canvas. Strange enough: if loaded from canvass i have to flip both horizontally and vertically...

 

http://www.babylonjs-playground.com/#11GAIH#8

[Fabri]

Excellent jschwuch!

I like your solution to flip the image via canvas. I updated your good code changing flipping just horizontally (you were right about it):  
        context2.translate(canvas2.width,0);
        context2.scale(-1, 1);

and setting option invertY to "true" during Texture creation:  var tex = new BABYLON.Texture("data:flipped", scene, true, true,........

http://www.babylonjs-playground.com/#11GAIH#11

If BJS authors would add an option for Texture similar to invertY also to "invertX" or something similiar then all  extra effort we are doing to flip would be avoided...may be...just an idea...

What do you think??

Thanks a lot for strong support!!!

 

[jschwuch]

Ahh! I missed the invertY option  Good to know. Looked into the source and flipping Y is done with an option for the pixelstorage directly in WebGL. WebGL doesn't have an option there to also flip X, so i suspect it won't be supported. And I wouldn't want to implement it with a canvas there as this might cause some performance problems. Maybe we should think about an other mode (like FLIPPED_FIXED_EQUIRECTANGULAR_MODE) for the texture so we can flip the texture coordinates directly in the shader. Maybe a dev could say a word or two about that idea, I'd implement it if it's a reasonable approach.

[GameMonetize]

Do you think this is a common need? If yes I'm ok for the new mode

[Wingnut]

Hey DK... welcome back from caroling!  Everyone missed you.

Nice job with the forum catch-up, thanks for that.

DK, if YOU would have needed this, how would YOU have done it?

Perhaps with a .isBackground Layer ...where texture.uOffset bound-to camera.alpha?

Okay, maybe not that way. 

What would deltakosh have done... for a quick, one-time, no-framework-change -solve for this issue?

(If ya got time.)  (thx)

Right-handed was certainly easy, but that seems like it could have FRR.  (far-reaching ramifications)  Thoughts?

[Fabri]

Thank you all, my new friends, I really appreciate your consideration and support!!

Well, I don't know if such new mode/feature is a common need as regards gaming development,

but for those users that would use BJS in a different kind of web application

(for example to create a 3D user interface to navigate a virtual environment and then to show some real 360 photographic details....without mentioning augmented reality experiments....)

such feature could be useful (or simply fundamental).

So I think that it depends on BJS future objectives and how much universal and multi-purpose aims to become in future ...if it wants to expand its horizon or not??

I'm really interested in all your opinions about that !!!  

[GameMonetize]

We are not locked to gaming but to everything that we can do in 3D

@WingnutThanks for the kind words. I would have use a canvas to update the image for a one shot approach

[Wingnut]

  Hope ya'll had a swell holidays.  DK, did you bury your little girl in Christmas gift wrappings?  FUN!

Hey Fabri... it isn't JUST about expanded horizons, but it's also about adding yet another parameter to BJS Texture constructor.  One more thing to teach and learn.  That's why invertX parameter is weighed for usefulness.  Will it be used often enough for warranting a parameter addition?  hmm. 

We could get fancy... and bog the Texture code.  We already have invertY parameter.

In the invertY tester code...

IF (invertY instanceof Boolean)  then apply as boolean.

IF (invertY instanceof Vector2)  then apply as 2D transform.  (an X and Y rotation)

Nah, that's shoddy overloading... demented. 

How 'bout a new dataType... called a Twolean or Duolean...  two booleans side by side.  So if invertY were set to truetrue or 11 (Two-lean), both X and Y would be inverted.

Nahhhhhhhhhhhh.  hehe.  Goofy.  But I'm trying trying trying... to avoid that added parameter to Texture constructor. 

Man, canvas2.toDataURL()... @jschwuch just smoothly moving from canvas/Image() to BJS texture... nice.  Who IS this guy/gal?  That's some pretty advanced stuff for a 9-post noob.  Phew!  Cool!

A one-image skybox.  hmm.  "Equirectangular projection", eh?  hmm.  Looks like it might have a big future.  img.src is dripping with security blockades, though.  So, it's back to the old way... so we can view some cool web-shared equi-pics... without being security-blocked.    Fun!

[jschwuch]

Hey, thanks @Wingnut for your kind words

This guy is new to BJS but has some years of pure JS experience. I'm currently using BJS for a production project at work, really doing everything learning by doing (had computer graphics lectures at university though) and abusing BJS way too often  (like calculating transformation matrices myself and setting them on private fields "Ouch!"). But I'm planning to stay here a bit longer and do some private projects too. And I'm interested in developing BJS once I learned enough too feel comfortable with it.

Now back to topic. I wouldn't use another parameter for BJS Texture constructor either. Because WebGL doesn't support it when creating the texture and we would have to use a canvas in the BJS Texture constructor to do the flipping. It's most likely much more efficient to do the flipping via the shader. That's why I supposed another coordinateMode for Textures, like this:

skyboxMaterial.reflectionTexture.coordinatesMode = BABYLON.Texture.FLIPPED_FIXED_EQUIRECTANGULAR_MODE;

@Deltakosh I'm supporting this functionality too for uses like 360 photos and such. Question is, should I jump in at the deep end and code the changes myself and send a pull request or will you (or some other experienced developer) do the changes?

[GameMonetize]

If you can do it it is better. Else you will have to wait for my immense list of todo to be cleared first

[GameMonetize]

But it should not be too hard, here are some pointers:

Add the constant there:

-https://github.com/BabylonJS/Babylon.js/blob/76c2745cd76d096ff04b12802a3c83bb91049155/src/Materials/Textures/babylon.texture.ts#L15

-https://github.com/BabylonJS/Babylon.js/blob/c675f483a04e3ade06f9fdf30332823473403aae/src/Materials/babylon.standardMaterial.ts#L48

-https://github.com/BabylonJS/Babylon.js/blob/c675f483a04e3ade06f9fdf30332823473403aae/src/Materials/babylon.pbrMaterial.ts#L44

Use it there:

-https://github.com/BabylonJS/Babylon.js/blob/c675f483a04e3ade06f9fdf30332823473403aae/src/Materials/babylon.pbrMaterial.ts#L732

-https://github.com/BabylonJS/Babylon.js/blob/c675f483a04e3ade06f9fdf30332823473403aae/src/Materials/babylon.standardMaterial.ts#L378

Update shaders here:

- https://github.com/BabylonJS/Babylon.js/blob/dea49de07ac07a2bf7896aaa0e13c26e0f674c9e/src/Shaders/ShadersInclude/reflectionFunction.fx#L3

-https://github.com/BabylonJS/Babylon.js/blob/064a186c8aa0c092c85cb43c2c3565bfdd244325/src/Shaders/default.vertex.fx#L87

[jschwuch]

Ok. I'll look into this tomorrow, I'm sure I'll get this to work somehow. Thanks for the hints where to look  I'll keep you all up to date  

[MrVR]

Hi @Fabri , hey is it possible to do this with Video Textures as well ?

[Fabri]

Hi @MrVR, yours is a brillant idea! I really don't know if it is possible use (so I let experts to reply about it) a Video Texture for a panorama view in order to recreate a realistic animated virtual environment...but I agree with you that it would be incredible!!! The only test I could do is to insert a video texture in our sample static panorama: http://www.babylonjs-playground.com/#11GAIH#19

Let's wait for our experts suggestions!

 

  

[Fabri]

Good news @MrVR,  I found that using video texture for panoramas it's possible!!!! 

Look at https://threejs.org/examples/webgl_video_panorama_equirectangular.html

So, now we need to wait for BJS "big dogs" (as @Wingnut call them) to know how to do with BJS...