Advanced Baking of Reflections Script

[Rodrix3]

I am investigating BabylonJS capabilities for creating realistic reflections and I need the community feedback to create a possible improvement.

 

I want to build an automatic script that will create realistic reflections for every object on scene. If it works, I will share my code, but first I need some conceptual approval!

This is my idea:

   - Loop though every mesh on scene

  - For each mesh, create a reflection probe with an update of 1 so that the frame is only rendered once. The reflection probe shall contain all other meshes on scene except the current mesh.

  In this way we would get N cubemaps, being N the number of meshes on scene.

 - Next for each mesh we would set the reflection texture to the one generated by its reflection probe. And set the option to use local cubemaps.

  For the *cubemap position* we would use the center of the mesh (assuming a rectangular bounding box). And for the cubemap bounding box size we would set it to the bounding box size of the mesh.

 

All reflections would be position relative and calculated once only at startup.

What do you guys think of this approach? Did I miss anything? Could it be improved?

 

Thanks so much!

p.s: @Deltakosh and @Pryme8 I have seen you talking about local cubemaps on recent forum posts. Any ideas? Am I getting the idea of the local cubemaps right? Thanks

P.s.s: for the texture size we would establish a minimum and maximum. Get all meshes on scene and get maximum bounding box and get minimum. Based on that we will do a ratio and the system will decide different textures sizes according to mesh relative size Inc comparison with the rest.

 

  

 

 

[Guest]

Well this looks good to me

It could be really memory intense if you have a lot of objects but it is worth a try

[Rodrix3]

@Deltakosh here is the first proof of concept

https://www.babylonjs-playground.com/#J0D279#6

The idea is that you could add this line on any project to generate real reflections for the whole scene:

//Just adding this line generates real reflections for all meshes

addRealReflections(scene);

 

If we are able to optimize this, I think it could be a great asset to Babylon.

 

To begi,n I need some help. I am not sure how to define the boundingBoxSize, so we can set local reflections (which is the new feature from 3.2). I used extendSize, but it doesn't seem I am getting the right result.
What exact property for getBoundingInfo().boundingBox should I use to get the size?
 

Line 17: scene.meshes.material.reflectionTexture.boundingBoxSize = scene.meshes.getBoundingInfo().boundingBox.extendSize;

 

My ideas to continue:

Imagine if we could create a load screen to wait fo reflections to be created (right now the screen gets frozen until reflection probes are done).
Imagine if we could generate them once, and save to disk as hdr (and not have the user never wait.. they would be generated once by the developer and then loaded into the projects), and we could make this script into a pre-deployment utility to bake all reflections.

.. well those are some ideas.

What do you think? Maybe I am using local probes in a different way than intended and this whole idea won't work..

Let me know your thoughts

Thanks so much!

 P.S: I hope other babyloners can chip in too to see if we can make something great together

 

 

 

[Rodrix3]

@Deltakosh, any thoughts?
..and any idea how to set  the boundingBoxSize properly?

I am not sure how to get the actual size of the mesh, and so if you look at my demo the reflections don't look as they should..

 

 

[Guest]

The main problem is that local reflection map are mostly intended for flat surface and not for spherical ones

 

But it should be better than that anyway:

* The bounding box size is expected in world space (extendsSize is local space you must use the world version)

* You must also set the boundingBoxPosition

[Rodrix3]

Thanks @Deltakosh

I updated it!

https://www.babylonjs-playground.com/#J0D279#11

I added these two lines and added a cube on the center to test reflections on a flat surface:

scene.meshes.material.reflectionTexture.boundingBoxSize = scene.meshes.getBoundingInfo().boundingBox.extendSizeWorld;

scene.meshes.material.reflectionTexture.boundingBoxPosition = scene.meshes.position;

 

(Are they ok?)

 

The results don't seem as expected....Also after changing to extendSizeWorld the reflection on the floor disappeared. Maybe we are not measuring the box size correctly or the position? (Is the position world coord?). 
Thanks! ..I hope we can make this work. It would be really interesting.

 

[Rodrix3]

The results are much better than not using local reflection probes, but I hope we can improve it.

I want to test this for a bigger scene. Is there any way I can put make each reflection probe to load one at a time instead all at once and put be able to display on  loading screen  the progress for each probe?

This is the comparison between local and non local reflection probes. There is an improvement, but I am quite sure most likely I haven't set the position or size of bounding box correctly, so please help me double check it.

[Rodrix3]

Here is an update with some tweaking:

https://www.babylonjs-playground.com/#J0D279#14

The results seems to be pretty GOOD and now reflections are making more sense:

. I think this idea has a lot of potential; however, I really need the community's help. I can't do this alone.
Deltakosh is always there on every post - and I thank you!

..but could others chip in so you can help me solve this faster?

IMPORTANT: Even if you are not an advanced Babylon user and you are reading this you could help me with some basics (I come from ThreeJs):

• How can I get the size of a bounding box that wraps a mesh (I need a vector3 with x,y,z sides of the box)?
  mesh.getBoundingInfo().boundingBox.extendSizeWorld <= Correct?
   
• How can I get the world position of a mesh?
  mesh.position <= correct? Or is this local position in relation to root?
   

• How can I get the size of a mesh by wrapping it with a sphere in world coord?
  meshes.getBoundingInfo().boundingSphere.radiusWorld <= correct?
   
• Is there any way to apply blur to a cubeTexture object? Or is there any way we can make the reflection appear blurred?
  reflectionTexture.blurKernel  and reflectionTexture.adaptiveblurKernel => do nothing (specs says this applies to mirror texture, but nothing about cubetexture).
  Is there any shader expert who knows how to do this via shaders? Or is there any other way to it?
  Could we convert it to equirectangular, blur it, and then reconvert it into a cubetexture? Is there a simpler way?
  We are on PBR, but roughness doesn't seem to blur the reflection texture.
   
• Is there any way to convert a cubeTexture to a equirectangular format (so it can be seen on 2d) and saved to disk? or easier question: is there a way to save a cubeTexture (from a reflection probe) to disk? In dds format or whatever format.
  The purpose would be to save the cubeTexture to disk, and not have it generated. We could create a tool to bake all reflections, and then save to disk. 
  So we would calculate all reflections once, save to disk, and on deployment just provide the textures that would be loaded as cubemaps from equirectangular / spherical / or any other projection that can be saved in one file.
   
•   Is there any way to get callbacks on BABYLON.ReflectionProbe and know when each has finished rendering?
  I am using probe.refreshRate = BABYLON.RenderTargetTexture.REFRESHRATE_RENDER_ONCE;
  .. I would like to have on the load screen (Babylon loader) a text that shows how many probes are pending to render. Is there any way to integrate the Reflection Probes into the Babylon Loader?
   
•    Is there anyone who can explain more about reflectionTexture.boundingBoxSize?
  I can't find any information about it. I tried setting it to  mesh.getBoundingInfo().boundingBox.extendSizeWorld() and I get bad results (reflections appear too small):
  Arbitrarily I set it to getBoundingInfo().boundingBox.extendSizeWorld.scale(6) and now results are better; but I need to really understand what it means and what I am doing,
  so I can set this value dynamically for all objects on scene correctly. 
   
•   If you can't answer any of the questions but you like what I am doing, also please let me know!
  And if you are an advanced user and you think my concept is a bad idea, please also let me know so I don't lose so much time in this investigation, or so we can together make it different and better!

Thanks everyone!   

 

[Pryme8]

Wow, good work.  When I get some time ill review and lend input.

[hcmetal]

Just sharing my 2 cents here. I think this will be a solid functionality that will make Babylon even more powerful.

Even if the reflection update is not 'real-time-ish' due to performance constrains, having dynamic HDR map generation will be a great efficiency boost.

I am a total newbie to Babylon, but have some experience with traditional ArchViz. 

So I might not be able to help with the code, but I would like to try this out and provide test cases ( archviz scene with interior and exterior ) once this is functional in Babylon.

@Rodrix3, thanks for your good work. Really looking forward to this.