solved layerMask is ignored when alwaysSelectAsActiveMesh is True

[alexoy]

Hello!

I've got a strange behavior of a layerMask parameter. If I understand it right:

1. mesh.alwaysSelectAsActiveMesh = true - this is about mesh visibility and related calculations only. When we load a scene and this mesh is behind a player - it is not active, but becomes active when you turn around, and some things are calculated only when the mesh is active, right?

2. mesh.layerMask - this is about multiple cameras only, right? When you want some meshes be visible on specific camera only - you put needed layerMask for that.

 

What I have here - https://www.babylonjs-playground.com/index.html#2GXKNW#30 - here are 2 cameras, 1'st normal standard, 2'nd takes the right half of the canvas and is in front of the first one.

Why all meshes are visible on both cameras?

If you comment out lines with alwaysSelectAsActiveMesh - 2'nd camera will show a sphere only, that is an expected result for me. That means alwaysSelectAsActiveMesh makes layerMask useless.

[aWeirdo]

Hi @alexoy 

alwaysSelectAsActiveMesh means that the mesh is Always active, Thus the layerMask can't disable it from a viewport, and i believe the mesh isn't culled either when out of viewport.
You shouldn't really need this unless a mesh should always be active (even when out of viewport / camera view), but if you do need them to always be active, yet respect camera layerMask, i suggest overwriting the culling logic with your own game logic. (find, copy & edit camera culling function(s) to follow your game's logic & re-load function(s) after the babylon.js core)

[alexoy]

alwaysSelectAsActiveMesh was mentioned somewhere about scene optimizations, with things like freeze(), freezeWorldMatrix() etc. I use it too, because without it my scene freezes when I turn around and see the meshes for the first time. Probably because new meshes become active and BJS does some things when it sees the mesh for the first time. This camera freezing is very noticeable, so I can't just leave it as is.

What can I do then? I set all the meshes as active now.

[aWeirdo]

@alexoy
As this behaviour is not how alwaysSelectAsActiveMesh was designed to act, you have to overwrite a core function.
The following will force layerMask to be respected. Please test performance in your project as i'm not sure how it will act with your freezing issues.
Line 118 in PG;

- if (mesh.alwaysSelectAsActiveMesh || mesh.isVisible && mesh.visibility > 0 && ((mesh.layerMask & this.activeCamera.layerMask) !== 0) && mesh.isInFrustum(this._frustumPlanes)) {

+ if ( (mesh.alwaysSelectAsActiveMesh && ((mesh.layerMask & this.activeCamera.layerMask) !== 0)) || mesh.isVisible && mesh.visibility > 0 && ((mesh.layerMask & this.activeCamera.layerMask) !== 0) && mesh.isInFrustum(this._frustumPlanes)) {

Copy the whole function from PG, lines 52 -> 146 and load in your project after the Babylon.js core file. (3.2)
https://www.babylonjs-playground.com/index.html#2GXKNW#31

[alexoy]

@aWeirdo, big thanks! Performance looks good, both actual and potential FPS are similar. Potential FPS digit changes too fast to be able to say more precisely, but numbers look more or less the same. In a debug layer - maybe it's possible to set a refresh rate lower? It will be much easier to read a potential FPS

[aWeirdo]

@alexoy
Glad to hear it works!

I rarely use the inspector i'm afraid, and i don't know any of it's coding,
but i made a github issue about slowing down the update rate; 

https://github.com/BabylonJS/Babylon.js/issues/4302