Cutout shadows in custom shader

[mainequin]

Hey Folks!

In our scene we have some images with transparency
on quads. The transparency is just used to mask the
silhouettes of the image, so we have no larger half-
transparent areas. So far we used the default shader
for this, which produced nice silhouettes.
All these Objekt are registered in a shadow generator
from which we extract the shadow map an pass it in
a custom shader.

After writing our own shader for the images we ran
into a problem: The shadow generator now only takes
the quad into account, not the transparency of the
image. Therefor the shadows of these images are
square.

When constructing the custom shader we tell baby-
lon to use alpha blending and we also tell the shadow-
generator to use transparency shadows. The diffuse
texture is simply loaded with a texture sampler in-
cluding the alpha channel. We even tried to clip frag-
ments under a certain value but the shadow genera-
tor still ignores it.

We looked through the default shader but couldn't
find anything that's helping us in this matter. We
found the opacity map, which to our understanding
is just a separate, black and white masking texture.

So our question is: What steps are necessiary to
generate shadow silhouettes based on the alpha
channel of an objects texture? As we said, clipping
with a threshold or cutout for that matter would be
enough.

Thank you for your time
-Mainequin Team

[Wingnut]

Hi Mainequin & Team!

I don't know if this applies, but a user once asked about something similar, and we made THIS playground.

https://www.babylonjs-playground.com/#1Z89KY#10

I'm not a very good coder, so mistakes and unnecessary settings are likely.  If this playground is NOT on-topic, then feel free to modify it until it is so.  Most forum helpers like to see the issue repro'd (reproduced) in a BJS playground scene.  Then everyone can experiment with it, easier.

Let me try to summarize/restate.  You have an array of mesh, similar to a wall-of-monitors, with a single texture/image that spans all the tiles.  You would like the shadow generator to treat this tiled mesh-array as a single mesh/texture, and cast shadows from the non-alpha parts of the texture.  Did I restate that correctly?  (I hope so) 

I think the above-reffed playground can show how to do that... perhaps.  Without mesh-merging all the tiles, I think you will need to push ALL tiles into the shadowList.  It might be the .useAlphaFromDiffuseTexture that is important, here.

Anyway, we're rolling with a testing playground, at least.  Sorry if I misunderstood the issue.

[mainequin]

Dear Wingnut,

thank you for your answer! It took some time but
we peeled the relevant code from our project and
used the code from your example playground to
illustrate our problem. Basiaclly we want to repro-
duce what you did using the default shader with
our custom shader.

http://www.babylonjs-playground.com/#VZKI0U
#12

As you can see in this example, Felix correctly re-
ceives shadows, but does not cast them accord-
ing to his silhouette. Changing the "hasAlpha" set-
tings of the texture does not change this behavi-
our. Our guess is, that the depth texture of the
shadow generator does not account for the trans-
parent parts of the texture.
As stated in our last post we could not find rele-
vant code in the standard shader that differs
from our approach, therefor we think it coult be
a setting in the shadow generator.

Thank you very much for your time
- Mainequin Team

[mainequin]

Is there no solution for this problem?

[GameMonetize]

Sorry for the late reply

 

Here is the fix:

https://www.babylonjs-playground.com/#VZKI0U#15

 

Check lines #202, #209