Shadows in Custom Shader

[mainequin]

Dear Babylon JS community,
we as a company have decided, that we want to use Babylon JS for a larger project.
For this we have specific requirements for what the shaders have to be able to do.
I will first state the problem I am trying to solve and then give the context, for possible
alternative solutions.

PROBLEMS:
For our more complex shader computations we want to integrate shadows from at
least one shadow-generator in a custom shader. For reasons of confidentiality I can
not submit our current project code, that is why I created this test playground:
http://www.babylonjs-playground.com/#VZKI0U
We want to get the influence of all shadows on a fragment as a float value in the
shader for further computations. For this we encountered the following problems:
- Mapping to shadow-map coordinates seems to be wrong
- using functions like computeShadow() from #include<shadowsFragmentFunctions>
  yields not-declared-error
- computeShadow() yields always 1.0 as a result

COURSE OF EVENTS:
We started playing around with the standart material and shadow generators and
quickly got them to work. we wrote a small utility function for setting up the shadow
generators, which you can find at the top of the linked playground code. After this
we played around with uploading textures into our custom shaders and were able
to create the desired effects. We looked into uploading the shadow-map and the
shadow-generator parameters into the shader, which was sucessful. You can find
the uploads at line 113-115 of the linked playground code. Since we do not want
to write the mapping to shadow map coordinates ourselves, we looked if there is
already existing code, which we found in the shadowsVertex.fx, shadowsFragment.fx
and shadowsFragmentFunctions.fx files. While trying to get the mapping right, we
encountered the aformentioned problems. We were not able to get correct results
regarding the shadow-uv-coordinates, shaderincludes like the above mentioned
#include<shadowsFragmentFunctions> yields a "computeShadow() has not been
declared" error when used in the code after the statement and what code we
currently copied from these files seems to always yield 1.0 as a result for the sha-
dow intensity. We are turning to you now, because we are at a point where we
cannot find the errors in our approach/code anymore. We are required to use
Babylon JS version 2.5 in our project. Although it didn't seem to make a difference
for the shader code we looked through I wanted to mention it.

CONTEXT:
Our scene is basically shadeless, with multiple materials per object, distributed via
a mask. Therefor we combine a precomputed light texture (for individual objects)
with a diffuse texture and multiple material textures blended via a mask texture.
Since we require no lighting computation we just want the shadow values to get
some visual depth in the scene. Therefor the standart material seems to be not
sufficient for our purposes, hence the reliance on a custom shader.
I saw code that created a custom material with the standart shaders and then re-
placed parts of the vertex and fragment code via a function. We would be ready
to do this kind of code insertion, if it yields correct shadow information. Sadly I
cannot find the example project for this anymore, so if you could provide a link
to a simmiliar source it would be much appreciated.

Thank you sincerely for your time and help
With best regards from the green heart of Germany
The Mainequin Team

[GameMonetize]

Hello and welcome!

I found several things:

- You used blur shadow maps but the code uses to read it was not taking it in account (comment line #9)

- You forgot to add the world uniform when constructing the ShaderMaterial (line #109)

- SHADOWFULLFLOAT was not define (line #110). You actually need to test for float texture support using engine.geCaps()

- bias was wrong (no more required in shader code)

The working PG:

http://www.babylonjs-playground.com/#VZKI0U#3

 

Hope this helps

[mainequin]

Dear Deltakosh,
thank you for your quick answer!
Today we integrated your corrections into our shader code.
We received shadow-acne, which we fixed by adding bias
again. The computeShadowWithPCF-function looks at
neighboring coordinates in a distance defined by a Poisson-
disk and fades the shadow strength based on visibility, as
far as we understand. If all four samples are in the dark, we
receive full shadow. So since this is done in the shader, the
filter mode in the shadow generator should be irrelevant for
this.

We fixed the upload of the shadow-parameters as Vector3,
so now darkness, bias etc. are correctly uploaded.

We copied the computeShadowWithPCF-function and in-
creased the samples of the Poisson-disk, so that it fits our
needs.

Here is the final project: http://www.babylonjs-playground.com/#VZKI0U#7

Thank you again for your help and time!

[GameMonetize]

Excellent!!! Glad that it worked for you