Sprite loses rotation after applying filter to it

[mts]

Hello!

I am trying to implement an array of various animations, some of which require shaders.
I am working on an image transition in using a displacement shader from an empty texture -> image.

The sprite is rotated when first added to the container. However, after I apply my shader, the sprite reverts to default rotation.

I have asked a question on Stackoverflow, in order to not repeat myself I will post the link here:

https://stackoverflow.com/questions/64175920/pixi-js-sprite-loses-rotation-after-applying-filter

Thank you very much!

[ivan.popelyshev]

Hello and welcome to the forums!

Filters are applied to portions of screen, container where filter is applied is rendered to different framebuffer, then shader is uses  that rectangle as input. in your case you dont use the input, you use textures from uniforms. If you dont want to go through all types of coordinats and transforms for filters like here: https://github.com/pixijs/pixi.js/wiki/v5-Creating-filters , you have to use something else.

There's no "uSampler" usage, means you dont even need a filter, you can use mesh-shader: https://pixijs.io/examples/#/mesh-and-shaders/triangle-textured.js

Stackoverflow is useless for pixi.

 

[mts]

Thanks Ivan for your reply.

 

Unfortunately I don't see how to use mesh for this, since it looks more complicated than the sprite. Also, doing anchor.set on it causes error where anchor is undefined.
When I try to apply my shader to this mesh, I get the following message:

GeometrySystem.ts:276 Uncaught Error: shader and geometry incompatible, geometry missing the "aTextureCoord" attribute
    at r.checkCompatibility (GeometrySystem.ts:276)
    at r.initGeometryVao (GeometrySystem.ts:317)
    at r.bind (GeometrySystem.ts:180)
    at r._renderDefault (Mesh.ts:330)
    at r._render (Mesh.ts:297)
    at r.e.render (Container.ts:545)
    at e.render (Container.ts:550)
    at e.render (Container.ts:550)
    at r.render (Renderer.ts:404)
    at t.render (Application.ts:119)

Here is what I am trying to accomplish:

I am expecting the gradient to actually rotate with the sprite it's applied on.

This is the shader I am using:

precision highp float;
          uniform vec3 colors[4];
          uniform float steps[4];
          uniform float centerX;
          uniform float centerY;
          uniform vec4 inputSize;
          uniform vec4 inputPixel;
          uniform vec4 outputFrame;
          varying vec2 vTextureCoord;
          uniform sampler2D uSampler;
          uniform vec4 rotation;

          void main() {
            vec2 uv = (vTextureCoord * inputSize.xy / outputFrame.zw);
            vec2 gradXY = abs(uv - vec2(centerX, centerY));  // 0.5 is centerX, centerY
            float dist = pow(max(gradXY.x, gradXY.y) * 2.0, 2.0);
            float start = steps[0];
            for (int i = 1; i < 4; i++) {
                float end = steps[i]; 
                if (dist >= start && dist <= end) {
                    gl_FragColor = texture2D(uSampler, vTextureCoord) * vec4(mix(colors[i - 1], colors[i], (dist-start) / (end-start)), 1.);
                    break;
                }
                start = end;
            }
          }

Uniforms passed:

{
        centerX: 0.5,
        centerY: 0.5,
        colors: [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],
        steps: [0, 0.2921, 0.3452, 1]
      }

Can you direct me how to achieve this?

Edited October 12, 2020 by mts

[ivan.popelyshev]

then you have to use local sprite coords, like DisplacementFilter does (look at the source)

> since it looks more complicated than the sprite. 

Yes, it will be awesome if you'll get filter working but as i see, you dont get the trick with coords and matrices yet. It works in screen coords, vTextureCoord is screen coord, uSampler is sprite that is rendered already on screen, its rectangle which has rotated sprite inside. If you want rotated coords - look how DisplacementFilter does it, just ignore the part where it rotates displacement itself and passes extra rotation inside.

I'm sorry we dont have fast way to make sprite with uniforms, but mesh is defeinitely easier than Filter. I advice you to take some time to learn it, and not go through filter route, for your case filter has overhead both in performance and understanding.

Edited October 12, 2020 by ivan.popelyshev

[mts]

Can you give me some instructions for the mesh in this case? My end use case is to make a sprite, or a rectangle, or a mesh as you say that will have this black/white gradient applied to be used as mask.
Can I use a mesh as a mask? If so, do I pass aVertexPosition to it in format of four rectangle coordinate pairs? I tried, but can you tell me what is the error I am having? Will vTextureCoord work in the mesh?
Should I be using a renderTexture or some other construct with the mesh?

I noticed I am not able to set an anchor to a mesh. Is this normal?

Also, generally, filters combined with sprite rotation, scaling, do not work the way I expect them to. I am guessing it's because I don't understand which matrix does what in the shader and how to use them to get what I need.

[mts]

Hi Ivan, sorry to bother you again.

As you advised, I adapted(tried) the displacement filter example. However, I get the same result, although I noticed that the rotation uniform indeed changes according to the sprite.rotation. So I am missing something here.

What I get as a result now:

How I adapted the code:

...
const sprite = new PIXI.Sprite.from(
      'https://static4.depositphotos.com/1006994/298/v/450/depositphotos_2983099-stock-illustration-grunge-design.jpg'
    );
    sprite.anchor.set(0.5);
    sprite.x = renderer.width / 2;
    sprite.y = renderer.height / 2;
    sprite.rotation = Math.PI/6;
    setTimeout(() => {
      sprite.filters = [generateGradientFilter(sprite)];
    }, 300);
...

function gradientVertexShader() {

      return `
          attribute vec2 aVertexPosition;
          uniform mat3 projectionMatrix;
          uniform mat3 filterMatrix;
          varying vec2 vTextureCoord;
          varying vec2 vFilterCoord;
          uniform vec4 inputSize;
          uniform vec4 outputFrame;

          vec4 filterVertexPosition( void ) {
              vec2 position = aVertexPosition * max(outputFrame.zw, vec2(0.)) + outputFrame.xy;

              return vec4((projectionMatrix * vec3(position, 1.0)).xy, 0.0, 1.0);
          }

          vec2 filterTextureCoord( void ) {
              return aVertexPosition * (outputFrame.zw * inputSize.zw);
          }

          void main(void) {
            gl_Position = filterVertexPosition();
            vTextureCoord = filterTextureCoord();
            vFilterCoord = ( filterMatrix * vec3( vTextureCoord, 1.0)  ).xy;
          }
      `;
    }

    function generateGradientFilter(sprite) {
      // rectangular gradient
      const squareGradientShader = `
          precision highp float;
          uniform vec3 colors[4];
          uniform float steps[4];
          uniform float centerX;
          uniform float centerY;
          uniform vec4 inputSize;
          uniform vec4 outputFrame;
          varying vec2 vTextureCoord;
          uniform sampler2D uSampler;
          uniform vec2 scale;
          uniform mat2 rotation;
          uniform vec4 inputClamp;
          varying vec2 vFilterCoord;

          void main() {
            vec4 map =  texture2D(uSampler, vFilterCoord);
            map -= 0.5;
            map.xy = scale * inputSize.zw * (rotation * map.xy);
            vec2 uv = vTextureCoord * inputSize.xy / outputFrame.zw;
            //vec2 uv = vTextureCoord;
            vec2 gradXY = abs(uv - vec2(centerX, centerY));  // 0.5 is centerX, centerY
            float dist = pow(max(gradXY.x, gradXY.y) * 2.0, 2.0);
            float start = steps[0];
            for (int i = 1; i < 4; i++) {
                float end = steps[i]; 
                if (dist >= start && dist <= end) {
                    gl_FragColor = vec4(mix(colors[i - 1], colors[i], (dist-start) / (end-start)), 1.);
                    break;
                }
                start = end;
            }
            gl_FragColor = texture2D(uSampler, clamp(vec2(vTextureCoord.x + map.x, vTextureCoord.y + map.y), inputClamp.xy, inputClamp.zw)) * gl_FragColor;
          }         
      `;
      
      const filter = new PIXI.Filter(gradientVertexShader(), squareGradientShader, {
        centerX: 0.5,
        centerY: 0.5,
        colors: [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],
        steps: [0, 0.2921, 0.3452, 1],
        scale: { x: sprite.scale.x, y: sprite.scale.y },
        rotation: new Float32Array([1, 0, 0, 1]),
      });
      filter.apply = function (filterManager, input, output, clearMode)  {

        // fill maskMatrix with _normalized sprite texture coords_
        const maskMatrix = new PIXI.Matrix();
        this.uniforms.filterMatrix = filterManager.calculateSpriteMatrix(maskMatrix, sprite);
        // Extract rotation from world transform
        const wt = sprite.worldTransform;
        const lenX = Math.sqrt((wt.a * wt.a) + (wt.b * wt.b));
        const lenY = Math.sqrt((wt.c * wt.c) + (wt.d * wt.d));
        if (lenX !== 0 && lenY !== 0)
        {
            this.uniforms.rotation[0] = wt.a / lenX;
            this.uniforms.rotation[1] = wt.b / lenX;
            this.uniforms.rotation[2] = wt.c / lenY;
            this.uniforms.rotation[3] = wt.d / lenY;
            console.log('rotation', this.uniforms.rotation);
        }
        // draw the filter...
        filterManager.applyFilter(this, input, output, clearMode);
      }
      return filter;
    }

What is suspicious to me:

vec4 map =  texture2D(uSampler, vFilterCoord);

I don't have a map here so I used uSampler. Is this ok?

vec2 uv = vTextureCoord * inputSize.xy / outputFrame.zw;

I copied frorm here https://github.com/pixijs/pixi.js/wiki/v5-Creating-filters. Not clear what it does but it was guesswork which seemed to get me closer.

One thing I could maybe try is to use mapSampler by creating a RenderTexturer and running the gradient filter on it then using that as a map. Although I have no idea if that could work.

Can you also explain this:

map -= 0.5;
map.xy = scale * inputSize.zw * (rotation * map.xy);

Why -0.5? What do the multiplications do?

 

Can you help on this?

Edited October 13, 2020 by mts

[ivan.popelyshev]

> rotation uniform indeed changes according to the sprite.rotation

That's what I talked about - rotation is used in DisplcamentFilter to rotate data insite displacementtexture. you need a filterMatrix, not rotation.

Sorry, I dont have time for this, im buried with tasks at the moment. If you want, you can wait till weekend and I'll make you example based on Mesh.

[mts]

Thanks Ivan for you answer again! I understand that you are very busy and that you don't have time.

I would be very grateful if you could take a look at this post again on the weekend and help me understand better the
excellent PIXI.js.

[ivan.popelyshev]

Here we go, 75 lines total: https://jsfiddle.net/Hackerham/cke3gasx/27/

1. mesh geometry 200x200 size with center (0,0)

2. mesh material (shader, basically) specifying program for it. Same program can be used for many shader instances, just specify different uniform objects for them

3. texture, when loaded, changes size of mesh. Calculated size is 200,200 based on points in buffer, so, for 800x600  scale of element becomes 4,3

Edited October 19, 2020 by ivan.popelyshev