Fastest way to do masking?

[AndrewJS]

Pixi usually performs really well even with heavy scenes. I can put hundreds of objects, even thousands of particle sprites and get 60 FPS on mobile.

I typically only use a single masked object in a scene but recently needed to use more and I was surprised at how badly Pixi performed with masked objects. For every masked sprite, I lost about 5 FPS on mobile (both Android and iOS on old and new phones, newer phones actually performed worse in some cases). By the time I added 4 masked sprites, I was down to 40 FPS, adding 10-20 masked sprites dropped to 20 FPS with really bad stuttering. As soon as I switch the mask off by removing the assignment, it goes from 20 FPS to 60 FPS, even with 200 of the same sprites.

It doesn't seem like it should take so much resources to do masking given all the other effects that are possible. Tinting for example, adds a value to every pixel and costs nothing. Masking is just checking a pixel in one object and setting the equivalent pixel in the other object.

Is there a faster alternative to using object.mask = mask? Is there a graphics buffer I can use to set the pixel values myself, e.g if I could create an array of pixel values and generate a texture buffer from that. Javascript is pretty fast with arrays.

The main thing that bothers me is that the low performance happens with static masks. I could understand the performance hit when the sprite is animating relative to the mask but not when they are both static. Why doesn't it buffer the masked sprite and use that over and over like a normal sprite?

I found a thread that describes the same issue, unfortunately I'm stuck on v3 for now:

https://github.com/pixijs/pixi.js/issues/2163

I just found the following with a possible alternative using multiply blending:

https://github.com/pixijs/pixi.js/issues/3735

To isolate it, it suggests using a voidfilter. Is this the fastest way to do masking in Pixi? If so, is there example code for this?

Say that I did at some point want to draw a texture pixel by pixel, one way would be to draw a tinted 1x1 pixel sprite into a render texture. Is there a better way than this e.g set values in a buffer and convert it to a texture?

Is there a way to read a pixel color/alpha value from a sprite or texture. There seems to be an extract function for WebGL and Canvas but it looks like this extracts the viewport. It would be good to be able to render sprites to a rendertexture and be able to read the pixel values of that texture using pixel co-ordinates.

[ivan.popelyshev]

it doesnt matter if they static or not, pixi lacks of any flags about updating bounds, coordinates, and fields. Except a few TransformStatic optimizations. Yes, flash did better, but code of open-source lib can be ruined with all those static optimizations. Even "cacheAsBitmap" is FUBAR and works only, and look at its code, it looks clean! It has so many hidden bugs that i cant give any guarantees for users.

Pixi has "generateTexture" that can be used to improve static objects performance  That thing will work 100%, however you'll have many problems in the code, like, calculating bounds and offsets for this thing.

Yeah, VoidFilter exists for v3 in a legacy plugin: https://github.com/ivanpopelyshev/pixi-legacy

That topic requires some knowledge on pixi internals. Don't think that you can just ask for solution and whatever i say magically works for you, you have to find the one for your case from a number of them.

One of solutions for v4 is https://github.com/gameofbombs/pixi-heaven/ , it is possible to port "spriteMask" field and custom renderer for v3 if you know how both v3 and v4 work.

[ivan.popelyshev]

Yeah, there are examples. This one requires extra plugin, however that plugin is used only to draw lights inside a texture instead: https://pixijs.io/examples/#/layers/lighting.js

The problem is ,  ALL mask solutiont might be slow on mobiles. EsotericSoftware Spine runtime even calculates intersection of sprites and mask geometry manually. The only good enough solution is heaven, but, well, good luck with porting it on v3

[ivan.popelyshev]

You can also extract texture from videomemory and porocess it and put it back, its easy in v5, slightly less easier in v4 and v3. You have to understand how TextureManager works and hack it to make it possible uploading of custom array. It will also affect performance because any operation that takes information from videomemory back to regular memory is synchronous.

[ivan.popelyshev]

Actually, all my can be voided because i dont own mobile devices. I guess, you have to test everything in v4 and then port back whatever works better.

[AndrewJS]

13 hours ago, ivan.popelyshev said:

You can also extract texture from videomemory and porocess it and put it back, its easy in v5, slightly less easier in v4 and v3. You have to understand how TextureManager works and hack it to make it possible uploading of custom array. It will also affect performance because any operation that takes information from videomemory back to regular memory is synchronous.

Quote

EsotericSoftware Spine runtime even calculates intersection of sprites and mask geometry manually.

These options sound viable. I have managed to extract the pixel values from sprite and mask, how do I create a texture from a buffer of pixel values? There seems to be a BufferResource:

https://github.com/db-noop/pixi.js/commit/53f98973b16710131646431385f2c82c4e5a21b9

Which Pixi version has the fromBuffer option? I can't find BufferResource to be able to create a texture from a buffer:

https://pixijs.download/v5.0.0-alpha.2/docs/PIXI.resources.BufferResource.html

I downloaded the source on dev branch but can't find BufferResource there. "Next" branches are all 404:

https://github.com/pixijs/pixi.js/tree/dev/src

[AndrewJS]

I managed to get a slower method working but not usable in real-time for animation as it takes about 0.2-0.5 seconds to generate:

- draw sprites into a rendertexture

- draw masks into a rendertexture (same size as first rendertexture);

- get pixel buffers for each using renderTexture.getPixels()

- iterate over the buffer and draw a new PIXI Graphic using beginFill/endFill per pixel (drawRect 1x1) using color from sprite buffer and alpha from the mask buffer

- convert the graphic to a texture using renderTexture.generateTexture(Graphic)

This lets me manually create a static masked sprite in v3 and hundreds of sprites can run at 60 FPS. The slowest parts seem to be rendering the graphics into the buffer. What I could do for animation is render the whole mask into a larger buffer and then just read a portion of when generating the masked graphic. This would probably get it down to around 0.05-0.1 seconds per frame.

This is still too high though, needs to be around 10ms (0.01s) tops . I need a faster way of converting a buffer of pixel values into a texture. Is there a low-level way of doing this like a WebGL call?

Is Spine using direct WebGL calls for masking?

[ivan.popelyshev]

Yes, go lower. I think that "iterate over the buffer" thing is really excessive, it can be done with other operations faster. 
I told you all possible options but its too wide, and I think you are overwhelmed by it

However, I still don't understand your problem. Can you make a small demo that doesn't perform good enough for you?

[AndrewJS]

On 6/29/2018 at 12:35 AM, ivan.popelyshev said:

Yes, go lower. I think that "iterate over the buffer" thing is really excessive, it can be done with other operations faster. 
I told you all possible options but its too wide, and I think you are overwhelmed by it

However, I still don't understand your problem. Can you make a small demo that doesn't perform good enough for you?

I made up a small demo scene. The problem is on mobile, if you drop it on a server somewhere and load it on any mobile device, you will see the performance issue. On desktop, it supports over 500 masked sprites no problem. On mobile, it drops 30FPS with just 4 sprites. The demo has the number set to 16 (maxSprites = 16) and the FPS drops to under 10FPS. When you turn masking off by commenting out gate.mask = gate_mask, mobile is smooth up to over 100 sprites.

If there's a faster way to do masking, even if it's WebGL-only perhaps with a shader, then I could use that to maintain performance on mobile.

pixi-mask.zip

[ivan.popelyshev]

If you need only rectangular masks, you can just clone the texture for every sprite and change its frame, then call updateUvs() or whatever method exists in v3 for updated texture frame. This is what people use for HPbar-s on their characters

 

[AndrewJS]

7 minutes ago, ivan.popelyshev said:

If you need only rectangular masks, you can just clone the texture for every sprite and change its frame, then call updateUvs() or whatever method exists in v3 for updated texture frame. This is what people use for HPbar-s on their characters

That's good to know for directly masking textures with rectangles. Unfortunately I need to do shapes e.g circular arc masks and it needs to work on containers with multiple sprites.

Can you describe what the masking process is doing in WebGL? If you set a tint value, I assume it just adds a value to every pixel. When you set a mask, can't it just copy the pixels from a mask into the alpha buffer of the masked object very quickly? What is slowing it down so much?

[ivan.popelyshev]

500 drawcalls is too much for mobiles. So, you have to render multiple elements with one drawcall, and that means very difficult shader.  I told you already about https://github.com/gameofbombs/pixi-heaven and its masking shader, so the only way to do 500 masks is to study this plugin and its "maskSprite" features with pixi-v4, then port SpriteMaskedRenderer to v3.

Its time to clone that thing and understand how does it work.

[AndrewJS]

Just now, ivan.popelyshev said:

500 drawcalls is too much for mobiles. So, you have to render multiple elements with one drawcall, and that means very difficult shader.  I told you already about https://github.com/gameofbombs/pixi-heaven and its masking shader, so the only way to do 500 masks is to study this plugin and its "maskSprite" features with pixi-v4, then port SpriteMaskedRenderer to v3.

I'm only doing 4 masks, not 500. PIXI drops 30FPS with 4 masks on mobile.

How much faster is SpriteMaskedrenderer than the PIXI one? If it's a lot faster, why doesn't PIXI use that rendering method?

[ivan.popelyshev]

pixi-heaven is a collection of things that I had to code to help people with their project but i dont have time to negotiate any of those algorithms to be merged into vanilla pixi because the code is difficult. Sometimes it takes 1 hour to make a breakthrough and tens of hours through several months to negotiate about merge.

I have several repositories full of things that people asked about "why isnt this merged in pixi".

I suggest you read MaskManager, FilterMakager and SpriteMaskFilter source code in pixi, and meditate on it.

[ivan.popelyshev]

Heaven masks dont use extra FrameBuffers and stencil on mobile, things that are very slow there, but they can only be applied to one sprite over another sprite.

[cursedcoder]

As for pixi-heaven - it reduced mobile draw calls from 500+ to 150~ 

[Jonny Shaw]

Sorry to bump this but just wondered if this had improved significantly with V5, or an alternative approach is recommended?  I noticed that masks seem to act slightly differently in V5 someway in their stacking order.
A system I had set up in V4 for instance with a Kawase blur filter on a masked container that animated.  The filter would also blur the mask (which was the intended idea) but with V5 only the contents of the container were masked.

Also seems that using graphics for masks on V5 doesn't work on mobile for me?  In a project I'm working on at the moment I have a scene with 5 masks that are simple rectangular graphics and the height of the masks is tweened on occasion.
Because the graphics arent working on mobile two approaches I thought of...

1. Rendering all the graphics to a single texture on the fly and having that texture used as a single sprite mask. 

2. Just switching the masks up for simple sprites and keeping the rest as is with 5 masks?

Is there any benefit on draw calls for instance if I had to mix 1 with 2, ie having one rendered texture but using it as 5 masks as opposed to 5 separate images for the masks?
Also was interested to see what @cursedcoder mentioned about pixi-heaven - is this still applicable with V5 as I understand pixi-heaven still needs a bit of work with 5?
 

Edited February 3, 2020 by Jonny Shaw

[ivan.popelyshev]

> Also seems that using graphics for masks on V5 doesn't work on mobile for me?  In a project I'm working on at the moment I have a scene with 5 masks that are simple rectangular graphics and the height of the masks is tweened on occasion.

It should work. If it doesnt work - make a demo and tell us a device that doesnt work with it. We made several changes in v5 releases regarding masks, and one of them - scissor masks for rects work inside the filters too. Its not easy to debug that stuff because we have not enough reports from people.

For 5 objects there cant be benefits, 5 is a very small number.

Pixi-heaven sprite masks weren't ported to v5 but i can port them provided you can actually test it and report back.

[ivan.popelyshev]

Also you can use https://spector.babylonjs.com/ to capture one frame and see how exactly masks are handled. Again, I remind you - it might be different for pixi 5.1 and 5.2. There might be bugs.

Edited February 3, 2020 by ivan.popelyshev

[Jonny Shaw]

4 hours ago, ivan.popelyshev said:

> Also seems that using graphics for masks on V5 doesn't work on mobile for me?  In a project I'm working on at the moment I have a scene with 5 masks that are simple rectangular graphics and the height of the masks is tweened on occasion.

It should work. If it doesnt work - make a demo and tell us a device that doesnt work with it. We made several changes in v5 releases regarding masks, and one of them - scissor masks for rects work inside the filters too. Its not easy to debug that stuff because we have not enough reports from people.

For 5 objects there cant be benefits, 5 is a very small number.

Pixi-heaven sprite masks weren't ported to v5 but i can port them provided you can actually test it and report back.

Brilliant thanks Ivan, will take a look in detail first, it maybe something else that's causing the error on mobile and report back.  Device currently testing on is a Samsung Note 8 (though will try others)

[ivan.popelyshev]

OK, i recommend to test it for pixi 5.1.5 and 5.2.1

[Jonny Shaw]

2 hours ago, ivan.popelyshev said:

OK, i recommend to test it for pixi 5.1.5 and 5.2.1

Thanks Ivan!  I was on 5.14 checked that this morning and was the same issue.  Updated to 5.21 and all seems fine now on mobile