Performance Issues With Filter-based Dye System

[Roomy]

Hey!

I'm making an MMORPG-type game in an engine that uses Pixi.js and I was hoping to get some insight on how to fix a problem I'm encountering. I'd like to clarify I don't necessarily think the filter performance is the problem- I'm moreso looking for alternatives to my current method.

I use Spine2d for my models, which works great. I recently introduced a Dye system that's extremely simple. It uses filters to recolor pieces of clothing based on their connected Spine2d slots. As far as Spine models go, mine is pretty barebones with very few meshes and only ~100 slots.

In order for this dye system to work, though, I need to apply a filter to every slot... which means upwards of a thousand filters (or filter-targeted sprites) being rendered simultaneously when we have 20-30+ people on screen at the same time.

I've tried lots of things. If I batch the filters (read: use the same filter for dyes of the same color values), there's next to no performance difference. AKA filter count doesn't seem to matter, it's just that there are hundreds-thousands of sprites with an attached filter that's the issue.

Rendering the skeletons on their own without filters is perfectly fine. Rendering the entire skeleton with a given filter is fine too. But my current method is clearly not working that well in high cost scenarios and I was hoping to get some insight on that and how to fix it, possibly, or on a better method to use than this one.

Thanks in advance!

PS: The performance is unchanged even if there's no code in the filter itself, meaning the overhead is likely the issue and not the contents of the filter code.

Edited February 22, 2022 by Roomy

[Exca]

Using hundreds of filters will create a huge performance hit. There's no way how you could get thousands of filters to work in real time without performance issues.

Options to fix the problem are:

- Use filters in advance to prerender the filtered sprites into memory (preferably into a single base render target and then use that as an atlas) and then use them as regular textures to replace the original textures. This method limits the amount of colours possible as at some point you will run out of memory.

- Instead of filters, use a custom shader to render the spine slots and have it do the color transformation.

I have no experience in how to override spine rendering so can't help much with that.

Might be that there's also some other way to fix the issue, but cant think of any.

[Roomy]

Yeah, it's fine for most cases (people usually only have <20 people on screen, so it doesn't usually come up), but I'm looking to fix it so I have some breathing room & can add NPCs and so on.

I don't know if using texture atlases is going to be very easy / plausible with Spine since there's a whole procedure for loading atlases prior to displaying skeletons, etc., etc.- combined with the memory usage + garbage collection it probably isn't worth it.

I was thinking about circumventing filters and using shaders exactly as you mentioned there, but I'm not sure if I know how at the moment- there's a lot more material on Pixi filters than shaders. Does anyone (or do you) have any insight on that?

EDIT: It looks like a renderer plugin is closest to what I need for this: https://github.com/pixijs/pixijs/issues/6049

Edited February 22, 2022 by Roomy

[Roomy]

UPDATE:

It looks like the renderer plugin system is exactly what I was looking for. My only remaining issue is passing in the R/G/B recolor fields as uniforms, so I suppose that's what I'll refocus my question on.

How could I pass values per sprite into a batch plugin's fragment shader? I'm having trouble finding how to pass uniforms in. Talk of passing attributes, maybe?

Edited February 22, 2022 by Roomy

[Exca]

What kind of color adjustments were you doing initially? As you can do same adjustments that could be done in spine in script also.

If the default adjustments arent enough then creating custom attributes would be best way I think. Or you could reuse some of the existing attributes if your shader doesnt use them.

[Roomy]

Actually, I just came to the same conclusion a few minutes ago, I think.

This is what the filter fragment looks like:

var dyeFrag = `
precision mediump float;

varying vec2 vTextureCoord;

uniform sampler2D uSampler;
uniform vec3 uRed;
uniform vec3 uGreen;
uniform vec3 uBlue;
uniform vec3 uColBright;
uniform float uBright;

void main() {
  vec4 maincol = texture2D(uSampler, vTextureCoord);
  vec3 firstcol = vec3(maincol.r);
  vec3 secondcol = vec3(maincol.g);
  vec3 thirdcol = vec3(maincol.b);
  
  if (maincol.r > maincol.g && maincol.r > maincol.b) {
  	firstcol = vec3(maincol.r * uRed) * uColBright.r;
  } else if (maincol.g > maincol.b && maincol.g > maincol.r) {
  	secondcol = vec3(maincol.g * uGreen) * uColBright.g;
  } else if (maincol.b > maincol.g && maincol.b > maincol.r) {
 	thirdcol = vec3(maincol.b * uBlue) * uColBright.b;
  }
  
  gl_FragColor = vec4((firstcol+secondcol+thirdcol)*uBright, maincol.a);
}`

So I think I might be able to just interpret the color (tint?) of the sprite as stand-ins for the uniforms here. Not totally sure how to do that just yet but I think I have the idea-ish. I believe vColor is the color provided by the uniforms by default in the plugin code (?)

EDIT:
I've made a test setup to try and get this stuff working before I load it into my game (takes forever to reboot and etc.)
https://www.pixiplayground.com/#/edit/WADgQ_StS_wSnpPcIg4xg
I am 100% sure of what I need to do now (send over five attributes as stand-ins for the ^ above uniforms), just need to figure out how to do that.

Edited February 22, 2022 by Roomy

[Roomy]

Alright, I got it working as I wanted in the above playground url, but I'm still encountering some weirdness.

The biggest thing is that there's almost no performance difference between this and filters, which is a shame- I might be doing something wrong, though.

Quote

No Filters, No Custom Batch
197 Skeletons
Hovers 49-51 FPS

Filters, No Custom Batch
197 Skeletons
Hovers 34-36 FPS

No Filters, Custom Batch
197 Skeletons
Hovers 39-41 FPS

No Filters, Barebones Custom Batch
197 Skeletons
Hovers 39-41 FPS

As with filters, it seems like it's the overhead that takes 99% of the actual workload and not the frag/verts themselves. There's also the matter of it looking really odd and messed up in-game but I figure that's probably something I'm doing wrong elsewhere. Unfortunately, it's not rendering certain sprites correctly (or at all in some cases), which is leading me to believe that's why it's ~5 FPS better than the filters anyway.

Can anyone offer some insight into if it's possible to squeeze any extra performance out of this situation, or if I'm implementing it incorrectly?

[Exca]

Can you check if you are really getting batch rendering? Use spectorjs (or similar) tool to capture a snapshot of the rendering and check the drawcalls if they happen per sprite or does it render lot of sprites at a same time.

https://spector.babylonjs.com/ (installing as a chrome extension is easiest way to use).

[Roomy]

It seems to be drawing per sprite, yeah, that'd definitely explain it. If I can ask, how would I go about enabling proper batching?

I've zipped the capture data and attached it if you'd like to take a look.

capture 09_42_26.rar

Edited February 22, 2022 by Roomy

[Exca]

I'll check the data later today. About batching I'm not very familiar with it's internal workings and batching. Hopefully someone else has more info about that.

[ivan.popelyshev]

I have but i wont answer because im in my holidays for 3 months already ))) Use ShukantPal's pixi-batch-renderer, it has variation that allows uniforms in sprites, just assign same set (object) of uniforms to it. Or my uniform batcher from pixi-projection

[Exca]

On 2/22/2022 at 4:44 PM, Roomy said:

It seems to be drawing per sprite, yeah, that'd definitely explain it. If I can ask, how would I go about enabling proper batching?

I've zipped the capture data and attached it if you'd like to take a look.

capture 09_42_26.rar 1.8 MB · 0 downloads

Checked the data now. Getting batching to work would help a lot. Having 1k+ drawcalls is a lot and as a consequence having many drawcalls means a lot of commands (11k+), these would drop also with batching. 

[Roomy]

Sorry for the delay, I was messing around with steam achievements in the interim.

So, I went through the effort of implementing pixi-batch-renderer, but it was encountering issues calculating vertices (according to the error it gave). Regardless, though, I don't know if batch rendering will fix the problem- the uniforms are going to be unique for almost all slots (with an exception for groups of maybe 5-6). The total would only drop by maybe 100-200 from 1000.

Is there a way to accelerate this without batching, or is it possible to batch with unique uniforms per sprite?

EDIT: Gonna check out your pixi-heaven stuff though, it looks promising for stuff I'm trying to do

Edited February 27, 2022 by Roomy

[Roomy]

I successfully imported the pixi heaven stuff (which will be useful in a lot of other areas too), but it has the same issue as before: there's pretty much no difference in performance between when the 'filters' are baked into the batch renderer (and pull from uniforms there) versus when they're just normal filters.

After further testing though, it's starting to become pretty clear the performance difference between a lot of skeletons with filters and without is very minimal. I'm not sure if filters are really the core issue so much as the basic rendering process itself (probably due to not batching them). I'll have to dig into it more

Edited February 27, 2022 by Roomy

[Exca]

With filters you will force yourself off from batching. In that sense they are a problem when you get batching to work for spines.