RenderTexture.getPixels -- is there a setPixels? Also, webgl custom shader lag/perf questions

[timetocode]

RenderTexture has an awesome function called getPixels, which returns a Uint8Array of rgba pixel data.

 

Is there a way to create a pixi object from a Uint8Array of pixels?

 

I'm writing an image processor which works in a webworker. The Uint8Array is already perfect for passing to/from the webworker, I'm just not sure how to use that pixel data again when it comes back from the worker.

 

 

On a related note, the main reason I'm looking into using the webworker is because I can't quite get webgl filters (fragment shaders) to perform fast enough for me. I am using a ColorReplaceFilter and applying it to a sprite approximately 12-15 times to create my desired artwork. This creates a lag of 40-80 ms on my system. I need to get this number down to less than 16 ms. It is only the step of rendering via  rendertexture with filters that causes this lag. Fifteen might sound like a lot of filters, and I could certainly engineer a single filter that does the work of 15, but the majority of the performance hit comes from applying 1 filter, adding subsequent filters does not slow things down very much.

 

I don't really know how webgl works, but seems to me like something is blocking the main thread and creating this lag. What is the source of this?

I feel like there might be an easy solution somewhere. If the problem is that the filters need sent from the cpu to the gpu, perhaps I could send them one at a time (not sure how). If the problem is that PIXI.RenderTexture.render blocks the main thread until webgl has processed all the filters and rendering is complete, perhaps there could be a PIXI.RenderTexture.asyncRender that uses a callback and allows for some image processing magic.

 

Any ideas?

[timetocode]

I was testing the webgl ColorReplaceFilter version of my image processor some more, and I can confirm the ~40-80 ms lag occurs when rendering in whichever code section sets the filters for the first time. E.g  sprite.filters = myFilters; someRenderTexture.render(sprite); is where the lag occurs. Any subsequent rendering that uses the same filters occurs much faster (0-4 ms). Once again I don't know much about webgl, but it sounds like there may be a way for this all be blazing fast.

 

In the end my requirement is that players can dye/recolor all of their gear.

So far I've got this functional, but with a noticeable stutter when I recolorize any gear, or a new character walks on screen (and thus has their totally unique spritesheet generated for the first time). I accomplish this by composing a unique spritesheet **at runtime** containing a character and all of its piece of gear, all of which can be colored with a unique palette. I have my base spritesheets which contain my base artwork, but then I create spritesheets at runtime from all the base pieces of artwork, depending which pieces of art are needed.

Characters can also change their appearance in real time. Example: If a player were to change their helmet, I would clear the section of their sprite sheet that had the old helmet, then I would render their new helmet to a temporary render texture and apply filters to coloriize it (e.g. plain copper, or plain iron,  or customly dyed hot pink etc), and then I would take this colored helm and render it back onto their original spritesheet in place of the old helm. My rendering/animation logic knows where on these dynamic sprite sheets the helm is located, so the new helm will simply appear on this character from here forth.

I could use a different approach, but I feel like the above logic is for the most part pretty good -- I'm just looking at reducing the performance hit for the imageprocessing task of recoloring each little piece of artwork before it gets stitched back onto a spritesheet. I'm open to using whatever tricks are available.

[xerver]

Warning: Wall of text incoming! I tried to respond inline to a lot of your comments I hope it helps!

 

Is there a way to create a pixi object from a Uint8Array of pixels?

Not directly, you can use putImageData on a canvas you create and use that as a texture. As far as setting the values back into a render texture, it is possible in webgl but we haven't implemented it. Feel free to open a feature request on GitHub.
 

I'm writing an image processor which works in a webworker. The Uint8Array is already perfect for passing to/from the webworker, I'm just not sure how to use that pixel data again when it comes back from the worker.

Great idea, the best way would likely be to keep a canvas around that is used as a texture. You can use putImageData to that canvas repeatedly, and then just call `texture.update()` when you are ready for Pixi to update the texture.
 

I am using a ColorReplaceFilter and applying it to a sprite approximately 12-15 times to create my desired artwork.

Hmm, what does applying it to a sprite 12-15 times mean? You are creating a new one each time? If so, don't do that just use one and change the values it uses. Keep in mind when using a filter on an object it removes that object from the batching system. That means that each filtered object will be a draw call, and likely even "split" the sprites it is siblings of into 2 batches, ultimately turning 1 draw call into 3. Beware!
 

In the end my requirement is that players can dye/recolor all of their gear.

Can you use the tint feature for this instead? Tinting doesn't break batching, and uses the normal fast sprite shader. It also only takes a single render pass where filters require multiple FBOs and multiple passes. It obviously isn't the same as replacing colors, but if you make armor pieces separate sprites then you can tint individually and all will be batched into a single draw call anyway (assuming they all are from the same source image).
 

and thus has their totally unique spritesheet generated for the first time

Hmm, that is likely part of your problem. Sounds like you have a lot of different textures. Your lag is probably coming from uploading texture to the GPU dynamically so often. Try to make as many objects share a texture as possible to reduce draw calls and texture uploads.
 

I have my base spritesheets which contain my base artwork, but then I create spritesheets at runtime from all the base pieces of artwork, depending which pieces of art are needed.

I feel like you could instead do this by using the tint feature like I mentioned before. That way you are only uploading the one base texture to the GPU and all your sprites are batched up.
 

Example: If a player were to change their helmet, I would clear the section of their sprite sheet that had the old helmet, then I would render their new helmet to a temporary render texture and apply filters to coloriize it (e.g. plain copper, or plain iron, or customly dyed hot pink etc), and then I would take this colored helm and render it back onto their original spritesheet in place of the old helm.

Yeah instead if each armor piece was an individual sprite this entire process would go away and be trivial. You would just assign a new Texture to the helm sprite, and set the tint. The texture would just be a frame into the base art sheet you have. Much easier (and faster).


Hopefully this post helps, key take-aways are:

• Less sprite sheets usually is faster performance
• Using a lot of Render Textures means a lot of memory (FBOs) and a lot of binding them (GPU texture swapping) which is expensive
• Using filters and different sheets break batching, if you only need to recolor just use tinting.

 

Obviously everything I'm saying is only based on the limited information in this post, I haven't seen your code. But with that disclaimer, I strongly believe your best optimization is to use only the single base art sheet for all your character sprites, make each armor piece another separate sprite that is just a child of the main character, then just set the tint property of each armor piece as needed. I think this will make your app lightning fast! Good luck!

[timetocode]

Thanks for the reply Xerver, a lot of options to work with!

 

I will experiment with putImageData for all my image processing, and see what I can accomplish. Depending on how quickly this operation occurs, I may be able to make just about anything with it. I'll share my results when I do.

 

As far as tinting goes, I can't quite use it right out-of-the-box. My issue with tinting is that I haven't ever found it to produce images that visually pop. Tinting is not truly recoloring. An exception to that is high constrast monochromatic designs, where a tint is basically a color selection. When there are numerous colors in the art, a tint (both in digital and physical art) reduces contrast, and for artists that don't highlight with pure white or shade with pure black, the tint conceals a lot of the detail.

 

There's one way in which I might be able to use tinting. If I cut up my drawings so that every color was a separate image, I could then tint all of these separate images and put them together to make the final character art. This would increase the number of sprites that I used by quite a bit but I could still use a single spritesheet and it would not interfere with batching. Pixi does appear to be extremely fast when it gets to use batching. This would avoid the washed out nature of tints, because it wouldn't ever be tinting the whole thing... in fact I would make all the base images pure white, and use the tint to completely select their color. I may have to experiment with this. I guess I'll try to use the v3 bunnymark example, and give a unique tint to each bunny and see how many I can make before it chokes. I'll also give this a try tomorrow and share my results.

 

As far as having a unique spritesheet per character goes, I too am concerned that this simply is too much. If in the end I have to limit the number of colors and permutations for my game characters, then so be it. I do like the customization however, so I'm still going to exhaust the above options before giving up on full customization. My character "spritesheets" currently are between 128x128 px and 256x256 px big. The characters themselves are pretty tiny. If I were to bake these into a larger 2048x2048 atlas-esque texture, I could still fit quite a few of them on there.. on the order of 64 to 256 unique characters in just one atlas... which makes me feel like in the end its a reasonable amount of graphical data for one game, but the fact that I expect to be able to programmatically change the graphics inside of the atlas, well maybe that is too much.

 

Thank you for all the options. Super helpful!!!

[timetocode]

Summary of tests

 

Okay, so I've tried the above options and here's some real rough benchmarking/notes. First let me say that this is not your average use case, this is for someone who wants to take some source art and programmatically replace every color, and also be able to change these colors arbitrarily again during runtime. This is some serious image processing! This is not tinting, though one of the methods sorta hacks tinting into accomplishing the goal.

 

Manipulating pixelData via canvas, creating multiple spritesheets (one per art)

In this approach I took my character art, which consists of varying bodyparts (see image below) and then wrote them to a new texture via PIXI.RenderTexture. I do his for the sake of duplicating the character art from my original spritesheet, which also has a lot of other art. Now I have a renderTexture with just a single character worth of art. While drawing these sprites to the renderTexture, I also save PIXI.Rectangles recording their exact positions in the new sheet. Near the end of this process I register all of this new generated art with the PIXI.utils.TextureCache. Using one of the new pieces of art back out of the cache follows a syntax like: new Sprite(PIXI.utils.TextureCache['generated-character#533-/arms/front/frame3.png']). The actual logic for going from a renderTexture to color replacement via canvas and back follows:

	var pixels = base.getPixels() // where base is a renderTexture containing whatever image you want	var pixelData = new Uint8ClampedArray(pixels.buffer) // clone it, might be an extra step	var theme = generateOrcTheme()	replaceColors(pixelData, theme)	/* // replace colors has logic similar to THIS:	for (var i = 0; i < pixelData.length; i+=4) {		var r = pixelData[i + 0]		var g = pixelData[i + 1]		var b = pixelData[i + 2]		var a = pixelData[i + 3]		if (r === 20 && g === 20 && b === 20) {			pixelData[i + 0] = getRandomInt(0, 255)			pixelData[i + 1] = getRandomInt(0, 255)			pixelData[i + 2] = getRandomInt(0, 255)		}		if (r === 180 && g === 180 && b === 180) {			pixelData[i + 0] = getRandomInt(0, 255)			pixelData[i + 1] = getRandomInt(0, 255)			pixelData[i + 2] = getRandomInt(0, 255)		}		if (r === 220 && g === 220 && b === 220) {			pixelData[i + 0] = getRandomInt(0, 255)			pixelData[i + 1] = getRandomInt(0, 255)			pixelData[i + 2] = getRandomInt(0, 255)		}		//console.log('RGBA', r, g, b, a)	}	*/	var imageData = new ImageData(pixelData, 256, 256)	var buffer = document.createElement('canvas')	var bufferContext = buffer.getContext('2d')	bufferContext.putImageData(imageData, 0 ,0)	var blah = PIXI.Texture.fromCanvas(buffer)	var sprite = new PIXI.Sprite(blah)	PIXI.utils.TextureCache[textureIdPrefix] = sprite.texture

There's another step which was simply too long to include... which is where I took all of the PIXI.Rectangles that I had saved while originally making the renderTexture and then used them all to register the new art back with the TextureCache. Here's an unfinished example spritesheet created by the above:

I drew the character to the left so you can see approximately what it looks like when all rendered together. It also has 7-frame run animations for N,S,E,W and attack animations. By moving all of the body parts individually (rather than baking them into static frames) I was able to make a neat little bouncy bob to the torso/head/arms while the character jogs. This spritesheet is missing its gear section (armor, weaps, etc).

 

Benchmarks; (a 2012 gaming laptop) cpu i7-3610QM @ 2.3GHz, gpu geforce 660m, Win7, Chrome 

Generation: It takes 2-4 ms to generate a single spritesheet. My performance target was <16ms, which it meets by an acceptable margin. Potential improvement: the TypedArray color replacement math should be compatible with WebWorkers (yay multicore/thread javascript).

Limits of having one 256x256 spritesheet per character: I was able to generate and animate ~1200 unique characters and spritesheet before the FPS dropped from 60 to 55. My performance target for this was >50 which it far far exceeded. Honestly being able to make this many spritesheets came as a surprise. Each animation per character involves 6-15 sprites being drawn. Targeting computers like mine, one should limit spritesheet generation like this to 1000 to maintain full FPS. Hypothetical improvement: 64 spritesheets like this could be put into one 2048x2048 texture.

 

Onto the tinting approach:

 

Tinting sprites, turning every color of each character into its own frame

Due to success with the plain color math via canvas approach above, I did not fully implement the sprite tint-hack, but I did perform a logistical assessment.

 

The idea here is that pixi tinting can change colors and is very fast, but that tinting a whole image won't get the end product that I'm going for. This post is already too much of a wall of text for me to explain why tinting makes things ugly, so you'll just have to take my word  . To deal with this, the idea is that each time we use tint, we tint a single color in an image, effectively creating a color replacement feature. How do we do this? Well, first off it requires separating out an image into all of its colors. This will only work with limited palette art, for example my art has 3 shades for skin and a dark outline, a total of 4 colors and totally feasible to divide into 4 images. Then, you turn each of these images WHITE, and from then on color them exclusively via the tint feature. So if my skin midtone was #ffe6a3, I turn the source image white (#FFFFFF) and then load it up in pixi and tint it to #ffe6a3 and it looks exactly how it did before. That's the whole trick! Now onto some really rough benchmarks...

 

My character (the one shown above) is drawn in 6 pieces: head, arms, torso, feet, hair, weapon. Each of these pieces uses maybe 3-6 colors. Using the upper end of this range this would mean that to produce a fully rendered character would take 36 tinted and correctly positioned images. My character is pretty much naked, so lets double that and say that 72 variable images would cover the full customization potential of my game. You might say, hey sounds like that could maybe be optimized a bit -- but let's hold off on that, because this tint-hack, at least on paper, has tons of performance to spare. I copy and pasted the pixi v3 bunnymark code, and replaced the bunnies with random bodyparts from my characters. I also removed the fancy particle batch stuff, and made the benchmark use a regular Container. I assigned each bodypart a totally random tint (Math.random() * 0xffffff). The results? PIXI comfortably rendered 10,000 tinted sprites on my laptop before any signs of lag. To render my goal of 50 unique characters onscreen at a time would take a max of 3600 tinted sprites.

 

If I were starting over again with a pixel-art aesthetic, I would probably use this tint hack approach. However, it would take me several hours (understatement) to cut up all of my art by color. I also happen to have a bunch of "plain old math" image manipulation logic left over from another project, so I'm going to stick to the pixelData manipulation approach. Take this praise of the tint-hack with a heap of salt though, because I did not finish implementing this so there could be a gotcha somewhere in here.

 

Compared to the pixel manipulation approach...

Advantage: it would use only one spritesheet (might not matter as much as i thought)

Advantage: can recolor anything in realtime super super fast (the other method would have to re-manipulate the pixel data)

Situational disadvantages: ONLY feasible for art with a very limited palette (like mine!), not a candidate for a WebWorker

 

Via a WebGL Fragment Shader

Here is a shader which also performs color replacements: ColorReplaceFilter.js. It is hard to benchmark this filter, because the main performance problem arises from custom filters not being candidates for batching. If you only want to replace the color of a few things, and you don't plan to change the colors all of the time, USE THIS APPROACH! Your color replacements will be done in one line of code.

 

Some approximates about performance: the very first time the filter is used on a texture (when render is called) the filter will get sent to the gpu/webgl which will create 10-40ms of lag. Subsequent renderings with the same filter are very fast. This filter as is, can only replace one color with one other color. My need to periodically generate new character or recolor existing characters prohibited me from using this shader. Potential improvements: The filter could be modified to have multiple color replacements rather than creating multiple instances of the same filter. The filter can have its uniforms changed to perform color changes rather than being swapped with a new filter (this should work already, I just haven't tested it).

 

Hopefully someone finds this R&D valuable. Happy coding and thanks for the help!

 

Oh and here's 1000 generated naked orcs for no reason:

[coolblue]

Hi, I spent a while hacking in circles trying to get tinting to look good also.

I had a sprite that I wanted to scale and color for different species, so I made a base sprite that was a nice looking, shaded grey object and my plan was to scale and tint it to avoid the (quite considerable) overheads of multiple textures.

 

I'm very new to this so it took me a while to get my brain around the problem...

 

Basically tinting doesn't manage saturation (the s in hsv): you only get a uniform multiply across the color channels and that's why it always looks drab.

You really need to control the relative channel values to manage saturation.  That's how you get nice, colorized light and shade...

 

For my strategy to work I need a filter that can feed the s channel of my base sprite into the s channel of the colorized clones: that would be awesome!

I guess that can be done with a colorMatrix filter but I don't get how filters work yet and I don't know if it would be expensive or not but I would be interested to know if it could be done this way?

 

The way I solved the problem was to (after hacking my way through the gum-bleeding jargon) figure out what a sprite sheet is and how to implement it with PIXI.

The result was astonishing (well, to me anyway): http://bl.ocks.org/cool-Blue/a4530669c16faced0d57