How do you ignore transparency when picking meshes?

[ZackMFleischman]

Hi! 

So I'm ultimately trying to pick some meshes with textures that have transparency, but when they overlap, the transparent parts of the mesh still get picked.

Playground: http://www.babylonjs-playground.com/#1UCP5L

If you open up the console in the playground and click the 2 black circles in the center of the overlapping "impact" textures, you'll notice that it always picks the 1st texture (although you're clicking on 2 separate "visible" textures if you take transparency into account). 

I thought perhaps I could test to see if the texture color at the UV coordinates of the picked mesh is transparent, and if it was, I could temporarily mark that mesh as not pickable and pick again at the same location to get the mesh under it until I got something that isn't transparent (and then restore the isPickable state of everything).

I still think that could work (although it seems terribly inefficient as I have to do several picks unnecessarily), but I frustratingly can't actually find a good way to get the texture color given the texture and some UV indices (obtained through the pickedInfo). Am I just missing something obvious? Is there not simply an analogous textureObject.getTextureColorAtUV(u, v) function? 

I'm also open to other suggestions to get more accurate picking with transparency taken into account.

Thanks so much!

Zack

[NasimiAsl]

mesh.isPickable = true | false

http://www.babylonjs-playground.com/#4SUQZ#0 

http://www.babylonjs-playground.com/#4SUQZ#1

 

[ZackMFleischman]

I appreciate you taking some time @NasimiAsl but I'm not asking how to ignore a mesh that has transparency.

For clarity, I'm asking 2 things:

1.) How do you ignore ONLY the completely transparent part of a mesh when picking?

2.) How do you get the RGBA color of a texture given a specific UV coordinate into the texture?

[NasimiAsl]

@ZackMFleischman you need GPU Picking system or just wanna read pixel from rendered Result?

 

 

[Pryme8]

I would think you would have to do a multiple pick loop, where you check the alpha of the first picked meshs uv at that point and if its 0 then set that mesh to pickable false and do another pick at the same location.

[ZackMFleischman]

@NasimiAsl If a GPU Picking system would help I'm not opposed to it, but I don't know of one that exists or what that would entail. 
As far as reading the pixel from the rendered result, I'm referring to reading the color data from a Texture at a UV coordinate in the range (u=[0,1], v=[0,1]). Whether or not the texture is rendered is irrelevant. I want to know what it's color is at certain coordinates. I'm surprised that I can't just access a 2D array like this: textureObject.colorData[v].

@Pryme8 Yes, that is the algorithm I described above. I believe I could accomplish that if I were able to check the alpha at the meshes uv...BUT I don't know HOW to check that alpha value (as I've mentioned a few times now). HOWEVER, this algorithm seems terribly inefficient due to multiple picks on the order of the number of meshes the cast ray would intersect. I feel like there is a way to construct an algorithm that should only have to execute the picking process once, and I'm open to any ideas people might have towards this.

Thanks again guys.

(ps, no idea why this underlined everything but I can't turn it off with an edit)

[fenomas]

@ZackMFleischman

I don't think browsers implement a way of reading a pixel color out of an image. I believe the only way is to create a canvas element (doesn't have to be part of the DOM), draw the image into it, and then read the pixel back out of the canvas. It would be costly to do this over and over, but if you only need checks against a single texture you could draw the image once and read out of it cheaply when needed.

As for efficiency, an algorithm is only inefficient if it does work that didn't need to be done. If I follow you here, you need to know where the pick hits a given mesh before you can determine whether that mesh should be ignored. So I'd expect that doing multiple picks is unavoidable.

[Pryme8]

Multipicks are standard 

[GameMonetize]

Hey

1.) How do you ignore ONLY the completely transparent part of a mesh when picking? You can define your own predicate to determine which meshes can be picked. You can either set mesh.isPickable = false for transparent mesh or use https://github.com/BabylonJS/Babylon.js/blob/master/src/babylon.scene.ts#L2906 (When calling scene.pick(x, y, predicate))

2.) You first need to load your texture into an image object (DOM object). The using a canvas, draw your picture inside it in order to get pixel colors (https://dev.opera.com/articles/html5-canvas-basics/#pixelbasedmanipulation). Then with UV it is easy to get the RGBA of the pixel you want.

[Pryme8]

2) is where the butter is

[ZackMFleischman]

@fenomas @Deltakosh @Pryme8

Thanks guys!

For testing whether a part of an image is transparent I ended up doing as @fenomas and @Deltakosh said and did the following:

1. Created an offscreen canvas element with a 2D context and draw the image to it.
2. Get the Image Data via `context.getImageData` and thus get access to the raw pixel data.

 

For only picking meshes when I've picked a non-transparent area of the mesh I did a pick loop as suggested:

1. Pick with all meshes set to `isPickable`.
2. Test the UV coordinates of the texture of the pickedMesh to see if it's transparent. (via `pickInfo.getTextureCoordinates`)
3. If it's transparent, mark that mesh's `isPickable` to false and recurse.
4. Eventually we reach a mesh that isn't transparent at the picked point, or there is no mesh. We set `isPickable` back to true for all the ones we flipped and return that pickInfo.

 

You can see the results in the playground here: http://www.babylonjs-playground.com/#1UCP5L#5

If you click the image on the left, the background will turn red.

If you click the image on the right, the background will turn blue.

If you click the background (including in the middle of the images where it's transparent!) the background will turn back to green.

 

Thanks again for your help. Big fan of this community.

[Vousk-prod.]

On 13/11/2016 at 4:21 AM, ZackMFleischman said:

You can see the results in the playground here: http://www.babylonjs-playground.com/#1UCP5L#5

If you click the image on the left, the background will turn red.

If you click the image on the right, the background will turn blue.

If you click the background (including in the middle of the images where it's transparent!) the background will turn back to green.

Hmm, that's strange, it doesn't fully work for me (Firefox).

If I click the square or outside the square, it turns green. But if I click the black drawing, it turns usually red on the left and blue on the right, but it also sometimes turns red on the right, and blue on the left.

[ZackMFleischman]

@Vousk-prod. This is by design. The black drawing is actually 2 "impact" images overlaid on each other. I chose it because it has transparency. 

Thinking back I probably should have chosen 2 different images but I was just hacking something up to demonstrate the point. If you click the image on the left it turns red, if you click the image on the right it turns blue. This includes the parts of the images that overlap.

 

[Vousk-prod.]

Oh, I see. I should have had a look at the code to get that