How to add thickness to a 2D textured plane?

[Nesh108]

Hello!

As you might know already, I am working on a voxel project and I am trying to add thickness to 2D sprites.
Currently they are shown as 2D sprites in billboard mode. That is fine when they are laying on the ground but when the player picks them up, they need to have some sort of thickness to them.

What I would like to achieve is something like the attached image: a 2D sprite used on a 3D plane. The top and bottom are the sprite but the sides are the continuation of the edges of the sprite.

How can I do this?

 

[GameMonetize]

Hey, perhaps you should use billboard then. You can billboard a cube in this case.

See the triangle here: https://www.babylonjs-playground.com/#1ET2K6#3

[BitOfGold]

I know it sounds weird, but you could try a voxel editor instead of a pixel image:
https://ephtracy.github.io/

This can output .obj files.
I'm sure the arrow on the top image is a mesh, not a thickness shader.
 

[Nesh108]

7 hours ago, Deltakosh said:

Hey, perhaps you should use billboard then. You can billboard a cube in this case.

See the triangle here: https://www.babylonjs-playground.com/#1ET2K6#3

The problem with the billboard is that, apparently, it cannot be attached and follow a bone. I was using the mesh with `BABYLON.Mesh.BILLBOARDMODE_Y` and the mesh just stood in place, without it, it would follow the bone properly.

6 hours ago, BitOfGold said:

I know it sounds weird, but you could try a voxel editor instead of a pixel image:
https://ephtracy.github.io/

This can output .obj files.
I'm sure the arrow on the top image is a mesh, not a thickness shader.
 

Since I am already using those sprites elsewhere, I would rather being able to re-use them. If anything else fails, I will go for the obj way instead.

There must be a way to tell babylon to make a sprite with Top and Bottom being the image with alpha and the sides being just the sides of those two textures. Right?

[GameMonetize]

sprites are not meshes at all. but pure 2d images rendered on top of 3D. Hence my suggestion for a billboard.

Billboard can be parented so it should work with bones attach. Can you create a repro on the PG? I'll try to check what is going wrong

[Nesh108]

@Deltakosh: yeah but if you place a sprite onto a plane, it becomes a mesh, correct?  

To clarify the problem, see the attached image. That's what happens when I keep my code and instead of use a plane to show my texture, I use a box with the size I want.
I would like to be able to splat 2 planes one over the other at a certain distance (depth) and fill the sides with pixels from the border of the mesh. Do I really have to go through the pain of making them 3D in an editor?

I think it should be fairly simple to do in Three.JS, let me check!

 

EDIT: I think that's how it is done in Three.JS:

var mesh = new ExtrudedMesh(Weapon.Image, this.class.offset.x, this.class.offset.y, this.class.offset.width, this.class.offset.height);

On 4/28/2017 at 4:14 AM, BitOfGold said:

I know it sounds weird, but you could try a voxel editor instead of a pixel image:
https://ephtracy.github.io/

This can output .obj files.
I'm sure the arrow on the top image is a mesh, not a thickness shader.
 

[GameMonetize]

Something like that?

http://doc.babylonjs.com/tutorials/parametric_shapes#extrusion

[Wingnut]

On 5/1/2017 at 10:42 PM, Nesh108 said:

fill the sides with pixels from the border of the mesh

Hi Nesh.  Did you mean to say... "fill the sides with pixels from the border of the texture" ?

Just checking.  You also said "but the sides are the continuation of the edges of the sprite."... in the first post... so I think I understand.

The sprite texture-edges need to wrap-onto the sides of the "thickness".  I don't think we have anything that can do this, yet.  Your arrow picture (1st post) is a great example.  But, I think that arrow was made with a 3D modeling app.  Still, let's talk, and talk, and talk.

I asked about this shape-extrusion myself, once upon a time.  I wanted to make "silhouettes" (probably the wrong word).  They are used in one version of the intro to the Red Green Show.  https://www.youtube.com/watch?v=Xxl-6HIGCuA

About 1:05 into the video, the "silhouette" show-open happens... and you can see that each of those "sprites" that the camera flies-past... has SOME thickness - easily seen on the camper-trailer.  These things are like pine tree air fresheners... for cars. 

Somewhat similar to these things... https://s-media-cache-ak0.pinimg.com/originals/2b/f4/57/2bf457cdde48035f71000d91958529b8.jpg

Essentially, we would create an extrudeShape mesh... whose SHAPE... is created from a 'texture edge".  We need... one of these things:

- A somewhat-thick black border-line around the sprite (or white border-line), which we can "find" with edgeFinder algorithm during fuzzy-logic analysis of the context2d image buffer colors.  (ouch, that sentence made my brain tumor hurt!)

Actually, any color border can be used... IF our edge-finder code can find it... within the image buffer.

- OR... use a transparent png, etc - an image-shape that is surrounded by transparency.  Transparent background area is removed ignored by the edgeFinder, and just the main texture "shape-content" (data) is retained.

Having either of those... is imperative... to edge-finding. (I think)

We would use that edge-data as the "shape" for an extrudeShape.  Oh man, not easy.  The arrow that Nesh showed us... has SOME of its black border-edge... wrapped onto the sides (onto the thickness dimension).  The black borderline should map 50% (depth) across the side-edge.  The reversed image on the OTHER side of this fit-to-sprite mesh... will wrap ITS black borderline, too, covering the other 50% of the side depth.

Uscale and vScale 102% should take care of that wrap for us... IF the UVs on the extrudeShape... are set in a certain way.

Cookie cutter!  Makes a mesh... shaped like a sprite... with settable mesh-thickness and settable edge-resolution (how many vector2's in the shape path for the extrudeShape).  (How accurate is the edge-tracking.)

Wow, huh?  I did SOMETHING similar... using bird doodoo, err, bird data (sorry for the bad extrude-caps. Remove caps with line 73 = 0).

Yep, thick-sprites.  Image-analysis edge-detecting stuff.  Black ops facial-recognition tech. wow!

Sorry for rattling-on, but I, too, have a dream to make Red Green silhouettes/cutouts.    I don't think this is going to be easy. 

[Nesh108]

@WingnutThat's exactly what I meant

Anyway, since there seems to be nothing available in Babylonjs at the moment, I thought of working on it. I think I got the theory ready but I am stuck at trying to get the ImageData/Pixels from a given Texture. Is that even possible in Babylon? I have been searching the forum for a while now and found nothing conclusive.

There was an attempt of yours here: http://www.babylonjs-playground.com/#2KUCI but that doesn't seem to work either. I am kind of used to OpenCV where working with the pixel data is the daily bread  

Any thoughts?

[Nesh108]

@Wingnut After some research, I found that it's possible to access the WebGLTexture of a BABYLON.Texture with 'tex._texture' but I can't seem to be able to access its frame buffer as shown here https://github.com/BabylonJS/Babylon.js/blob/master/src/babylon.engine.ts#L1243. There it seems possible to retrieve the pixel data by simply getting a WebGLTexture and using 'webgltex._framebuffer'. No luck though...

Anyway, here is what I plan to do if I am able to extract a matrix of Color4 from a Texture:

Step1: GetTextureOutline(pixels: Color4[], width: number, height: number): MyPixelList

Go through each pixel and check if it has an alpha different from 0 ( = not invisible). If:

- True: check if the pixels TOP, LEFT, RIGHT and BOTTOM also have alpha different from 0 in it. If False, store the coordinates and RBGA value, else skip it.
- False: Skip it.

In this way, we can return an array with only the outline of the texture and the color values from each pixel in the outline.

Step2: ExtrudeOutline(texture: BABYLON.Texture, outlinePixels: MyPixelList, thickness: number): BABYLON.Mesh

Here we can simply create 2 planes with the texture at <thickness> distance from each other and both mirrored (so we don't have to double side them). Next we create a mesh given the MyPixelList:

Go through each pixel and convert it into VertexData with its color. Next, we merge the 3 meshes (2 planes and the outside "ring" made by the vertex data) with BABYLON.MergeMeshes(topPlane, bottomPlane, outlineMesh). Ta-Da!~~

 

Disclaimer: it might not be correct, just my current thought that I cannot test since I cannot extract the pixel data from the texture  

[Wingnut]

Hi Nesh... good to hear that we got the goal... accurately defined.  I fell asleep for a few hours... just woke again.

https://www.babylonjs-playground.com/#1P3JT4#3

Just another imageData PG.  Somebody made their own image for dynamicTexture... solid green... with alpha.

Here's another... displaceMap with some context2d manipulating... that I worked-on:  https://www.babylonjs-playground.com/#P9UZG#12

See line 46:  var buffer = context.getImageData(0, 0, heightMapWidth, heightMapHeight).data;  (you can use canvas.width and canvas.height as those two 'range' parameters, instead of the heightMap variables).  Keep in mind that textures (and BJS ViewPorts) have their 0,0 in lower left corner, as opposed to HTML elements which have their 0,0 in upper left.

After we/you get the data into a buffer like that, I'm lost.  How to convert all the "edge" pixels that you find in the buffer... into a REASONABLE AMOUNT of vector2's to use as a shape for an extrudeShape... I have no clue whatsoever. 

If I were you, I would scour the web for JS-based image-manipulation funcs.  Discover all the other mad scientists who have "attacked" an image buffer... with demented image analysis code. 

Let's say you traversed the buffer, and every place you think you found an edge... write 255, 0, 0, 255 back into the buffer at THAT location.  Make it red.  The jack-o-lantern image is all black and white, so you will be able to see your red dots, once you WRITE the buffer back into image (something similar to line 42).  All the context2d commands/tools... are listed here.  Context2D is a DOM object, and not a BJS object.

You're a brave man, Nesh!  I'll get some coffee... and I'll have a medical team standing-by for ya, in case something blows-up.  Wear your safety gear and check life insurance policies.    There's 43 results for PG search for 'drawImage'.  Might be worth investigating.

Others... have done various "context2d work", when using BJS dynamicTextures.  DynamicTextures allow access to the context2D image that they were made-from.  Dt's, used as a texture on a simple plane... will be easier to work-with than displaceMaps.  There's 17 results for PG search for dynamicTexture AND drawImage.    Things to be learned.  But that jt4#3 PG at the top of this post... that might be a good starter PG, too.

Others may have ideas and comments.  More soon.

[Nesh108]

@Wingnut That's a step closer but DynamicTexture doesn't allow to create a texture from a URL... Darn... Here is what I trying to base it off: http://www.babylonjs-playground.com/#1ZHE0O#2

 

[Wingnut]

Well, you could NOT use a dataURL for now... temporarily.

THEN, you would be allowed to continue work on your edge-finding (perhaps using jack-o-lantern image).

Meantime, Mister Wingnut and other forum helpers... will try to fig a way to use a dataURL in a dynamicTexture.  We'll get out the "jaws-of-life" device that they use at car accidents, and make a big hole in the side of DT's, and cram it in there.   With enough force... you know... stuff happens.  (usually somebody loses a finger)   heh.

I'll keep thinking.  Hopefully, smarter people than I... will comment soon.

[GameMonetize]

the DynamicTexture is a canvas and you can load dataUrl into a canvas:

http://www.html5canvastutorials.com/advanced/html5-canvas-load-image-data-url/

 

[Nesh108]

@Deltakosh Thanks, I tried following the example but using 'drawImage' on the context doesn't seem to work: https://www.babylonjs-playground.com/#1P3JT4#5

The idea now is to be able to draw my donut over the plane and then edit the pixel within it. But now I can't seem to be able to do it. Almost there though!

[GameMonetize]

Here we are: https://www.babylonjs-playground.com/#1P3JT4#6

 

the update has to be called AFTER drawing the image

[Nesh108]

@Deltakosh Great stuff! Thanks!!

@Wingnut Here is the first part of the algorithm: https://www.babylonjs-playground.com/#1P3JT4#9. Not too hard to be honest, after being able to manipulate the pixels it's a piece of cake.

After this, just have to displace the 2nd plane by the wanted thickness: https://www.babylonjs-playground.com/#1P3JT4#10

Connect each pixel with a line and then merge the mesh into a single one: https://www.babylonjs-playground.com/#1P3JT4#11

Now there are some problems left:

1. To fit the lines and match the planes, I had to use some values to shift stuff around. I am probably missing out some basic calculations that I need to do. Gotta think about them (too tired now  )
2. Merging the 2 planes and the lines end up in a total mesh (sorry for the lame joke, I had to): https://www.babylonjs-playground.com/#1P3JT4#14
3. And for some reason, the color of the lines doesn't seem to be applied properly. Weird.

We are almost going to be able to achieve your RedGreen Silhouettes  @Wingnut

 

[Wingnut]

hahah  Excellent work!!!

I actually pee'd a little with excitement! 

I can't wait to see the outline-to-vector2 convertor.  Yum!  (Wingy grabs the WD-44 Processor Lube and a crash helmet) 

And I think there might be trouble ahead... up on El CAPitan  (extrude-capping issues)

SO everyone... SING... "There's... a... flea, on the hair, on the wart, on the ass, on the frog, on the stump, in the hole, in the middle of the doughnut"  (...whose area won't be removed because perhaps extrudeShape can't do doughnut holes).  You'll likely find a work-around, cuz ya got good mad scientist vibes. 

[Nesh108]

Well, let's see if someone can help with problems number 2 and 3. Maybe @Deltakosh can enlighten us about them!

For the first one, I think I have to take into account the pixel size and the canvas size as well. I gotta think harder, @Wingnut

 

[Nesh108]

Update: I think to properly create the mesh around the texture outline, the points need to be ordered into contiguous lines. I will research around to see if there are some proper ways to handle that.

@Deltakosh I think the line created by CreateLines has a problem concerning the colors. Try this:

 

var line1 = BABYLON.Mesh.CreateLines("line1", [new BABYLON.Vector3(0,0,0), new BABYLON.Vector3(1,0,0)], scene, true);
line1.color = new BABYLON.Color3(0,10,1);

var line2 = BABYLON.Mesh.CreateLines("line2", [new BABYLON.Vector3(0,1,0), new BABYLON.Vector3(2,0,0)], scene, true);
line2.color = new BABYLON.Color3(0,80,20);

var line3 = BABYLON.Mesh.CreateLines("line3", [new BABYLON.Vector3(0,2,0), new BABYLON.Vector3(3,0,0)], scene, true);
line3.color = new BABYLON.Color3(0,1,50);

var line4 = BABYLON.Mesh.CreateLines("line4", [new BABYLON.Vector3(0,3,0), new BABYLON.Vector3(3,0,0)], scene, true);
line4.color = new BABYLON.Color3(0,255,100);

var line5 = BABYLON.Mesh.CreateLines("line5", [new BABYLON.Vector3(0,4,0), new BABYLON.Vector3(3,0,0)], scene, true);
line5.color = new BABYLON.Color3(0,200,200);

var line6 = BABYLON.Mesh.CreateLines("line6", [new BABYLON.Vector3(0,5,0), new BABYLON.Vector3(3,0,0)], scene, true);
line6.color = new BABYLON.Color3(0,2,255);

All the same colors to me :/ (or is it that I am color-blind and I never knew?  )

Basically, the problem would be that the colors are not actually a 0-255 range but rather a 0-1 range. Is it by design perhaps?

[jerome]

It's by design.

Just change all your values to : val / 255

[Nesh108]

@jerome I actually just came back from a walk and realized that I was silly not to try the 0-1 range. Now it works perfectly  (pretty weird though that this color is of a different range)

@Wingnut@Deltakosh New update! Now I got the contiguous lines of the outline, the last step is to create the surrounding faces from the vertexData. Here is my attempt, almost there: https://www.babylonjs-playground.com/#9GPMUY#6

Do you see the problem?

[Wingnut]

Cool.  Ya got some vertexData fired-up there, eh?  Nice.  Mad scientist workbench.    Way over MY head, but I'm enjoying the show.

You're a genius, N!  (Wingy checks the first aid kit, making sure it has plenty of brain tumor cream.)  heh

[Nesh108]

@Wingnut Hoorray! https://www.babylonjs-playground.com/#9GPMUY#11

The final 2 problems:

1. How the heck to I properly merge the 2 planes and the new mesh into 1 object? MergeMeshes doesn't seem to work :/

2. How to convert a normal Texture into a DynamicTexture?

Other than that, pretty happy (although there are still those 550/670 magic numbers, pretty annoying).

 

 

[BitOfGold]

That looks so mouth-wateringly cool!