Decals

[fenomas]

Is there any current or planned support for decals - i.e. projecting a texture onto a mesh?

 

I don't see anything in the APIs, but from looking at how shadows work it seems like the engine might already have most of the needed moving parts.

 

If there's already been discussion of this a pointer would be great.

[GameMonetize]

Hey,

 

you can set texture.coordinatesMode = BABYLON.Texture.PROJECTION_MODE as a good first step

[fenomas]

Hey,

 

Thanks for the reply, but it doesn't give any hints what the answer to my question is.

Also, there don't seem to be any docs or tutorials that discuss projection mode in any way. 

 

Could you maybe wrap one or two more words around it? 

[Wingnut]

hahaha!  Boy, have I been in YOUR shoes a few times, Fenny.  Careful... Deltakosh will label you as a "puppy", and that is a fate worse than herpes.   

Metaphorically speaking, you are expected to know how to split atoms... after your public tour of the nuclear power plant.  heh.

I'm just having some fun, here.  Actually, it is me that is supposed to TRY to make sure we have documentation for all framework features, and I have fallen behind in that responsibility.  Unfortunately, I need to do the very same research as you are needing to do, Fenomas... just so I can write some more documentation on this subject.  (but I love it)

First, let's get a very accurate definition of "decal".  Let me try one... and Fenny, you can correct it, adjust it, or do your own.

Decal - a texture that is "placed upon" another texture/material... without needing texture coordinates for it.  It can be easily sized so that it APPEARS to be similar to a sponsor "sticker" placed on the side of a race car.  The sticker texture then moves with the car, and does not shift its location upon the car whatsoever.  It becomes part of the diffuseTexture of the car... and it can be placed after the initial rendering.  (How's that?) 

Now let me tell you what I've found so far:

According to our MSDN-based tutorial called Unleash the StandardMaterial (far down in the "ReflectionTexture" section), we can read that Spherical, Cubic, Planar, and Projection... are called "kinds of reflection".  This MIGHT imply that these "kinds" or "modes" are ONLY used with reflectionTextures.

But that might not be true.  We would probably need a github search for PROJECTION_MODE to learn how/when it is used or taken into consideration by other BJS systems.  (primarily... material/texture systems, of course)

Now for my pathetic test playground... http://playground.babylonjs.com/#1215MA

I was trying to "project" (decal) the picture of the officer... onto the side of an already-textured sphere.  It works... but it's seemingly not sizeable, and it moves-around on the sphere because it is a reflection texture.  This is not really what you/we are seeking.  What WE want... is more like a slide projector (spot light that "shoots" an image).  We want the projected texture to move with the mesh (diffuse-ish), and not with the reflection.  The slide projector/texture projector... is attached to the car... or at least CAN be.

Does any of this sound correct?  I hope so.  Otherwise I need a definition correction for the term "decal". 

Here is an old Bryce render that I THINK shows what is wanted...  http://webpages.charter.net/wingthing/img/projector.jpg  (the ISS projected onto the side of a hill).  Slide projector.  Movie projector.  Spotlight with picture attached.  Yay! 

My projection (in that picture) is blurry, but a GOOD projector should be able to blur, or not. 

Stickers.  Extra textures, placed upon mesh anytime.  No need to have them in place prior to initial scene render.  Sound right?

Ok, here we go. Time to try to 'crowbar' secrets and cool ideas from Deltakosh and other renderGods.  Here's a method I haven't tried yet...

Deltakosh, please... help us!  We don't know what we'll do without your beacon of guidance!!  Please... think about the starving CHILDREN (and their love of puppies)!!!  (noooo, they won't EAT the puppies, geez!)

Blech, I'm not very creative or fluent with my begging-to-avoid-legwork methods, am I? 

Deltakosh COULD BE blowing breadcrumbs off-of his table, and seeing if we can conjure a solve on our own.  I do like this trait about him.  When a fart is difficult to pass, and fart-assistance is elusive, then the "hey, I did it all by myself" fart is more appreciated.    Or, something like that, anyway.

I am the opposite.  If a user asks a question, I like to drown them in TMI.    But I have much more free time than DK.

Yep, the decal.  It's related to the spotlight image projector that I have wanted to accomplish for awhile, yet not the same.  One is lightMap, the other is textureMap. 

For some reason, I think the textureMap version is easier to accomplish, but without uvs on the mesh FOR the decal, the decal seems to reject sizings and mesh-clamp. 

I think we need to work in the "diffuse" area of town, not in the "reflect" area.  Thoughts?  (sorry for the long post)

[jerome]

public static EXPLICIT_MODE = 0;public static SPHERICAL_MODE = 1;public static PLANAR_MODE = 2;public static CUBIC_MODE = 3;public static PROJECTION_MODE = 4;public static SKYBOX_MODE = 5;

Well, there are some constants in the texture.ts file.

I guess they each represent a pre-determined way to compute UVs.

So PROJECTION_MODE might be the way to project an image onto a volume... different from PLANAR_MODE which would be the planar painting of a mesh (no volumic perspective taken in account)

 

You're right Wingy, it's more a reflection only feature, I guess...

Different from what fenomas was talking about, I think. I imagine decal is more about :

given a mesh, an axis or vector and a texture... how to apply this one onto the mesh following the axis ?

[Wingnut]

"volumic perspective".  Nice!

 

(Wingy tries to grind-off a fresh tumor with his Black & Decker power sander.) 

[fenomas]

I think we need to work in the "diffuse" area of town, not in the "reflect" area.  Thoughts?  (sorry for the long post)

 

Hey, thanks for the very in-depth reply!

 

I'm not sure about the terminology, but what I'm asking about is like your projection example. Since such effects are quite similar to shadows, except that one projects a static texture rather than a rendered silhouette, I thought there might be code in the shadows implementation that could be used.

 

Regarding bread crumbs, I know the devs are busy and I'm not asking anyone to write code for me. Just asking if the code that would be necessary is already in the engine, planned for the future, half yes and half no, etc.

 

Finally, for the puppy/fart stuff I'm afraid you lost me, but I'm glad you're having fun. 

[fenomas]

You're right Wingy, it's more a reflection only feature, I guess...Different from what fenomas was talking about, I think. I imagine decal is more about :given a mesh, an axis or vector and a texture... how to apply this one onto the mesh following the axis ?

 

Yes, I looked into it a fair bit and projection mode is only ever used for reflections. (I only checked the default shader.) Whether it could be used to simulate a projection, I've no idea.

[Wingnut]

You're probably better-off lost... with the puppy/fart stuff.    You are not being a puppy at all, Fen. 

 

Puppy is a term that DK and others use to describe folks who want everything done FOR them.  (I got some puppy in me)

 

I was having fun with Deltakosh. I think he is sometimes quick to call folks 'puppy'... cuz he' a friggin' God of 3D and BJS ops.

 

Sometimes it seems that he wants us to know how to do things... that are very foreign to us.  But it's all in good fun. 

 

If you place dynamite on Deltakosh in just the right places... you can blast very valuable information from him.  And sometimes, he has no idea how to accomplish a challenge... because he is a toolmaker, and does not know ALL of the things that are possible with those tools.  Often... the framework contributors are doing what you stated, Fen.  Trying to get the doors open between users and the graphics drivers... bleeding edge... a somewhat un-tested frontier.

 

So, yes, I was using you to have fun with my buddy Deltakosh (who is really quite a fantastic guy).  I hardly know him.  I'm not sure that he isn't a robot...  some kind of Azure Cloud-driven walking talking thinking coding bio-bot. 

 

You probably understand the fart analogy, too, now.  Things are more satisfying when we can do them with no assistance from the big dogs. 

 

Now, where were we before we found out that most of Wingy's comedy failed? 

 

Oh yeah... them things.  Light-projected texture-placing.... and "put it anywhere in space" texture-placing that Jerome recently brought-up.  Jerome, your way is the same as putting it on a plane somewhere in space, right?  It won't follow the contours of an object mesh.  The side of a race car has contours... and we would like it to follow those.

 

But... placing it at an arbitrary point in the space UPON THE CAR's CURRENT TEXTURE... now... that's sound yummy.  But it has to follow contours of the mesh... somehow.  (I think)

 

Actually, both "big dog" methods are needed.  We need the slide projector spotlight thing, and we need the non-light "stick this texture onto ANYTHING" system.  I think.  One texture based, one light based.  Eh?  *scratch scratch*

 

Yeah, Fen... good question.  Do we currently have the tools to do this?  Fun challenge.  Be well.

[Wingnut]

Oh, I just thought of something.  Maybe... what we should be looking-at... is being able to contour a basic plane (actually, a mid-rez createGround plane)... identical-to a section-of another mesh.  We would spawn a plane that was "bent" just like the place we want to put the decal.  Then texture that bent-up plane however we like, and then stick it to the side of the car. 

 

Wow.  Sub-mesh spawning.  hmm.  I bet Jerome could pull it off.  He's a plot-God!

[jerome]

I guess the problem is more this one :

you've got a mesh,

you've got a plane (defined by a vector and a point at the surface of the mesh),

you've got a rectangular image

 

You need then to make a plane projection of each visible mesh vertex (in terms of backfaceculling) onto this plane... well, turning some 3D coordinates into 2D coordinates.

These will then be your computed image UVs (after the right translation in the image 2D system and scaling to 0-1)

 

Plane projection is not that complex, just some maths... If the projected texture doesn't need to move on the mesh surface, it could be done once CPU side, else I guess a shader should be necessary.

Selecting the visible vertices only is a mystery to me.   

 

 

Yes, this should be the same algo than the one used to cast shadows.... unless another trick  is used  : "in 3D everything is fake" (DK's video)

[jerome]

Btw, for the ribbon, I implemented a texturing rule quite different.

It's not based on a plane projection but on distance between vertices.

As a ribbon is designed by many successive paths side by side and paths are themselves successive Vector3, I compute both :

• distances of each vertex on a path from the path beginning (sum of every vector norms until each point),
• distances of each vertex at position i along j paths (pathA_point i -> pathB_point i -> pathC_point i -> ... ->pathj_point i, same sum method)

(not sure I'm very clear   but it works ... )

 

Scaled then to percentages, this give U and V to be applied to the texture.

This give a volumic stretching effect : the whole image is stretched on the given vertices according to their relative distances.

[Wingnut]

nod.  (Thanks Jerome)

Picture this, if you please.  http://playground.babylonjs.com/#27RX3A

Well that was easy to picture with the help of the playground, eh? 

Imagine shooting a distance-measuring ray... from each point on the green decal plane... to the sphere.  Once those distances are collected, they can be used to contour (bend) the decal plane.  Once you have that plane contoured, it will make a perfectly fine decal.  Just put any texture on it, and you have your complete contour-matching "sticker".

A 3d laser scanner! 

But, there's user problems.  If any distance-checking ray... misses the sphere... the "contour cloning" will fail.

Also, if the user is trying to make a decal that wraps around a tube/cylinder, the decal cannot wrap to the back side.  The distance raycasts can't "see" the backside of the cylinder.

*** Actually, it CAN, but you would have to rotate/position (orbit) the decal... precise amounts... around the scanned cylinder...  WHILE the ray-scan was being done.  Essentially, you would fly the decal plane in an orbit around the cylinder... as it's being distance-scanned.  (phew)

But in the case where the user wants a decal wrapped around a cylinder, the user could just create ANOTHER textured cylinder and place it around the first (like a belt). 

-----------

I'm talking about non-lightMap/shadowMap options, here.  The light/shadow map option is still a good one, I suspect.  But shadows and spotlights have blurry edges.  But maybe, they are coded that way... for effect.  *shrug*

I'm not really AT ALL qualified to be yapping in this thread.  hehe.  Hell, I'm just the social coordinator for this joint.  Nobody said the job would involve math, and plotting, and challenging questions from Fenomas. 

[fenomas]

Gentle reminder that it would be great if a dev could answer this question:

 

Just asking if the code that would be necessary [to project a texture onto a mesh] is already in the engine, planned for the future, half yes and half no, etc.

[jerome]

half maybe

[GameMonetize]

Sorry I was not clear enough

 

To project a texture, you can use a reflectionTexture on your standardMaterial and set  texture.coordinatesMode = BABYLON.Texture.PROJECTION_MODE

[fenomas]

To project a texture, you can use a reflectionTexture on your standardMaterial and set  texture.coordinatesMode = BABYLON.Texture.PROJECTION_MODE

 

Okay, then the texture gets drawn in screen coordinates.

 

Are you suggesting this could be done to project the texture onto a particular part of the mesh? I looked at the internals but I don't see any way to control where the projection is drawn.

[GameMonetize]

Yes you're right, so far we do not have a way to control the part where texture is drawn

[fenomas]

Okay, so basically there is no code for what I'm asking about.

 

Out of curiosity, what is projection mode meant to be used for?

That is, what is the use case for overlaying a texture onto a mesh in untransformed screen coordinates?

[GameMonetize]

Shadow maps mainly, mirrors as well

 

And I disagree with you, there is a lot to do what you're asking for:

- You can set your texture to the reflectionTexture of a stdMaterial

- Then define coordinatesMode to projection

- Overload this function: https://github.com/BabylonJS/Babylon.js/blob/e98c6d93d80e3d7be61114be5b228784c26c6f84/Babylon/Materials/Textures/babylon.texture.js#L114

 

Here is the code used by the shader:

https://github.com/BabylonJS/Babylon.js/blob/master/Babylon/Shaders/default.fragment.fx#L191

 

As you can see it use the reflection matrix to compute the right UV. 

 

So basically, you just have to change the getReflectionMatrix of your texture and you're done

[fenomas]

Hmmph, well why didn't you say so in the first place? 

 

 

http://www.babylonjs-playground.com/#UCORW

 

[GameMonetize]

because I love to discuss with you guys

 

More seriously, I have so many questions to answer so sometimes I just give some pointers

 

I'll work on a DecalTexture to simplify this (Or perhaps this is something you want to PR?)

[fenomas]

I know you're busy, I meant just before where you said there was no way to control the positioning.

 

I mean, I like a puzzle as much as the next guy, but I read every single google hit for babylon and "projection_mode", and none of them says anything except that it exists. So it's not really something where a little hint will just lead to the answer. To be frank, I still have no idea what projection mode is or what it's meant to be doing or how it differs from other modes.

 

I'll work on a DecalTexture to simplify this (Or perhaps this is something you want to PR?)

 

If it can be generalized, sure, I think it would be a useful API. By generalized I mean something like, the user specifies a mesh, a texture, a vec3 along which to project. Or would you need an up-vector as well? Anyway you see what I mean.

 

It's definitely not something I can do though, I just hammered that demo out by brute force and I assume it wouldn't work for arbitrary meshes. I don't even know what those hard-coded matrix values are doing, or, again, what the damned mode is meant to be doing in the first place.

 

If you do add an API it might be better to call it ReflectionDecal or something like that, to distinguish it from what people would expect "real" decals to be (i.e. this).

[GameMonetize]

What's the difference with the example you linked and what you got in the PG?

[fenomas]

In the link example the decals have their own reflections and so on. I imagine it's probably creating a new geometry for each decal.

 

I guess the reflection method is presumably faster (if it generalizes to arbitrary meshes?). But impossible to have more than one decal per mesh, which is unfortunate since people tend to use decals for bullet marks and so on.