video alpha channel approaches

[Dickensian]

Hi,

What is the best way to tackle displaying a video texture with a video alpha channel?

Dickensian

[GameMonetize]

Hey...Are you aware of a video codec that supports alpha?

[Dickensian]

Hi,

thanks for your reply

i've tried the Chrome support for WebM with alpha which works but is chrome desktop only, and I've read previous posts that people have done using half the video frame as a luminosity mask with html5 canvas libraries. I've not explicitly looked at multiplexed streams, but I guess they'd be another option.

Even if we could encode a matte colour and mask on that with a defringe filter a single RGB video texture could work. The challenge with using two non-multiplexed sources for the texture and the matte is around maintaining the sync.

Should I be looking at a custom shader or do you recommend another approach to achieve this; I'd like to ultimately have an actor walk on canvas and point at the 3D model and talk about it for an educational demo I'm working on in my free time.

Dickensian

 

[GameMonetize]

Yes I guess the best option is to use a custom shader with a videoTexture. This way you can control how to apply the mask

Here is an article about customshaders in babylon.js: https://blogs.msdn.microsoft.com/eternalcoding/2014/04/17/what-do-you-mean-by-shaders-learn-how-to-create-shaders-with-babylon-js/

[Dickensian]

Great link, thank you

i'll pursue this and post back when I know more.

BTW did you know that a lot of the tutorial links from Google search results are hitting 401's since the site changes, if an admin could set up some 301 redirects based on the logs it would be a great help to noobs like me.

Dickensian

[MasterSplinter]

I would also look at how shadertoy has done some of their implementations.

 

https://www.shadertoy.com/

[dbawel]

So, has anyone been able to recognize alpha channels (RGBA video) on supported video formats such as webm in babylon.js? I doubt it, but it's worth asking. Has anyone such as @Dickensian who started this topic built a shader to recognize a color range in RGB pixels and convert to transparent pixels? I beileve this might be the quickest route to a demo, but then I would love to have the support of alpha channels RGBA on video formats such as webm and forthcoming formats. Any ideas beyond the above?

OK @Wingnut you love a challenge. And just so you know, I've spent days trolling the web and found extensions and possible solutions, but nothing really worth noting here - as I'd really like to add this function distinctly to BJS.

Cheers,

DB

EDIT - What about using the alphaindex of a mesh? Is this a crazy approach?

[tranlong021988]

This is a solution using chromakey Shader for video texture: https://makc3d.wordpress.com/2014/04/01/transparent-video-texture-in-three-js/

It's working with threejs but you can implement Chroma Key Shader to BabylonJS with a little modification. 

[dbawel]

@tranlong021988 -

Thank you, this is very helpful and perhaps not too difficult to impliment in BJS. I just need to review the materials structure in the babylon-master file. If I get a working version of a material similar to this, I'll post an update.

Cheers,

DB

[adam]

9 hours ago, tranlong021988 said:

This is a solution using chromakey Shader for video texture: https://makc3d.wordpress.com/2014/04/01/transparent-video-texture-in-three-js/

demo link on that page is NSFW

[Dickensian]

@dbawel I've not had time to look at this unfortunately, although the shader that @tranlong021988  linked to looks promising for integration into Babylon.js.

@dbawel I'm looking to make this work for a not-for-profit educational tool, if you'd like help with peer review on your code, please publish a link to your working repo and I'd be happy to contribute some effort to it.

Thanks,

Dickensian.

[dbawel]

@Dickensian -

We have a simultanious multiuser media design application we are alpha testing. Please message me personally for more info if you are interested in learning more. Perhaps you might have a group which can help test and us build a better app. It is currently going to Weta and to Lockheed Martin for testing. Video compositing is for a secondary feature, and not for first version release.

@adam - Please let me know what is NSFW about the link, as I've not found any issues with it thus far. If so, I should be more on my game.

DB

[adam]

The dancing girl in the demo lifts up her shirt.

[dbawel]

@adam - Yes, I did notice that and thought she did a fantastic job. I thought you meant that there was some invasive code in the script.

[Dickensian]

...and they wonder why there's a shortage of developers of the fairer gender in our profession.

[dbawel]

Because we don't look so good lifting up our shirts! I know that when I put on my tight shorts and shirt, it's not a pretty sight.

[Dickensian]

so, about the chromakey shader in Babylon.js; have you had any joy? :-)

[Pryme8]

A shader that can recognize a chroma key.  I'd look at the video shader and then see if you can pass a color and a threshold value to the shader then modify the shader to turn the pxls that it's displaying a alpha accordingly.

[Dickensian]

<script id="fragmentShader" type="x-shader/x-fragment">
	uniform sampler2D texture;
	uniform vec3 color;
	varying vec2 vUv;
	void main()
	{
		vec3 tColor = texture2D( texture, vUv ).rgb;
		float a = (length(tColor - color) - 0.5) * 7.0;
		
		gl_FragColor = vec4(tColor, a);
	}
</script>

source: https://github.com/makc/makc.github.io/blob/master/three.js/chromakey/index.html 

@Pryme8 would you impliment the shader like this example from a three.js project or differently? Are you aware of any approaches to defringe the edges, where the greescreen cast can show through?

[Pryme8]

Um yeah I can possible write something up for you tonight or tomorrow, I'll just modify Babylons video shader and upload the source for you, and yes I'll include a bounding box for the alpha but I'm not sure why you would need that.  Anyways I'll post updates once I have them unless someone beats me to it.

[adam]

1 hour ago, Pryme8 said:

the shader to turn the pxls that it's displaying a alpha accordingly

I might be wrong, but I think @NasimiAsl recommends using "discard".

[Pryme8]

I'll have to look that up

[NasimiAsl]

we really need fix alpha blend problem 

we don't use that in BJS for now so maybe need some team start work that (i wanna join but i know less in this stuff )

gl.blendEquation( gl["FUNC_ADD"] )

gl.blendFunc( sourceBlendType  ,  destinationBlendType , blendEquation );

gl.enable(gl.BLEND);

gl.depthMask(false);

you can see parameter value from thire

https://msdn.microsoft.com/en-us/library/dn302371(v=vs.85).aspx

we just need list of alpha Object (for find gl Render Object ) and refresh that when we change camera position

http://delphic.me.uk/webglalpha.html

Demo : http://delphic.me.uk/Fury/demos/AlphaTest/

but about discard is good solution

other plan for that ( play video in 3d scene like lizer draw ) you just draw light color 

http://www.babylonjs-playground.com/#3KRGA#1

http://www.babylonjs-playground.com/#3KRGA#2

i dont know why sample 1 work fine but sample 2 is wrong ( alpha sorting problem )

anyway i think if you work with single face maybe you can use alpha without any problem (if you don't care for Low FPS )

http://www.babylonjs-playground.com/#3KRGA#3

http://www.babylonjs-playground.com/#3KRGA#4

 

 

 

[Pryme8]

I can not find the babylon.js git on the videoShader

http://makc.github.io/three.js/chromakey/

yeah looks like using a chromatic is a valid approach.

also @NasimiAsl
I striped a bunch of stuff out of your example http://www.babylonjs-playground.com/#2BBTZV

how did you do the alpha calculations?  I was gonna edit the GLSL files but it looks like your able to do it with your shader builder include?
can you explain whats going on in your example on how your generating the basic alpha transparency.  I know exactly how I will do the chromatic key I just need to be able to see the videoShaders fragment and vertex scripts, its a really easy clamp calculation and the only area of concern is the dithering area between key and objects on the video.  Should not be to bad and I will add a threshold to adjust how fast that fades.

***UPDATE EDIT,
still cant find the video shader, but I have confirmed that discard is going to be the method of choice, just setting the alpha makes the texture colors dimmer.

If you look at the discard example on the CYOS and modify the line for the discard to read:

if (color.g > 0.45) {
        discard;
    }else{
        color.r = 1.0;
        color.g = 0.0;
        color.b = 0.0;
    }
you can see that its pretty clean, and would work for general purposes, and the color correction at the end could be tailored. now to just load a video in and have a variable for the color you want to discard and how much of it.

[NasimiAsl]

1 hour ago, Pryme8 said:

ow your generating the basic alpha transparency

in Each Step of ShaderBuilder Commands I change the 'result' variable

'result' is your Fragment Color you Made be before and you most change that if you want change the color

for use Enable Alpha you just need Add '.Transparency()' step in your  commands

and for set Alpha parameter you just need write a line

 .InLine('result.w = 0.1;')  or InLine('result.w = sin(time*01);')

 

http://www.babylonjs-playground.com/#2BBTZV#1

for understanding InLine you just write your shader in source and in last make one vec4 and set that in result

http://www.babylonjs-playground.com/#2BBTZV#2