Streaming Mpeg-Dash Video Texture *Now Working w/Webcam Demo*

[MasterSplinter]

Very close to getting a mpeg-dash streaming texture to work.  I had to create a streamingVideoTexture object to look for a videoelement rather than pick a url because of the createUrlObject method.  The one portion I'm having issues with -- I was able to get this to work correctly in vanilla webGL but for whatever reason babylon doesn't seem to like my settings.

In vanilla JS my video texture settings should be something like this:

// gl.NEAREST is also allowed, instead of gl.LINEAR, as neither mipmap.
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
// Prevents s-coordinate wrapping (repeating).
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
// Prevents t-coordinate wrapping (repeating).
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);

However, it seems babylon is forcing some other mode --

Any thoughts?  Thanks!

 

-P

 

[GameMonetize]

You can set wrapping like this:

   refTexture.wrapU = BABYLON.Texture.CLAMP_ADDRESSMODE;
    refTexture.wrapV = BABYLON.Texture.CLAMP_ADDRESSMODE;

[MasterSplinter]

Thanks Delta,

Hopefully I'll get this working tomorrow morning and post my branch in the afternoon --   I know I read someone else who wanted to work with streaming textures.  

[MasterSplinter]

VideoTexture is referred to by name I couldn't create a new object and had to modify the existing VideoTexture class.  It may make sense to allow someone to some method of selecting a videoelement in the future so you can stream video.

Here are the changes I had to make to get this to work.  Thanks yall!!

constructor(name: string, videoId: string, scene : Scene, generateMipMaps = false, invertY = false, samplingMode: number = Texture.TRILINEAR_SAMPLINGMODE) {
    super(null, scene, !generateMipMaps, invertY);

    this.name = name;

    this.video = document.getElementById(videoId);
    this.video.autoplay = false;
    this.video.loop = true;

    this.video.addEventListener("canplaythrough", () => {
        if (Tools.IsExponentOfTwo(this.video.videoWidth) && Tools.IsExponentOfTwo(this.video.videoHeight)) {
            this.wrapU = Texture.WRAP_ADDRESSMODE;
            this.wrapV = Texture.WRAP_ADDRESSMODE;
        } else {
            this.wrapU = Texture.CLAMP_ADDRESSMODE;
            this.wrapV = Texture.CLAMP_ADDRESSMODE;
            generateMipMaps = false;
        }

        this._texture = scene.getEngine().createDynamicTexture(this.video.videoWidth, this.video.videoHeight, generateMipMaps, samplingMode, false);
        this._texture.isReady = true;
    });

    /*urls.forEach(url => {
        //Backwards-compatibility for typescript 1. from 1.3 it should say "SOURCE". see here - https://github.com/Microsoft/TypeScript/issues/1850
        var source = <HTMLSourceElement> document.createElement("source");
        source.src = url;
        this.video.appendChild(source);
    });*/

    this._lastUpdate = Tools.Now;

A second glance it looks like there is a get request on the stream and it won't load without it fully caching..  MAy have to do some more investigating for performance reasons.

 

Helllllo webCAM babylon this weekend

[dbawel]

Hey MS,

I have read several posts with users requesting streaming video support. I persoanally am very interested in this, as I will begin building an app next month in which streaming video to multiple users is key. We are already doing this from our node.js server, and providing real time controls for all users, so the experience is shared as well as the controls across any # of users. However, I haven't attempted streaming onto a videoTexture, but only HTML5 video objects. So this is perfect timing for myself, and I'm certain many others as well. If you are able to provide an example when you are ready, this would be very helpful to the community.

Thank you for looking at this and for working to impliment.

Cheers,

DB

[GameMonetize]

You should think about doing a PR (which could be even more useful!)

[GameMonetize]

I did the update actually

[dbawel]

MasterSplinter,

Do you have a link to the latest demo of this? I would love to take a look at the last working version, and thanks to everyone who took part in achieving such an awesome function. In my opinion, it is the aded features such as this which defines babylon.js outside of all f the rest of the WebGL frameworks, and why I exclusively use babylon.js in practically every project now. Im still trying to convice companies such as Magic Leap that WebGL is the furture of all graphic content (at least in the next decade), and surprised they are still divided on this. However, functions out of the box such as streaming video show that there is no other framework or platform that exists which has the functionality as well as the whole support of most every device and platform. Not to mention that its open architecture makes this limitless in its potential to graphically produce anything.

DK and his collegues must be very happy with what they have built. However, I have to say that it wasn't by chance since the design of the framework is so very well planned and executed; providing an architecture which apparently has no limits beyond the constraints placed by the hardware, device manufacturers, and platform developers. 

Thanks,

DB

[Srisub]

Hey guys, i prototyped an app combining webRTC-driven HTMLVideoElement paired with a VideoTexture in THREE.js. I am trying to switch to Babylon.js because its a better game engine, and am mostly there, except for the hurdle that VideoTexture is really about static video URLs, not streaming or live video. As i can see from this thread, that seems to be WIP.

Can someone please point to any code for streaming/live video - basically grabbing the live content coming into a HTMLVideoElement and attaching it to a node material's texture. 

[GameMonetize]

Hey

 

can you try using bjs 2.4? This should work now if you do not provide urls

[Srisub]

I have been using 2.4 (preview release). I noticed the following in the latest 2.4 code, which i believe is why it didnt work for.

function VideoTexture(name, urls, scene, generateMipMaps, invertY, samplingMode) {
            var _this = this;
            if (generateMipMaps === void 0) { generateMipMaps = false; }
            if (invertY === void 0) { invertY = false; }
            if (samplingMode === void 0) { samplingMode = BABYLON.Texture.TRILINEAR_SAMPLINGMODE; }
            _super.call(this, null, scene, !generateMipMaps, invertY);
            this._autoLaunch = true;
            this.name = name;
            this.video = document.createElement("video");   /////----> I *already* have a video element that WebRTC streams into, but VideoTexture
            this.video.autoplay = false;                    /////       creates a new one. I dont pass in urls, but want to pass in the video element
            this.video.loop = true;

 

 

[GameMonetize]

Ok I'm ok to update the videoTexture object to support this. I'll work on this tonight

[GameMonetize]

Just updated the repo: you can now send a video element instead of the urls

[MasterSplinter]

3 hours ago, Deltakosh said:

Just updated the repo: you can now send a video element instead of the urls

Heeey -- I was just about to make these updates and then I saw you had already done so.  However, one thing I noticed.  The canplaythrough event only fires after a streamed source has played 100% through the video.  This could cause problems if you were to use a webcam or something.  I would suggest removing that condition or attaching to an event that fires from streamed sources.  (which I tested a bunch and couldn't find one that works -- I just flat out removed that for playing HLS and mpeg-dash).

Just branched and added PR.

[MasterSplinter]

Okay,

Just made a webcam demo for ya'll to look at it... I kinda did this quick with some magic copy-pasta action.

My version has a slightly different version of BABYLON.VideoTexture as I removed on video ready whatever its called event behavior.

https://rawgit.com/wpdildine/webCAMbabylon/master/index.html

As added bonus this works on my cell phone!

[GameMonetize]

Fantastic...Please PR

[MasterSplinter]

I added my changes.  If I have a moment later today I'll make that example a playground example.

[GameMonetize]

yep this is cool..Can you please just refactor common code?

[MasterSplinter]

Yup send it up in a second.

[MasterSplinter]

I think this is... i mean its a cors issue...

http://www.babylonjs-playground.com/#1NYKLE#8

[GameMonetize]

sounds like..

but your update works

[dbawel]

Hi,

I saw that Srishub posted the following:

this.video.autoplay = false;

I am not familiar with this function. Am I to understand that this will stop autoplay of a video texture on PC? As most of you know, autoplay is enabled on PC, but not on Android. So I am not close to my laptop in the next 2 days to test, but if users on this forum know that this will stop the autoplay on PC, I won't need to test when I return from Mexico. I'm throwing back tequilla right now since it is after 12pm - and in Mexico, that is late to start drinking.

Adios,

DB

[Srisub]

Hey guys, thanks so much for this. I tried it yby ting it to a video element that receives a WebRTC webcam video, and i got this error. 

[.CommandBufferContext]RENDER WARNING: texture bound to texture unit 0 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering.

I am also gtting very weird U-V mapping results (texture looks jumbled) but dont know if thats a blender issue or the above one. Wil keep digging in..

[Srisub]

Hey guys, i tried using this, and it keeps giving me the power-of-2 warning, and doesnt display the (live webrtc) video texture.

Warning:
[.CommandBufferContext]RENDER WARNING: texture bound to texture unit 0 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering.

Code - i even tried explicitly setting to clamp_addressmode, and linear filter.

var texture = new BABYLON.VideoTexture( node.name + "_Video", video, this.scene, false, false, BABYLON.Texture.BILINEAR_SAMPLINGMODE );
texture.wrapU = BABYLON.Texture.CLAMP_ADDRESSMODE;
texture.wrapV = BABYLON.Texture.CLAMP_ADDRESSMODE;
texture._generateMipMaps = false;

[Srisub]

In fact, i tried the playground example link above:

http://www.babylonjs-playground.com/#1NYKLE#6

And i got tons of the same error!

babylon.js:3 BJS - [15:18:10]: Babylon.js engine (v2.4.0-alpha) launched
VM170:4 getUserMedia() no longer works on insecure origins. To use this feature, you should consider switching your application to a secure origin, such as HTTPS. See https://goo.gl/rStTGz for more details.
256/#1NYKLE#6:1 [.CommandBufferContext]RENDER WARNING: texture bound to texture unit 0 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering.
/#1NYKLE#6:1 WebGL: too many errors, no more errors will be reported to the console for this context.