Using Webcam with Pixijs v4

[zeh]

I'm trying to create a Sprite using PixiJS v4 (or v5?) but feeding a texture from a Webcam (a MediaStream) into it. It doesn't seem to be possible.

An old 2017 thread has a link to an example that should work. Unfortunately 1) this example doesn't work anymore, likely because of HTML change since then and 2) even if it worked, the solution also seems pretty convoluted (copying frame by frame?) so I'd like to avoid something like that if possible.

On my case, I've tried creating a <Video> element and setting a source from that...

const stream; // This is a MediaStream that comes from navigator.mediaDevices.getUserMedia()
const video = document.createElement("video");
video.autoplay = true;
video.srcObject = stream;
document.documentElement.appendChild(video);

This works well, since we can set the srcObject (not the src) of a <Video> to a stream and it works magically.

But the problem arises when I try adding that to Pixi. I can do this:

const texture = Texture.fromVideo(video, SCALE_MODES.LINEAR, true, false);
const sprite = new Sprite(texture);

Which is, I pass the <Video> instance (since I don't have a url) to the texture. But it doesn't work; Pixi itself is kinda silent, but them I get an initial Chrome render error:

[.WebGL-0000017A07139680]GL ERROR :GL_INVALID_OPERATION : glGenerateMipmap: Can not generate mips

And additional errors on every frame:

[.WebGL-0000017A07139680]RENDER WARNING: texture bound to texture unit 8 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering.

Despite what the error says, I don't think it has anything to do with the bounds of the texture. Even if I grab a 256x256 texture from the camera, it stops working.

For comparison, if I just set the <Video> src to a normal file...

const video = document.createElement("video");
video.src = "myvideo.mp4";
document.documentElement.appendChild(video);

...then the above code works fine and I'm able to create a Texture from a <Video> tag.

I've tried a different approach, creating a video URL out of a media stream (which, technically, should work):

const stream; // This is a MediaStream that comes from navigator.mediaDevices.getUserMedia()
const videoURL = URL.createObjectURL(stream);
const texture = Texture.fromVideo(videoURL, SCALE_MODES.LINEAR, true, false);
const sprite = new Sprite(texture);
this.addChild(sprite);

But I get a nasty error on the "URL.createObjectURL" line:

Uncaught TypeError: Failed to execute 'createObjectURL' on 'URL': No function was found that matched the signature provided.

So, anyway.

Does anyone know of a way to create a texture from a webcam input in PixiJS?

[zeh]

Udpate with some progress: somehow, I need to call video.play() before I attempt to create the texture (despite autoplay).

video.play();

This seems to make it work for the first update. Then we have to update on every frame...

video.addEventListener("canplay", () => {
	const texture = Texture.fromVideo(video, SCALE_MODES.LINEAR, true, false);
	const sprite = new Sprite(texture);
	this.addChild(sprite);
	
	// Wait a frame before start updating, otherwise we still get the WebGL error
	requestAnimationFrame(() => {
		ticker.shared.add(() => {
			texture.update();
		});
	});
});

But it still seems to break pretty easily.

I could wrap around all of that and make it cleaner to reuse, but I'm still curious if there's a better solution.

[ivan.popelyshev]

Looking by errors, you have to switch off mipmaps before pixi updates videotexture first time. "texture.baseTexture.mipmap=false" That's all.

[zeh]

Thanks for the suggestion @ivan.popelyshev. Unfortunately "texture.baseTexture.mipmap = false" (right after calling Texture.fromVideo()) didn't do much... the error is still there if I don't do video.play() beforehand.

I'll keep it though, hopefully it'll make the whole endeavor more stable.

I do wonder if there's a more elegant overall solution however.

[ivan.popelyshev]

try pixi-v5 (pixijs.download/dev/pixi.js, pixijs.download/dev/docs/index.html) , its in "dev" branch, it has TextureResource thingy that can help to make your own custom logic for using texImage2D.

If you dont know whats texImage2D and how to upload stuff in videomemory, well, i dont think i can help you, its just looks like a problem that REQUIRES fiddling with pure WebGL

Of course you can try hack v4 and do something there, hack VideoBaseTexture. 

I just don't know how to help you, sorry

[ivan.popelyshev]

Actually, that's an idea: can you make it using pure webgl and not pixi first? Then we can fix pixi that way it does same operations you do!

[zeh]

Thanks @ivan.popelyshev. Right now I'm using Pixi v5 from [email protected] from npm, it seems to be working fine with the same workarounds...

My current "high-level" solution (just a Sprite with a bunch of events) seem to be working well so for now the solution above suffices as I'm just trying to get this work moving...

If I do have time later I'll try creating a new MediaStreamTextureResource or some such and contribute to v5. I have no problem dealing with pure WebGL, but I'd like to understand the Pixi internals first before I put something together to ensure it's following the standard architecture and can be cleanly merged.

[ivan.popelyshev]

Nice!

TextureResource thingy: https://github.com/pixijs/pixi.js/tree/dev/packages/core/src/textures/resources , if you create your own resource , you can pass it to BaseTexture constructor, and your code will be called from TextureSystem when its time to upload the texture to the memory. It wasn't easy to push it to pixi, and that's the first time this part of renderer is exposed in html5 rendering library

[zeh]

Cool, thanks again. I'm reading through it and it seems fairly approachable.

Just as a heads up right now a first roadblock seems to be that the TypeScript definitions for Pixiv5 are not up-to-date (e.g. resources.VideoResource is not present, even though it exists; other resources.* are properly defined) so there's a little bit to fix until I get there.

For reference, after a ~30 min test, this seems to work as a new MediaStreamResource, simply by extending VideoResource:

import { Ticker, resources } from 'pixi.js';

/**
 * Resource type for HTMLVideoElement.
 * @class
 * @extends PIXI.resources.VideoResource
 * @memberof PIXI.resources
 * @param {MediaStream} source - Media stream to use
 * @param {object} [options] - Options to use
 * @param {number} [options.updateFPS=0] - How many times a second to update the texture from the video.
 * Leave at 0 to update at every render.
 */
export default class MediaStreamResource extends resources.VideoResource
{
	constructor(source, options)
	{
		options = options || {};

		const videoElement = document.createElement('video');

		videoElement.srcObject = source;
		videoElement.play();

		super(videoElement, {
			autoPlay: true,
			autoLoad: true,
			...options.updateFPS,
		});
	}
}

And the usage:

const stream = ...; // MediaStream
const res = new MediaStreamResource(stream, {});
const baseTexture = new BaseTexture(res, { mipmap: false });
const mediaTexture = new Texture(baseTexture);
const mediaSprite = new Sprite(mediaTexture);
this.addChild(mediaSprite);

But there's still some errors to deal with (a weird DOMException that doesn't break anything, some Video-specific methods that are useless for MediaStreams, etc). Maybe a better solution would be to copy&paste VideoResource (extending BaseImageResource instead of VideoResource) and remove unneeded behavior. It'd be a bit of duplication but probably the correct solution.

One alternative solution is also just detecting a MediaStream "source" given to VideoResource and using that instead. In the same way where you can pass a source URL.

Anyway, I'll take a more serious stab at it soon.