solved-or-something Web Audio API definition differences

[JCPalmer]

I was attempting to use MediaRecorder to mix all the BABYLON.Sounds that called play() into a single opus audio track, at the relative time they started playing.  I am starting from this stackoverflow topic.  Am writing in Typescript, and found MediaRecorder is not yet defined.  I can live with that.

The bigger problem is an AudioContext is defined very inconsistently.  I will need createMediaStreamDestination(), which is defined by Mozilla, but not a method in Typescript.  I think, ok maybe do a PR of the missing things, but better check what WWW says.  WTF, not one them matches another!  What am I missing?

This is part of to a rewrite / typescript conversion of Whammy, calling Double Whammy.  I know whammy only works on chrome, so got to test this works there before proceeding with the larger project.  I have verified opus is a valid webM audio format though, and think the video should play everywhere.

[Guest]

Pinging our audio master: @davrous

[JCPalmer]

I realize that this was pretty much a bitch thread, but checking what chrome supports for MediaRecorder, found some interesting stuff in addition to opus support.

In order to get the codecs=vp8 for the video,  have to execute "canvas.toDataURL('image/webp', this.quality);" for each frame.  Now wondering out loud, whether if just could make a video only webM like whammy currently does, then in the second pass add that as another source when mixing the audio sounds?  Certainly would be less messy.  Could also output as "video/webm;codecs=vp9,opus". vp9 might have interframe compression, which I know vp8 does not have.

[davrous]

Hi,

I'm sorry but I'm not sure to understand what you're waiting from us. Can you please clarify what you'd like to achieve and what we are missing to help you doing it?

Thanks,

David

[JCPalmer]

@davrous, actually nothing on the d.ts front, unless you can clarify why lib.d.ts, mozzilla, and W3C are out of sync with each other.  The project is a week in, and the video track rewrite is getting close to completion, with improvements beyond being Typescript & OO.

I am beginning to think all integration to BJS should be outside of this actual code base.  Args passed like BABYLON.Scene & BABYLON.Sound would be replaced with HTMLCanvasElement & and an interface of { getAudioBuffer() & play() }.  Would make it more useful to contexts outside of BJS, if I decide to publish the repo.  Have kind of put that decision off.  It has to work first.

My initial integration will be with the TimelineControl after render in my QI animation extension, and in the place all sound.play()'s are called.  Sound is tightly integrated in QI for sync requirements of speech.  For general BJS use, adding a simple after render for adding video frames would require nothing to change in the framework.

Sound.play() on the other hand might need something like this added:

if (typeOf(WEBM.AudioMixer) !== "undefined"  && WEBM.AudioMixer.recording) {
    WEBM.AudioMixer.addTack(this);
}

Otherwise, the recorder will never know the track was played, & exactly when it started.  I do not think the framework currently provides the audit trail needed to determine those 2 things.  I have not completely fleshed this out, but will if I decide to publish to the public, which would include a dblWhammy.d.ts if that were the case.

[JCPalmer]

Well, I guess I should mark this solved or something.  I cannot fix different organizations branching off from one another.

This project seems better & better each day, regardless.  Remember when I refactored 3D right into the BABYLON.Camera base class in 2015 (see topic "Making ALL cameras 3D").  I also added a couple of new rigs, one being stereoscopic side by side parallel.   Back then, I was actually trying to screen cast a live scene with a Android tablet to a Sony 3D TV.  This recording process can record that as well without changing a thing!  @Pixar "look out, I a comin fer yer daughter".

I am not yet operational for the video, but I took a sample WEBM from the Whammy repo & put it on a thumb drive (a clock).  I plugged the drive into the TV & it almost worked.  When I selected the file, it said it was a "unknown format",  but put up a square video with 4 seconds.  There was garbage in the square, but that means it knew what a WEBM video was.  It just did not know the vp8 codec.  I would bet a beer it knows vp9 though (there are only 2 in WEBM).

Full implications of this, I cannot calculate (well yet).  Going to have to see what YouTube actually wants.  Whether you can submit both 2D & 3D videos on the same channel, what resolutions work, & if 24 FPS is supported.  24 is the one for movies, right @dbawel ?

If I get the right answers, stereoscopic might get an update.  I am also looking at a number of low priced programs which will do format conversion / compression once the original is rendered.  I will be able to examine changes to the shader more easily, because I can just play one file then compare to another video of the same thing.  Much tougher when you have to have multiple directories of the scene with different versions of the framework.  I just ran out of time before.

FYI, while most may think BJS is for games not video, I am pursing a sort of "Pirates of the Caribbean"  business model.  The channel supports the game, & the game supports the channel.  That is also why I spent so much time on speech.

[dbawel]

@JCPalmer

It is such a subjective task, that perhaps you're right in putting it to rest for now.

As for film, we've moved beyond 24fps and practically all films are filming and post processing at 60fps. Not only that, but most films are currently shooting at 5K or above, and several are shooting at 10K - so it's a fast evolving world, and I doubt WebGL will be far behind with processing power on our devices growing exponentially.

Thanks for all of your effort.

Cheers,

DB

[JCPalmer]

Well, YouTube says it does WebM, but does not break out codecs. I guess I will put out a sample & see what happens.  If they support that many input types, I am guessing they run in through some type of conversion so that te site always shows some standard format.

From my view, a real time render / game engine can produce fixed rate output for video, but let's see how well what the movie industry uses works as a game engine.  Much work on my part, but I get a 2fer.  With QI's timelineControl, I assign what time BJS thinks it is, so I can do any speed.  Here is a test page where you can switch to fixed frame mode.   Ignore the mp4 encoding part.  This is a replacement for that.  It works whether you say 24 fps or 120 fps.  I am not going to exceed 60 fps though.

Sounds like the film industry is recording at such high rates, because it is so expensive to do production, that they do not want to miss anything.  In post, they can the best frames if forced to.  What is the output rate though?

[Magilla]

@JCPalmer YouTube does in fact do WebM. Is seems to be the only format that they provide 1080p+ videos in anymore. I've only noticed because when you use Flash Video downloaders on YT, the only 1080+ options are .webm, and download without sound. I don't know what codecs they support, but I suppose you could download a couple of webm files from YouTube to see what codecs they use for streaming back to the public.

[JCPalmer]

Thanks, @Magilla.  I needed to get an ebml viewer (ebml is what webM is) anyway, to see samples of how an video & audio track are organized together, and debug my output. I will try right clicking a YouTube vid, and hopefully get a menu option to save to file.

Here is a Java based viewer.  Assuming you have a Java runtime on your system just:

1. download ebml-viewer-2.0-bin.zip.
2. Extract to somewhere.
3. Double click ebml-viewr-2.0.jar
4. The first launch you will only get a toolbar.  It is trying to remember your window size from the last launch, but there isn't one.
5. Drag to window size you want.

Opening the clock.webm from above, I see it is V_VP8 ( highlighted row )

Think I am still going to have to submit samples to YouTube with different frame rates though.  webM does not really even store an FPS in the file.  Every frame has its own duration.  You can actually change the duration on a frame by frame basis.  That is one of the improvements on Whammy that I have done.  Though I will not be using this, DoubleWhammy can record using the standard render loop in real time.

Also need to see if YouTube is doing some processing to frame rate.  If you give them a 24 fps, & they modify to 60, then better to give them a 60.

btw, @dbawel, I am now pretty sure those 5k rates are for special effects.  They can take an explosion that looks like a glorified firecracker, record it at a really fast rate.  When they slow it done it makes it look really BIG & long duration.  That's why I asked if you know the OUTPUT rate of movies.  Thought 24 fps makes things look bigger, and 60 made things look like cheap video.

[JCPalmer]

Well, finally went to YouTube.  You cannot actually download AFIK, not surprised.  I'll find a WebM somewhere with audio to get a visual on the layout somewhere else.  It can display stats from a right-click though.  This is from an Incredibles trailer.

Output is vp9 / opus.  Wondering what happens if you give them vp8?  There is one at least one way of finding out, I guess.  I see 854x480@24 for the resolution.  Is that HD+?  Think the @24 is for 24 FPS.  Anyway, if 24 is good enough for the Incredibles, it is good enough for me.

[Magilla]

4 hours ago, JCPalmer said:

Well, finally went to YouTube.  You cannot actually download AFIK, not surprised.

YouTube themselves don't provide the ability to download. You need a 3rd party to help. Many people use browser extensions, although there are also 3rd-party websites that will provide download links for you as well.

[dbawel]

@JCPalmer

Indeed Webm is the most universal media format for all devices. As for the film industry, every movie you see in the theater is played at 60fps these days.

DB

[dbawel]

@JCPalmer

I love to get responses by you. I'm a big fan.

DB

[JCPalmer]

Well, I have gotten further.  My re-write of whammy video part is working.  I am also capturing the sounds played() during recording, and have basically merged them directly from the buffers into a combined buffer.  I tested the combined buffer, by writing it out as a .WAV, and all the sounds merged good.  Working on the merge of audio into the output.

During the many, many searches I have been doing.  I saw that there is an experimental feature of HTMLCanvas.captureStream().  It only works on Chrome & Firefox, and does have some problems:

• Both do not output valid EBML headers
• The chrome output does not run on anything, but chrome.

This is too new & untested for me, but someone else might take a look.  I made https://www.babylonjs-playground.com/#A7UIKS#1.  It works.  I tried using createMediaStreamDestination(),  to add the audio tracks in, but I get an error.  Un-comment lines 71 & 72 to see the problem.  Can anyone get this to work?