Cinematic recorder (Experimental)

[aWeirdo]

Hi all,
i've thought about making a simple scene recorder for a while now which could do light-weight cinematics out-of-the-box,

So today i had a bit of time and gave it a quick go.

The basic idea is that you can code your scene (move meshes, camera & play animations, visual effects, etc etc) to playout a cinematic, and use this for easy recording & download.
It could be anything from a ball rolling down a hill to a large scale battle recorded from a bird-view camera, all controlled by pre-made animations.

Use-cases include;
Large in-game cinematics that would otherwise cause FPS issues.
Promotional videos / cinematics. 

Currently, it uses whammy to convert webp frames into a webm https://github.com/antimatter15/whammy
 

Basic useage;

Quote

Small update; 
Uncomment line 133 to activate recording.
http://playground.babylonjs.com/#Z5RY9C#4

// Added _targetFps (simple implementation, only supports 30 & 60 fps so far.)

// Added _lengthInMs, preset the length/duration of the downloaded webm.(when set, & length is reached, endRecording is called automaticly.)

// Added renderLoop hijacking. (Sometimes bugs in the PG..)

// Slight quality issue with webp, might have to replace with other format in the future.

newCinematic.startRecording(targetFps<nullable>, lengthInMs<nullable>, downloadOnEnd<boolean><default: false>);

 

 

To-Do's;
- Add support for different frame-rates. (Partially done, 30 & 60 fps)
- Add sound support.
- Add renderLoop overwrite. (Partially done, missing re-activation of default renderloop)
- Include conversion.

[waverider]

really coool

[JackFalcon]

Cinematics: JSON configurations that triggers cinematics... 

Nvm....

[aWeirdo]

Hi @aFalcon 
webp is just a image format,
the recorder simply grabs an image of the rendered scene each frame and then converts it into a webm video file  

The goal at the moment is to provide an alternative to normal screen recorders, e.g. creating promotional cinematics, etc.
and hopefully, be able to cut off some "post-recording" editing time aswell.

But your idea is nice, full-fletched cinematic extention? to display in-game cinematics loaded via a json file i suppose?

[Dad72]

I like it a lot, it's a good idea and I plan to use it for one of my projects, Do you plan to put the code on GITHub?

Thank you for sharing.

[JackFalcon]

@aWeirdo, webp, really cool. 

Used Camtasia.

Thx.

[aWeirdo]

Small update; 
Uncomment line 133 to activate recording.
http://playground.babylonjs.com/#Z5RY9C#4

// Added _targetFps (simple implementation, only supports 30 & 60 fps so far.)

// Added _lengthInMs, preset the length/duration of the downloaded webm.

// Added renderLoop hijacking.

// Slight quality issue with webp, might have to replace with other format in the future.

newCinematic.startRecording(targetFps<nullable>, lengthInMs<nullable>, downloadOnEnd<boolean><default: false>);

@Dad72 no plans to putting it on Github atm, it's all available in the PG so far, WhammyVideo code is included at the bottom of the editor.

 

[JCPalmer]

This is interesting, thanks.  I have used a different antimatter15 repo (modified) to generate GIFs from my animation system, QI.   See a sample scene.  Remember to first size the window the size you wish the GIF to be, & then re-center the character.  You can practice without recording first.  It takes a while for the first frame to complete.

I think you may problems / quality issues with any BABYLON.Animation which are based on BABYLON.Now, like skeletons.  I implemented QI with an afterrender.  For the recording part, your renderloop replacement is equivalent, but mine defines time itself in the after render.  I can get very accurate recordings (either 25 or 50 fps for GIF).  You are hostage to using BABYLON.Now.  You might want to change BABYLON.Now so it will report time to support fixed rate rendering, similar to how QI.TimelineControl does.  Maybe for 3.2, @Deltakosh?

FYI, QI is very sound oriented.  I was originally going to try to convert a Java implementation of mp4, but settled on this.  I still need to write the frame # & millis since recording start to the console, of all sounds QI starts.  That way after using a conversion tool to mp4, I can add sound back in at the exact right spot, without trial & error.

[aWeirdo]

Hi @JCPalmer very interesting,

i was expecting problems with anything real-time based, having to overwrite with a per-frame solution,
as recording in real-time is quite tricky, grabbing a copy of the render each frame seems to be quite a heavy operation, so a render-hijack & slowly processing the scene seemed the best solution.

As for sound, i'm hoping i can simply copy the sound track & cache the start frame, then when compiling the video insert it at that frame directly, might meet some unexpected errors..

But anyways, i think i'm putting a lid on this for now, and return to it at a later date, i have some other things to do that takes priority.

[Dad72]

@aWeirdo HI, Why not use RecordRTC which supports video and audio. Whammy does not support audio to record a video. I would appreciate to be able to record music, audio effects of Babylon during the video recording.

How is what you would do with RecordRTC who seems really complete ? Do you think that it would be possible for you to use RecordRTC instead of Whammy? Thank you friend.

[JCPalmer]

How would you know what sound(s) are being played at the time any given video frame is being written in JAVASCRIPT?  Plus the frequency difference 60hz vs 8khz (bare minimum) is 2 very different operations.  Merging in post is much simpler.

[Dad72]

When the canvas is recorded, the sound that comes out of the speakers is also recorded, so at the same time an image is recorded. No ?

I do not understand what is meant by the 'Merging in post' ?

[JCPalmer]

That was a short for post- production, or after the video file was created.

I know how to record in Javascript using a microphone.  That is navigator.getUserMedia({audio: true}).  That is not going work for sounds being generated & played, as far as I can tell.

Figuring out a completely different format like RTC would take a lot of time.  I different starting repo would be required.  Perhaps try something like this. 

I do not see that integrating directly with the framework is absolutely required.  There are many solutions if you go the route of just recording the screen.

[Dad72]

Canvas2D-Recording does not work, it just records images like Whammy, but the sound miss it.

I would have to find something in JS to merge sound into the video, but I do not know if it exists.