Small Freezes (GC?) on strong computer

[royibernthal]

The scene:

I have a scene with 21 textured very low poly (300-400 poly count) models with skeletons and animations.

Out of which I null the skeleton of 20 of the models, in order for bjs to avoid calculating their bones. (done to increase performance)

Whenever I'm playing the animation of any of the 20 models (happens seldom), I'm temporarily setting their skeleton back for the period of the animation.

The remaining model plays animation in a loop.

 

I have a pretty strong computer (runs latest hardcore 3d games).

 

The problem:

The game mostly runs very smoothly, but in the beginning, I suspect before each model was visible on the screen and/or each animation was played for the first time, or something of the sort, there are a few freezes.

A freeze is mostly 0.1-0.3 seconds but can even get to 3-5 seconds.

After a few of these freezes occur (typically 2-3 freezes after game initialization), the game keeps playing smoothly with no more freezes.

Throughout the game different models are displayed, but always in the format I displayed above (20 mostly with no skeleton, seldom play animations, 1 plays animation in loop).

I'm suspecting the reason behind these freezes is garbage collection - which for some reason happens only a few times and then stops.

Could bjs be doing lazy initialization of certain things only when they need to be used and after each initialization which happens on the run there's enough garbage to trigger a garbage collection? e.g. bjs initializing certain things only before playing an animation for the first time. Such a thing would explain the freezes.

Hopefully I gave you enough info to solve this without a PG / profiling.

 

[RaananW]

If it only happens when loading meshes, it is due to the buffer initialization, which is, very very sadly, synchronous.

you can notice such pauses when loading the scenes on babylonjs.com as well - see that the loading symbol stops turning and freezes for a very short period of times. 

Long pauses are caused due to large number of vertices (depends on the amount of calculations needed. For example when loading an .obj file, babylon needs to convert OBJ Vertex format to Babylon vertex format. This is also synchronous and depends on the amount of data).

This is just an assumption thou! If you have a way to actually show your scene, it will be easier to tell you if that's the problem.

[royibernthal]

Sharing the scene would be problematic for me as it'd reveal confidential information, so as much as I would like to, I can't at the moment.

I have a preloader that loads the meshes before I'm showing anything, after everything is loaded and the game starts - there are freezes.

btw, are you by any chance originally from Israel?

[RaananW]

With a name like this? My parents either didn't love me at all, OR they are somehow Israeli related . So, yes, originally.

About the scene - Will be hard to check the freezes without seeing it. If you are sure the meshes are preloaded, and sure that the meshes buffers (and WebGL buffers) are created pre-render, then something else works synchronous and blocking your scene for a while. Are you making any heavy calculations / array traversals / object initialization / Hummus making / etc that might create a (relative) heavy load?

[Nabroski]

You could enable OpenGL see if you get a better results. This is not a end client solution, but then you have a direction to work with. TextureBufferSize.  And when not, you have to do some scene optimizations. 
Gook Luck.

[royibernthal]

Yeah your name gave you up

I'm starting the game once the AssetsManager finishes loading all assets, including meshes. Is it possible that the mesh buffers and WebGL buffers are not already created at that point?

I'm not making any heavy calculations / hummus. Once the scene is created I'm not initializing anything else, unless my actions cause initializations under the hood which I don't know about. As for array traversals - going over an array of 20 meshes every frame, which is negligible.

In my scene I'm changing every frame the position and scaling of the meshes, and swapping meshes with other (already loaded) meshes. That's about all I'm doing, could it be an issue?

[RaananW]

Nope, shouldn't be an issue. 

Again, these are only suggestions as it's hard to debug without actually debugging:

There is a way to initialize (and load) meshes when they are first visible. Might be something you sry without knowing? Can you check that all meshes are ready prior to the first frame render?

[GameMonetize]

The lags that you are seeing are probably caused by the shaders.

Here is how things work internally:

- The very first time a mesh uses a new material, the shader is compiled. This is (not my fault :)) a synchronous task

- It will freeze everything on your screen for a few milliseconds

- Next renders will be faster as compilation is kept in memory

 

My advice: do not start rendering until all material.isReady(mesh) are not true.

[BitOfGold]

Just one idea: when I'm watching video on my other monitors with a good encoding (like x265 HEVC), all webgl FPS suffers periodic lags in every 1-2s.
Sometimes even a facebook video in another browser causes lags.
Just one thing to clear out as NOT beeing the problem...

[fenomas]

No need to theorize about this stuff; get to know your browser's profiling features. Modern browsers can tell you exactly how long they spend on GCs, rendering, specific JS functions, etc.

In Chrome, you want to open devtools, go to either the Timeline or Profile tab, and hit the circle in the upper left to start/stop recording a profile.

 

[royibernthal]

There's one freeze that always happens - right before I set the skeletons of 5 of the meshes and play animations in all of them in parallel, for the first time. (profile below in reply to fenomas)

 

Raanan - Yeah I understand these are only suggestions based on the info available to you, if I't'll turn out impossible to solve it like that I might be able to share the source code in private, if that works for you.

Would that be possible given I'm explicitly preloading all meshes using AssetsManager before rendering them?

By checking if all meshes are ready do you mean what Deltakosh suggested? If so see my reply to him below.

 

Deltakosh - I checked mesh.material.isReady(mesh) for all meshes and they're all true before I begin testing.

 

BitOfGold - I have no video open on my computer while testing.

 

fenomas - I have zero experience with the browser's profiling features, but let's see if I got this right:

I think that the problem is with requestAnimationFrame which at the freeze point took 506.86ms and dropped the fps to 1.89 from an average of 36.52 (out of a max of 60).

There are also Minor GCs all over the place, the max for a single GC I think is 25.11ms.

Just in case, I uploaded the recorded profile for you to take a look as well.

profile.json

The freeze point in this profile is the one I mentioned in the first line of this reply.

[GameMonetize]

Beware, you must do the test AFTER attaching the skeletons to the meshes as this will trigger a new compilation (The shader has to add support for bones hence a recompilation)

[royibernthal]

Now for the sake of testing I'm no longer setting/unsetting the skeletons, the skeletons are always set.

Would setting them, waiting for shader compilation (material.isReady?), and unsetting them be the right way to initialize everything before rendering? In order to later be able to freely set/unset them.

Would unsetting a skeleton once the shader has already added support for bones even improve the performance or is doing such a thing completely redundant?

 

That same freeze no longer happens when skeletons are always set, however when I play the game a little longer another freeze happens before an animation that occurs a little later on.

The following come one after the other in the new freeze point:

• requestAnimationFrame - 526.66ms
• Non-incremental GC - 305.31ms (Too many bytes allocated)
• Minor GC - 28.32ms (Too many bytes allocated)
• Non-incremental GC - 210.21ms (Dead global revived)

Uploaded again the recorded profile:

https://www.dropbox.com/s/cro3ve00k00dsa7/profile.json?dl=0

 

In another profiling a shorter freeze occurred, completely unrelated to animation:

• Incremental GC - 212.61ms

Recorded profile for that one:

profile.json

[GameMonetize]

Quote

Would setting them, waiting for shader compilation (material.isReady?), and unsetting them be the right way to initialize everything before rendering? In order to later be able to freely set/unset them.

Yes

Quote

Would unsetting a skeleton once the shader has already added support for bones even improve the performance or is doing such a thing completely redundant?

Yes as a mesgh with skeleton is slower (because of skeleton computation)

[royibernthal]

I'm trying to grasp what's going on.

Immediately after AssetsManager finishes loading meshes, skeletons are automatically set on the loaded meshes. mesh.isReady() and mesh.material.isReady(mesh) both return true. (allegedly reporting that shaders with support for bones have been compiled?)

Following that I null the skeletons, in order to avoid skeleton computation when not needed.

Then when I set them back before animating the models the freeze/lag occurs as if the shaders support for bones is just now being compiled, and previously the mesh.isReady() and mesh.material.isReady(mesh) falsely reported true.

Does it make sense?

 

Also, were you able to deduce anything from the profiles I described in my previous reply? The problem seems to be more complicated than compiling shaders with support for bones, or alternatively, there is more than one problem causing lags.

[GameMonetize]

Quote

Following that I null the skeletons, in order to avoid skeleton computation when not needed.

You then need to wait for the material to be ready for this configuration as well

[royibernthal]

I'm now waiting for material to be ready for both configurations - first waiting for with bones, then for without bones.

There are no longer freezes before playing animations for the first time, but there are freezes immediately when beginning to render meshes without bones.

My guess is my check for material ready is valid for with bones, but not for without bones, and here's why:

At first, by default, meshes are loaded with bones, I simply avoid nulling their skeletons until the material is ready (material starts as not ready when AssetsManager finishes loading the meshes).

Once the material is ready for that configuration, I'm nulling the skeletons, if I check for material ready immediately after nulling the skeletons it'll still return true, as if for the previous configuration (with bones). It still returns true after a tiny setTimeout.

When starting to render, apparently the configuration without bones still hasn't been compiled, and results in small freezes. (or at least that's my interpretation of the freezes)

Am I correct to assume that the configuration without bones hasn't been compiled yet even though checking for material ready after nulling the skeletons returns true? If so, is there a way for me to make sure that bjs is already tracking the compilation of the new configuration without the bones so that I know when checking for material ready will give me a relevant result?

 

EDIT: I started the rendering the meshes with skeletons by mistake, which is probably the reason there was enough time to compile the shaders with bones support, and not lag before starting an animation.

However now when I start rendering the meshes on the screen without skeletons (after checking of course for material ready for both configurations), there are freezes again.

All this leads me to believe that when meshes stay in a certain configuration enough time (e.g. with bones) it'll be compiled and there won't be future freezes, but that I can't really trust material.isReady().

It's highly likely though that I fail to understand something or I keep missing something - in which case it'd be great if you could help me understand what's going on.

[GameMonetize]

To dig into this more deeply can you provide a simple repro on the PG?

We could then analyze what is going on

[royibernthal]

Would sending a link to the source code in private work for you?

I can't make it public because it contains confidential information, and I doubt I can effectively create a simplified repro in a PG without revealing that information.

[GameMonetize]

Not really. I'm not always available and having a public PG is really useful for the community to help.

[GameMonetize]

Something like that:

http://www.babylonjs-playground.com/#11BH6Z#189

[royibernthal]

Yes I understand, what I see in the PG you linked to is more or less what I'm doing but there's more going on.

I don't know how to simplify the repro while still maintaining the issue and without revealing my confidential information, otherwise I would have already made a PG.

I can ZIP the source, once you get a few minutes to answer you can simply unzip and run it immediately, not much different from running a PG online, takes an extra 1-2 minutes to download and that's it.

Sorry for insisting, I really can't find a way around sharing it in private, hopefully you'll find the request reasonable.

[GameMonetize]

It is reasonable  send it to [email protected] (But please make the sample as simple as possible, I don't want to spend ages debugging through no relevant code)

[royibernthal]

Thanks for understanding  Sent