BJS 3.0 is packing on the pounds!

[JCPalmer]

I know I have been a little obsessive about the size of geometry files.  In fact, the latest version of Tower of Babel exporter in dev is taking it to the next level.  A hint for one of the tricks is 0's are FREE for typed arrays (think about matrix weights / indices).

BJS 3.0's waist line has just top the 1300 kb.  It is starting to get my attention as being a pig.  Mind you the recent breaking out of stuff into other files is good.  All the stuff in the main file is also good.  Unfortunately, no one ever really needs more than 2 cameras, is not using all 4 types of lights, shadows, using actions / animations, post processes, physics, all the different types of textures & materials, and on and on in the same scene.

WebGL being locked out social media, is sure to remain too obscure for CDN to really be of any use.  Almost worthless for just BJS which has 1 CDN with and 1 without https (not to mention a new version every 6 mos).

The build your own Babylon is more of a file combiner for the kiddy end of the pool.  Not a "static linker" type utility.

- - - - - - - - - - - - - - - -

Here is my idea.  Use one of the Browser's writing of profiling files to know what a given scene is actually using.  Firefox is writing a JSON file.  Worst case is the file can be regex mined to find all the objects with the name of BABYLON.xxxx that got instanced.

If that file could used as input, to generate a boolean for each line in files section of the config.json file.  Output a modified config.json file to gulp where only the files with a hit get included.  There would be issues with objects in files like Math.ts.  There might be some mandatory files, though I think math.ts is too big and should be broken out.

I am not ready to do this, but thought I would bring it up before I was going to do it.  If anyone wants to work on this, reply to thread.  If I end up doing this myself, I'll probably cut many corners, so I would be unlikely to publish it.

[BitOfGold]

That's a good idea, I don't know it can be achieved.
I compile a custom babylon, (with Tools/custom.config.js), but not easy to separate used functions by file.

For example, I don't use any of babylon's physics or collision functions - I do it with my own entity/component framework, with a collision and a rigidbody component using cannon.js.

but if I remove "../../src/Collisions/babylon.collisionCoordinator.js", I get an error.
Update, now I checked and it works  

And it also differs by project. I don't use VR function (yet), but what if I would like to use it?
Than i would need two custom babylon versions... Ot jut that few VR-related files separately minified and included.

And my stripped down babylon.custom.js is 1 170 702 in size, so I reduced onyl 13%.

About CDN: There are free or cheap CDN-s like cloudflare, really helps with load times around the world, I think any CDN is better than no CDN.

[BitOfGold]

The good news is, the gzipped version that gets downloaded from the server is just 249KB.
Size is not really a big problem I think.
If size is a problem there are other frameworks like TWGL:
https://github.com/greggman/twgl.js/

But, BABYLON is such a great library, it would take years to get anything done in a lesser library (for me), so who cares about a few kilobytes...
 

[JCPalmer]

I thought of reading the standard config.js and generating a custom.config.json.  Part of the reason for only the 13% is Engine, Scene, & Math are so big.  Are you trying to accommodate multiple scenes with your custom build?

The idea is to automate as much of the effort of making the config file as possible.  Co-opting this work flow makes this a much smaller effort.  The config file has the name of the output file, so each scene can have their own js file.

Maybe a html in the Gulp directory.  Have a text field for the performance data file with a file chooser button.  After picked, javascript in the html file then populates  a scrollable listboxes that contain file items to discard.  The ones in the keep were found in the performance data were removed.

The discard list might be contain check-able items. You check ones to keep anyway.

Also have a text field of the BJS file name with a generate button next to it.  Once generated, gulp & test. If you go too aggressive, un-check as required.  To keep what you picked also write out a 2nd copy of config file with same name as

[brianzinn]

ES6 modules do this automatically with tree shaking.  So, the javascript will only contain what you are using.  ie: whether you are using a material or not will decide how the JS is generated.  Webpack2 and Rollup do this automatically: https://github.com/rollup/rollup#tree-shaking

Projects like d3 (v4) have split into modules to allow this (https://github.com/d3/d3/blob/master/CHANGES.md).

[GameMonetize]

I'm interested to see how we can help here. Don't have time for it but definitely if something can be done to help on engine side I'll do it

[Hersir]

I alos vote for ES6 modules, to have better support for node / webpack .... Also plugins should be registered not just auto used from global scope (cannon for example).

[JCPalmer]

Thanks for the heads up about ES6.  I know nothing about it.  Specification of the target, currently ES5, is in the Gulp.js file not the config file. I changed to ES6 for the hell of it.  Got errors at compile, Gulp then crashed.  I you guys want to PR changing to ES6, think you have significant work ahead of you.

I also think that this will not work for people using .babylon files.  How will tree-shaker know Arcrotate cam is needed if it is JSON?  I generate JS files with inline geometry, but I am pretty alone.

I know much about data mining,  I think I can get something to work using the current plan.  I might need advice / help trying to refactor some code out of the very large Engine.ts or Scene.ts, if there might be a significant size reduction.  That would be after I get this self contained (including brief directions right in the page) html file operational.  It will not be a work of art.  I can tell you that.

[JCPalmer]

Ok, I have made some progress.  Looking at the profile of a simple scene, I am getting hits.  Looking at the console.logs that I currently have coming out:

including "../../src/Math/babylon.math.js" using search of 'math|mathtools|color3|color4|vector2|vector3|vector4|size|quaternion|matrix|plane|viewport|frustum|bezierCurve'
including "../../src/Tools/babylon.tools.tga.js" using search of 'tools'
including "../../src/Tools/babylon.tools.js" using search of 'tools'
including "../../src/babylon.engine.js" using search of 'engine'
including "../../src/Cameras/Inputs/babylon.freecamera.input.mouse.js" using search of 'freecamera'
including "../../src/Cameras/Inputs/babylon.freecamera.input.keyboard.js" using search of 'freecamera'
including "../../src/Cameras/Inputs/babylon.freecamera.input.touch.js" using search of 'freecamera'
including "../../src/Cameras/Inputs/babylon.freecamera.input.deviceorientation.js" using search of 'freecamera'
including "../../src/Cameras/Inputs/babylon.freecamera.input.gamepad.js" using search of 'freecamera'
including "../../src/Cameras/babylon.freeCamera.js" using search of 'freeCamera'
including "../../src/babylon.scene.js" using search of 'scene'
including "../../src/Tools/babylon.tools.dds.js" using search of 'tools'
including "../../src/Cameras/Inputs/babylon.freecamera.input.virtualjoystick.js" using search of 'freecamera'
including "../../src/Tools/HDR/babylon.tools.cubemapToSphericalPolynomial.js" using search of 'tools'
including "../../src/Tools/HDR/babylon.tools.panoramaToCubemap.js" using search of 'tools'
including "../../src/Tools/HDR/babylon.tools.hdr.js" using search of 'tools'
including "../../src/Tools/HDR/babylon.tools.pmremgenerator.js" using search of 'tools'

I am getting good hits for the Freecamera inputs from the 'freecamera', made from new RegExp('babylon\.\\w+') on the list of filenames.  This is also sweeps up some files starting with the name 'tools'.  @Deltakosh, expect the names of tga, dds, cubemapToSphericalPolynomial, panoramaToCubemap, hdr, & pmremgenerator to have 'tools' stripped from their file name.

Am not generating a config file to the downloads directory yet.  That is not as important as getting the mining correct first.  I notice I am not getting hits for StandardMaterial or Texture.  Investigating.  I already have the feature to add on extra search criteria.  The math test above was added with:

appendSecondarySearches("math", "mathtools|color3|color4|vector2|vector3|vector4|size|quaternion|matrix|plane|viewport|frustum|bezierCurve");

Math has many more classes.  Think math should be broken out to at least 'Math.core.ts' and 'supplementalMath.ts'.  Math is very big.  There is tons of stuff rarely used, but some stuff is mandatory.

[JCPalmer]

Ok, Got a much better search.  Actually it is an additional search.  It finds all uses of prototype functions.  It is finding many internal things like depthCullingState and other things.  It takes 31 files to build a .js.  Still need to verify it can find static calls.

[GameMonetize]

This is really promising and we could perhaps be able to wire it in the gulp process. Something like gulp build /use:myscene.babylon

[JCPalmer]

I am going towards the .html file implementation for now.  It will be very bare bones, and only work on FireFox with FireFox performance files.  The most important thing is searches that get an accurate list of files with no extras.  I will PR at that point.

2nd priority should probably break out Math.ts, which can be done with zero compatibility issues.  See where we are at that point.  I am testing with one of the simplest of my scenes.  My goal is 300kb.  After zip, would be under the 64kb limit for a single packet.  Using a CDN with that would be fast, including the reduced parse time due to a smaller file.  If a class cannot be in multiple .ts files, some damage to user direct calls to Engine might result, if some stuff was pulled out and disbursed to dependent files.

Changing to a .babylon as input could have problems. First, I do not use a .babylon, so that would be useless for me.  Also, getting info from a running scene allows for mixing .babylon with user code, & extension code for input.  Am sure people say "blah, blah, blah, Why can I not use a .babylon as input?".  Life is hard, and then you die.

[GameMonetize]

I'm not opposed to have math.vector2.ts. math.vector3.ts and so on

 

But I'm pretty sure you want gain a lot as there are all required to almost all scenes

[JCPalmer]

On 4/7/2017 at 8:15 PM, Deltakosh said:

I'm not opposed to have math.vector2.ts. math.vector3.ts and so on

But I'm pretty sure you want gain a lot as there are all required to almost all scenes

I think you meant babylon.vector2.ts right?  They would already be in the math directory.  My visual read of the 5000 line file is that about 1000 lines might not always be called. 

I got a final running scene with a 688kb file.  Having to drag in 70 kb that was not detected.  Though part of the was due to sub-classing Bone, skeleton, and Action in the QI extension, there were other areas that could be minorly adjusted.  The simplest is not having to drag in all of the lights. MaterialHelper.PrepareDefinesForLights  uses an instanceof check to force the inclusion.  The base class Light.getClassName() could be used to avoid this.  There are also other things getting dragged in for little reason, because this area had never really been explored.

I am getting into an area that no one has been before.  It was very difficult before to use this tool.  Now that I can see things, pounding very slightly in many areas over time will start to add up.

Also, you have asked more than once "Can this can be integrated into the build process?  My answer was and still probably not. There is always the possibility that there will be things that are needed, but do not get recorded, like sub-classes that are never called. Starting to think that you are asking the wrong question.  Maybe, can a custom build process be integrated into this?

This process is not great and might be improved.  In my limited time remaining, probably 2 days before I have to move on (starting Wed), will look at trying to touch off Gulp from firefox.  The other browsers stand little chance of reading files.  FYI, this is what it looks like today:

[GameMonetize]

This is cool. I'll dig more into it when I'll have more time

[JCPalmer]

Ok,  I removed some mandatory files.  Will not have time to investigate if Gulp can be called by the html file.  I have improved the process/ directions though. 

• Adding a temp alert before Engine is instanced is cleaner than reloading the page after starting recording.
• To avoid copying files: change Firefox download option & specify a directory for build in the additional text field

[GameMonetize]

What is the size of the output now?

[JCPalmer]

The final size was 688.  It did not budge from cleaning up Lights, loadscreen, and one other.  Reason is in the 2 days between 18kb of uglified space got added to the required files.  This thing is growing by the day.

The clean up had as much to do about trying to get things under control with as few special rules added as possible.  Probably need to do some final testing as BJS 3.0 goes into bug fixes only mode.  One other weakness of the custom build is it occurs against pre-production code.  Want to get good rules for the final print.  Then one could use the 3.0 tagged zip file to build against production 3.0 while 3.1 goes into alpha.

[GameMonetize]

makes sense

[jeremybyington]

Custom builds are potentially very handy for production releases if during the development the full version can create a profile of all components being used -- not just from any .babylon files but from the actual JS. Maybe a stack trace and a source map could build a profile...but in the end I think letting the human brain decide with a series of checkboxes and a submit button would be most useful. A CDN makes it more likely that someone's browser already has it cached...and also, many of the websites I build have images that are larger than the 250kb gzipped size so I don't even see it as a big deal, but thanks for the option!

[KramSurfer]

Very cool concept.  I like the above idea alot, where in development it creates a profile, then the release has just the needed items. 

Else even a little more break out of the bigger items and specific classes that are special use.   VR, mobile device, or maybe some of the animation systems ( I know I'm not using in my current project ) would be helpful and would help lighten it. 

And remember to compare this to the Unity3D html5 export output to realize how awesome this is... :-)

[GameMonetize]

I recently introduced worloads to thw build process to help:

http://doc.babylonjs.com/generals/how_to_start#custom-builds

[JCPalmer]

Yes, that has in the near term put this process out of commission.  I will not attempt to adjust this until 3.0 enters beta.  Too much labor to do it now, and then test before release as well.

[GameMonetize]

We should enter beta early june

[JCPalmer]

We have entered beta, I am redo-ing this.  As there is only 1 config in the repo since this was first made, I was going to read the config file from Github rather than the local file.  This will also allow browsers other than Firefox for this.

I was thinking of putting up a select / dropdown of the repo branch.  This is merely a directory from a URL standpoint.  In the dropdown, was going to add:

• Preview
• Production

There is not currently a Production Branch in the repo, just version tags.  If you were really going to generate a stripped down runtime for your deployment would you really want it to be some Alpha code?  Granted, you will need to switch the branch on your local repo, but that is better than having to download a separate zip of the tag.  Do not think this will increase repo much, as that version of files should be there already.

So when there is a final print of a version, delete the Production branch (will not be one for 3.0), then create a new Production Branch.  Sound OK?