Multiplayer Test with NullEngine()

[jodo]

Hi Guys! I haven't been active here for quite some time and haven't done anything with Babylon.JS for quite a long time. But after checking out the changelogs, I saw that we can now run Babylon.js Server Side, how awesome is that!

So I had to squeeze in some time and implement a proof of concept multiplayer simulation with Client and Server side physics engine. It's quite basic. The Client can control a ball by spinning it forward or backward (with W and S). By changing the camera angle (with A and D) you can change the direction of the impulse. With Space you can jump around. To check out how it behaves with multiple players you can either ask someone to also visit the site at the same time or just open a new tab in your browser. 

Technical it is rather simple. Server and Client communicate via Websockets. The client applies impulses to it's ball, these parameters for these impulses are sent to the server. The server applies these also and keeps the state for the whole world up to date. Each render loop the server sends the current state to all the clients (ideally 60 Hz). The clients then correct the position, direction and velocity of all objects including their own ball if needed. I haven't tried it out with higher delays, but I would suspect the result will be quite "jumpy". Interpolation for correction and prediction of movement is not (yet) implemented. 
Added Server Update Rate and Ping to see lags and delay in perspective to these metrics. 

Here is the code: https://github.com/j-o-d-o/multiplayer-babylon-js-game

Here is the Demo: http://185.82.21.82:8700/

 

Here is a great article about Server-Client Game Networking techniques: http://www.gabrielgambetta.com/client-server-game-architecture.html which was somewhat the motivation to implement this proof of concept.

[GameMonetize]

Woot!!! This is kinda cool!

 

[Threedy]

Well done! I tried this a couple years ago, but I ran cannon.js separately on the server (at that time I don't think there was any Babylon.js implementation on the server). I see that you are using Oimo.js which should perform a bit better than Cannon (with the cost of less features). At the time, once I had >6 clients joining, the game becomes very laggy (even including interpolation). I see that the same also happens in your demo. Do you have any plans to tackle this issue? I couldn't figure it out and thought that it was simply the physics engine .

Looking forward for more updates!

[jodo]

@Threedy Yeah, the server side NullEngine() just came out with v3.1 about a month ago (according to the 'what's new.md' file), so you had a much much harder task 6 month ago ; )

I haven't got time yet to go into detailed performance debugging. But I would say it is a network problem and not a physics engine / simulation issue. 10 objects are definitely no problem at all for the physics engine, even on bad hardware. There is almost no CPU used on the Server, but pretty much all of Memory is allocated. (Also, other small test applications are running on it). And it is a really small server (just 1GB of RAM). I am not a network specialist, but I assume decreasing the server update rate could help as well as decreasing the size of each update message. 

When 10 users are connected, all RAM is used and there is about 800kb/s outgoing traffic. You can see that the ping stays pretty much the same for the first few connections and at some point, just skyrockets to 2000 ms and higher. I assume TCP packages start getting lost while 10*60 new packages are Queuing up per second which quickly stacks up to such a high ping. (Someone please correct me in case I am talking garbage here) ; ). 

I also assume (yes, lots of assuming here) that using UDP would decrease the network problem, but as we know, it isn't possible with browsers. 

[Gijs]

Ow, for a minute I thought I overloaded the server

But thanks for sharing this, it's interesting and helpful.

[Threedy]

5 hours ago, jodo said:

@Threedy Yeah, the server side NullEngine() just came out with v3.1 about a month ago (according to the 'what's new.md' file), so you had a much much harder task 6 month ago ; )

I haven't got time yet to go into detailed performance debugging. But I would say it is a network problem and not a physics engine / simulation issue. 10 objects are definitely no problem at all for the physics engine, even on bad hardware. There is almost no CPU used on the Server, but pretty much all of Memory is allocated. (Also, other small test applications are running on it). And it is a really small server (just 1GB of RAM). I am not a network specialist, but I assume decreasing the server update rate could help as well as decreasing the size of each update message. 

When 10 users are connected, all RAM is used and there is about 800kb/s outgoing traffic. You can see that the ping stays pretty much the same for the first few connections and at some point, just skyrockets to 2000 ms and higher. I assume TCP packages start getting lost while 10*60 new packages are Queuing up per second which quickly stacks up to such a high ping. (Someone please correct me in case I am talking garbage here) ; ). 

I also assume (yes, lots of assuming here) that using UDP would decrease the network problem, but as we know, it isn't possible with browsers. 

Hmm you're right. I most likely have messed up something there.

Anyway, looking forward to your project and hopefully more updates!

[timetocode]

Hey sorry to post 6  months after you wrote this, but I'm a network programmer (mostly) who has recently discovered BJS + NullEngine and I was wondering how to get the physics in NullEngine working. 

My code is very similar to yours, I'm not sure what I'm doing wrong. I made a thread about it. I've also posted a buncha optimization tricks below if anyone is interested.

 

I'm the author of nengi.js which is a networking layer / engine. Performance has been my focus, and depending drastically on the game and how its programmed (and the hardware), it has been possible to get 50-300 concurrent players, and 100s or 1000s of other entities in a single instance. My only shipped products are in 2D so far. I don't have much experience in 3D, but representing a change in xyz and rotation xyz is gonna be a bit heavier than the simpler data in 2D. I would still guesstimate that 20-150 CCU are possible depending on many variables, and that the game client can run at 60-144 fps while the server runs at 20 fps.

The first and largest is to bundle the game state for any given frame from the server into a single snapshot. So rather than sending one client 20 messages about the 20 objects around it, we can create one message with 20 objects in it. Even while remaining in JSON  this can push games from ~5-15 CCU to 25-40 CCU. The reason this is so massive is that socket.send is expensive to call (even with very little data in it). This also makes that whole clientside prediction stuff easier later on.

Reducing server tick rates is good for performance, and not as bad for the game as it may seem. Most high performance first person shooters even in 2018 use a server tick rate of 20. I've seen one that uses 33. This will require entity interpolation on the clientside to keep the game running at requestAnimationFrame rates. The client can save the snapshots it receives from the server each frame, and when it has 2 or more of them, it can begin moving the clientside representation of the objects to positions that are lerped between the two snapshots. This allows us to have 60 or 144 fps (or whatever) movement on the client.

The id, x, y, z, rotX, rotY, and rotZ are probably 99% of the data sent for a 3d multiplayer game. So these are candidates for optimization. The id is the single most sent property. In binary an id can be a UInt16 usually. In JSON/text anything short will do, just avoid UUIDs. The position/rotation in binary will be great as Float32s. In JSON perhaps just trimming a few decimal places is as good as it gets (but still pretty big!).

Sending input (commands, keystrokes, etc) from client to server is rarely a choke point, so not much optimization needs to occur here. Despite how counterinuitive it may sound, it is possible to move a player at 60 fps or 144 fps even if the server tick rate is 20. The trick to doing this is to collect the client's inputs across a frame (in whatever format you desire, e.g. w: true, a: false, s: false, d: true, facing:xyz rotations) and then to stamp the clientside deltaTime for that frame to the data. Upon receiving the data on the server, the server can then deterministically move the player.  For example if W was true, the server can set the entity to face whaterver direction the client said it was facing, calculate a normalized forward vector, and then move by speed * client.command.deltaTime. This is a deterministic calculation thanks to having deltaTime stamped on. The client itself (if performing client-side prediction) can process this identical command locally, thus moving by the same amount. Even if the server tickrate is 20, and the client tickrate is 60, all this means is that the server each time it ticks will have on average 3 pending frames of data per client which it processes in order.

Preventing the client from cheating by sending duplicate commands or changing the deltaTime to be a large number (to move faster or teleport) can be accomplished by summing the deltaTimes that come through each frame. The sum of the deltaTimes should always be less than (but close to) the total time since the client has connected (cause that's what deltaTime truly is) otherwise it means a client has sent the server more than 5 minutes of input in 5 minutes of time (or whatever). This either means space and time are coming down around us, or someone is cheating.