Scaling game server on node.js

[LoudSilence]

Hi everyone!

I'm working on the fast paced multiplayer browser pvp game with following gamedev stack.

Angular Website -> global place for the game, responsible for: user account state, game UI, user info etc. (REST and WebSocket connections)
Client Game Engine -> based on ECS and PIXI rendering with custom gameloop, client-side- predition and all that stuff. (WebRTC Geckos.io connections.)
Nest.js -> backend for angular website, chat, matchmaking, creating services with Server Game Engines when players are ready to play. (REST api and WebSocket connections)
Authoritative Server Game Engine -> shared a lot with client, based on ECS etc (WebRTC Geckos.io connections.)
MongoDB -> DB for everything.

Any ideas / guides on how to scale things up? 

I read about load balancers using reverse-proxy on nginx and pm2, node.js task partitioning and offloading also microservices with redis.

But still dont know if i'm understaning this correctly. 
For example:
2 players want to play with each other -> new instance of ecs world will be created with all physics etc, 
this instance has a server game loop with examplary 50 ticks per second, taking from 10% to 100% cpu usage depending on how many entities are currently in game world.
If i understand correctly when server game loop is running its blocking the node.js event loop so no other task can be done.
So if another 2 players would want to play i would need to create new game server instance in the best case scenario on the second core in the worst on the new physical server.
To make this work I would need to have a load balancer and some kind of management over creating and removing new server game engine instances.
Maybe game engine should be microservice and Nest.js backend would be the main service which would delegate users to game engine instances?

It's very odd looking to me that on 8 core CPU only 16 players could play in the same time.
Is there any diffrent way i should go with this to enable more CCU playing? Maybe serverless?
Any advice on how to scale server side gameloops which are resource demanding on node would be appreciated :).

Sorry for my english and overall knowledge (new in networking and backend world!)
 

Edited April 11, 2021 by LoudSilence

[b10b]

@LoudSilence hey, these are interesting (and specialist) questions - you may do best finding someone who is already hosting similar and asking them directly?

For what it's worth these are some things I learned when building similar for a client:

• Yes, Nginx as a proxy for NodeJS is a win-win
• Using PM2 to manage multiple NodeJS apps is also a win - and check out the multiple instances mode if going multi-core
• Weigh up multi-core vs multi-server - within a virtual environment I'd always favor the latter (8 lite servers rather than a single 8-core server gives more redundancy for the same performance and price)
• Load balancer SSL termination is likely going to be beneficial, sticky session management also (yes Nginx can be used in this role, or your host platform may offer a front end to similar "product")
• Consider not "ticking" the server game loop, investigate a functional reactive approach that can absorb any delta (time and inputs)
• Each "room" need not have a dedicated thread (as long as it's not choking the CPU, which it must never do)
• Consider some auto-scaling provisioning environments to manage overall traffic and spin up new servers ahead of needed (containers, docker etc will help for this) - although not on day one!
• This is opinionated - but I rarely see the need for (remote) Redis if there's another (remote) DB already in the mix - pick one.  For local memory a good NodeJS data structure is just fine imo.
• Don't code with JS  

Mastery of all represents years, so pick off the low hanging fruit.  Don't over engineer a solution upfront - get something demonstrable and fix problems "just in time" - scale with audience: alpha, closed beta, beta, release etc.

Alternatively, do none of the above.  Find a SAS that specializes in similar and adopt their stack and payment plans, then alter your game concept to fit.  For example https://www.colyseus.io/arena ?  I'm sure there are others?

[LoudSilence]

@b10bThanks a lot for your response!

I must admit i asked many broad topic questions, also i did some research and started learning docker and kubernetes. As you said it will be more useful later than now but still I liked the concepts

Back to your response, the only thing i don't quite understand is to "Consider not "ticking" the server game loop, investigate a functional reactive approach that can absorb any delta (time and inputs)"

What I understand from it, Your advice is to have multiple game instances on one e.g. "World service", then the World could call update function in every game instance passing delta time etc like in normal gameloop.
But then what if one update call will took too much time, all other game instances will experience lag.

I think the proper way is to assume the max usage and then decide how many game instances create in one process remembering to leave some free calculation space but
still in my project the game can have very little amount of entities like 10/20 but in some 2-3 seconds later it can be 1000 that's why it is problematic to assume cpu usage that will be enough to run the game smoothly.

Edited April 12, 2021 by LoudSilence

[b10b]

17 hours ago, LoudSilence said:

Back to your response, the only thing i don't quite understand is to "Consider not "ticking" the server game loop, investigate a functional reactive approach that can absorb any delta (time and inputs)"

You mentioned ticking at 50fps on the server.  I'd question whether that frequency is necessary even within a physics sim.  It may be that a much lower rate is acceptable, say 15fps - if combined with client side interpolation.  For example, round trip time alone may limit actual frequency to closer to 10fps on some clients, visuals are going to need a minimum of 30fps - so interpolation is likely essential anyways?

As I don't know the specifics of your game loop it's hard to say, or indeed how much load each thread genuinely needs.  Just a gut reaction says that the simulated world should / could be lean - use client processing whenever possible, and server to periodically check what occurred elsewhere is legit (authoritative).  The advantage of a functional reactive state is that update frequency can theoretically be independent of client while the outcome somewhat similar.

Tell us more about your project, what kind of game is it, do you have a demo?  Thanks.

[LoudSilence]

20 hours ago, b10b said:

Tell us more about your project, what kind of game is it, do you have a demo?  Thanks.

Sadly I don't have demo yet (only local development), but I can say some more about the game.
First of all this game is mainly for my learning purposes so I write my own code as much as I can. For example (ECS engine, physics, collision detections, etc)
Some core concepts about the game are: 
2D topdown multiplayer advanced machine sim with bullet hell like gameplay, i dont know for now if it will be 1v1 only or bigger teams like 2v2, 3v3 etc.
Main thing is to have big amout of bullets, weapons etc to kill your opponent but I want it to play smooth (144fps+) on better pc's without input lag etc.
 

20 hours ago, b10b said:

You mentioned ticking at 50fps on the server.  I'd question whether that frequency is necessary even within a physics sim.  It may be that a much lower rate is acceptable, say 15fps - if combined with client side interpolation.  For example, round trip time alone may limit actual frequency to closer to 10fps on some clients, visuals are going to need a minimum of 30fps - so interpolation is likely essential anyways?

Because i've already implemented client side interpolation and prediction I think I should now test how low i can get with that frequency util i see unwanted effects in the gameplay.
After some tests i can say currently my local machine can host something around 40 players concurrently without issue but when they'll spam some more bullets this number decreases to 20'ish.

 

 

 

[b10b]

Sounds cool, let me play!

I sometimes play an FPS where botting is prevalent and it sure is f$!strating, but for a "learning purposes" project perhaps being botproof isn't the highest priority?

Maybe the design choice is whether to consider each bullet a distinct server authoritative entity, or whether to consider each "blast" (spawning multiple bullets) to be the server authoritative entity.  Some degree of trust from the client is likely acceptable, so long as it mostly correlates with others?

 

 

[LoudSilence]

Its nice to hear!

Currently every bullet is separate entity but I'll definetely try other concepts, may be for special kinds of weapons like bullet streams/gatlings? 
I will give some more info later after tests and further development etc, but anyway once again I thank You very much for help and sharing