How does a game server run multiple games at once?

[rocket04]

I can't imagine this is an unusual question, yet I've searched all over and am not finding any answers on this. I found two threads where people seem to be asking this same question, but seems like it got lost in the other stuff they were asking because they didn't get any answer as far as I can tell:

When running client side, you have a game loop and it takes care of everything for that one client. But on the server, I can't imagine you have one game loop running all your games. Here, I'm assuming something like a sports game, where you have multiple games going on at the same time involving two players each, not an MMO. So what do you need to do, spawn a mini game engine server for each game that has its own loop?

For some background, what I'm trying to do is something similar to how Championship Manager was, simple 2D soccer simulation. For those unfamiliar, you basically just act as the coach, so there's no real-time interaction with the players, you just select tactics and the engine reacts to them. I see a bunch of stuff about Node.js as a game server, which has me a bit confused since I thought Node.js was single-threaded, so how would it handle multiple concurrent games?

[BobF]

5 hours ago, rocket04 said:

But on the server, I can't imagine you have one game loop running all your games.

Typically, the server would simply communicate player input between the players. It might also track the game state. Your soccer game already accepts player input from the keyboard/mouse and then produces some resulting output. You would modify the game to also accept remote player input from the network. When the game receives input from the local player it not only produces a game result, it also communicates that player input (and possibly the game result) to the remote player via the game server to which both players are connected. I'm glossing over a lot of details, but that's a high level view.

 

5 hours ago, rocket04 said:

I thought Node.js was single-threaded, so how would it handle multiple concurrent games?

In the simple case, the node server is just transferring player input data between players. It doesn't need to be multi-threaded because it handles player input messages independently one message at a time. Info about which players are connected to whom can be stored in a db and retrieved as each incoming message is processed. Again this is very high level and it's not the only way to do things.

If you're unsure about rolling your own multi-player solution then you might want to consider a multi-player game service that does most of the work for you. Photon is one example.

[rocket04]

1 hour ago, BobF said:

Typically, the server would simply communicate player input between the players. It might also track the game state. Your soccer game already accepts player input from the keyboard/mouse and then produces some resulting output. You would modify the game to also accept remote player input from the network. When the game receives input from the local player it not only produces a game result, it also communicates that player input (and possibly the game result) to the remote player via the game server to which both players are connected. I'm glossing over a lot of details, but that's a high level view.

I think I'm doing a bad job at communicating what I need to know. I think I have a handle on the communication between the client and server, which will likely be one of the easiest parts because there aren't even keyboard and mouse inputs that change the game real time. You pause the game, pick tactics/make substitutions, hit play again and the game moves on. What I'm confused about is the game simulation happening on the server. You've got a game loop going that's running the AI so players make decisions, velocities get updated, etc. But say I have 20 people playing one-on-one, so 10 games going on running AI and that then need to broadcast the velocities and positions to the game. How are those 10 game simulations running at the same time? What is the architecture that makes that happen? I'll give an example off the top of my head that I bet is the wrong answer, but say I was doing a server in Java, I could spawn a new thread on the game server every time a new game needs to be created. You create a Game class implementing Runnable or extending Thread and inside that class you have your game loop. I'm pretty sure that's not a good way of going about it because as far as I know threads are expensive. So does something need to spawn a new process for a game engine for each game? Or something else? Mind you, I provided an example in Java because that's a language I worked with in the past and used threads in, but I am most interested in what would be done with a Node.js server. But I have a feeling the language doesn't matter, if I can grasp the concept for one language, I'll probably be able to apply it to another.

Thanks for taking the time to reply, btw, very much appreciated.

[WombatTurkey]

7 hours ago, rocket04 said:

I think I'm doing a bad job at communicating what I need to know. I think I have a handle on the communication between the client and server, which will likely be one of the easiest parts because there aren't even keyboard and mouse inputs that change the game real time. You pause the game, pick tactics/make substitutions, hit play again and the game moves on. What I'm confused about is the game simulation happening on the server. You've got a game loop going that's running the AI so players make decisions, velocities get updated, etc. But say I have 20 people playing one-on-one, so 10 games going on running AI and that then need to broadcast the velocities and positions to the game. How are those 10 game simulations running at the same time? What is the architecture that makes that happen? I'll give an example off the top of my head that I bet is the wrong answer, but say I was doing a server in Java, I could spawn a new thread on the game server every time a new game needs to be created. You create a Game class implementing Runnable or extending Thread and inside that class you have your game loop. I'm pretty sure that's not a good way of going about it because as far as I know threads are expensive. So does something need to spawn a new process for a game engine for each game? Or something else? Mind you, I provided an example in Java because that's a language I worked with in the past and used threads in, but I am most interested in what would be done with a Node.js server. But I have a feeling the language doesn't matter, if I can grasp the concept for one language, I'll probably be able to apply it to another.

Thanks for taking the time to reply, btw, very much appreciated.

I have no idea about the Java version, sorry.   However, it really depends on your server architecture / what back-end framework you have.

Nodejs makes it super easy. You need to spawn multiple instances of your app running on different ports and assign them to a load balancer (using nginx or haproxy). (Example: 9300, 9301, 9302) -- This module runs node as a daemon and does that. https://github.com/Unitech/pm2

Then, since Nodejs doesn't share memory, you need to use an in-memory db system (node-redis). This enables you to share memory across the nodes, but more importantly it lets you spawn more instances of your gameserver to take help with scaling / utilizing multi-core servers  Make sure you install Bluebird (or any Async library) to help with control flow. This helps tremendously if your game server ever reaches 20, to 50k + lines of code. 

Pertinent to the game loop question. I think everyone kind of does it differently. But, @GoldFire has far more expertise in this area. Personally, I run a game loop on each instance handling the monster AI. But see, my architecture is a bit different. My game is instance based so when a player clicks to join a game of which is on another nodejs instance... I silently transfer that player (making a new websocket connection) -- So the player's can communicate data between either other on the same instance instead of relying on Redis` Pub/Sub across instances. This way achieves less latency and overhead.

But, you should be keeping a list of all online users and what instance they are on, inside Redis. This way, when a player is sending a global chat message, you just grab the list from Redis and dependent on how many instances are running, you send a publish signal to them with the message.

For example: I have 3 node instances, #1 - 9300, #2 - 9301, and #3 - 9302.

3 Players are in a game killing monsters, chatting to each other, etc on #1.

55 Players are in a game killing monsters, chatting to each other, etc on #2

12 Players are in a game killing monsters, chatting to each other, etc on #3

Now, a player named Muffin is on instance #3 and has a friend called Akira, who is on instance #1. Muffin wants to send her a message. This is why Pub Subbing is so important. Muffin can send a signal: "TXT Hello Akira" and the Websocket server can now grab Akira's data from Redis and find out what instance she is on, and then the instance that Muffin is on sends a publish signal with "Hello Akira" to Akira's instance. Each node instance has a subscribe channel for their port # (game instance) --  You can read more about Pub Subbing here: https://gist.github.com/reu/5342276  (See how it says subscribe "test"), for a game server, (well for mine anyway) -- I use subscribe "9300" for the port number. 

You might be wondering why I threw in the Pub Sub thing, well, it's very and I mean very important if you want to scale with node. It makes it so easy and fun, good luck!

[mattstyles]

Node is great for running multiple processes, I'd go for the route of spawning multiple processes for each 'thing' you need to do, there is minimal overhead in doing so, the only downside is that there is a little bit of architectural know-how to get this all working nicely but it's a very attractive way to work, not least because it helps to detach separate parts of your application which is great for being robust, adding redundancy, testing, development, performance etc etc the list goes on and on to the advantages of this style of approach.

The opposite monolithic approach you've already covered. Inside your main process you spawn multiple instances of separate 'Games' and then for each update you loop through all active games and call their `update/whatever` functions. Node is single-threaded by design as it is designed to be easy and lightweight, for the minute don't worry too much about it, it'll jump quite happily between `update` functions and it is fairly hard to block the thread but you're right to flag that as a concern, for the easiest solution see wombats answer!

Part of the decision to have node single threaded is because multi-threaded programming is hard, the nuts and bolts of it are seriously hard and it is very easy to allow programmers to slip into deadly holes, node tries (as much as a language can) to help programmers fall into the pit of success. Part of the premise is that the largest bottle neck is not calculations but I/O, by it nature node will not block for I/O operations, hence all the callbacks (promises are taking over again). It's possible to block the thread of course, but node/JS has many constructs that discourages programmers even choosing blocking solutions. Whilst waiting for I/O the node process could very well be off doing something else, it takes the approach that a lot of the time multiple threads and stuff are idle so it tries to fill the gaps and its main loop is very tight (its naff-all lines of code, hard to be any tighter, its just responding to events after all).

 

Good luck with your game, just a quick note, if your game (or some of the games) involve 2 multiple players then pausing to change tactics/subs/etc is going to be pretty annoying for the other player, you might want to consider rethinking that so that the game in-fact is real time. Your client will be detached from the AI running the simulation anyway so the users can be free to 'fiddle' with your menus/etc, you only send a message to the simulation when that user has initiated an action i.e. a substitution.

[rocket04]

4 hours ago, WombatTurkey said:

Nodejs makes it super easy. You need to spawn multiple instances of your app

So am assuming something like this: https://nodejs.org/api/child_process.html#child_process_asynchronous_process_creation ? Then once you've spawned the processes, I assume they have a way of talking back to the main process if needed?

I'm pretty sure I will end up using a bunch of libs for a lot of things, but I think it's always a good idea when getting into a new project to actually understand the moving parts. I don't want to just start a collage of libraries I know nothing about. I'll probably even create a good part of the engine myself and then rip it all out to use something more robust. At least I have the luxury of time since this is just for fun and I have no time constraint whatsoever. Thanks for all the useful advice, I'll take all that into consideration. Bluebird should be easy to pick up, I'm pretty familiar with promises from my regular day job, although haven't used that particular framework. And I'm supposed to start setting up Redis clusters for work as well, so I guess that experience will come in handy for the game too!

 

3 hours ago, mattstyles said:

he opposite monolithic approach you've already covered. Inside your main process you spawn multiple instances of separate 'Games' and then for each update you loop through all active games and call their `update/whatever` functions. Node is single-threaded by design as it is designed to be easy and lightweight, for the minute don't worry too much about it, it'll jump quite happily between `update` functions and it is fairly hard to block the thread but you're right to flag that as a concern, for the easiest solution see wombats answer!

Wow, so seems you're saying that it would not be completely inconceivable to do something along the lines of:

while (gameServerOn === true)
    foreach(games as game)
        game.update()

Where in the Game object you have your game loop? If so, interesting, I would have thought that wouldn't work in a million years with anything more than a few games, but maybe I'm vastly underestimating the processing power we have these days. Not that I intend to go that route, but found it interesting that you didn't seem to peg it as categorically bad/not doable.

 

3 hours ago, mattstyles said:

Good luck with your game, just a quick note, if your game (or some of the games) involve 2 multiple players then pausing to change tactics/subs/etc is going to be pretty annoying for the other player, you might want to consider rethinking that so that the game in-fact is real time. Your client will be detached from the AI running the simulation anyway so the users can be free to 'fiddle' with your menus/etc, you only send a message to the simulation when that user has initiated an action i.e. a substitution.

The now defunct Championship Manager Online actually did pause, and you only had 30 seconds to make your changes. Surprisingly, in the hundreds (probably thousands) of games I played, not a single time was I annoyed by somebody constantly pausing. Just never happened. Having said that, I think you're right, I see no reason why I couldn't make it real-time, it's still much easier than people who actually have to send input commands for moving players.

Thanks so much for all the tips!

[Gio]

Just a couple of observations as I believe your question has been answered already, but

1. Node.js doesn't have to be single-threaded. It can be, in many cases it makes sense for it to be, but it's easily multi-threadable too.
2. Don't use a while loop like that. While single-threaded, node.js needs "idle cycles" to work properly. It's during those idle cycles that scheduled events can take place. Instead, do something like this when creating a new game on the server (oversimplifying now, but still):

   setInterval(function()
   {
       myGame.update();
   }, timeStep);
   

3. Only worry about scaling when scaling is likely to be an issue... how many games do you have? How many can you support with a single process? Now I can't find the article that I read some time ago, but some guy was claiming that he was supporting 600,000 sockets (players) on a single node.js instance. I know I can easily support a few hundred on a very low-specced (256MB RAM!) virtual machine. If you are only going to have, say, a few dozen concurrent players, don't bother overcomplicating your architecture making it super-scalable, just keep it simple and single-threaded. Redis is a good suggestion, but with only a bunch of concurrent players, it may be unnecessary.
    

[mattstyles]

1 hour ago, Gio said:

Node.js doesn't have to be single-threaded. It can be

No it can't, just one thread. You can spawn other processes but thats not threading, node has no mechanisms for threading the CPU. It can attach to C++ processes to do so but this is a hack and most of the (very very few) libs that tried this a few years ago gave up. You could tell each process to use a separate cpu though, if available.

3 hours ago, rocket04 said:

So am assuming something like this: https://nodejs.org/api/child_process.html#child_process_asynchronous_process_creation ? Then once you've spawned the processes, I assume they have a way of talking back to the main process if needed?

There is a protocol for sending messages back and forth, a very simple event emitter style protocol, almost identical API to web sockets or communicating with service workers. But I dont think @wombat was suggesting to do that all in node, rather spawn complete new instances rather than child processes, although doing it all in node is viable too.

And yeah, I agree with @Gio, don't use a while loop, when I said its very hard to actually block the node thread you've embarrassed me by picking an unterminated loop that will block the thread! I'd go with Gio's approach of creating a master game loop that calls each live instance update function, you can get clever about this by limiting the amount of time each child gets to do its update but that gets pretty darned tricky.

Just for your own interest node's event loop (basically the whole core of node) is almost identical, its just a loop that pops actions off the stack and executes them. You're effectively doing the same, adding and removing games from a list of active games as they start/finish and processing that loop.

I was interested in the actual performance concerns of running multiple node processes on the same core and came across this article which should be interesting for you, its a little old now and node has fairly recently undergone some large changes but I think it all still holds true, at least conceptually.

[Gio]

Not sure what you mean, I've always done "threading" in node, i.e. execute separate node.js programs in different processes that run in parallel (that you can assign to separate CPU's, but could also have more than one per CPU). Never had a problem with it.

var cluster = require('cluster');
var numWorkers = 7;
cluster.setupMaster({exec: 'myWorkerProgram.js'});
for (var i = 0; i < numWorkers; i++) 
{
  cluster.fork();
}

Sharing memory between threads is not possible directly, that's where you'd need Redis or something like that.

Though I can see where you're coming from, technically I think it ends up being a separate node.js instance handling each process individually, so to be precise it's not a multithreaded instance, it's several single-threaded ones. In practice it's a simple system for running programs in parallel, call it what you like...

 

[rocket04]

7 hours ago, Gio said:

• Don't use a while loop like that. While single-threaded, node.js needs "idle cycles" to work properly. It's during those idle cycles that scheduled events can take place. Instead, do something like this when creating a new game on the server (oversimplifying now, but still):

  
  setInterval(function()
  {
      myGame.update();
  }, timeStep);
  

I feel rather stupid right now. I'm so used to using javascript only on the client side that when it comes to the server side I forget I can still use its async features. Thanks for the reminder. And thanks again all for the useful info, I now feel much better equipped to tackle the game server. Funny how I googled until I was blue in the face and did not manage to find the kind of answers I was provided with here so quickly. I should have posted days ago!

[BobF]

9 hours ago, Gio said:

Node.js doesn't have to be single-threaded. It can be, in many cases it makes sense for it to be, but it's easily multi-threadable too.

You're thinking of processes, not threads, those two concepts are not synonymous. Your JS app can have multiple processes, but in node (and browsers) those processes are strictly single-threaded.

 

9 hours ago, Gio said:

• Don't use a while loop like that. While single-threaded, node.js needs "idle cycles" to work properly. It's during those idle cycles that scheduled events can take place. Instead, do something like this when creating a new game on the server (oversimplifying now, but still):

  
  setInterval(function()
  {
      myGame.update();
  }, timeStep);
  

Right!

Better yet, get rid of the setInterval and only update the game when a new message arrives from a player (client). setInterval should only be needed if the server AI needs to push new data, such as updated player positions, to the players between player actions. If the players can reasonably keep themselves updated between player actions then it may make more sense to do that in the client. That will save a lot of server cost once as the game scales.