[Out Topic] Data Collection and Game Analytics

[Nodragem]

Hello!

I did not find a section of the website that was about general HTML5 gamedev questions ... so I am asking the BabylonJS community, cause I am using BabylonJS at the moment. Please move the thread if there is a more relevant place for it to be.

 

I am looking for a solution that allows me to save the progress of my players in the cloud, create leaderboard (or community statistic), record events, actions and choices made by the players, etc...

 

I had a look at GameAnalytics here: https://gameanalytics.com

It sounded very promising until I realised (please tell me if I am wrong) that you can't fetch data and even less data about the current user ? but maybe I missed something in the API? 

 

I had a look at GameSparks, which seems to allow me to do anything, but I would like a free solution that can be used for a small project for now.

 

I had a look at MongoDB Atlas and mLAb and Heroku+mLAB, which are giving you access to a MongoDB server of 500Mb for free. That seems to allow me to do everything I want, but probably with more programming; the main issue is that I don't know how far I will go with 500Mb.

 

So I am just wondering what you are using on your side for collecting/storing data, whatever if it is to store the content of the inventory of the player or making statistics on player's habits. 

 

[Sebavan]

Unfortunately I did not personnaly use game analytics since a while ?

[brianzinn]

My initial goal was to go cheap and serverless and something I can run locally for development.  I did not end up with a free solution - I am running Azure functions for my API and a relational DB (it was much cheaper than DocumentDb).  I may switch my website (App Service) to be serverless at some point - Azure now allows websites to run from their CDN (like AWS S3 has for ages).  Not currently doing proper analytics, but I do track things like levels attempted/completed/etc.  In retrospect it may have been cheaper and easier to just host a website somewhere - hope that helps a bit   I think the best would be an affordable usage based turnkey solution that can be configured to store and query different data structures with user management/authentication.

[Dad72]

You can use PHP to save to a database or save to a Json file.

Then retrieve the information with PHP again.

[Nodragem]

So like, on Heroku, I could not write on the server with PHP right? I guess you would use a SQL database? 

I need to check if SQL database are less expensive than MongoDB.

I will run some test on how much info we can store in 500MB with MongoDB and I will report the results here.

[brianzinn]

Here is an open source game running on heroku and mlab (javascript server with mongoDB. websockets for 4 player peer communcations):
https://github.com/HoneyBadgers2/ColoniesOfCotton/

[timetocode]

Something that can be neat for clientside events is adding custom events to google analytics. Those are easy to make, and then they end up in the same data set with the rest of the stuff. Obv. not a viable candidate for player data, just analytics

For serverside persistence of player data, this isn't going to be particularly helpful, but I just do it myself on a vps (vultr and linode lately). I usually need authentication as part of my games, and I do that with a node + express + passport + postgres. At that point I already have a server with psql installed so I just add a table or some columns for the game-specific data. Despite doing this, I have mixed reservations about whether the work is worth it. It can be a lot cheaper than something like aws or a standalone sql provider (that probably uses aws behind the scenes..) but it is a lot of work.

Edit:

I'd also like to mention that if the data stored about players is just like a finite number of columns with some counts like "totalKills" it ends up being a very small amount of data. On bruh.io, our data was like that, and during its most popular days (when breaking 1000+ ccu at peak every day) our needs were easily met by the smallest dedicated instance from elephantsql. In total that game has ~320k registered users and 25 million rounds played.

By comparison I've tried to estimate the storage needs for a first person shooter style game where every single kill and item pick up is saved as a time stamped event. If a game like that was played the same amount as bruh, it is hard to say exactly, but we'd be going from 100 GB being total overkill to wanting a several TB.

[Nodragem]

thanks for the feedback, that is very interesting!

In fact, for now, the game will be played by like, 10 persons ^^ but we want to record almost everything about what they do if possible (that is part of a research project).

I guess I will just start with a free database, then if the project is distributed to a bigger audience, I will change to a paid database.

[Dad72]

Why not save the information in a Json file. it's very very simple with PHP.

A file_put_contents($root, json_encode ($json)); to save and file_get_contents($root) to read.

[Nodragem]

@Dad72, I'm going to ask the most newbie question ever but I m just not sure: is that saving the JSON file on the server (like Heroku)? If I understand, you are not saving in a database? If saving on my Heroku server, how much memory can I use?

[Dad72]

I do not know what a Heroku server is.

If you have an FTP, you can send files (js, html, php ...)

In this case, you can with PHP code write in .json, .dat, .txt files or whatever you want. The files will go to your server by filling in the path.

Level use of the memoir, it consumes virtually nothing. The use of SQL is much more expensive.

Do you already use PHP? it's a really powerful language when you pair it with Ajax

I can give you complete examples with PHP to write and read a Json file.

[Nodragem]

Thank you for your explanation  I will check what's a FTP server (I am very new to web development).

Never used PHP, but I am hoping there is a node.js equivalent to access files from a FTP server. I would like to keep my project in one language for simplicity.

[Dad72]

For an FTP server, you can use Filezilla. This software connects to your web server to send files from your computer. This can be any file.

Then if you want to run a php file, you call it by your server address and the file name and the php code is run. Or you use Ajax which it loads to call the php file to run it in the background.

PHP is very easy to use and very powerful. You can with, generate files of all kinds, generate images. PHP is used to generate the sending of registration form, connection for example.

I use it in all my projects to read and write in files to record, in your case, it would be very useful to record the data of your players and much more.

PHP is a lot easier than JavaScript, it's an essential language on the Web when you want to boost its pages or web projects.

The only thing is that your pages that use php should no longer have the .html extension but .php, but you call it as an html page. The Apache server will take care of executing the PHP code on the server and no longer by the browser as with Javascript. This is why the use of ajax + php can work wonders.

Learning PHP will be very beneficial to you for this project and other projects.On the FTP server, it allows you to send your files from your computer to your web server (hosting or dedicated)

Also most of the web hosting allows free databases in limited numbers. Only one database is sufficient. Then you can still use PHP to communicate with a MYSQL database if you prefer rather than write to json files.

I hope this helps you.

[QuintusHegie]

On 11/29/2018 at 2:01 PM, Dad72 said:

I use it in all my projects to read and write in files to record, in your case, it would be very useful to record the data of your players and much more.

Do you have any pointers or ideas on securely connecting Javascript to a PHP/MySQL backend? E.g. how do I know if my client player isn't submitting cheat high scores? And how to identify my client player (that set the high score) properly? E.g. using Facebook or LinkedIn or Google account?

[Dad72]

You can not use cheat code with PHP. The code is not visible in the source code compare to Javascript. PHP is very secure.

To identify a player, you can use the session with PHP which is secure. I do not use a Facebook or other account, users log in and a session containing their nickname is registered, and all that remains is to retrieve the login account information from the database.

 

[timetocode]

Social logins work for identifying a player. The APIs typically provide a special id to store in your own database, and this id will be provided via api each time the player comes back.

A single player type of a game played in a browser means that the entire game's state is not in control of the developer. This is running on the player's computer, and they can alter the code and do whatever they want. Scores submitted by a game like this cannot ever be trusted. That doesn't stop developers from making plenty of games like this, but inherently there is no way to know that any submitted score is valid without doing something vastly more sophisticated. Usually people just periodically clean out their leaderboards of all the fake scores. 

An example of something more sophisticated would be a game that submitted all of its player input. This is uncommon (maybe non-existent) in single player games, but for example a game like Tetris could be made where the game records every move the player makes. At the end of the round the game client could submit this long list of moves to the server, and the server could re-simulate the same game and generate the player's score. The player could still be a hacker submitting fraudulent moves, so the server would still have to validate some things... but this provides a basis for knowing whether submitted information is potentially valid whereas receiving a json message like { playerId: 219, score: 9999999 } leaves very little to validate. All we can really do with that data is perhaps guess that they are a cheater. Meanwhile to hack a game that submits a move list that then gets simulated on the server is very difficult -- it requires essentially writing a bot that can play the game like a human (but better).

I'm not saying anyone should make a system like this, but this "deterministic simulation built from player input" is the element that prevents cheating in real time strategy games. A similar concept reduces the cheating in server-authoritative first person shooters (and is why aim bots exist).