Making an efficient and healthy db call for a real-time browser game

Question

I'm doing a calculation for a real-time browser game, where users can invest some of their balances to the site. I need to update their invest's based on AN_AMOUNT_FROM_SITE and their percentage of shares (see PROFIT line). This function needs to run every 20-100 ms because AN_AMOUNT_FROM_SITE is coming consistently.

When this process finish, I'm saving a few unimportant data in callback().

When FUNDS collection has a few hundred document, this process will be very slow. Are there any ways I can reduce this process with Mongo to make it more efficient?

FUNDS.aggregate().group({ //get sum of invested amounts
 _id: '$id1',
 invest_total: { $sum: '$invest_amount'}}).exec(function ( e1, d ) {
 if( !e1 && d ){
 FUNDS.find().exec(function( e2, invests ){ //find all invests
 if ( !e2 && invests ){ 
 //calculate PROFIT value to each invest according to their rates (it can be +/-)
 for (var i = 0; i < invests.length; i++) { 
 var invest_amount = invests[i].invest_amount,
 invest_id = invests[i].invest_id,
 PROFIT = AN_AMOUNT_FROM_SITE * invest_amount / d[0].invest_total; // 
 //update each invest 
 FUNDS.findOneAndUpdate({ 'invest_id': invest_id }, { $inc: { 'invest_profit': profit } }, function(e3) {
 callback()
 });
 };
 }
 })
 }
})

Answer 1

The power of the aggregation framework is it's ability to iterate over the dataset in various useful ways without incurring extra round trips between the database and the app.

Your code uses one stage for aggregation (grouping by id1), and then jumps out of aggregation to iterate over the entire FUNDS collection for every user. That is very expensive, giving you exponential time complexity or worse.

The code below makes the database do all the heavy lifting in 3 aggregation framework stages, and returns an array with all the necessary data.

1. pivot over the user_id, calculating invest_amount, saving unique records in an array called records
2. unwind records. Now you have your original documents, but with a useful extra field.
3. calculate profit

Then we end up with an array of objects which can be iterated over, applying the the $inc update.

FUNDS.aggregate([
 {
 "$group": {
 "_id": '$id1',
 "invest_total": {"$sum": "$invest_amount"},
 "records": {"$push": "$$ROOT"}
 }
 },
 {
 "$unwind": "$records"
 },
 {
 "$project": {
 "invest_id": "$records.invest_id",
 "user_id": "$records.id1",
 "invest_total": "$invest_total",
 "invest_amount": "$records.invest_amount",
 "profit": {
 "$divide": [
 { "$multiply": ["$records.invest_amount", AN_AMOUNT_FROM_SITE ] },
 "$invest_total"
 ]
 }
 }
 }
]).exec(function(err,docs){
 docs.forEach(function(d){
 FUNDS.findOneAndUpdate(
 { "invest_id": d.invest_id},
 { "$inc": { 'invest_profit': d.profit } },
 function(e3){
 callback();
 }
 );
 });
});

• Lazy: Thank you for your awesome explanation and help, this was what I want.. But Mongodb doesnt let me use $$ROOT when pushing documents to records: FieldPath field names may not start with '$'.
• Lazy: I noticed I was using old version of mongodb.