Node async control flow

Question

I am trying to work with twitter data in node and am running into some road blocks that I think should be related to node style coding. The block of code is meant to grab the tweets, check if the text is in mongo and if not insert it.

The first stumbling block I find is that on trying to print out i to the console it will always iterate through each i before it starts iterating through the cursor. I think if I could clear that up it may help me going forward. Is this enough info to help me out?

What I have is:

T.get('statuses/user_timeline', options , function(err, data) {
 var db = MongoClient.connect('mongodb://127.0.0.1:27017/test', function(err, db) {
 if(err)
 throw err;
 console.log("connected to the mongoDB !");
 myCollection = db.collection('test_collection2');
 for (var i = 0; i < data.length ; i++) {
 //let's wrap this in a loop
 docu = data[i];
 //console.dir(data);
 console.dir(i);
 var cursor = myCollection.find({text : data[i].text}).limit(1);
 cursor.each(function(err, doc) {
 if (doc != null) {
 console.dir('doc is not null');
 console.dir(doc.text);
 } else {
 console.dir('doc is null - inserting');
 myCollection.insert(docu, function(err, records){
 console.log("Record added as "+records.text);
 });
 }
 })
 }
 });
})

• phtrivier: What is myCollection.find(...).limit(...) supposed to return ? Is it really a cursor, or a promise resolved when the request is done ?
• Giuseppe Crinò: A cursor. Though, even if I can't test now, it's likely .find will return before cursor could be used for iteration. imho, the best practing would be to use ...find({...}, function (err, res) { ... }).limit(1);
• phtrivier: The doc seems to suggest using cursor.toArray(function(err, doc) { ...})) mongodb.github.io/node-mongodb-native/api-generated/… , but I don't know how the API behaves, so just my 2 cents...

Answer 1

The problem is just because Javascript is async. The loop is finished before the find function in Mongo gives you a return value.

I would do the Following, or something similar - just to explaine the concept:

T.get('statuses/user_timeline', options , function(err, data) {
 var db = MongoClient.connect('mongodb://127.0.0.1:27017/test', function(err, db) {
 if(err)
 throw err;
 console.log("connected to the mongoDB !");
 myCollection = db.collection('test_collection2');
 var myfunction = function(correct_i,docu){
 var cursor = myCollection.find({text : data[correct_i].text}).limit(1);
 cursor.each(function(err, doc) {
 if (doc != null) {
 console.dir('doc is not null');
 console.dir(doc.text);
 } else {
 console.dir('doc is null - inserting');
 myCollection.insert(docu, function(err, records){
 console.log("Record added as "+records.text);
 });
 }
 })
 };
 for (var i = 0; i < data.length ; i++) {
 //let's wrap this in a loop
 docu = data[i];
 //console.dir(data);
 console.dir(i);
 myfunction(i,docu);
 }
 });
})

This way each lookup/find to Mongo will have the correct i. because the function gets it as an parameter

• Giuseppe Crinò: You can pass a callback to .find, mongodb.github.io/node-mongodb-native/api-generated/…
• Sven Delueg: the comment of @giuscri might be the cleaner solution