Async foreach in nodejs

Question

In node.js, I know array.forEach is blocking.

I need to loop through an array and build a string like this:

var rarray = ['test', 'test2'];
var rstring = '';
 rarray.forEach(function(e, i){
 rstring = rstring + i + e;
});
return rstring;

how do i do it asynchronously?

• Yalamber: May be using async library?

Answer 1

In such a case, you probably don't need to transform this code into its async version.

Hypothetically, or for very very large arrays, heres's a way to transform your code into something asynchronous, or at least something that will get back to the main event loop periodically.

var rarray = ['test', 'test2'];
var rstring = '';
var max = rarray.length;
var current = 0;
process.nextTick(function next() {
 rstring = rstring + rarray[current++];
 if (current >= max) {
 console.log("done", rstring);
 } else {
 process.nextTick(next);
 }
});

In practice you'd wrap this in a function, and replace console.log by calling the completion callback.

• Yalamber: How about using modules like async?
• Pascal Belloncle: Something like async helps to sequence asynchronous calls, not to make something synchronous to be asynchronous. That's where process.nextTick() helps.

Answer 2

I might add that what you are trying to do is actually called reduce. you can write it this way (doc)

var res = array.reduce(function(prev, current, index) {
 return prev + index + current ;
}, '');

doing it asynchronously could be done this way

var array = ['one', 'two'];
function reduceAsync(collection, initial, process, callback) {
 var i = 0;
 var res = initial;
 function DO(err, result) {
 if(err) return callback(err);
 if(i > collection.length) return callback(null, res);
 var index = i++;
 var value = collection[index];
 process(res, value, index, collection, DO);
 }
 DO(null, res);
}
reduceAsync(array, '', function(previous, current, index, collection, callback) {
 setTimeout(function() {
 callback(null, previous + index + current);
 }, 10); // wait 10 ms 
}, function finalResult(err, result) {
 console.log(result);
})

or, you know, you could use async.reduce

Answer 3

Pascal's answer is essentially a form of cooperative multithreading (See Wikipedia: Thread).

It's hard to say without measuring, but I would guess that preemptive multithreading would have better throughput for things of this nature, because the compiler has the opportunity to do loop optimizations (not sure that would happen specifically with the code above), and the operating system is likely better at deciding exactly how often to task-switch. It looks like you can do Web Worker Threads in node.js.

Anybody have benchmarks?

Answer 4

If you want to use the async module with the code above, you would still end up with synchronous code. The async module allows you to avoid callback hell and manage your callbacks. It doesn't make something synchronous async. To achieve that you need to use process.nextTick like in Pascal's answer.

If you were doing some extra async processing on each item in your array and you want to aggregate the result from each operation while preserving order then you could use async.eachSeries like this:

var async = require('async');
var rarray = ['test', 'test2'];
var rstring = '', i = 0;
async.eachSeries(rarray, function(item, callback){
 rstring += (i++) + item;
 callback();
 }, function(err){
 console.log(rstring);
 }
);

If you are not fussed about the order of things, then you could use async.each and it will execute your async processing function in parallel.