Build Status

A stream that transforms chunks to batches form the stream. NPM

A way to use a Node.JS Transform stream that batches chunks into an array (default is 10). Objects/chunks will still come in the same order, just in batched arrays.

Built using through2 and has the same API with the addition of a batchSize option.

Non-objectMode streams are supported for completeness.

Written by Nima Gardideh (halfmoon.ws) and used in production by Taplytics.

npm install --save through2-batch

Process rows from a CSV in batches.

var through2Batch = require('through2-batch');
fs.createReadStream('data.csv')
 .pipe(csv2())
 .pipe(through2Batch.obj(
 {batchSize: 100},
 function (batch, enc, callback) {
 var self = this;
 console.log(batch.length); // 100
 someThingAsync(batch, function (newChunk) {
 self.push(newChunk);
 });
 }));

Don't specify a transform fn, and let the batches be processed by another stream

var through2Batch = require('through2-batch');
fs.createReadStream('data.csv')
 .pipe(csv2())
 .pipe(through2Batch.obj())
 .pipe(getSomeOtherStreamProcessingBatches());

Fixed or improved stuff? Great! Send me a pull request through GitHub or get in touch on Twitter @ngardideh.