Buffered read from a large list

Question

There is a large list of email addresses and there is a limitation of max 100 email addresses to process once. This is the code I have written.

 //Create a list of email addresses to test
 List<string> emails = new List<string>();
 for (int i = 0; i < 10000; i++)
 {
 emails.Add(string.Format("email{0}@domain.com", i));
 }
 //buffered read
 int remainingCount = emails.Count;
 int maxToReadOnce = 100;
 int startIndex = 0;
 while(remainingCount > 0)
 {
 int countToRead = (remainingCount > maxToReadOnce) ? maxToReadOnce : remainingCount;
 List<string> myemails = emails.GetRange(startIndex, countToRead);
 startIndex = startIndex + countToRead;
 remainingCount = remainingCount - countToRead;
 Process(myemails); // Method that process the email addresses 
 }

Answer 1

Your code is pretty well written and I can't see anything too much I'd change...

(remainingCount > maxToReadOnce) ? maxToReadOnce : remainingCount

might be clearer as

Math.Min(maxToReadOnce, remainingCount) 

but other than that it all looks really sensible.

---

That said, by mixing the batching into the processing code both are harder to read. Batching (splitting/sharding/whatever you want to call it) comes up a lot - I have an extension method lying around for it:

public static IEnumerable<IEnumerable<T>> Batch<T>(this IEnumerable<T> enumerable, int batchSize)
{
 var c = 0;
 var batch = new List<T>();
 foreach (var item in enumerable)
 {
 batch.Add(item);
 if (++c % batchSize == 0)
 {
 yield return batch;
 batch = new List<T>();
 }
 }
 if (batch.Count != 0)
 {
 yield return batch;
 }
}

There are a whole bunch of variations on the above - feel free to make any changes... The ++c % batchSize == 0 has always bothered me a bit but I like the terseness of it.

With that extension method in place you can simplify your code to:

foreach (var batchOfEmails in emails.Batch(maxToReadOnce))
{
 Process(batchOfEmails);
}

(if you don't like var the type will be IEnumerable<IEnumerable<string>>)

• Dimuthu: Mixing batching and processing code together was my concern too. Thanks for the clean code.
• mjolka: "The ++c % batchSize == 0 has always bothered me a bit" -- you can replace that with batch.Count == batchSize (and remove c entirely).