Sending many unique emails at same time

Question

I have an asp.net core app that sends emails with a link. When they click the link they can start taking an online test without logging in. The link contains a number that is validated by the controller who then returns their test.

I log into my app and create a project. Here I enter a list of emails in a textbox separated by commas. In the backend I put this textbox string into a list like so:

string emailList;
List<string> list = emailList.Split(',').ToList();
//removes whitespace in list:
for (int i = 0; i < list.Count(); i++)
{
 list[i] = string.Join("", list[i].Split(default(string[]), 
 StringSplitOptions.RemoveEmptyEntries));
}

The only way i can see on how to send many unique emails would be to loop over this list and then use the normal SendEmail method in mimekit:

string message = "Press this link to take the test";
int number = 0;
foreach(var email in emailList)
{
 //gets a unique number stored in a table for each person:
 number = GetNumber(email); 
 message = $"http://www.mywebsite.com/Test/{number}";
 await _emailSender.SendEmailAsync(email, "Subject", message);
}

Now the emailSender uses the normal mimekit method to send one email for each in the list.

This takes a long time for a big list. Are there any other ways to do it, or can I write this more effectively? What could be the bottlenecks of my solution? Any feedback would be great, from my bad async code to security . I suspect that I use too much async when I should return only a task, but i'm not sure how to do it. I also wonder how I can even test this on a larger scale as I don't have a big emaillist to use.

Here is the mimekit code to send the email:

public async Task SendEmailAsync(string email, string subject, string message)
{
 var melding = new MimeMessage();
 melding.From.Add(new MailboxAddress("Frank", _options.Option1));
 melding.To.Add(new MailboxAddress(email));
 melding.Subject = subject;
 melding.Body = new TextPart("plain")
 {
 Text = message
 };
 Try
 {
 await SendMessage(melding);
 }
 catch(Exception ex)
 {
 //log error
 throw new ApplicationException($"something went wrong:'{ex.Message}'.");
 }
}
private async Task SendMessage(MimeMessage message)
{
 using (var client = new SmtpClient())
 {
 await client.ConnectAsync("smtp.WebSiteLive.net", 587, SecureSocketOptions.None);
 client.AuthenticationMechanisms.Remove("XOAUTH2"); // Must be removed for Gmail SMTP
 await client.AuthenticateAsync(_options.Option1, _options.Option2);
 await client.SendAsync(message);
 await client.DisconnectAsync(true);
 } 
}

any help much apprechiated!!

• BCdotWEB: It depends on the volume, but once you need to start sending out significant amounts of emails you'll need to look into using an outside service that specifically deals with such issues and offers an API (like for instance sendgrid.com ). See also stackoverflow.com/questions/4369178/…
• Johan Herstad: you are probably right. I just don't want to pay for it, haha.
• Peter Taylor: Sign up for SendGrid via Azure and you get 25000 free e-mails per month. I'm not aware of any minimum spend on Azure to qualify.

Answer 1

Populate mails in batches and process batch by batch instead of sending mail one by one, this provides better scalability and also have more control on the usage of number of threads {cancellation[if required]}.

 For example:
 Number of emails to send: 103
 BatchSize :5 (Make it configurable)
 We have in total : 20 batches of size 5 and 1 batch of size 3

Steps are as follows

 a. Process first batch.
 b. Wait till batch completes execution
 c. Take next batch for process
 d. Repeat the process till all the batches.

I have two approaches to achieve this, based on the requirement you can use any of the approach

1. Use async/await and Task => Create separate task for sending each mail in each batch, which uses multiple threads.

   Process Batch 1 => 5 users, Create 5 tasks and execute => Wait for all tasks to complete Process Batch 2 => 5 users, Create 5 tasks and execute => Wait for all tasks to complete ...

2. Use only async/await => which will use almost same thread for processing.

Consider the Email Class as below, and _batchSize

 public class EmailClass
 {
 public string Email { get; set; }
 public string Message { get; set; }
 public string Subject { get; set; }
 }
 private int _batchSize = 5;
 List<EmailClass> _emailCollection = new List<EmailClass>();

When the user clicks on Send button

 private async void button_Click(object sender, RoutedEventArgs e)
 {
 for (int ii = 0; ii < 22; ii++)
 {
 _emailCollection.Add(new EmailClass()
 {
 Email = "Email_" + ii + "@gmail.com",
 Message = "http://google.com",
 Subject = "Subject_" + ii
 });
 }
 await SendBulkMail();
 }

Divide the whole bunch of mails, into chunks of _batchSize, process each chunk one by one.

 private async Task SendBulkMail()
 {
 int remainingBatches = _emailCollection.Count % _batchSize;
 int numberOfBatches = (_emailCollection.Count - remainingBatches) / _batchSize;
 Debug.WriteLine(string.Format("TotalCount:{0} NumberOfBatches:{1}", _emailCollection.Count, numberOfBatches));
 for (int ii = 0; ii < numberOfBatches; ii++)
 {
 Debug.WriteLine(DateTime.Now.ToString() + " CurrentChunk : " + ii);
 await SendBatchMail(0, ii, _batchSize);
 }
 if (remainingBatches != 0)
 {
 await SendBatchMail(_emailCollection.Count - remainingBatches, 0, _emailCollection.Count);
 }
 }

Solution 1: Create Task for each mail while processing each block

 private async Task SendBatchMail(int initalIndex, int batchIndex, int size)
 {
 List<Task> chunkTasks = new List<Task>();
 chunkTasks.Clear();
 for (int jj = initalIndex; jj < size; jj++)
 {
 var index = _batchSize * batchIndex + jj;
 chunkTasks.Add(Task.Factory.StartNew(() => {
 SendEmailAsync(_emailCollection[index].Email, _emailCollection[index].Subject, _emailCollection[index].Message);
 }));
 }
 await Task.Run(() =>
 {
 Task.WaitAll(chunkTasks.ToArray());
 });
 }
 public void SendEmailAsync(string email, string subject, string message)
 {
 Debug.WriteLine("\t" + DateTime.Now.ToString() + " " + Thread.CurrentThread.ManagedThreadId + " Sending mail : " + email);
 try
 {
 SendMessage(email + subject + message);
 }
 catch (Exception ex)
 {
 }
 }
 private void SendMessage(string message)
 {
 Debug.WriteLine("\t\t" + DateTime.Now.ToString() + " " +Thread.CurrentThread.ManagedThreadId + " Message sending : => " + message);
 int sleepTime = new Random().Next(5) * 1000;
 Thread.Sleep(sleepTime);
 Debug.WriteLine("\t\t" + DateTime.Now.ToString() + " " +Thread.CurrentThread.ManagedThreadId + " Message sent : => " + message);
 }

Solution 1 output : Separate threads[may or may not] is used for sending each mail in every batch

 TotalCount:22 NumberOfBatches:4
 4/12/2018 11:07:02 PM CurrentChunk : 0
 4/12/2018 11:07:02 PM 13 Sending mail : [email protected]
 4/12/2018 11:07:02 PM 15 Sending mail : [email protected]
 4/12/2018 11:07:02 PM 15 Message sending : => [email protected]_1http://google.com
 4/12/2018 11:07:02 PM 11 Sending mail : [email protected]
 4/12/2018 11:07:02 PM 11 Message sending : => [email protected]_2http://google.com
 4/12/2018 11:07:02 PM 13 Message sending : => [email protected]_0http://google.com
 4/12/2018 11:07:02 PM 14 Sending mail : [email protected]
 4/12/2018 11:07:02 PM 14 Message sending : => [email protected]_3http://google.com
 4/12/2018 11:07:02 PM 14 Message sent : => [email protected]_3http://google.com
 4/12/2018 11:07:02 PM 14 Sending mail : [email protected]
 4/12/2018 11:07:02 PM 14 Message sending : => [email protected]_4http://google.com
 4/12/2018 11:07:02 PM 14 Message sent : => [email protected]_4http://google.com
 4/12/2018 11:07:05 PM 15 Message sent : => [email protected]_1http://google.com
 4/12/2018 11:07:05 PM 11 Message sent : => [email protected]_2http://google.com
 4/12/2018 11:07:05 PM 13 Message sent : => [email protected]_0http://google.com
 4/12/2018 11:07:05 PM CurrentChunk : 1
 4/12/2018 11:07:05 PM 16 Sending mail : [email protected]
 4/12/2018 11:07:05 PM 14 Sending mail : [email protected]
 4/12/2018 11:07:05 PM 13 Sending mail : [email protected]
 4/12/2018 11:07:06 PM 15 Sending mail : [email protected]
 4/12/2018 11:07:06 PM 15 Message sending : => [email protected]_8http://google.com
 4/12/2018 11:07:06 PM 11 Sending mail : [email protected]
 4/12/2018 11:07:06 PM 11 Message sending : => [email protected]_9http://google.com
 4/12/2018 11:07:06 PM 14 Message sending : => [email protected]_7http://google.com
 4/12/2018 11:07:06 PM 13 Message sending : => [email protected]_6http://google.com
 4/12/2018 11:07:06 PM 15 Message sent : => [email protected]_8http://google.com
 4/12/2018 11:07:06 PM 16 Message sending : => [email protected]_5http://google.com
 4/12/2018 11:07:06 PM 16 Message sent : => [email protected]_5http://google.com
 4/12/2018 11:07:10 PM 11 Message sent : => [email protected]_9http://google.com
 4/12/2018 11:07:10 PM 14 Message sent : => [email protected]_7http://google.com
 4/12/2018 11:07:10 PM 13 Message sent : => [email protected]_6http://google.com

Solution 2 : Use only async/await pattern

 private async Task SendBatchMail(int initalIndex, int batchIndex, int size)
 {
 List<Task> chunkTasks = new List<Task>();
 chunkTasks.Clear();
 for (int jj = initalIndex; jj < size; jj++)
 {
 var index = _batchSize * batchIndex + jj;
 chunkTasks.Add(SendEmailAsync(_emailCollection[index].Email, _emailCollection[index].Subject, _emailCollection[index].Message));
 }
 await Task.Run(() =>
 {
 Task.WaitAll(chunkTasks.ToArray());
 });
 }
 public async Task SendEmailAsync(string email, string subject, string message)
 {
 Debug.WriteLine("\t" + DateTime.Now.ToString() + " " + Thread.CurrentThread.ManagedThreadId + " Sending mail : " + email);
 try
 {
 await SendMessage(email + subject + message);
 }
 catch (Exception ex)
 {
 }
 }
 private async Task SendMessage(string message)
 {
 Debug.WriteLine("\t\t" + DateTime.Now.ToString() + " " +Thread.CurrentThread.ManagedThreadId + " Message sending : => " + message);
 await Task.Run(() =>
 {
 int sleepTime = new Random().Next(5) * 1000;
 Thread.Sleep(sleepTime);
 });
 Debug.WriteLine("\t\t" + DateTime.Now.ToString() + " " +Thread.CurrentThread.ManagedThreadId + " Message sent : => " + message);
 }

Solution 2 Output (Almost one thread is used for processing of all batches)

 4/12/2018 11:01:05 PM CurrentChunk : 0
 4/12/2018 11:01:05 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:05 PM 10 Message sending : => [email protected]_0http://google.com
 4/12/2018 11:01:05 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:05 PM 10 Message sending : => [email protected]_1http://google.com
 4/12/2018 11:01:05 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:05 PM 10 Message sending : => [email protected]_2http://google.com
 4/12/2018 11:01:05 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:05 PM 10 Message sending : => [email protected]_3http://google.com
 4/12/2018 11:01:05 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:05 PM 10 Message sending : => [email protected]_4http://google.com
 4/12/2018 11:01:05 PM 10 Message sent : => [email protected]_2http://google.com
 4/12/2018 11:01:05 PM 10 Message sent : => [email protected]_3http://google.com
 4/12/2018 11:01:05 PM 10 Message sent : => [email protected]_4http://google.com
 4/12/2018 11:01:08 PM 10 Message sent : => [email protected]_0http://google.com
 4/12/2018 11:01:08 PM 10 Message sent : => [email protected]_1http://google.com
 4/12/2018 11:01:08 PM CurrentChunk : 1
 4/12/2018 11:01:08 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:08 PM 10 Message sending : => [email protected]_5http://google.com
 4/12/2018 11:01:08 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:08 PM 10 Message sending : => [email protected]_6http://google.com
 4/12/2018 11:01:08 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:08 PM 10 Message sending : => [email protected]_7http://google.com
 4/12/2018 11:01:08 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:08 PM 10 Message sending : => [email protected]_8http://google.com
 4/12/2018 11:01:08 PM 10 Sending mail : [email protected]
 4/12/2018 11:01:08 PM 10 Message sending : => [email protected]_9http://google.com
 4/12/2018 11:01:09 PM 10 Message sent : => [email protected]_9http://google.com
 4/12/2018 11:01:10 PM 10 Message sent : => [email protected]_7http://google.com
 4/12/2018 11:01:10 PM 10 Message sent : => [email protected]_8http://google.com
 4/12/2018 11:01:11 PM 10 Message sent : => [email protected]_5http://google.com
 4/12/2018 11:01:12 PM 10 Message sent : => [email protected]_6http://google.com

• Johan Herstad: Really enjoyed reading your solutions. Thanks alot for your feedback! I will start to rewrite my solution on monday

Answer 2

 using (var client = new SmtpClient())
 {
 await client.ConnectAsync("smtp.WebSiteLive.net", 587, SecureSocketOptions.None);
 client.AuthenticationMechanisms.Remove("XOAUTH2"); // Must be removed for Gmail SMTP
 await client.AuthenticateAsync(_options.Option1, _options.Option2);
 ...
 await client.DisconnectAsync(true);
 } 

Look at how much work you're doing per message. If you reuse the client, you save setup and teardown time on every message after the first one.

If you reuse the client then you'll need to handle errors and be prepared to close it, open a new one, and continue where you left off; so there is a speed-complexity trade-off.

• Johan Herstad: good point! I could maybe loop the emailmessages inside the client call and use the sendmessage there for each one.

Answer 3

string emailList;
List<string> list = emailList.Split(',').ToList();
//removes whitespace in list:
for (int i = 0; i < list.Count(); i++)
{
 list[i] = string.Join("", list[i].Split(default(string[]),
 StringSplitOptions.RemoveEmptyEntries));
}

This is a really creative way of removing whitespace. But one could just use Trim:

emailList.Split(',').Select(x => x.Trim()).ToList();

---

string emailList;
List<string> list = emailList.Split(',').ToList();

This naming convention however isn't so inventive anymore. You name the emailString as emailList but use list instead of emails for a list of emails. Exactly the other way round.

• Johan Herstad: Thanks for feedback! I knew about Trim, just couldn't find a way to use it. I feel dumb not using a more simple solution. You are right about the naming, I will implement that too!