How to queue background tasks in ASP.NET Web API

Question

I have a webapi that is designed to process reports in a queue fashion. The steps the application takes are as follows:

• Receive content
• Map the content to an object and place it into the queue
• Poll for pending items in the queue
• Process items in queue one at a time

I was thinking to use Entity Framework to create a database of queued items, such as:

public class EFBatchItem
{
 [Key]
 public string BatchId { get; set; }
 public DateTime DateCreated { get; set; }
 public DateTime DateCompleted { get; set; }
 public string BatchItem { get; set; }
 public BatchStatus Status { get; set; }
}

My question - Is there a more efficient way, using NServiceBus, BlockingCollection or ConcurrentQeueue, than to constantly poll the database and pull out pending items one by one? I have not used queues before.

One thought is to create a queue of tasks, and on a separate thread process all pending tasks. Somewhat similar to Most efficient way to process a queue with threads but I want to ensure that I am going the most efficient route.

EDIT: A big question I have here is the best way to display the progress to the user. Once the user submits content, he gets taken to a new page and can view the status by the batch identifier. Is MSMQ necessary, or NServiceBus, in order to notify the user? This seems like a variation of the REquest/Acknowledge/Push paradigm?

• efkah: my solution would be a lock statement around the part of code where you write to the database.
• appsecguy: Thanks for the tip - hadn't thought of that, and would probably have caused a lot of issues later on :) Any thoughts on the method of queuing? Should I bother trying a BlockingCollection, or just the constant database polling? Updated the question with a quick edit

Answer 1

IMHO, your ASP.NET Web API application shouldn't run those background tasks by itself. It should only be responsible to receive the request, stick it inside the queue (as you indicated) and return the response indicating the success or failure of the received message. You can use variety of messaging systems such as RabbitMQ for this approach.

As for the notification, you have a few options. You can have an endpoint which the client can check whether the processing is completed or not. Alternatively, you could provide a streaming API endpoint which your client can subscribe to. This way, the client doesn't have to poll the server; the server can notify the clients which are connected. ASP.NET Web API has a great way of doing this. The following blog post explains how:

• Native HTML5 push notifications with ASP.NET Web API and Knockout.js

You can also consider SignalR for this type of server to client notifications.

The reason why background tasks are hard for ASP.NET Web API application is that you're responsible to keep the AppDomain alive. This is a hassle especially when you are hosting under IIS. The following blog posts explains really good what I mean:

• Returning Early from ASP.NET Requests
• The Dangers of Implementing Recurring Background Tasks In ASP.NET

• appsecguy: Well, the background task is essentially running several ASP.NET functions to generate a report automatically. Would you then recommend a separate application, and to pass the content to that?
• tugberk: @pendraggon87 what do u mean by "ASP.NET functions"?
• appsecguy: Essentially, I am receiving content, and then using the Interop libraries to generate a word document. I had made a web application that allowed users to generate the document via a button click after inputting a lot of information, but I am working now on just making a system where I can generate the report by receiving JSON data. So I have all my code written to write the report. Are you recommending I put that code in a separate application?
• tugberk: @pendraggon87 what I would do is that I could get the data through the my ASP.NET Web API application, stick the details into the queue and then let the workers generate the word doc. for notifications, I would pick one of the above described options.
• appsecguy: Thanks! And would you recommend using BackgroundWorker for the workers? (I have not had much experience at all in dealing with queues, etc. I currently just have my EF database of items and using the database as a queue based on which rows have the Pending status.

Answer 2

It's a NuGet package and it's called HangFire - https://github.com/HangfireIO/Hangfire. The tasks persist even beyond apppool recycling.

• Phil: thanks for this link, I've never come across it before but could be just what I need!

Answer 3

I like @Todd's solution. Just for the sake of completeness, but there's another option not mentioned yet:

HostingEnvironment.QueueBackgroundWorkItem(cancellationToken =>
{
 // Some long-running job
});

Note:

"When ASP.NET has to recycle, it will notify the background work (by setting a CancellationToken) and will then wait up to 30 seconds for the work to complete. If the background work doesn’t complete in that time frame, the work will mysteriously disappear."

And avoid using service methods with an injected DbContext here as this won't work.

Sources: MariusSchulz and StephenCleary

Answer 4

If you use Asp.net Core,you can use IHostService.