Retry Architecture

Question

This has been on my mind as I developed several applications with this feature.

Suppose that an application is required to process incoming requests in an asynchronous manner. Take the example of a notification system (other agents will submit a request for the notification to notify people via email or text, etc.) In this case this application requires to call other external systems (smtp server in our example). These external systems might be down temporarily so a retry mechanism is required (up to a certain number of retries).

There are libraries that offer a way for retry, such as Polly. The idea is that the application will retry X times with D delay. But the problem with this is that the request processing is done in memory throughout the retry process, making it resource inefficient.

What would be a plausible pattern for this sort of problems? What are some considerations or platform I should look into? What did you do when you faced similar problem?

Every time I faced this problem, I solved it with a table that contains the tasks that need to be processed. I process them in batches, and update their statuses (NEW, IN_PROGRESS, ERROR). This mechanism is good for having one instance, but once I have multiple instances, then locking the table is necessary so that no two instances process the same request. It seems that there is a better solution for this problem.

• Robert Harvey: Use a fresh copy of the table for each instance.
• Jules: You seemto be trying to solve the problem that message-queue middleware solves, e.g. ActiveMQ. Have you looked into systems like that?
• Husain: @RobertHarvey Could you please explain what you mean? If I use a fresh copy, would not that mean that I will process the same request twice?
• Robert Harvey: If you use a separate list for each instance/request, you won't have threading problems, since each thread will only touch its own list.
• Laiv: If I try message X from the queue and it fails, where do I put it given that there are more messages that are coming in. Into different queue; from where a scheduled jobs will withdraw the messages. Looks like you are thinking in a single table, queue and process. You can have many of them as you need it. Having a single table/queue/job is like having a whole dining room eating from the same plate.

Answer 1

In your example, if sending the notification fails, and I didn't want to use a third-party library to retry sending it, I'd add it to a queue with the time it should be retried.

You could then initiate a retry of all the queued notifications in a couple ways. You could have a single thread which periodically polls the queue for any waiting tasks. If there are any then it checks if it is time to process it and if so then dequeues it and processes it. Rather than periodically polling, when you add a notification to the retry queue you could configure the thread to poll after the minimum retry time of all the notifications in the queue.

You would have to lock on the queue because you're queueing notifications and iterating through the queue possibly concurrently from different threads.

Lots of languages provide some sort of scheduled event construct you could use as well, and that'd be my first choice if you can find one. I know you can use a scheduled ExecutorService in Java, but . NET likely has something similar.

If your notifications are particularly large and disk bandwidth isn't a primary concern for you then you could write the notification content to a file or database entry and just keep a reference to that file or database entry in the queue.

• Husain: Thanks you very much @max for the elaborate answer. It helps much than a small comment. But here is my problem which I want to solve. If I am following your reasoning here, once the retry instance takes the task from the queue, the task is poped from the queue. So, If I come and turn off the retry instance, the task from the queue is lost. Is there a feature on queue that keep the message until processed or something like that? Is there a way to make sure even the task in process is not lost?
• Max: @Husain You can use the peek method in C# to get the next element in the queue without removing it. You could then remove the task from the queue only after it is cancelled or completes. But it is probably better to retry multiple tasks concurrently, in which case you can iterate through the queue. Then like before only remove the element from the queue once it finishes. Maybe you change the state of the notification to "pending" while it is being processed.
• Max: I'm glad the answer helps. Feel free to accept it as the answer since it helped and there are no other answers! Thanks.