Designing long running transactions

Question

I have to conduct tasks 1-6 in an orchestrated async manner. every task has a receive queue and respond queue. These tasks are long-running...can take several days to finish. The orchestrator is listening on respond queues and making decisions to send message to the next tasks receive queue.

each task is a rest service call to an external system , receives results and runs some rules to calculate outcome.

Orchestrator{ Task1 Task2 Task3 Task4 Task5 Task6 Success }

Since every tasks happens in an order...this works.

However, i might have a situation that some tasks can happen in parallel . And if those parallel tasks all succeed then i can proceed to success

Orchestrator{ Task1 Task2 Task3 (Upon completion of Task 3. Task 4,5,6 need to execute in parallel) Task4 Task5 Task6 Success }

Is there some open source framework to write this ? i currently use a spring boot based stack.

What's the best way of handling situation like this : some tasks in order and some in parallel . message queues for ordered tasks and a pub/sub for parallel tasks ? what's the best way to aggregate parallel task results (which also can take a while since this is a long running SAGA) ?

Any help would appreciated.

Answer 1

Your requirements are trivially covered by a Durable Execuction based orchestrator like temporal.io. Here is the Java implementation of your use case:

public void parallelTasks() {
 activities.task1();
 activities.task2();
 activities.task3();
 Promise<Void> t4 = Async.procedure(activities::task4);
 Promise<Void> t5 = Async.procedure(activities::task5);
 Promise<Void> t6 = Async.procedure(activities::task6);
 Promise.allOf(t4, t5, t6).get();
}

Note that tasks can be as long as needed, even months. No external queues are required as Temporal uses them internally to invoke the tasks.