Aggregating events in event-driven system

Question

Lets say there are two services. One generates event A and the other event B. We need to build a new service that implements the function C = func(A, B) , which produces the result C.

But as A and B are asynchronous events, they may reach (and invoke) the new service separately. Depending on what is required, it may be apt for the new service to:

1. either wait for both events,
2. or implement different variations of the function: C = func(A,C') or C = func(B,C') where C' is the previous state of C (assuming that it is even possible to build that function)
3. or query back to retrieve the complimentary event, e.g. if A invoked the service, then query for B.

Question:

Lets say the requirement is that we update the state of C ASAP, and A and B have a significant time gap in occurrence - that rules out option #1.

Additionally assume it is hard/impossible to build different variations of the function (#2).

That leaves us with option #3, which probably has drawbacks because it may be breaking the event driven paradigm (or maybe not, I am not sure).

How should we solve this problem? I expected this to be a common problem with a recommended pattern, but I could not find it. Any suggestions or pointers on where I could find the answer, or if the question is wrong, how should I look at it?

Answer 1

Whether it breaks the event-driven paradigm should not be your primary concern. The question is if this solves your business problem.

There are a number of questions that need to be answered:

• If A and B are events, it doesn't really make sense to query them. But I believe you mean they represent some state, and an event is raised when the state changes.

• Will querying service B yield a different result than simply taking the latest received event? In other words, could you simply call func(A, B') where B' comes from a cache? What do you do if the cache is empty? And do you need to invalidate it at some point?

• A query will cause the client of service C to block until service B responds. Will that be a problem? What if service B is unreachable?

• What happens if you receive a new A or B event while you are querying?

These are general problems of asynchronous systems and finding the right trade-offs is very specific to the requirements of your particular application.

Fundamentally, you just can't guarantee to get the most up-to-date state if your state is distributed. That means you either deal with temporary discrepancies (read up on 'eventual consistency') or you find a way to make the relevant parts synchronous (giving up the advantages of the asynchronous approach).

• Avinav: thanks, yes i did assume 'they represent some state, and an event is raised when the state changes', but again that assumption is not really inherent to this problem. I also think this problem is related to event-driven ETL, confluent.io/blog/changing-face-etl but a lot of blogs do not talk about this scenario, where data enrichment or data transformation is from other events. like you said, i have to weight in trade-offs for my specific problem, but I like the idea service C caches only latest A or B, and deal with empty cache.