How do I negotiate messages to avoid errors resulting from duplicate messages?

Question

My system looks the following: enter image description here

I have N message producers, N RabbitMQ instances, N message consumers and 1 service. Producers produce the same messages while consumers consume the same messages. Here is the problem. Consumers must agree on who calls the service API, since calling the service API twice with the same message leads to an error. And I don't know how could consumers decide who calls the service.

I had some ideas. We could create a RabbitMQ cluster (with a plugin that detects duplicates) and unite all the consumers with the cluster. Consumers could send messages to the same queue using "direct" routing which will lead to "round-robin" message processing. But I'm not sure if there is a reliable and well-tested plugin that detects duplicated messages in RabbitMQ clusters.

Is there any other technique to solve this problem? I have heard about Apache Camel, but I can't say if it can save my day.

• Robert Harvey: What problem? Is it not scaling in its current form?
• Tony: Yes, I can see two comments made by you. I've edited my question. Please take a look.
• Robert Harvey: If each consumer/producer line is independent, how does a different producer know the message has been duplicated? I'm not sure I quite understand why you're getting duplicate messages.
• Tony: Producers are not independent. They all are connected to the Bitcoin p2p network. So they all have the same stream of messages(blocks).
• Robert Harvey: Does having multiple producer/consumer queues provide any benefits? Or could you get away with just using one? You did say that they each has the same stream of messages anyway.

Answer 1

Your receiver should be idempotent and deal with duplication. Look at Idempotent Receiver Pattern. As you mentioned Apache Camel implement some features to deal with duplication.