-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Subscribe to a Topic, and then Batch the messages #1826
-
On our system, we use MQTT.js in our service.
We subscribe to a shared topic (MQTT Broker is VerneMQ).
Then on each message, we add it to a Queue/Messaging Service (BullMQ Pro and Redis)
What I was wondering if the following is possible:
When we ingest the messages, and process them, instead of adding each message to the BullMQ Queue separately, I was wondering, if there is a way to ingest the messages, check one thing (Topic needs to contain string v1.0), and then create an Array, and use the Bulk Add Jobs Functionality for BullMQ Pro.
So processing each message, and then bulk add all of them in one command.
We want to receive messages, for like 50,000 Devices, which send a message every 10s.
So for 50,000 Devices, it could mean 50,000 Messages in one second.
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 2 comments 1 reply
-
I was wondering, if there is a way to ingest the messages, check one thing (Topic needs to contain string v1.0), and then create an Array, and use the Bulk Add Jobs Functionality for BullMQ Pro.
@wernermorgenstern Sorry but It's not very clear to me what you would like be able to do on MQTTjs side.
Beta Was this translation helpful? Give feedback.
All reactions
-
What I am asking, if there is a way in the Handler Function, to process messages individually, and then collect them together, and then run one command, when a certain Threshold is reached, to act on a Group of processed messages.
Beta Was this translation helpful? Give feedback.
All reactions
-
just create an array, push messages into it when they arrive and then when the array size you want is reached trigger the command?
Beta Was this translation helpful? Give feedback.