#ZBUS = MQ + RPC
Zbus strives to make Message Queue and Remote Procedure Call fast, light-weighted and easy to build your own elastic and micro-service oriented bus for many different platforms. Simply put, ZBUS = MQ + RPC.
QQ Discussion: 467741880
Star this project if you like zbus:)
##1. Features
- Fast Message Queue in persisted mode, capable of Unicast, Multicast and Broadcast messaging models.
- Language agnostic RPC support out of box.
- High Availability inside, easy to add more servers.
- Simple HTTP-alike control protocol(Browser direct control), easy to extend clients.
- TCP/HTTP/WebSocket, all in one single port, including monitor, capable of DMZ deployment.
- Multiple platforms support, JAVA/.NET/JavaScript/Python/C_C++/GO/PHP...
- Extremely light-weighted(<400K + a very few dependencies like netty.jar)
##2. Architecture
Producer = Broker = Consumer
###2.1 Topic
###2.2 ConsumeGroup
ConsumeGroup plays an important role of MQ in zbus. It is mainly designed to support different messaging models in applications.
Regarding to a topic, no matter how many consume-groups exists, there is only one copy of messages queue storing messages received from Producer(s), Consumer(s) can read the message out from topic via a single consume-group or multiple ones, Consumers and consume-groups combination helps to build different messaging models, typically including Unicast, Broadcast, and Multicast.
- Unicast -- Consumers share only one consume-group, message in topic is designed to be consumed by only one consumer
- Broadcast -- Consumers use privately owned consume-group, message in topic is then consumed by every consumer
- Multicast -- Multiple groups of Consumers, in each group, consumers share one same consume-group.
###2.3 MqClient
MqClient is a TCP based client to control topic and consume-group in MqServer:
-
produce message to MqServer as Producer role
-
consume message from MqServer as Consumer role
-
declare create/update topic and consume-group, capable of change consuming policy
-
query topic and consume-group
-
remove topic and consume-group
-
empty topic and consume-group
###2.4 MqServer
###2.5 Broker
Producer-Broker-Consumer PBC model is the high level view of a Messaging Queue system. Broker refers to a MQ server, however, from the client point of view, Broker in client is an abstraction of connection(pooling) to MqServer(s).
Broker manages a set of MqClientPools each of which connects to a MqServer in group, and all , creating the RouteTable of topics.
Broker also detects the connectivity of remote MqServers, removes the dead ones.
Broker can work in dynamic or static way
- dynamic, configure with tracker address(or list of trackers)
- static, just manual call addServer.
Broker defers the MqServer selection algorithm to Producer and Consumer, both of which could be configured with special selection algorithm.
BrokerRouteTable
serverTable: serverAddressKey => ServerInfo
topicTable: topicName => [TopicInfo List]
votesTable: serverAddressKey => [Voted TrackerServer List]
+ updateVotes(trackerInfo)
trackerInfo = {
serverAddress: {address: xx, sslEnabled: xxx}
trackedServerList: [{address: xx1, sslEnabled: xxx},{address: xx2, sslEnabled: xxx}]
}
1. for each trackedServer in trackedServerlist:
add(no changes if exists) vote with trackerAddress
2. for each server in votesTable: //remove server not in this tracker's tracking list
votedTrackerSet = votesTable[server]
if server not in trackerInfo.trackedServerList:
votedTrackerSet.remove(trackerAddress)
3. rebuild topicTable
+ addServer(serverInfo)
rebuild topicTable after added of serverInfo
+ removeServer(serverAddress)
rebuild topicTable after removal of serverInfo
##3. Protocol
Common headers
cmd: <cmd>
topic: <topic>
token: [token]
produce
cmd: produce
tag: [tag] //tag of message, used for consume-group filter
body: [body]
consume
cmd: consume
consume_group: [group_name]
consume_window: [window_size] //default to null, means 1
declare
cmd: declare
consume_group: [consume-group name] //short name=> group
topic_mask: [topic mask value]
group_mask: [consume-group mask value]
group_filter: [message filter for group] //filter on message's tag
//locate the group's start point
group_start_copy: [consume-group name] //copy from
group_start_time: [consume-group start time]
group_start_offset: [consume-group start offset]
group_start_msgid: [consume-group start offset's msgid] //validate for offset value
query
cmd: query
topic: [topic] //if not set, result is the server info
consume_group: [consume-group]
remove
cmd: remove
consume_group: [consume-group] //if not set, remove whole topic including groups belonging to the topic
empty
cmd: empty
consume_group: [consume-group] //if not set, empty whole topic including groups belonging to the topic
To be browser friendly, URL request and parameters are parsed if header key-value not populated, however, header key-value always take precedence over URL parse result.
URL Pattern: /<cmd>/[topic]/[group]/[?k=v list]
/produce/topic
/consume/topic/[group]
/declare/topic/[group]
/remove/topic/[group]
/empty/topic/[group]
/query/[topic]/[group] * topic can be optional in order to query whole server
/track_sub
/track_pub
/rpc/topic/method/arg1/arg2.../[?module=xxx&&appid=xxx&&token=xxx] *exception: rpc not follow
Exampels:
http://localhost:15555/consume/MyTopic consume one message from topic=MyTopic, consume-group default to same name as topic name
http://localhost:15555/consume/MyTopic/group1 consume one message from group1 in MyTopic
http://localhost:15555/declare/MyTopic declare a topic named MyTopic, consume-group with same MyTopic should be created as well.
http://localhost:15555/declare/MyTopic/group1 declare a topic named MyTopic, and consume-group named group1.
http://localhost:15555/declare/MyTopic/group1?group_filter=abc&&group_mask=16 same as above, but with consume-group filter set to abc, consume-group mask set to 16
http://localhost:15555/query/MyTopic query topic info named MyTopic
http://localhost:15555/query/MyTopic/group1 query consume-group info named group1 in MyTopic
http://localhost:15555/remove/MyTopic remove topic named MyTopic, which will remove all the consume-groups included
http://localhost:15555/remove/MyTopic/group1 remove consume-group named group1 in MyTopic
http://localhost:15555/rpc/myrpc/plus/1/2 invoke remote method plus with parameter 1 and 2, remote service registered as myrpc topic
##4. Monitoring