Message ReSequencing in a Distributed Publisher-Subscriber System

Message ReSequencing in a Distributed Publisher-Subscriber System

In a typical cloud based distributed environment with many message publishers and subscribers, the message could be processed by any of the subscriber and this is usually not predictable. In certain scenarios, it could be necessary to have a group of messages processed in sequence though they reached the subscribers randomly.

One solution to this is described below. The solutions tries to satisfy the following requirements :

1.) Make sure there is no tie-up / hard-link between subscribers and publishers: Any message can be received by any subscriber and any publisher can push any message.

2.) When the messages are not grouped, the system continues to behave as usual / before.

3.) When group messages are detected at subscriber, change in overall processing time should be minimal.

4.) When group message are detected at subscriber, no blocking operation should be performed and the subscriber should continue to be available for receiving and processing other messages.

Each group message is expected to contain the following items in addition to the message pay load itself :

            a.) Group Message ID.

            b.) Total Number of Messages in group

            c.) Group Message Sequence Number.

e.g.:-

            a.) GROUP_1

            b.) 4 //GROUP_1 contains 4 messages in total

            c.) 2 //this means that this message is 2nd in the group GROUP_1.

This solution employs subscribers working in a distributed cooperative manner. As a group message is received by a subscriber, it queries the distributed hash table to check if any other subscriber is working on the same group number. If yes, the message received is pushed onto that subscriber. (A push endpoint is expected to be available for each subscriber. This list of end points too could be maintained in the distributed hash table indicated earlier.) If there is no entry in the distributed hash table for this group, the subscriber adds itself into the distributed hash linking the message group id with itself.

GroupHash[GROUP_1] = SUBSCRIBER_ID

Additionally, the message received is added into a local data structure/bag of the receiving subscriber.

When a new message is received in the group_message_queue, the following steps are executed by the watcher@subscriber owning the bag.

            a.) check if all messages for the group has been received.

If all messages have been received, the messages are sorted based on the Group_Message_Sequence_Number and processed one after the other or as the logic demands for the group. Distributed hash entry is cleared for this group GroupHash[GROUP_1] = "" once the processing of message group is complete.

The data structure/bag maintained by the subscriber would be typically filled in the following scenarios :

            a.) message pushed from another subscriber.

            b.) message pushed by the local listener since no other subscriber is working on this group.

Partitioning within groups can be employed if required by employing the same strategy for sub-groups. In this case, it could be required by subscriber to wait for the sub-group messages to be processed before the proceeding with the group messages.

The system can be enhanced such that the subscribers internally check the load of active group message subscriber (as pointed by GroupHash) before pushing the message. This way, the subscriber that received the message can take over the ownership of group message voluntarily; especially if the message received was the last of the group message expected, requiring a message process flow.