Before you read this post, I urge you to familiarize yourself with distributed system (if you are a newbie).
- Please stop calling databases CP or AP
- Notes on distributed systems
- You can’t sacrifice partition tolerance
So without further ado, let’s go back to the subject matter of Pastry DHT as implemented in Project Iris.
Pastry DHT in Project Iris solves the same problem of cluster membership just like what gossip protocols do. But instead of gossip communication, Iris maintains the state of the cluster in a distributed hash table. Since Iris is peer-to-peer, the key-value pairs are stored in a redundant fashion so there is no single point of failure.
When one node joins or leaves the cluster, that information is known to the cluster without a need for a separate key-value store. This is how Project Iris lends itself to easy operation. However, there is a caveat. Project Iris at the time of this writing only works with IPv4 networks.
So what happens when a node fails and its underlying data are lost.
It is important to note that Iris is a message transport, and as such, needs to operate as stateless as possible. That is, data in motion needs to be as transient as possible by offloading it to a stateful message queue as fast as possible. In this case, you need not worry about message loss because the message queue is responsible for message persistence and retry.
So it is a matter of separation of concern.
The message queue may even resort to a distributed key value store if the message persistence needs to be replicated as well.
In a sense, this is how your push-based processing must flow:
Message producer –> Message Transport (Iris) –> Message Queue (NSQ) –> Message Consumer.
Note that Iris as message transport is optional. You may also use mangos, Go implementation of Scalable Protocols
You can see more of this in detail at http://itmarketplace.net/messaging.