A Tale of Two Processing Modes

There are only two types of processing modes.

  1. Push-based (aka stream-based) processing – Websockets, NSQ, ReactiveX (Kefir.js, etc), Apache Storm, etc
  2. Pull-based (request/reply) processing – the Web is the most popular example of request/reply pattern

Push-based processing

It is important to note that push-based processing can be further subdivided into two:

  • real-time push-based processing – WebSockets, Apache Storm, Apache Flink
  • batch push-based processing – Hadoop

Pull-based processing

  • HTTP
  • RESTful
  • RPC

I don’t know where you stand regarding the RPC debate but there is one main issue to be addressed:

How do you cope with message failure?

With request/reply, there is a timeout mechanism so the requester will be notified in case something had gone wrong on the reply side of the communication.

With push-based processing, there must be a mechanism to persist the failed message (in memory or on disk) in such a way that the process can be retried.

This is the essence of distributed computing.

Once you understand this, it will dawn on you that at-least-once message delivery now makes sense.

Advertisements

One Comment Add yours

Subjectivity aside, leave a reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s