Notes on NSQ and Project Iris

In preparation for publishing my book on Project Iris and NSQ, here are some notes as I gather more research along the way. Stay tuned.

But first, here is the rationale:

Amidst the proliferation of microservices frameworks and APIs, I would like to present the case of messaging as the foundation for building a microservice-based architecture. No API sprawl. Just protocols.

Why messaging?

First, messaging provides patterns (or topologies) beyond request/reply (the most familiar example being the Web).

Second, messaging enables the ability to retry processing brought about by properties of distributed computing (read the fallacies).

Once you understand those two reasons behind messaging, it will dawn on you that the messaging-based approach follows a unidirectional computing pipeline. That is, messages may branch out in accordance with its processing but the business process workflow generally flows like a pipeline.

In computing, we are taught about I-P-O (input-process-output).

In practice, it is more about I-P-O-R (input-process-output-routing).

In this line of thinking, you sort out asynchronous operations from the synchronous ones. In the case of Web applications, you can still use whatever language or software you are using. The only difference this time is, once you figured out the asynchronous operations, you can feed those async ops to a message queue so you don’t have to reinvent those retry/backoff procedures.

To illustrate, let’s say you have version 1 of your business API. So you design those API endpoints on your Web app. With a messaging-based approach, those API endpoints become topics on a message queue. You still code the synchronous operations within the context of your Web app, but now, you offload the asynchronous operations to a message queue. The message queue has one or more message consumers (or workers) which you can scale depending on requirements.

So once a message consumer is done with the computing (I-P-O), the optional result can be routed to another message queue (an endpoint) which forms the chain of computing according to your business logic.

This is the essence of pipeline computing.

In the book, I use Project Iris as message transport and NSQ as message queue. In most cases, Iris can replicate the messaging patterns available with Scalability Protocols (although SP has the advantage of direct communication via IP addresses).

However, Iris has the unprecedented notion of name-based clustering (without using DNS) so this is where it really shines. You will get a taste of how really easy it is to setup a cluster using Iris. There is no need for etcd, Consul or Zookeeper. On the other hand, NSQ has built-in service discovery so the bulk of your code resides among message queue consumers.

Project Iris

  • Iris clients always connect to a local Iris daemon
  • Broadcast, Request and Tunnel all require a cluster name parameter
  • There are only three scenarios or topologies when setting up an Iris cluster. A node refers to a physical or virtual machine. Your cluster can then be a combination of any of these topologies. Of course, a used port cannot be reused

Scenario 1 (one node – one cluster)

iris -net accounting -rsa /home/id_rsa

Scenario 2 (one node – one cluster – two or more different ports)

iris -net accounting -rsa /home/id_rsa

iris -net accounting -rsa /home/id_rsa -port=44444

Scenario 3 (one cluster – two or more nodes)

iris -net accounting -rsa /home/id_rsa

iris -net accounting -rsa /home/id_rsa



Subjectivity aside, leave a reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s