Developer’s Guide to NSQ

I have not settled yet with the title of my NSQ reference book. In the process of writing it, these are some articles that I have completed already. (Use #nsqio hashtag on Twitter)

Take note that this is still a work in progress so follow this post for updates!

Meanwhile, this is the tentative Table of Contents (still in flux, may change over time).

Basics

Getting Started

  • Prerequisite
  • Topics and Channels
  • Basic Example
  • Balancing Example
  • Broadcast Example
  • Simulating Consumer Error

Scenarios

One Node

  • Scenario 1 – Run nsqd and nsqadmin on one node
  • Scenario 2 – Scenario 1 + run nsq producer on the same node
  • Scenario 3 – Scenario 2 + run nsq consumer on the same node

Two Nodes

  • Scenario 1 – Run nsqd on one node and nsq producer on another
  • Scenario 2 – Run nsqd on one node and nsqadmin on another
  • Scenario 3 – Run nsqd on one node and nsqlookupd on another

NSQ Scenarios

Assumption: The following scenarios assume you are running nsqd and/or its components on one or more Turnkey Linux VMs (virtual machines) using host-only networking.

Scenario 1: Run nsqd and nsqadmin on one node

Assumption: ip address = 192.168.56.101

1) Run nsqd

nsqd

Notes:

  • by default, nsqd listens on all network interfaces
  • by default, nsqd listens on port 4151 for HTTP clients like nsqadmin

2) Run nsqadmin daemon

  • nsqadmin requires –nsqd-http-address or –lookupd-http-address

nsqadmin --nsqd-http-address=192.168.56.101:4151

3) Run nsqadmin UI

On your Web browser, type

http://192.168.56.101:4171

Advertisements

3 Comments Add yours

  1. lucmichalski says:

    So, what is your take between NSQ, Mangos and IRIS ? How would it be working inside a Kubernetes deployment ?

    My case:
    It is to use the survey pattern with 5 to 10 respondents (cpp or go client
    9, that’s where the raw request from mangos could be useful. Also it would be important to load balance the respondents and get a registry for the upstreams of clients. An interesting case would be to administrate and store with etcd the surveyors/respondents groups in an etcd key/value store (like vulcand the go proxy) and be able to use http_to_nsq to gather the result of respondents with a timeout per routes/endpoints (the surveyor).

    I saw the following interesting repo:
    https://github.com/dahernan/gopherdiscovery
    https://github.com/glycerine/goq
    https://daniel-j-h.github.io/post/distributed-search-nanomsg-bond/ (cpp)

    And, what about using capnproto or flatbuffers working additionally ?

    I am surprised that nobody is moving forward with nanomsg. ^^ What are the key difference with NSQ that would make the surveyor/respondent pattern not suitable with nanomsg ?

    Do you think that the Iris Project will go somewhere with their license terms ?

    Thanks in advance for your insights and your takes on the topic.

    Like

    1. ITJUMPSTART says:

      First and foremost, the choice of NSQ, mangos and Iris is purely personal. Everyone has their own opinion but this is my take.

      From what I see, NSQ, mangos and Iris were all designed with 1) physical/virtual machine in mind, and 2), were designed to operate in a private subnet. As such, you have to take the huge undertaking of making it work in a container or cloud computing environment.

      I am not against Docker, Kubernetes and the like.

      I concur with Matt Jaynes when he said, “At the moment, you need more systems expertise to use Docker, not less” (https://valdhaus.co/writings/docker-misconceptions).

      My take is, use Kubernetes once you reach the limits of physical/virtual machine. But that’s because I personally favor deploying with just Go binaries in a physical/virtual machine. If you program in other languages, that is where the container system shines due to its process and environment isolation.

      As to the survey pattern, if I understand your question correctly, I think the use case that you are asking for is about giving fault-tolerance to a load-balance cluster of servers. That is where Iris request/reply shines. Iris implements Pastry DHT so there is no need for a separate key-value store. The requests/responses are load balanced. If one of the node dies, the Iris overlay network knows it and it would be taken off the list in the cluster. This is useful for stateless servers like Web application servers. So in this case, you need to cluster all the relevant respondent servers, and Iris would take care of it automatically (no need for mangos survey pattern). When a new node joins the cluster, the requests will be load balanced. No need for Consul or SWIM protocol like serf.

      As to capnproto or flatbuffers, if you are using mangos, Iris or NSQ, it does not matter what data format you are using since all three expect simply a slice of bytes. It is up to you to serialize and deserialize it.

      Iris, mangos and NSQ are meta-frameworks so they don’t get in the way how you organize your code or workflow.

      To summarize.

      mangos – use this message transport if you want communication with other socket-based programs using direct addressing (IP address)

      Iris – use this message transport if you want communication with clusters of servers but using name-based addressing (like DNS but without DNS)

      NSQ – this is the job queue that is either the ultimate destination of your messages or as an endpoint in your message processing pipeline.

      Iris needs contributions especially production deployment stories, benchmarks and the like. Personally, I’d like Project Iris to take off because this is a very promising project especially about the aspect concerning usability.

      Like

  2. lucmichalski says:

    For the surveyor pattern, I see much more the ability to nest several output result from the same the input… More specifically, in computer vision, you can compare a picture to 3 different visual recognition framework and get a unique json object. Load balancing is not the main idea.

    I saw your file with iris and oxy, maybe to build a middleware with vulcand would a good idea. (https://github.com/vulcand/vulcand-docs/blob/master/documentation/source/middlewares.rst)

    Wait for your book with tremedous pleasure !

    Happy New Year !

    Like

Subjectivity aside, leave a reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s