NSQ Tutorial

I am looking for NSQ tutorial beyond what Traun Leyden and Guillaume Charmes have provided in their respective posts. In particular, I am looking for a way to decouple nsqd, nsqlookupd, nsq producer, nsq consumer. After a few days wrestling with the documentation and a few Turnkey Linux virtual machines (using only VirtualBox, no Vagrant…

The Myth of the Full Stack Developer

I don’t need to rehash what Andy Shora had already posted in his blog. What I am about to discuss here is that, a full-stack developer is a unicorn. Instead, we must accept the reality that in IT, it is either you are a developer or in operations. Further down the line, a developer can…

Web Services vs Microservices

Microservices start where Web services where end. Web services Web services are meant to be exposed to the public interactive request/reply (pull-based). However, it can be also push-based (e.g. Web sockets) synchronous long-lived process (HTTP-based) transactional user-facing aka user response loop transactions are meant to be processed by an RDBMS (think of ATM transaction) Microservices…

Go Developer’s Guide to Distributed System

The IrisMQ Guide is now available at github.com/irismq/book Iris is peer-to-peer messaging transport software (with built-in clustering) that was developed pre-Docker era. (Courtesy: Dan Frank)

Distributed System Checklist

Concerns you need to address in distributed systems Distributed Computing Distribution (physical machine/ VM / container) Security Network Storage Messaging (synchronous request/reply vs asynchronous Scalable Protocols) Clustering (IP-address-based vs name-based) Timing (there is no now) Data format (JSON, Protocol Buffers, etc) Consistency (strong vs eventual) Service discovery (nsqlookupd) Fault-tolerance (rate limiting, circuit breaking, etc) Operations…

Notes on NSQ and Project Iris

In preparation for publishing my book on Project Iris and NSQ, here are some notes as I gather more research along the way. Stay tuned. But first, here is the rationale: Amidst the proliferation of microservices frameworks and APIs, I would like to present the case of messaging as the foundation for building a microservice-based…

The Road to Computing

The road to computing is paved with dangers. Beware of pitfalls, potholes, ravines and disasters ahead. Even sacred cows! Class-based OOP The pitfalls of class-based OOP as mentioned by Eric Elliott does not apply merely to JavaScript but to all programming languages which support that paradigm. Golang eschew it in favor of interface-based composition. You may…

Instance-based vs Name-based Clustering

There are two main methods on how to setup a cluster. First is old-school IP address-based (or instance-based) clustering. You can see it almost everywhere from HAProxy, MySQL replication, Caddy proxy, nginx, etc. At least with Consul, an agent only needs to learn about one existing member. Second is name-based clustering which is being used by Project Iris….

Why Pastry DHT?

Before you read this post, I urge you to familiarize yourself with distributed system (if you are a newbie). Please stop calling databases CP or AP Notes on distributed systems You can’t sacrifice partition tolerance So without further ado, let’s go back to the subject matter of Pastry DHT as implemented in Project Iris. Pastry…

A Tale of Two Processing Modes

There are only two types of processing modes. Push-based (aka stream-based) processing – Websockets, NSQ, ReactiveX (Kefir.js, etc), Apache Storm, etc Pull-based (request/reply) processing – the Web is the most popular example of request/reply pattern Push-based processing publish/subscribe broadcast anycast unicast observable message queue message-driven It is important to note that push-based processing can be…

Message Producer and Consumer

The following terms are synonymous with message producer and consumer. Message producer – sender, upstream client Message consumer – receiver, downstream client  

Message Delivery and Idempotency

With NSQ, the messaging pipeline goes like this: Message producer —> NSQ —> Message consumer Message producer The message producer and consumer can be written in any programming language that is supported by NSQ. Once a producer posted its message to NSQ, it is the responsibility of NSQ to deliver the message at least once….

Reverse Proxy using Project Iris

Project Iris uses Pastry DHT so you don’t need a separate key-value store. Node membership and load balancing are automatically handled. In the following example, I used vulcand/oxy to build a reverse proxy backed by Project Iris so you can cluster your Web app servers in a seamless way. No restart of reverse proxy is…

Addressing Message Loss in NSQ

While reading NSQ design, it says no SPOF but actually, it is referring to no single point of failure on the NSQ consumer side (not the nsqd itself). Here is one problem with nsqd: This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process….

Messaging Patterns in Go

In a bid to popularize messaging-based approach to building microservices, I am launching Messaging Patterns in Go. EventBus – use this for intra-process programs (that is, those standalone programs that do not communicate with other programs) as alternative to using channels. If you are familiar with Node.js, you will feel at home using EventBus but…

Developer’s Guide to NSQ

I have not settled yet with the title of my NSQ reference book. In the process of writing it, these are some articles that I have completed already. (Use #nsqio hashtag on Twitter) Take note that this is still a work in progress so follow this post for updates! Meanwhile, this is the tentative Table…

I’m writing a book about NSQ

Update: Within two months of feverish writing, I have come to the conclusion that writing NSQ apps is relatively easy for the developer. It dawned on me that the real meat of the issue is about distributed system. In short, this project is deprecated in favor of the IrisMQ Guide which is available at github.com/irismq….