Tag: distributed app

I was looking for something new to play with the other day and somehow ended up with the thing called a service mesh. Pretty interesting concept, I can tell you. Not a game changing, or world peace bringing, but still nice intellectual concept with several scenarios where it can make life much simpler. Let’s have a look. Continue reading “Playing with a service mesh”

Much to my surprise, starting from the last week Kubernetes became the part of my job description. It’s no longer something just interesting to try, I actually have to understand it now. And as you probably could tell from my older k8s post, I’m not quite there. The post sort of builds a logical example (containerized web server) but something just doesn’t click.

I was trying to understand what’s missing, and it seems like the problem is in the tooling. You see, there’re two and a half ways to run something in Kubernetes. One is through ad-hoc commands, like kubectl run or kubectl expose. They are simple, but they also skip few important concepts happening in the background, so the whole picture stays unclear. Continue reading “Dissecting Kubernetes example”

Imagine your distributed app has two kinds of services: web and db. Both of them are replicated for higher availability, live on different hosts, go online and offline whenever they like. So, here’s a question: how do web‘s find db‘s?

Obvious solution would be to come up with some sort of reliable key-value storage, and whenever service comes online, it would register itself with the address in the store. But what happens when service goes offline? It probably could notify the store just before that, but c’mon, it’s internet: things can go offline without any warning. OK, then we could implement some sort of service health checks to ensure that they are still available… By the way, did you notice how quickly the simple idea of using external store for service discovery started to become a reasonably large infrastructure project?

How do you usually configure an app? Over the decades our industry came up with multiple approaches, like providing command line arguments, various config files, registry settings and environment variables. Even hardcoding certain options into the app itself also works sometimes. Whenever we need to reconfigure the app we’d just go to its host, change a setting or two, and the job is done.

And now imagine a challenge: you have a cluster of services that come and go, they have different versions and locations, and you need to reconfigure all of them. How would you do that?

What is Apache Kafka

Official definition of Apache Kafka is distributed streaming platform, which starts to make sense only after reading at least few chapters of its documentation. However, idea behind it is relatively simple. In large distributed apps we have many services that produce messages: logs, monitoring events, audit entries – any type of records. On the other hand there’s similar amount of services that consume that data. Kafka brings these parties together: it accepts data from producers, reliably stores it in topics and allows consumers to subscribe to them. In other words, Kafka is a love child of distributed storage and messaging system.

As I promised last time, it’s time to check out RabbitMQ feature we can consider advanced – clustering. RabbitMQ cluster is a set of individual nodes that share the same users, queues, exchanges and runtime parameters. New nodes can come and go, be located at different continents, yet for the connected client they will look like one entity.

Clustering is not the same as replication or high availability. Yes, users and whatever is usually necessary for node to work will be copied across all nodes. Queues, however, will reside on the node they were initially created, though they will be accessible from any node. If one node goes down, its queues go with him.

RabbitMQ is an example of full blown Message Queue that somehow remained simple to use. Unlike ZeroMQ, which is embeddable into the services that use it, RabbitMQ is a broker. It’s an intermediary messaging service with own users, permissions, encryption, configurable durability and delivery acknowledgements, clustering, high availability, and bazillion of other features you might never need. RabbitMQ is built on top of Erlang and inherits its known resilience with compatibility to virtually any OS.

In the following article we’ll try to get a sense of how messaging with RabbitMQ feels like. I’ve chosen Ubuntu (in a Docker container) as a platform, but it could’ve been anything else. Continue reading “Quick intro to RabbitMQ”

Last time we built three client-server Node.js apps that were talking to each other using ZeroMQ. However, running both client and server on localhost is a little bit lame. Let’s put them into containers! They’ll still be lame, but now with Docker around.

So, here’s the plan: let’s see what we need to do to last week’s fire-and-forget ZeroMQ app, so its client and server can work and communicate from within Docker containers. Continue reading “Using ZeroMQ with Docker”

ZeroMQ is small, fast and very easy to use messaging library, which works equally well both within the same process and over the network. Despite being written in C++, it has bindings for most of the languages you can come up with. And it’s free. Hurrah!