Roberto Ostinelli's Blog

There are different reasons why you might consider having multiple databases in your Ruby on Rails application. In my specific case scenario, I needed to store large quantities of data representing user behavior: clicks, pages visited, historical changes, and so on.

This kind of databases generally are not mission critical, and grow much faster (and larger) than most databases. Their requirements are often different: for instance, they need more storage space, are more tolerant in the face of hardware or software failures, and are write-intensive. For these reasons, sometimes it is interesting to separate them from your application’s primary database. Often, non-RDBMS databases are chosen for these kind of tasks, something which is however beyond the scope of this article.

I googled and read many different solutions, however I couldn’t find one that was able to fully cover how to:

Have different and isolated migrations and schemas for every database.

Use rails generators to create new migrations for every database, independently.

Offer database-specific rake tasks for the most common database operations (i.e. like the ones available for the primary database).

Due to my personal interests and history, I often find myself building applications in field of the Internet Of Things. Most of the times I end up using Erlang: it is based on the Actor’s Model and is an ideological (and practical) perfect match to manage IoT interactions.

I recently built an application where devices can connect to, and interact with each other. Every device is identified via a unique ID (its serial number) and based on this ID the devices can send and receive messages. Nothing new here: it’s a standard messaging platform, which supports a custom protocol.

Due to the large amount of devices that I needed to support, this application runs on a cluster of Erlang nodes. Once a device connects to one of those nodes, the related TCP socket events are handled by a process running on that node. To send a message to a specific device, you send a message to the process that handles the devices’s TCP socket.

While building this application, I was early in the process faced with a very common problem: I needed a global process registry that would allow me to globally register a process based on its serial number, so that messages can be sent from anywhere in the cluster. This registry would need to have the following main characteristics:

Distributed.

Fast write speeds (>10,000 / sec).

Handle naming conflict resolution.

Allow for adding/removal of nodes.

Therefore I started to search for possible solutions (which included posting to the Erlang Questions mailing list), and these came out as my options:

I have been developing Rails JSON API applications for quite some time now, and I’d like to share a few of my setups and discuss why I do things this way. I’m starting today a series of articles that will cover up pretty much the steps I take every time I bootstrap a new Rails JSON API application.

One of the first things I do is to ensure I’m optimizing Rails for speed. I basically optimize the framework itself, prior coding any specific application logic.

You may have heard before that “Premature optimization is the root of all evil“. However, “Premature optimization is a phrase used to describe a situation where a programmer lets performance considerations affect the design of a piece of code”, which “can result in a design that is not as clean as it could have been or code that is incorrect, because the code is complicated by the optimization and the programmer is distracted by optimizing” (source: WikiPedia). This is not what we’re doing here: we’re just going to apply a few changes to Rails, and then basically forget about those and start coding in a framework that is optimized to serve our API.

Many of Rails functionalities are simply not needed when building an API server, and by stripping down Rails to a bare minimum we can actually achieve pretty significant performance increases.

GIN is a fast, low-latency, low-memory footprint, web JSON-API framework with Test Driven Development helpers and patterns. It is a framework that I open sourced some time ago, as an experiment.

It all started when I heard so many good things about Lua that I wanted to see it in action and find a project where I could unleash its power. Being an API fan, it came natural for me to build a JSON-API server framework. And GIN was born.

GIN is currently in its early stage, but it already enables fast development, TDD and ease of maintenance. It is helpful when you need an extra-boost in performance and scalability, since it is entirely written in Lua and it runs embedded in a packaged version of nginx called OpenResty. For those not familiar with Lua, don’t let that scare you away: Lua is really easy to use, very fast and simple to get started with.

Controllers

The syntax of a controller is extremely simple. For instance, a simple controller that returns an application’s version information looks like:

As some of you already know, I’m the author of Misultin, an Erlang HTTP lightweight server library. I’m interested in HTTP servers, I spend quite some time trying them out and am always interested in comparing them from different perspectives.

Today I wanted to try the same benchmark against various HTTP server libraries:

I’ve chosen these libraries because they are the ones which currently interest me the most. Misultin, obviously since I wrote it; Mochiweb, since it’s a very solid library widely used in production (afaik it has been used or is still used to empower the Facebook Chat, amongst other things); Cowboy, a newly born lib whose programmer is very active in the Erlang community; NodeJS, since bringing javascript to the backend has opened up a new whole world of possibilities (code reusable in frontend, ease of access to various programmers,…); and finally, Tornadoweb, since Python still remains one of my favourites languages out there, and Tornadoweb has been excelling in loads of benchmarks and in production, empowering FriendFeed.

Basically, websockets allow a two-way asynchronous communication between browser and servers, filling the gap that some technologies such as ajax and comet have tried to fulfill in these recent years. If you want to try this out yourself, you will first need to grab a browser which implements websockets, such as Google Chrome.

Message passing between Erlang nodes is considerably slower than within the same node. This is normal, and is due to the fact that messages sent between nodes are actually copied from the area of the sender to that of the receiver, then sent over from one node to the other via TCP/IP.

I was getting interested when a server could achieve amazing performance over a single node, performance which got much lower once the server got distributed over two Erlang nodes. I therefore tried a small benchmark test to somehow measure the difference in message passing speed within/across Erlang nodes.

A Technical Introduction to OpenSpime

As early as 1968, influenced by visionary movies such as Stanley Kubrick’s 2001: A Space Odyssey, with its fictional computer character HAL 9000, we envisioned omniscient, intelligent machines that could easily contain the whole range of human knowledge. Then, in 1987, the Knowledge Navigator video from Apple Computer struck our imaginations and greatly boosted our expectations of what computers would be able to do for us in a few years’ time.

It is now 20 years later, but technology is still very far from fulfilling such hopes regarding virtual assistants:

Voice recognition isn’t immediate.

People adapt to machines’ lexicons rather than the other way around.

Computers cannot easily transcend the scope for which they have been programmed.

Self-learning machines are still experimental.

The generation of high-quality graphics representing emotionally rich, 3-D avatars is far from being real time, cheap, or extensively available.

The technology for fulfilling such high expectations simply does not yet exist. However, while we wait for technology to catch up with our dreams, the gap between our expectations and the reality of what we can now achieve has inspired two important trends in the development of virtual assistants: