A blog about technology and business from the perspective of a software engineer/consultant/exec in a small, long-standing tech firm.
A blog also about the trials and tribulations of making my way through the world of graphs and graph-driven databases. And any other technologies I feel like discussing. Because that's just how I roll.
Let's all learn something here!

Wednesday, 15 February 2012

On the Subject of NoSQL (and a bit about graph databases)

Pretty formal-sounding title, yeah?(I'm likely just suffering from title-writers' block.)So, before I dive head-long into an actual graph database, it's probably a good idea to briefly discuss what makes a graph database a graph database.RDBMSFor the past 30+ years, the world of databases has been primarily dominated by the colossus that is relational databases (i.e. RDBMS, or Relational DataBase Management System). RDBMSes are well-known and well-studied, but it suffices to say that they can be used to model almost any kind of information (that is, RDBMS can be used to model a broad, general set of data).I'm going to assume that the reader is familiar with basic RDBMS concepts like columns, rows and tables. Information in an RDBMS is grouped into similar entities that can have relationships between them. In this way, we can model just about any situation in this single kind of database. For example, you can model everything from a school (teachers, classrooms, students, schedules, etc.) to an online business (products, orders, inventory, etc.).Because each table consists of rows and each row is made up of columns (columns representing the types of information you want to store), we know what kind of data to expect in each table and database based on its schema. This is great for ensuring that you don't try to save a product's name when it's expecting the product's price.RDBMSes are also good for both looking up information as well as handling the storage of data. The concept of transactions is important; just as businesses need to deal with transactions every day, so to does a database that is used to enter in data of a transactional nature (e.g. payments, orders).Another important feature of any RDBMS is the ability to query the data. SQL (Structured Querying Language) is almost as old as RDBMSes themselves and is a powerful tool for looking up and modifying data in an RDBMS. It might not be the most efficient tool on its own given the potential complexities of how a specific database is laid out (see: query analyzers and optimizers), but it can be tuned to be a very powerful tool with the right indices setup and the right query. (SQL can also be prepared/compiled in some cases, but we won't get into that.)So why use one of these so-called NoSQL databases?NoSQLWithout going into too much detail about NoSQL (that's another post or ten on its own), NoSQL is really better off being called NoREL (i.e. non-relational model, though that's more in the traditional RDBMS sense).Whereas RDBMSes are geared towards being a general solution for almost any model, it does everything fairly well but requires specialized tuning in order to make it run really well, and it comes at the expense of some areas being better and others becoming "worse" as a result. For example, it's difficult to make a database that is tuned for efficiently handling a high volume of transactions be efficient at also handling speedy look-ups and reads for data.NoSQL databases provide specializations that RDBMS systems can't provide. They come out of the box ready to be super-good at one or two areas, but not to so great in others. Think of them as pre-tuned databases, and one size definitely does not fit all.NoSQL databases range in types from document-stores (e.g. CouchDB) to key-value stores (e.g. MongoDB) to graph databases (e.g. Neo4j), to name but a few types. (A decent breakdown of NoSQL database types can be found here and here.)You could likely tune an RDBMS to be quite good at a number of things, but it's a bit of a pain (trying to setup and tune proper indices is a painful and involved process).This is partly why I disagree wholly with those who say NoSQL heralds the death of RDBMS. Quite the contrary: I see NoSQL databases as being an excellent complement to RDBMS.Another advantage of NoSQL databases is the fact that most of them are schema-less; that is, they can store arbitrary information without the need to structure it. This can be very powerful when modelling heterogeneous information in a database. It can also allow for the evolution of your data models without the need to completely overhaul and change your database (anyone who's tried to do that before with an in-production database knows just how painful that is).NoSQL databases also tend to scale very, very well, often times easier than clustering together an RDBMS-based solution.One point worth mentioning is the fact that RDBMSes are traditionally known as what is called ACID compliant. ACID (which stands for Atomicity, Consistency, Integrity, Durability) is a very important concept for databases that handle transactional information (and most businesses do). A discussion on ACID is well outside the scope of this post, but, ACID compliancy is commonly lacking in most NoSQL databases (most of them subscribe to the principle of Eventual Consistency). This is definitely worth noting and keeping in mind when choosing a NoSQL database to use. A great example of an exception to this is the fact that Neo4j (a graph database) is, in fact, ACID compliant.(If anyone wants an actual discussion on ACID vs. Eventual Consistency and why it's important, let me know and I'll see about putting an article together.)Graph DatabasesAs far as graph databases are concerned (took me long enough to get here), I strongly suggest going to this link and checking out "What is a graph database?" and "Comparing Neo4j" tabs on the page. It does a wonderful job of explaining what a graph database actually is (big surprise) and how graph databases relate to other types of databases (both NoSQL and RDBMS).While you do need to typically index nodes and relationships for searching (e.g. full text searches over properties), strictly speaking a graph database is one that provides "index-free adjancency" (source: http://en.wikipedia.org/wiki/Graph_database). This means that each element (i.e. node) has a link to its adjacent elements to follow--no index look-ups are necessary.Graph databases are a great way of representing graphs (remember those things with nodes and relationships?). Graphs in a graph database typically consist of arbitrary nodes and relationships. Each node and relationship can have assigned to it an arbitrary number of properties. Properties are simply key-value pairs of information (e.g. "Name" = "Joe" and "Age" = 30).So you can easily represent a family tree, a network diagram, or even a social network with a graph database. For example, think of two nodes, each one representing a friend at work (call them Jason and Scott), and a relationship between them (representing their friendship). So, each node would have properties like, "Name"="Jason", "Age"=30, etc. The relationship between them could be labelled "KNOWS", and that relationship could have properties like "Since"="01/01/2001" and "At"="Acme Inc.". All of a sudden, we now have a way to track friends, find out who knows whom, when they met, and where they met.Now let's say somewhere down the line we learn something else about each person; say, Jason's birthday. It's very easy to add a new property to Jason's node.We begin to see the power of graph databases very quickly.We can use graph databases in many ways, including (but definitely not limited to):

Recommend products to buy based on a user's purchase history (follow a graph from a product someone has bought back through another user that's bought the same product and then on to another product that other user has also purchased).

Find out just how popular someone is (look at the number of relationships that person's node has).

See what geographic locations have the most users in it.

Find out if you know the CEO at a powerful company through a friend (you can always use more friends!).

This is why sites like Amazon and LinkedIn are so powerful. Think about how they might use a graph database.

Ok, that's enough for now. I keep thinking I can write short posts on this stuff, but, there's just so much!

As always, I'm sure there's plenty more I haven't covered, but if you'd like to see anything else put up here (or clarified), just let me know. I'm always looking for ways to improve how I organize and present this information.