January 22, 2009

Heterogeneous vs. Homogeneous System Architectures

I follow a certain philosophy when developing system architectures. I assume that very few systems will ever exist in a consistent form for more than a short period of time. What constitutes a “short period of time” differs depending on the specifics of each system, but in an effort to quantify it, I generally find that it falls somewhere between a week and a month.

The driving forces behind the need for an ever changing architecture are largely business requirement based. This is a side effect of the reality that software development, in most cases, is used as a supporting role within the business unit it serves. As business requirements (i.e. additional features, new products, etc.) pour forth, it is the developer’s job to evolve their software system to accommodate these requirements and provide a software based solution to whatever problems lay ahead.

Given that many businesses can be identified as having the above characteristics, I can now begin to explain why I believe that Heterogeneous System Architectures hold a significant advantage over Homogeneous System Architectures, in many distributed system cases.

An Experiential Use Case

I work daily on a mobile platform that mixes web applications, server applications, and XMPP servers together into an architecture that delivers on the business requirements, as set forth by our business development team and clients. This mobile platform is fairly distributed, with web applications on one set of machines, server applications on another, and XMPP servers on yet another. These three components interconnect, with the XMPP protocol and/or a database being the glue that binds them.

It’s important to point out that each individual web application and server application interfaces with XMPP or a database only, and that each is independent of the other – we’ve essentially developed a “shared-nothing” architecture. This loosely coupled, non-interdependent XMPP and database architecture has served our purposes well so far.

We currently run on the Windows platform and utilize the .NET Framework to build both our web applications and our server applications. The only exception to using Microsoft technologies within our platform is the usage of a Java based XMPP server (Jive Software’s OpenFire). When we were researching XMPP servers at the time, we found that there were really no exceptional .NET based XMPP servers with the support and maturity we needed. This lead us to our currently implemented Java based XMPP server solution. Given the nature of XMPP, and the openness of its protocol, we were able to find a couple solid .NET based XMPP client SDK’s with which to work with, that made interfacing with the Java XMPP servers trivial.

Just to drive my previous points home: turning to a Heterogeneous System Architecture wasn’t intentional; it was a side-effect of working with the 3rd-party software available at the time. This decision had essentially turned our all-Microsoft based Homogeneous System into a Heterogeneous system overnight.

One added benefit of this shared-nothing architecture, is that because it is based on an open protocol (XMPP) and uses databases with wide support throughout a variety of platforms and development languages, we are now able to mix and match platforms, differing technology stacks (LAMP), and development environments with minimal effort and with near zero incompatibility issues.

Differing Platforms, Technology Stacks, and Development Environments

Having available to us the ability to work with differing platforms, technology stacks, and development environments, all in an effort to find the best tool for the job (based on budgetary requirements, different developer skill sets, the availability of certain 3rd-party software, etc), gives us a real advantage in that our options have been vastly diversified.

Now, of course, the desire to utilize any solution should never be dependent solely on its availability as an option. Just because we can mix and match, doesn’t mean that it is within our best interest to do so. Our decision to derive value from this ability is highly dependent on a number of factors. For example, we’re a startup division within a larger company, our roots are startup based, and so is our budget. In going over some of this year’s software purchases, we’ve had to include Windows Server purchases/upgrades, SQL Server licenses, and Visual Studio development environment licenses. Those are just a few examples of common large ticket items that need to be considered when working with a startup’s budgetary concerns.

The flipside to the above software purchases are that our platform, although currently based on a majority of Microsoft software, is in no way dependent on that software existing at its current majority share. We could, for example, move to MySQL and eliminate the high cost of SQL Server licences, or we could move to a LAMP stack and decrease the costs associated with running our web servers on Windows Server. And again with the LAMP stack, we could opt to run a mature PHP based MVC framework instead of working with the ASP.NET MVC Framework which requires us to work through various Beta/RC1 bugs.

The above is just one budgetary example of where having the option to become more heterogeneous with our architecture may be in our best interest. Obviously, using one set of tools versus another has its trade-offs, and no one platform, technology stack, or development environment is the best in all situations. But, given how dynamic our business requirements have been over the last few years, I’m excited to say that we have a whole slew of options available to us, largely as a result of having a Heterogeneous System Architecture.

Heterogeneous System Impact on Human Resources

I strongly believe that the single greatest detractor from using a Heterogeneous approach to your system’s architecture is the human resource factor. When architecting a system, it is important to keep in mind the skill sets of your developers, the quantity of those developers, and the ability for those developers to work within a Heterogeneous environment.

It is far more common than not, to have a team of developers aligned with a particular technology. For example, a Microsoft system often has employed developers that follow the Microsoft path of technologies (.NET, SQL Server, Windows). Whereas with an Open Source system, it is more common to have a crew of developers that are well versed in working with Linux/Unix based technologies (Red Hat Linux, Apache, MySQL, PHP). The occurrence of this type of specialization amongst developers makes having a Heterogeneous Architecture somewhat harder to hire for.Along these same lines, another unfortunate commonality between developers aligned with a particular platform is that they are often reluctant to learn and evolve their skill set outside of their realm of specialization. Finding the ideal type of developer that simply loves all technology and has the capacity to apply their general development knowledge across many platforms, quickly and successfully, is a rare breed indeed. Understandably so, as it takes a very dedicated and smart individual to become specialized in more than one platform. Equally as difficult as finding the will within developers, is finding developers with more than an introductory level of experience with multiple platforms.

So, in cases where your system is going the Heterogeneous route, ideally you would try to hire the particularly “smart” technology passionate developers; developers who have no allegiance to any particular technology stack. These are the guys who just want to use the best tool for the job, and have fun doing it; these guys are the Rockstars of the development world. Now, it’s easy to say “just hire the Rockstars”, but this usually comes at an increased salary cost and it is therefore largely more difficult to fill positions of this type. The question then becomes, based on your particular business requirements, your current team, and the general financial outlook of your business, whether it is to your benefit to go with a Heterogeneous, and therefore extremely flexible, system, or a more easily manageable and more thoroughly supported, Homogeneous system.

Getting personal for a moment, I prefer the flexibility of a Heterogeneous system, especially when it comes to working within a distributed architecture. The freedom and sheer number of options available to solve the often more complex software problems associated with distributed systems, makes it worth the extra high developer requirements.

Wrap-Up

I attempt to weave into my posts, the common theme that all implementations involve trade-offs. I try to drive home the reality that there is no single “best” way to accomplish any sufficiently complex task. That every strategy that you implement, and every decision you make has consequences in addition to its benefits, is one of the more important fundamentals of building robust system architectures.

I also try to illustrate these concepts using real life examples and I hope that they help you to better relate their applicability to your situation, however different it may be. Please comment below on what your experiences have been with Heterogeneous versus Homogeneous Architectures. I’m always interested in hearing how others tackle this subject, and what rules they’ve placed as guidelines for the implementations of their systems, one way or the other.

Comments

You point about heterogeneity is well taken. At the end of the day, having well-defined interfaces and protocols matters more than the implementation. But why did you choose the .NET stack for the bulk of your systems? Was it a constraint of your 'parent' company? of the available developer skill pool?

I ask because the common wisdom for startups is LAMP, LAMP, LAMP. For obvious reasons, the costs of getting going with free software are minimal and many dorm room developers cut their teeth on those systems. On the flip side, if you're coming out of an enterprise IT environment, there's a good chance you've been on Microsoft.

So, as a startup, if you're building enterprise-oriented software, .NET has a compelling appeal. But if you're a startup building a highly scalable, available internet service (think Facebook, Twitter, Digg), and you are starting from scratch, then what do you think of the LAMP vs. .NET choice?

If you had to do it over without constraints, which would you choose and how would you calculate the cost basis of each choice?

You hit the nail right on the head. Parent company constraints were the driving force behind working with .NET. In fact, when I came on board, our start-up was already using .NET and it wasn't financially viable, nor was there time available at that point, to move to a LAMP architecture. Don't get me wrong, .NET has a long list of merits, and it has served us well with the exception of the costs involved.

This is one of the reasons why I'm excited that we have more multiple platform options open to us. We now have the opportunity to make gradual changes, or none at all even, if that is what we choose. And it's exciting for me personally in that I'm a LAMP nerd by preference, but a Microsoft developer in practice - I may be walking into the rare opportunity to combine both within the workplace.

To answer your last question, if I had been leading the founding team, I would have gone with a LAMP architecture based purely on our internal business requirements (which unfortunately I can't share). There's a time and place for both .NET and LAMP, and you really can't go wrong with either over the long-term, especially if money is no issue for you (i.e. you're heavily VC'd). Many .NET developers would even argue that the benefits of the .NET/Microsoft environment ends up paying for itself over the long-term. Anyways, I digress, it's really a judgment call based on preference and cash availability.

I'd been much more inclined towards a homogeneous set up for a while. It'd always been in my mind that you could employ a smaller, more skilled set of developers and admins that way.

I've slowly been adjusting my attitude recently on that front following observations at my previous employer. It was a bit more of an unusual environment there with the kinds of services we were offering to customers, but I really started to become conscious of the pros and cons of different solutions available under Windows and Linux. e.g. Exchange, for all it's Microsoft 'not quite standards compliant' quirkiness, is a very good and easy to set up office productivity tool.
It wasn't capable of doing what a smaller subset of staff required, however, so we ended up running Linux front end MX servers, based on exim, that were feeding e-mails on to the Exchange server after initial processing. Those Linux front ends allowed us to leverage some free anti-virus and anti-spam solutions too. Setting up a Linux based exim MX is a quick and easy task for your average sysadmin.
Re-creating all the required functionality of the Exchange server under Linux would have taken a stupid amount of resources away from money earning projects.
Being able to do a significant amount of spam and virus filtering on a Linux box also drastically reduced the number of Exchange servers we needed. In our experience exim is significantly more efficient at handling e-mail and due to the nature of business we received a significantly large amount of spam.

interesting thoughts, thank you for posting them. i would guess there are trade offs, of course. for example, how easy is it to debug your system end to end if/when a bad bug appears, vs. a system that is so homogeneous that you can just use the same debugger the whole way through? :-)

Great question! I've found that traditional debuggers are near useless for debugging most distributed system issues. Having Visual Studio debug across our system is near impossible given the types of components that make up our system (databases, web apps, server apps, 3rd party apps and SDK's) and the reality that Visual Studio or some similar debugger can't effectively debug all programs equally. In addition, many of the problems we face are more logic based which makes code debuggers less useful than you might at first think. It isn't until we've tracked down a bug, across the system, that we break out the debuggers and tweak the problematic component(s)(server app, web app, etc).

Since our architecture is disparate in nature, and very loosely coupled, there is little advantage to being able to debug with Visual Studio for everything versus Visual Studio for some things and Eclipse/Java for others. Just to be clear, this isn't on purpose so much as it is just the nature of a shared-nothing architecture.