5 things toxic to scalability

1. Object Relational Mappers

ORMs are popular among developers but not among performance experts. Why is that? Primarily these two engineers experience a web application from entirely different perspectives. One is building functionality, delivering features, and results are measured on fitting business requirements. Performance and scalability are often low priorities at this stage. ORMs allow developers to be much more productive, abstracting away the SQL difficulties of interacting with the backend datastore, and allowing them to concentrate on building the features and functionality.

Scalability is about application, architecture and infrastructure design, and careful management of server components.

On the performance side the picture is a bit different. By leaving SQL query writing to an ORM, you are faced with complex queries that the database cannot optimize well. What’s more ORMs don’t allow easy tweaking of queries, slowing down the tuning process further.

2. Synchronous, Serial, Coupled or Locking Processes

Locking in a web application operates something like traffic lights in the real world. Replacing a traffic light with a traffic circle often speeds up traffic dramatically. That’s because when you’re out somewhere in the country where there’s very little traffic, no one is waiting idly at a traffic light for no reason. What’s more even when there’s a lot of traffic, a traffic circle keeps things flowing. If you need locking, better to use InnoDB tables as they offer granular row level locking than table level locking like MyISAM tables.

Avoid things like semi-synchronous replication that will wait for a message from another node before allowing the code to continue. Such waits can add up in a highly transactional web application with many thousands of concurrent sessions.

Avoid any type of two-phase commit mechanism that we see in clustered databases quite often. Multi-phase commit provides a serialization point so that multiple nodes can agree on what data looks like, but they are toxic to scalability. Better to use technologies that employ an eventually consistent algorithm.

3. One Copy of Your Database

Without replication, you rely on only one copy of your database. In this configuration, you limit all of your webservers to using a single backend datastore, which becomes a funnel or bottleneck. It’s like a highway that is under construction, forcing all the cars to squeeze into one lane. It’s sure to slow things down. Better to build parallel roads to start with, and allow the application aka the drivers to choose alternate routes as their schedule and itinerary dictate.

4. Having No Metrics

Having no metrics in place is toxic to scalability because you can’t visualize what is happening on your systems. Without this visual cue, it is hard to get business units, developers and operations teams all on the same bandwagon about scalability issues. If teams are having trouble groking this, realize that these tools simple provide analytics for infrastructure.

There are tons of solutions too, that use SNMP and are non-invasive. Consider Cacti, Munin, OpenNMS, Ganglia and Zabbix to name a few. Metrics collections can involve business metrics like user registrations, accounts or widgets sold. And of course they should also include low level system cpu, memory, disk & network usage as well as database level activity like buffer pool, transaction log, locking sorting, temp table and queries per second activity.

5. Lack of Feature Flags

Applications built without feature flags make it much more difficult to degrade gracefully. If your site gets bombarded by a spike in web traffic and you aren’t magically able to scale and expand capacity, having inbuilt feature flags gives the operations team a way to dial down the load on the servers without the site going down. This can buy you time while you scale your webservers and/or database tier or even retrofit your application to allow multiple read and write databases.

Without these switches in place, you limit scalability and availability.

Thx Gustavo. Am a big fan of NewRelic. Checking out Boundary as we speak. Good tip.

Marquess

Totally agreed – and the NewRelic comment takes point 4 one step further. If you don’t have the tooling, trying to time-correlate any data that you may have from a bunch of disparate sources and inability to see what is happening on an aggregate and per-request basis, choosing where to spend effort in that critical triage phase is both a nightmare and often contentious. We get a lot of “help, we’re melting” calls and one of the first things we almost always need to do is improve the data and visibility situation. Gotta bring guns to a gun fight, not knives.

http://www.iheavy.com/blog/ Sean Hull

“Gotta bring guns to a gun fight”. Well said!

urkurk

An ORM standard like NHibernate greatly simplifies development especially when an app requires many tables (e.g. 100). If it’s correctly used with an appropriate db design it can speed up development while avoiding the tuning headaches. On my humble app I use redundant denormalized tables to avoid having to tune the normalized ones.
I’d rather code using an ORM than have a maintenance nightmare with tuned stored procedures. I’d go for a balanced but not really optimal design rather than something that potentially puts me in danger of not meeting the deadline because I’d have to tune every freakin stored proc the app uses.

hullsean

Hi @urkurk:disqus, yes I can see the appeal. On the dev side it makes for better code to be sure. However to say they’re not *optimal* is to simplify things. They produce code that asks the database engine to do orders of magnitude more work. Hence they do not scale well.

Applications built on these ORMs hit a serious wall. Once they do, the ORM is a rip and replace job, and a very messy one at that.

urkurk

Hi Sean,
I’ve gone thru several ORM (NHibernate) projects but I’m not an expert but I do know that there is a way to make queries generated by NHibernate tunable. It’s not a wall where you just stop and discard ORM. In fact there are elegant solutions to reach a certain amount of scalability when using NHibernate. From what I’ve read Amazon uses ORM (NHibernate) for production use which for me serves as evidence that it does work for someone serving millions of online customers everyday.

Rip and replace job? How? I think this can be avoided by having developers who understand how to use ORM. Just spending time using a profiler tool will show you that NHibernate for instance produces predictable queries. You do have to work closely with it to tune it but tuning is never easy anyway. But in cases where you need to use table hints, locking hints then that’s where you employ a different hybrid strategy.

You can definitely use an ORM like NHibernate and have some amount of scalability. But I’d be cautious using EF or other lightweight ORMs that don’t have the same reputation as NHibernate.

hullsean

Appreciate the comments urkurk

Would love to see a howto or blog post on all of this. Something along the lines of “Scaling hibernate for high traffic sites” or something similar. When I search for that I come up with a post on AKF partners website where they mention it and Active Record (one of Ruby’s ORM).

urkurk

I can’t promise but I’ll try to post an article but it certainly won’t be to scale for millions of users since my real world experience has only been for small to medium sized websites.

hullsean

Thx @urkurk:disqus What’s your blog address? Or perhaps you could cross post it here when finished, and we can all take part.

To be sure, @toddtrimmer:disqus … but at what cost? Many web applications don’t need to hold their standards so high.

Todd Trimmer

Well sure, but calling it consistent is dangerous doublespeak that can lull one into a false sense of confidence. If you understand and accept the risks, go ahead, but that phrase leads to misunderstanding right at the jump.

There are also other requirements that can dictate 2PC besides clustering. XA for cross-datasource transactions comes to mind.

http://www.iheavy.com/blog/ Sean Hull

Thx @toddtrimmer:disqus

Do you have a blog?

http://gatesvp.blogspot.com Gaëtan Voyer-Perrault

I don’t think that “ORM == toxic to scalability” really captures the range of trade-offs involved in ORMs.

At a basic level, ORMs enable a small group of developers to adequately manage interactions with a large number of tables. Is there a performance penalty? Absolutely. Does it suck up extra server resources? You bet. But it is dramatically easier to develop and modify than things like low-level stored procedure or views.

On the other end of the spectrum, developers _could_ be hand-tuning every query and writing everything by reading columns off of low-level DataReaders and hand-mapping those into objects to display. But this is a lot of work and it’s not clear how valuable that work really is. Even if you don’t have an “ORM” you still end up mapping DB columns into objects that you can safely pass around. So even if you don’t download Hibernate, you’re still doing a bunch of work mapping columns into objects. If you’re hand-coding every query into and out of the DB you are increasing the number of devs required to maintain the system.

The real key here is simply that ORM needs to be “by-passable”. I need to be able to write Stored Procedures for performance critical sections of the code while leveraging the default queries for other things. And things like MS’s Entity Framework completely allow for this. And teams have developed various “Micro-ORMs” and are getting better at using CQRS (http://martinfowler.com/bliki/CQRS.html) which clears up the differences between reading and writing data..

In that sense, I don’t really view most ORMs as toxic. I mostly feel they are “over-used”. ORM and CQRS can co-exist, but developers need to build this way. Dropping down to Views/Stored Procedures is completely allowed in modern ORMs, but it’s really about developers knowing when to make these trade-offs and funneling the code in the right direction.

http://www.iheavy.com/blog/ Sean Hull

hi Gaetan, thx so much for the in-depth comments.

Yes there are still opinions on both sides of the issue. I come from a DBA & operations background, so that may inform some of my biases.

http://gatesvp.blogspot.com Gaëtan Voyer-Perrault

Clearly, if you’re a DBA, then ORMs are the “tools of the devil”. 🙂

They lead to unreadable/unindexable queries, excess use of resources (all fields, all of the time), and N+1 errors that are uncaught by developers until they hit production. Plus, they try to move certain data integrity features out of the DB and into the code, leaving the DBA unable to protect their own data. And then there are generated migrations….

Despite knowing all of these ills, my current start-up day job still runs Microsoft’s EF because velocity of change is currently more important than optimization of server resources. That stated, we definitely _do_ have stored procedures and hand-written queries in key places, but we’re very specific about those places.

http://www.iheavy.com/blog/ Sean Hull

Hehe. Yes, reading that paragraph it was as if I wrote it myself!

Yes surely it is one of perspective. Glad to have your comments & criticisms too Gaetan. 🙂