Professional Internet technologist. Making the ‘net more human. http://moty.net

Feb 13, 2014

Moving Application Complexity to Data Layer

Low price point of SSD based storage could change how we think about developing software.

In fast paced Internet application development cycle, software complexity and change management becomes roadblock on the road to building latest and greatest app for consumer market.

With the recent popularization of various big data technologies it’s good to also revisit the smaller data sets (up to few TBs) from time to time to see how we operate on those and where the complexity should lie.

When storage was slow and expensive (hdd based) for performance reasons it was natural to push as much logic into the application layer as possible, finding the best algorithm for sorting, operating on data in software, applying all sorts of transformations in the application layer. But it didn’t come without cost. The biggest cost being maintenance. The hidden baggage that accrues with every line of code. Maintaining that layer of software can run into tens or hundreds of thousands of dollars spent when time comes to making even the smallest change. Problem is we’ve baked business logic into expensive to maintain application layer.

With silicon based storage dropping in prices a new approach comes to light. Application software should now be glue between user and data with as little (complex) transformation logic as possible. Yes, often there is no way around complex business logic placed in application layer but the old approach should be challenged every time new code is written rather than accepting it by default.

Cost may still seem high, with SSD storage roughly 10x price of HDD. But given the freedom to operate on raw data (made possible due to at least 10x gain in speed with silicon based storage) with cost saved over time unburdened by legacy application logic it can easily become a winning strategy.

With that we should also highlight the importance of traditional, relational databases. With highly normalized data sets made possible by relational databases we can slice and dice (given fast enough storage) at will in database. By choosing denormalized, noSQL data storage we are already baking business logic into the data set, defeating most of the flexibility and potential gained from fast storage.

Software is hard to maintain and Moore’s law of doubling efficiency every two years that exists in hardware stubbornly does not apply in software no matter what new software pattern or framework we come up with. In fact, the cost of writing software seems to be only going up. Let’s try to offset that by consciously choosing where we store complexity.