The Investing Edgeof Enterprise IT

Big data is a hot topic, especially here in Massachusetts – arguably the Big Data capital of the world with 12,000 people employed in big data jobs in more than 100 companies across the state, an unmatched talent pool from top universities, and home to EMC Corp.

But within those conversations, what’s interesting is that there has been a noticeable decline in the talk about extracting data and an increase in chatter about the scalability of it.

Of note, a Wall Street Journal piece on the recent buyout of Digg, one of the first social aggregation sites, explained that four years after being founded in 2004, Digg was valued at more than $160 million, but was sold last week for a mere $500,000. Why? For one thing, Digg was not able to effectively house the data entering their system every day.

During their re-launch in 2010, Digg realized they had problems with the scalability of their existing MySQL database software, and started a shift over to another open source system, Cassandra. But the new Digg launched before the new database was effectively installed, and their team was unable to fix the problems. This turned out to be a killer.

The data explosion of recent years has forced the hands of organizations everywhere, not just Digg. The ability to scale your internal database is no longer a nice-to-have; it’s a necessity. The companies that can store and manage vast amounts of data seamlessly will have a major advantage; those that do not, will struggle and may even perish.