Follow me on Twitter

How to be Big Data-native?

Big data has spawned a set of tools that deliver results beyond the buzz. It has started delivering real insights for companies, which result in more effective decisions.

When middleware natively supports big data, big data becomes more than just another option. It becomes the default. Let’s examine this idea:

Big Data Storage

Whenever you think of storage, (almost) everyone thinks of an RDBMS mysql, postgres, mariaDB, Oracle, etc. If you convert to supporting native Big Data, you turn into to NoSQL options that sacrifices SQL for scale. You start storing everything in Cassandra, HBase, mongoDB, redis, etc. You don’t have to dread the day your data volume becomes too big to handle. When it does all you do now is configure a new node and maybe tune the cluster a bit and you are done.

Big Data Analytics

If you are not looking at analytics, you should (Google for innovate or die). All analytics support for big data. If you thought about running some SQL or spread sheet macros, it’s time to move on. Start thinking Hadoop, Big Query, Drill, etc. Natively supporting Big Data analytics allows anyone who sets up the middleware to instantly enable their departments and teams to harvest their data silos, no matter how big they are.

Big Data Speed

There is no point of BigData storage and Big Data Analytics if you can’t collect big data fast enough. Looking at web scale transactions it is not unusual to have millions a second. Before middleware vendors start advocating they need to make sure the middleware is up to par. Computers are definitely fast enough so that even a single node can handle around 10,000 TPS.

The biggest concern I’ve seen is many companies claiming to be able to adapt to Big Data without actually supporting it natively. It is a nightmare to allow to scale for Big Data after choosing incompatible technologies. The concepts differ, the trade offs are different and the effectiveness is very low. Big data has reached the importance level to for middleware evaluators to add another section to their RFPs. And, yes that’s whether the middleware is “Big Data native”.