Big Data Bigger Opportunities

Posted in Operations & IT Articles, Total Reads: 3100
, Published on 03 January 2012

Advertisements

Drops of information from ubiquitous data sensors have created oceans of data for organizations. It is a point of concern for a few while the most see it as big opportunity... big data opportunity. Over the years the data that can be collected and used by Organizations has grown tremendously – in variety, the frequency at which data is acquired and the amount of data captured. Besides traditional data captured and processed by an organization’s MIS, there also is a wide variety of unstructured data which is a bit difficult to capture using the usual methods and hence does not get used while making decisions.

Big data is just a notional term which signifies voluminous amount of unstructured and semi-structured data which is difficult to handle for the current relational database systems. Though exact lower threshold is not defined for the volume of the data to be called big data, generally data in terabytes, petabytes and above.

It is not a sudden phenomenon, but a culmination of the increasing amount of unmanaged and unstructured data coming from countless information sensing devices and media like sensor networks, RFID, social networks, e-commerce, etc. Easy access to internet with a number of UGC (User Generated Content) has also been the driver for this enormous growth.

Earlier organizations saw this data as a liability and a matter of growing concern of storing such huge data. However, with passage of time most of them started realizing its potential as an asset which can bring lot of business intelligence. Big data brings with it the precious insights into the consumer behaviour, future trends and areas of numerous opportunities.

There always has been a cyclical relation in between advances in knowledge & advances in our capacity of measurement. All advances in measurement have led to the advancement of knowledge in some form or the other. Increased computing capacity & ability to measure data at more granular levels in the recent past have created opportunities to reveal patterns & improve our insights into business. These new insights in turn help us to identify the different parameters that should be monitored further extending our measurement capabilities and the resultant dataset.

The use of Big Data can be made to better an organization’s decision making. Through continuous experimentation, organizations can now come up with better predictions. They can now distinguish the causal factors from merely co-related ones. All these will lead to better forecasts, improved operational plans and more predictability in business. Use of Big Data will also help companies to think in terms of narrower segmentation. Moreover, Big Data processing is not a onetime activity but an on-going process.

But, what the companies want are pearls hidden under the ocean. The intelligence that is hidden within this data can be fathomed through analysis of this data; else it is really a liability and a missed opportunity.

Clearly traditional way of processing data will not be adequate to handle these new opportunities. A number of new technologies have been developed to help organizations respond to this challenge.

What are major players doing about this?

Most major software companies have been working on this opportunity and have already rolled out products to help organization leverage on Big Data. Oracle has already built a number of offerings around Big Data. Oracle recently introduced its NoSQL Database. NoSQL is a non-relational ship database management system that will allow its users to store massive amounts of data in a highly available distributed system. Oracle has also recently launched a number of other services aimed at providing users with complete support for BigData operations. Google’s BiqQuery which is still in its beta stage aims at providing users with a platform to allow users to carry out large scale internal data analytics. Apache’s open source projects- mostly based on HADOOP have also made significant progress in catering to the user’s requirement. Almost all major products now have incorporated some sort of interface with Hadoop. In fact Microsoft gave up its Dyrad in favour of Hadoop which will be integrated with Microsoft’s next SQL server release – SQL Server 2012. SAP too is trying to woo users with its HANA platform. IBM unveiled InfoSphere BigInsights and Streams software which will allows its users to gain fast insight into information flowing in and around their businesses. These software analyse traditional structured data found in databases along with unstructured data and process large chunks of data at a faster rate.

Opportunities for Indian IT

As mentioned already, use of Big Data to gain business insights is an on-going process and not a one-time activity. With increase in analytics, opportunities for Indian IT companies will surely increase. A recent report by IBM suggests that almost 90% of the world’s data has been created in the past 3 years. As our computing capacity increases, so will our need for data. In order to keep up with this growing demand, organizations will need to revamp their IT infrastructure if they are to make use of Big Data for their business. Scaling up to meet their ever increasing requirements is not an easy challenge. Furthermore investments needed to do this would seem prohibitive in the face of constantly evolving technology. The cloud seems to be a plausible solution to these problems. Big Data will encourage organizations to move to the cloud – at least partially if not completely. Analytics, Cloud and Infrastructure management services are key opportunities that IT service providers should try to make the most out of.

IT companies serving different sectors also can look for the opportunities that big data provides for each sector. Each sector has a different potential to reap the benefits of big data. The bubble chart below illustrates the same by taking US markets into consideration.

Source: McKinsey Global Institute report on Big Data.

What does this mean to IT service providers? This gives a direction to the companies while deciding the future investments and strategies for each vertical. BFSI a strong business provider again comes out to be a big time opportunity which can be grabbed with investments in that direction. Healthcare, pharmacy and transport are other sectors that can gain from initiatives to TAP Big Data – all of which form a major chunk of business for India IT service providers.

Cloud computing has already gained its status of next big thing owing to the requirements of processing of the enormous amount of data. Indian companies vouching on cloud computing as a new source of revenues can surely focus on the SaaS (Software as a Service) to come up with analytics solutions catering to the business intelligence needs of companies from various sectors looking to get benefited from the big data opportunities. The prophesy of cloud computing looks forward to comprehensive cloud services providers working on all the three fronts of Infrastructure, Service and Platforms, and big data is the key driver here.

This article has been authored by Gaurav Kawale & Sandeep Borkar from SJMSOM, IIT Bombay.