Big Data: The New Paradigm Of Effective Management

Posted in Operations & IT Articles, Total Reads: 3198
, Published on 19 November 2012

Advertisements

The Internet Revolution and explosion of digital data has put vast amounts of information into the hands of organizations; undoubtedly more than they can handle. To turn all this data into information in order to glean competitive edge, organizations are now turning to new skills and technologies and novel management strategies.

MIT Labs conducted an experiment in which they were able to estimate the sales of Macy’s on Black Friday – the start of Christmas shopping even before the retailer itself had recorded the sales. They were able to achieve this feat by using location data from the customers’ mobile phones when they were in the parking lot itself. This has become possible by the use of Big Data.

The amount of data in our world has been exploding at exponential rates and businesses now have access to large amounts of unstructured, unfiltered data that could change how they make decisions, and even how they compete. The analysis of this Big Data provides data scientists with trends and gives them insights that help the organizations to create value and innovation much sooner than conventional methods.

What is Big Data and how does it matter?

According to IBM, around 2.5 quintillion bytes of data is created everyday - so much so that 90 percent of world data has been created in the past two years. This data comes from both internal and external sources and through various mediums such as networks, sensors, mobile phones, posts from social media websites, digital pictures, audio and video or other unstructured sources. This is big data.

To put formally, big data is the data that exceeds the processing capabilities of conventional data structures. Three trends namely – Big Transaction Data, Big Interaction Data and Big Processing Data combine together to fashion what we know as big data.

Its relevance in today’s world is perhaps unparalleled because it has emerged as a cost-effective approach to quick and smart decisions. It helps organizations tackle complex problems that previously could not be solved. Executives can measure and consequently manage more accurately than before and can efficiently translate knowledge into improved decision making and performance.

Big Data also results in Big Return on Investment (ROI). Today’s commodity hardware, cloud architectures and open source software bring big data processing into the reach of the less well-resourced. Big data processing is eminently feasible for even the small garage startups, who can cheaply rent server time in the cloud. For instance, IBM solutions have resulted in significant improvements in various sectors.

When it comes to the scope, “big data” and analytics can be labeled as an umbrella term for a variety of tools and technologies for data storage, collection and processing. Input data can vary from banking transactions to chatter on social networking sites.

The “Three V’s” of Volume, Velocity and Variety are commonly used to characterize different aspects of big data.

Volume.In times when more data crosses the Internet every second than was stored in the entire Internet just 20 years ago, the ability to process massive volumes of data is one the largest attractions of big data analytics. Wal-Mart, for instance, collects more than 2.5 petabytes (one quadrillion bytes) of data every hour from its customer transactions.

For such large volumes of data, traditional relational database structures may prove to be ineffective. The solutions are parallel-processing systems – data warehouses or databases such as Greenplum – and Apache Hadoop-based solutions. Facebook is one the most well-known users of Hadoop.

The Three V’s of Big Data (Source: Wipro)

Velocity.For time-critical applications, the speed of data creation is more important than the volumes of data. Real-time information provides organizations the edge over its competitors and helps it make its operations more agile. Specialized companies such as financial trading firms rely on the timely analysis of critical data to make informed decisions. Social networking sites such as Twitter and Facebook rely on such data to provide recommendations to users in real-time.

Variety.Data comprising of big data can be both structured as well as unstructured. Most of the sources are relatively new. With the technology and telecom boom and advent of hand-held devices such as smartphones and tablets, the amount of data generated daily is enormous; tied to people, activities and locations. Social networking sites too generate more data everyday than the cumulative Internet data 20 years ago.

Big Data in Action

Big Data has numerous applications in the present day scenario. It constitutes Healthcare, Insurance, Communications, Financial Services, Engineering and many more.

Healthcare.The average amount of data per hospital is slated to increase from 167TB to 665TB in 2015(acc. To IBM) and big data analysis can improve patient care and reduce costs by providing relevant data at the right time.

Insurance.Government and Insurance agencies gather fraud related data related to their personal missions. With big data, these companies can apply advanced analytics to detect fraud quickly, before the payment of funds.

Using big data in organizations has its share of technical challenges. But the managerial challenges it poses are far more – from the changing roles of the senior management team to the changing culture in the organization itself. Big data in its application poses 5 management challenges.

Leadership.Contrary to the belief that big data analysis may dilute the roles of the leadership teams, the functions which they perform gain more importance. The leaders must be capable enough to pose the right questions to the analytics in order to get meaningful results which can lead to greater success in the coming times.

Talent Management.With its inception, big data also brings in the need for skilled professionals who are capable enough to visualize, clean and process the large mounds of data made available to them. Data scientists have come into prominence and are in great demand because of the expertise they bring to the table.

Technology.The tools to tackle the volume, velocity and variety of such data have also transformed over the years. Most of them are open source and easy on the pocket. Hadoop, for instance, is one such tool which is being used extensively by many large organizations.

Decision Making.With data being far away from the experts who work on it, the focus is on bringing people with the relevant expertise together and maximize cross-functional cooperation.

Company Culture.The traditional HiPPO (Highest Paid Person’s Opinion) approach relying on hunches and intuitions of the higher level management to take decisions also undergoes a change on application of big data analytics.

Final Words

Big data exceeds traditional analytics in terms of capability and relevance. Also, data-driven decisions are better decisions. Using big data enables managers to make decisions on the basis of evidence rather than intuition. Precisely for that reason it has the capability to revolutionize management. Even though there are several barriers to its success – with underdeveloped technology and expertise and many cultural challenges – the privacy of the data foremost among them – big data is here to stay.