Channel: Data Warehouse Appliances and Architecture - Krish Krishnan

Big Data and Business Innovation

Innovation has always been the spirit of our country and has been the key to success for a number of decades. If we were to turn through the pages of history, we would see there have been tipping points across time leading to innovations in every field that touches the human life, making the world a better place. A closer examination of any innovation reveals that there has been a lot of research and development using a large volume of data, based on which we have arrived at the solution. Many of the innovations – beginning with electricity, radio and wireless, automobiles and, more recently, the Internet and mobile devices – have been transformative both professionally and in our personal lives, and provided significant business benefits.

A successful innovation needs a powerful marketer and an organization that can invest in the creation of that innovation. This strategy works in most cases. In cases where the model does not flourish, the innovation itself does not last long.

In the late 20th century (circa 1992–2000), we witnessed the dramatic coming of the Internet (titled as the World Wide Web, the original Internet protocols have been used since 1973 by DARPA) and the infamous dot-com bubble. The arrival of this wave was strong but not clear enough to provide a business model that could be replicated and monetized. However, the first wave of innovation from the Internet saw the birth of the web browser, the search engine and the concept of an online marketplace. During this time, another innovation matured silently – the mobile communications platform.

Since the advent of the 21st century, we have seen advancement in several areas of technology. Some of the significant developments include:

Mobile communications infrastructure

Mobile devices (including phones and tablets)

The birth of Web 2.0 business models

The advent of social media

The improvement of search platforms

Adoption of open source platforms in data centers

Geospatial data integration

These massive shifts in technology are influencing the behaviors of people and businesses. From an individual perspective, consumers discovered the benefit of the Internet via mobile devices and found a virtual world where they could form virtual communities based on common interests. As these virtual communities developed, community-based problem solving became a central focus area for some communities, and a level of trust was established within the community. The idea grew so rapidly that many such communities were formed. Once social media platforms like Facebook, MySpace and other sharing sites started forming, the growth of the communities was viral, and soon there was a community to answer anything on any subject across the globe. The interest permeated to professional circles with sites such as LinkedIn and Plaxo. The growth of these communities led to the following impacts or influences on businesses:

Long Tail – The long tail is a term coined by Chris Anderson, where he explains how businesses started experiencing a longer and sustained tail of revenue and growth when they were able to market to a larger list of customers or prospects with niche products and services at competitive price points, providing better revenue. Another great example of long-tail involves fund-raising efforts and the use of social media in presidential elections in the U.S. in both in 2008 and 2012 by President Obama’s campaign.

Crowdsourcing – Through the formation of communities, businesses identified potential opportunities for involving the leaders and trendsetters in these communities to become their brand ambassadors and provide irreplaceable word-of-mouth marketing for their products and services. To support this, there are several initiatives sponsored by businesses, and some of them have even created innovative portals for the consumers to contribute to ideation and product design.

Gamification Strategies – Another popular sponsorship by businesses to foster innovation and create trust in their communities is the gamification strategy, where leaders are often rewarded for their contributions. By making such small strides, businesses are benefitting in the long run every day.

Open Innovation – Aligning with trends 3 and 4, there are companies that have started inviting the general public to solve some of their complex problems using mathematical models or other scientific or non-scientific techniques for prize money.

While these trends and their associated models of engaging in business are becoming clear, we have two significant problems that need to be understood:

There is a lot of data of different types available for consumption by business today. The data is produced or generated at furious pace and is commonly called big data.

Not all big data problems are solvable by strategies discussed so far.

This brings us to the critical question: How does Big Data Impact Innovation?

To understand this, let us look at the history of innovation. We have been delivering innovations since the beginning of time; however, until we discovered electronics and deployed the first generation of computers, the innovations were happening at a relatively slow pace. The ability to compute started speeding the innovation across industry verticals at a startling pace. However, the volume of data required for innovating constantly challenged us, as most of the data was not inside the computing environment in a structured format.

This problem was expensive to solve until recently. There are now technologies to process large volumes of data on commodity platforms for a fraction of the cost compared to the original costs. Platforms designed and built for handling scalability problems for search engines and social media platforms now provide the computing and storage platform for creating the enterprise compute and processing platform for large and multi-format, multi-structured datasets. The extensibility of this platform into the enterprise data repository is the “game changer” for many enterprises. Now you can access all the data needed for making informed business decisions, and this creates fertile ground for innovation of new business models and identification of “blue ocean” opportunities within the enterprise.

The availability of data provides the empowerment to start creating business scenarios and replay the outcomes using the data and the underlying infrastructure. Multiple scenarios called as experiments will provide outcomes; and by creating the near perfect experiment, you can predict the closest outcome to what your business expects. This experiment will allow you to create the right segmentation strategy for your customer, the right market for your product or the right cross-sell strategy for your call center. The biggest transformation that can be brought to bear is the overall approach of the business to its prospects or customers. Instead of asking the “lifetime value of the customer” or “the profitability segment of the prospect,” the question has shifted to “What is the value of me (the business) to the customer or prospect.” This type of introspection has provided the business with opportunities to adapt to different types of customers, offer personalized levels of marketing, and services and has directly increased the revenue from such an engagement.

There is a lot of complexity in this process that definitely is not trivial. However, if the business wants to increase its profitability and remain in business, they are forced to transform their thinking and behavior. This can be implemented in the most effective manner with the right set of insights and metrics, which can be provided by using big data platforms, all the data needed for such purposes, a collaboration platform and a robust set of analytical models.

As you can see from this article, we are just scratching the surface when it comes to implementing and monetizing big data. This is just the beginning, and the possibilities are infinite. Big data definitely empowers innovation and provides a scalable platform to create multiple successful strategies from one statistical model or one experiment. At the end of the day, the biggest risk is not doing anything.

Remember the bottom-line: For innovation to occur and thrive in a business, people are the biggest success factors, both from an executive and business user perspective. As you read this article and start thinking in this direction, remember the actions and outcomes need to be small, incremental steps. If not planned appropriately, it is easy to go overboard and attempt – unsuccessfully – to boil the ocean.

Krish KrishnanKrish Krishnan is a worldwide-recognized expert in the strategy, architecture, and implementation of high-performance data warehousing solutions and big data. He is a visionary data warehouse thought leader and is ranked as one of the top data warehouse consultants in the world. As an independent analyst, Krish regularly speaks at leading industry conferences and user groups. He has written prolifically in trade publications and eBooks, contributing over 150 articles, viewpoints, and case studies on big data, business intelligence, data warehousing, data warehouse appliances, and high-performance architectures. He co-authored Building the Unstructured Data Warehouse with Bill Inmon in 2011, and Morgan Kaufmann will publish his first independent writing project, Data Warehousing in the Age of Big Data, in August 2013.

With over 21 years of professional experience, Krish has solved complex solution architecture problems for global Fortune 1000 clients, and has designed and tuned some of the world’s largest data warehouses and business intelligence platforms. He is currently promoting the next generation of data warehousing, focusing on big data, semantic technologies, crowdsourcing, analytics, and platform engineering.

Krish is the president of Sixth Sense Advisors Inc., a Chicago-based company providing independent analyst, management consulting, strategy and innovation advisory and technology consulting services in big data, data warehousing, and business intelligence. He serves as a technology advisor to several companies, and is actively sought after by investors to assess startup companies in data management and associated emerging technology areas. He publishes with the BeyeNETWORK.com where he leads the Data Warehouse Appliances and Architecture Expert Channel.