Big Data

It’s been one of the most talked about and hyped tech concepts of the past few years.

In fact, it’s almost impossible to read any information about analytics – especially information from software companies – without hearing about Big Data.

But what does it really mean?

What is Big Data?

Companies have a lot of data.

Much of it is structured, meaning it can be organized in a database or Excel spreadsheet and sorted using simple criteria.

For example, a spreadsheet listing purchases would include columns for customer name, amount invoiced, product name, date, etc. All of those fields are standardized and can be filtered or sorted alphabetically or numerically.

However, a lot of the data companies have is unstructured, and the amount of unstructured data at organizations’ disposal is quickly increasing. This includes anything that doesn’t fit neatly into rows and columns, such as:

Collections of statements people are making about the company on social media

Audio recordings and transcriptions of calls with customer service agents

Website search indexes

Videos, images and data from various types of sensors

Big Data is any data that can’t be managed using conventional database tools.

In addition to just the type of information it contains, there are also other factors that make Big Data big.

Velocity – The interconnectivity of people and things via technology is generating data continuously, and businesses are attempting to capture and analyze it in real time.

Variety – Big Data comes from many different sources and includes many different types of information. Sets of data typically can’t be simply compared to one another using traditional tools.

Volume – As the phrase implies, Big Data includes, simply, a lot of data. Individual companies have access to a lot of it, with terabytes, petabytes, and soon zetabytes being stored and collected.

Veracity – With so much data from different sources being collected, there can be a lot of doubt about the accuracy and consistency of that data. However, in order to get actual value from Big Data, it’s important to have tools and processes in place to verify and clean up the data.

As with business intelligence, there are many varied definitions of Big Data floating around.

One issue is that many of the definitions are driven by software vendors and tend to drive up the hype of the technology, rather than focus on what Big Data actually means for businesses.

Big Data is a collection of data from traditional and digital sources inside and outside your company that represents a source for ongoing discovery and analysis.

Big Data is used to refer not just to specific types of data, but also the technologies used to manage them.

Big Data is an evolving term that describes any voluminous amount of structured, semi-structured and unstructured data that has the potential to be mined for information.

For our definition, we wanted to highlight what sets Big Data apart from other types of data, as well as touch on why it’s important for businesses:

Big Data is any data that can’t be managed using conventional database tools but can still be analyzed to create valuable insight for the organization.

Big Data: Challenges and Opportunities

The volume of Big Data is a critical point that’s creating a lot of new challenges for companies. As business and other aspects of life become more digital, there’s a lot of both structured and unstructured data being created all the time.

As companies try to mine intelligence from that data, that means they’ve got a lot of information to store, and that volume is growing exponentially.

According to predictions from IDC, the total amount of data stored in the digital universe will grow from 4.4 zetabytes in 2013 to 44 zetabytes in 2020. A zetabyte, for your reference, is one trillion gigabytes. Total data is doubling in size every year, and businesses are expected to create 44 times and much data in 2020 as they did in 2009.

Data with value – if you know how to capture it

Businesses will have a lot of opportunities to utilize and benefit from that data. Despite the fast growth, IDC estimates that, thanks to new tools and techniques, 37% of all the data a business has access to in 2020 will be useful if properly analyzed, compared to 22% in 2013.

Despite the challenges associated with storing and managing all of that data, the vast majority (89%) of database professionals consider Big Data to be a great opportunity to improve business results, rather than a costly data management problem, according to a 2013 TWDI survey.

Challenges remain

However, there are a lot of challenges still in the way before organizations realize all of Big Data’s potential benefits.

Businesses must adapt and deploy the right tools to be able to store, analyze and secure Big Data.

As mentioned above, traditional tools aren’t equipped to process this new data; new software and systems such as Hadoop are required. Also, specialized storage tools are needed, along with tools to extract and integrate Big Data from various sources.

As with the Extract, Transform and Load (ETL) tools used for business intelligence, there’s a lot of work to be done to gather Big Data, ensure its quality and get it ready for analysis.

Some of the other challenges standing in the way of Big Data benefits include:

Lack of skilled staff – Big Data has taken off fairly recently, and as such many companies report difficulty in finding staff with the skills necessary to manage this new data. In fact, that was the top challenge listed in TDWI’s report, cited by 40% of respondents. One strategy to combat that could be to offer employees Big Data training in order to grow the talent pool internally.

Data governance issues – With so much data available, it becomes even more critical to have a framework in place for deciding what data belongs in the system. However, just 30% of the companies surveyed by TDWI said that data governance teams were heavily involved in Big Data management.

Organizational readiness – As with business intelligence, successfully analyzing Big Data takes more than just installing software and other tools. The entire organization needs to be on the same page, and there must be a clearly articulated strategy built around actual business goals.

Big Data Use Cases

So why do companies use Big Data?

For the same reasons they use other forms of analytics and business intelligence. The goal is to analyze these new types of data and use the knowledge gained to make better decisions for the business.

Some of the most common use cases for Big Data analytics include:

1Getting a more complete view of the customer

In order to better serve current customers and find ways to attract new ones, businesses try to know as much about customers as possible. According to InformationWeek, one of the most common uses of Big Data is to use it to help augment traditional data sources like purchase history with clickstream data showing what customers have done on the company’s website, as well as data gathered from social media sites.

2Using external data for decision making

Big Data doesn’t just include the data generated by the company and its customers. There’s also a lot of information from a lot of other sources that can be valuable for making decisions or developing strategies. For example, crmsearch.com cites one hotel chain that analyzes weather data and information about flight cancellations to deliver mobile ads targeted to stranded travellers who need to book a room for the night. In a similar strategy, a pizza chain also uses weather and power outage data to target ads to customers who have lost power and are unable to cook.

3Spotting trends that may have gone unnoticed

Big Data can also be used to help organizations predict future trends and plan for the long term. CIO.com cites the example of a university health system analyzing health data in municipalities to spot trends in factors such as population growth and chronic disease diagnoses to gauge whether those regions are being adequately served by local healthcare facilities.

Big Data can help companies know more about their customers, their industry, and anything else they want to learn more about.

Big Data vs. Business Intelligence

Big Data and business intelligence are related and are often used in conjunction, but they’re not exactly the same.

They each have the same purpose and goal: to use available data in order to learn more about a business, its markets and its customers in order to make better, more successful decisions.

The difference lies in the types of data being analyzed, which affects the kinds of questions a business can ask.

When analyzing only structured data using business intelligence, you ask the system to run a report around certain parameters. You can dig into the report to get more detail, but you still need to request the specific information you’re looking for. Big Data analysis, on the other hand, is more about allowing the system to sift through the data to find new patterns and insights.

The tools and infrastructure used for Big Data and business intelligence also vary.

Business intelligence requires loading structured data from transactional systems and other sources into a traditional data warehouse, where it’s used to run queries and produce analytics. In Big Data, specialized applications such as Hadoop are required. Unstructured data is fed from various sources into that system where it’s accessed by Big Data applications.

Of course, business leaders don’t care very much about the specific differences between Big Data and business intelligence.

Different investments will be made depending on what the organization needs, but in the end, if the goal is to use the data available to make the best decisions possible for the business, it doesn’t matter whether that data is structured or unstructured, where it comes from, or what software is performing the analysis.

That’s why today, business intelligence and Big Data are often combined. That puts all the valuable information in the same place so users can get a complete look at their business with full, accurate insights.

Many vendors are now beginning to add Big Data capabilities to their business intelligence offerings, and many businesses use a data warehouse filled with both structured and unstructured data.

Big Data Skepticism

As with any widely hyped technology or product, companies need to ask themselves a really important question about Big Data: Do businesses really need it?

Despite the hype, the answer, of course, is that it depends. That’s not to say that Big Data is good or bad or valuable or not, just that organizations need to be careful in deciding if Big Data is right for them.

Here are a few questions businesses can ask when trying to make that decision:

1Is our data big enough?

As with traditional business intelligence, the point of Big Data is to use data to answer questions. So when making a decision, companies need to start by figuring out what questions they want answered, or at least in which specific areas they would like more insight, as well as what data they have or are able to access.

For example if all you want to know is who is buying what and where, you probably don’t need Big Data to find the answers. Structured data in a traditional database will do the trick. Also, remember that volume isn’t the only key attribute in the definition of Big Data. Having a lot of structured data doesn’t make it big.

2Is your data relevant to your business?

It’s important to tie the questions you have in mind and the data you will analyze to actual business results. Business leaders may ask questions out of curiosity, and data from new sources can certainly reveal information about an organization that is interesting, but implementing a Big Data solution is a complex, resource-intensive process, so it should deliver a real, measurable impact.

3Is your organization ready?

As with business intelligence, getting ready to use Big Data takes more than just installing software. Organizations need to be ready for their implementations to be successful. That includes both hardware as well as other considerations.

One aspect companies may struggle with is the sheer amount of information being collected and stored. That may require an upgrade in hardware or the purchase of additional servers. Hadoop is known for being able to run on “industry-standard” hardware; however, as InformationWeek points out, companies may find their own goals require different hardware configurations.

Organizations must also make sure they have the right processes in place for collecting the right data, ensuring its integrity and maintaining data governance. And perhaps more importantly, effectively using Big Data or any other kind of analytics may require a significant culture change.

However, according to the report from the Economist and Terradata, organizations tend to focus on the necessary cultural changes last, and technology and tools first. The report listed two key success factors necessary for building a culture ready to tackle Big Data and advanced analytics:

Top-down leadership – It’s critical that decision makers at the top make it clear that analytics will form the basis for decision-making. Tying Big Data initiatives to organization-wide decision making will help keep those initiatives on track and show the rest of the business that the analytics work is valuable.

Bottom-up engagement – Strong leadership will help bring employees throughout the organization on board. Other keys are making analytics tools easy to use, offering training and support, and aligning the work with day-to-day business goals and tasks.

Big Data represents the potential for new kinds of decision making, according to McKinsey & Company. Organizations and people used to making decisions based on assumptions must learn how to change course when their pre-established ideas are challenged by the data. Otherwise, there’s no point in investing in the technology to deliver those insights.

Talent is also a key consideration. Over the past few years, the number of people out there with Big Data skills and experience has grown, but it can still be tough to find the right talent. More than 80% of IT leaders believe there’s a significant shortage of employees with the skills need to plan and execute Big Data projects, according to a 2013 survey from TEKsystems. Companies will need to find out how to recruit those people, or offering training opportunities to grow those skills among the people they already have.

4Are your expectations realistic?

Companies should also have the right expectations regarding how long a Big Data project will take and how much it will cost. That’s often one of the biggest challenges. Most business leaders believe Big Data can help make better, more-informed decisions, according to a survey from IDG Research. However, only 23% of respondents say their Big Data projects have been fully successful, while 52% say they’ve been somewhat successful.

One way to help have the right expectations is to ensure close communication between IT and the business side from the very beginning. The business should be able to list clear goals, and IT should be able to explain what it will take to get there.

Given all the hype, the most important question business need to ask about Big Data is: Do we really need it?