3 Big Data Pitfalls To Avoid

Frost & Sullivan urges big data neophytes to get professional guidance in order to avoid three common deployment mistakes.

13 CIOs Share: My Big Mistakes

(Click image for larger view.)

Big data presents big opportunities, but newbies may need experienced professionals in order to avoid three common deployment mistakes. Three of the most common problems in big data deployments are incomplete data collection, false starts, and disruptive drains on IT and data-professional staff productivity, according to a recent report by the research and growth-strategy consulting firm Frost & Sullivan.

In fact, these problems are so pervasive and the associated risks so great that organizations new to big data should seek out assistance from professionals -- whether technology integrators or big data consultants. Tapping professionals "can easily cost the organization less than doing the job in house," writes Jeff Cotrupe, author of Frost & Sullivan's "The World Moves Fast, And Data Is Driving" report.

The three traps the report describes have a corrosive effect, not only big data projects and teams, but also on the IT and business leaders who give such projects the green light, putting their credibility at stake. Here are the few of the risks inside each pitfall:

Failure to capture critical data. With haste and inexperience, you might miss relevant data that could illuminate revenue opportunities or ways to reduce customer churn. If competitors start taking advantage of what you miss, the entire business could be vulnerable.

False starts. Taking multiple shots at big data will delay implementation. The impact of any delay will only be magnified if competitors beat you to a breakthrough.

Resource drains. IT and data-management teams are under pressure to maintain daily operations, deliver new reports and analyses, and incorporate new capabilities. Overburdening employees with too many roles or short-staffing the day-to-day work is not the way to go. In fact, many successful practitioners report that their big data teams are quite separate from preexisting BI, data warehousing, and data management teams.

Underscoring the many technical considerations that go into a successful big data project, the report outlines a multipoint data-management model. There are considerations around everything from knowledge management, master-data management, and big data architecture, to data integration, data warehousing, and search, and to analytics, collaboration, and security and rights management. A confusing array of more than 300 vendors offers products addressing these areas, although 20 to 25 of the largest vendors are capturing the lion's share of big data spending.

"Those considering a big data initiative are ill-served if they simply follow industry buzzwords, listen to a favorite vendor or two, and then map out their data management strategy from there," Frost & Sullivan advises.

The report details a litany of potential big data payoffs in areas ranging from network optimization, product marketing, and mobile commerce, to customer-experience management, customer-profitability management, and revenue assurance. Focusing on the telco industry, the report observes that communications services providers, for example, are beginning to adapt mobile commerce processes, "not just for mobile advertising but to all customer communications, because it is as crucial to reach customers on the go as it is to help brands advertise."

Other big data plays include analyzing customer behavior and attributes against responses to questions and offers, and also identifying influencers and customers who are influenced, using social network analysis techniques.

If you're not sure where to begin with these sorts of analyses, the advice to seek help is worth listening to. There are many advantages to leaning on external experts to start, while keeping internal staff focused on current IT and data-management priorities. But more importantly, experienced hands with big data experience are better prepared to deliver a successful project, says Frost & Sullivan.

As for those who insist on going it alone in the big data arena, perhaps the best advice we've heard comes from Scott Rose, VP of services at analytics consulting firm Think Big Analytics. "If you're going to be a pioneer," says Rose, "you better have some wilderness survival skills." In other words, don't expect quick wins.

Click here to get the full report. For additional thought leadership reports, please visit Frost.com.

Too many companies treat digital and mobile strategies as pet projects. Here are four ideas to shake up your company. Also in the Digital Disruption issue of InformationWeek: Six enduring truths about selecting enterprise software. (Free registration required.)

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of ... View Full Bio

Laurianne, one of the ways to address this, and possibly the best way, is by engaging a professional Big Data services team. This can easily cost you less than doing the job in-house when you consider the direct, indirect, and opportunity costs by helping avoid failure to capture mission-critical data, and delayed Big Data implementation. Both can undercut your ability to compete effectively and make you vulnerable to competitors, which leads to missed opportunities, lost revenue, and higher churn. You're absolutely on the mark re: resource drain on your team, in that most IT departments and, more recently, established Data Science teams, are under pressure to maintain daily operations and incorporate new business-enhancing technologies, without increasing staff. By hiring a team of experts, you can keep your staff doing the things they need to do and avoid costs associated with deferring or delaying other tasks.

At Stratecast we have developed 4 questions to ask a Big Data provider to ensure a successful deployment:

1. What is the data strategy?

2. What will the supporting IT systems infrastructure be?

3. What measures are you taking to ensure data protection and compliance?

4. What components and characteristics are you implementing to tune the Big Data platform for optimal performance?

We analyzed many key constructs, models, business drivers, and competitive dynamics--including listings of more than 300 providers, grouped by their areas of expertise, and mapped into the functions they provide within those models--in Stratecast's report The World Moves Fast, and Data is Driving/part 2: Competitive Strategies (BDA 1-04, November 2013).

I can tell you many IT projects suffer from the same pitfalls. It seems business management can't get their heads around the necessary people resources to apply technology to the processes of the business. Add a product or service sold by the business to customers and everything is lined up all well and good, but technology not so much. Existing IT personnel are usually consided a sunk cost readily available for more and more types of projects however a product manager responsible for a product sold by the company is never considered readily available for more and more product types. Okay, I can get off my horse now.

Okay, that covers a few things you want to avoid. Here are some "dos" based on a survey by Nemertes Research:

1. Have a separate big data budget. Nemertes finds that there's a strong correlation between having a dedicated big data budget and having success with big data. 60% of respondents say they have separate budgets and another 5% say they're headed in that direction. The remaining 35% do not have separate big data budgets, but those with separate budgets report about 20% higher success rates than those without separate budgets, says Nemertes.

2. Plan to manage the complete data lifecycle. Address everything from acquisition, classification, and management to analysis, visualization and end-of-life disposal, with security and compliance safeguards at every step along the way. The most successful big data practitioners address the complete lifecycle of big data. End-of-life planning is the step that big data practitioners most often ignore, but that can lead to bad decisions handled in a haphazard way.

3. Engage IT. Even if -- correction, particularly if -- you have a separate big data team, involving many people across IT makes good sense because big data initiatives are likely to touch on all parts of the IT infrastructure and organization -- data processing, database management, reporting, storage, networks, app development and so on.

4. Engage the lines of business. Initial big data projects should have a tight focus and scope to help ensure success, but long-range big data planning and strategy development should cast a wide net across business areas. The more successful respondents in Nemertes' research more often engaged business leaders in big data decision making and guidance.

5. Embrace a shared spending model. The more successful big data practitioners report that their firms have higher levels of technology spending outside of IT's control. About 28% of tech spending is outside of the IT budget at companies reporting big data success whereas only about 4% of spending is handled outside of IT among firms self reporting as less-successful with big data. The non-IT spending is often tied to software-as-a-service applications in sales and marketing areas or infrastructure-as-a-service offerings that might be used for analytic exploration.

ITís tried for years to simplify data analytics and business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.