Big Data Planning: Start Slow, Let It Grow

Don't begin with an end in mind. Allow yourself to be surprised by what big data reveals.

5 Big Wishes For Big Data Deployments

(click image for larger view and for slideshow)

"It's the start that stops most people," a quotation widely attributed to anonymous, is one I keep in mind when struggling to get started on a project. Whether it's researching a new car or motivating myself to play tennis again, the hardest step is often the first one.

For enterprise IT, big data projects are a classic tough starter. Companies can get overwhelmed when they over plan and insist on sticking to a regimen. Big data doesn't lend itself to step-by-step schemes and a predetermined finish line, according to Phil Simon, author of the book Too Big to Ignore: The Business Case for Big Data.

"Hadoop and other big data solutions represent a fundamentally more flexible, ad hoc and organic approach to data modeling," Simon writes. "Meeting a business need trumps following regimented, predefined models. You don't need to begin with the end in mind."

Before moving into the deep waters of a big data implementation, consider the following tips on how to start, grow organically and build momentum, culled from Simon's book and a recent InformationWeek big data research report.

Aim For Little Victories

Every big data project doesn't require the CIO to submit an RFP and earmark millions of dollars. The beauty of software such as open-source Hadoop, Simon writes, is that it lets organizations take on big data organically and affordably.

"Why not start a bit conservatively with reasonable investments in hardware, software and additional headcount?" Simon writes. "And don't forget that cloud solutions may essentially obviate the need for hardware purchases and upgrades."

Think in terms of achievable, short-term goals. When it comes to your company's customers, Simon writes, the basics include gathering unstructured data on current and former customers to predict which products will gain traction; improving your understanding of customer behavior on your website to reduce customer churn and to retain your most valuable customers; and improving your company's website design and product offerings based on customer behavior and specific Web traffic metrics so you can contact customers who are about to defect (without annoying them of course).

When it comes to employees, gather more structured data on current and former ones to predict which employees will be successful; gather unstructured data such as performance reviews and exit interview notes to minimize bad hires and employee turnover; and increase understanding of your current workforce through both structured and unstructured data so you can proactively communicate with valuable employees who are thinking of leaving the company.

Keep An Open Mind And Let Your Plan Evolve

It's quite possible that after you've homed in on the right data, you'll get some unexpected results. Keep an open mind to what big data analysis might reveal.

For example, Simon's book cites what big data trailblazer Wal-Mart learned back in 2004 about preparing its inventory for a hurricane. Logic would dictate stores should stock lots of batteries, flashlights, canned goods and bottled water. While its data analysis confirmed that thinking, Wal-Mart also discovered that less obvious products get a bump in sales before big storms.

Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.

Why should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.