Hallelujah or headache? For businesses, it’s all about being ready to ride this perfect storm of big data – and their understanding of what’s at stake.

“I think everything’s at stake,” said Eron Kelly, general manager of product marketing for Microsoft SQL Server. “Organizations that harness the power of big data will outperform their peers.”

Digital data is now more vast than all the world’s oceans (there is 2.7 zetabytes of data on the planet versus the 1.37 zettalitres of seawater) and it’s multiplying at breakneck speed. And it’s just as easy to drown in information if your organization doesn’t have the right tools to garner useful insights from big data, Kelly said.

Today, the promise of big data remains largely unfilled. Microsoft’s goal is to make big data accessible to the masses, Kelly said. Whether it’s businesses, educational institutions, healthcare companies or governments, if organizations don’t take full advantage of all of the information around them, they will fall behind.

As big data becomes more mainstream, the tools used to manage it must follow suit, which is why Microsoft has invested in providing a wide-ranging suite of tools for all types of users – the geneticist sequencing and comparing DNA, a bed-and-breakfast owner looking at occupancy rates alongside weather data, or a car manufacturer looking for ways to increase productivity and reduce inefficiencies.

“Those that are able to derive insights from data will make better decisions,” Kelly said. “They’ll be more efficient, and they’ll move whatever agenda it is that they have forward much faster than those that don’t.”

Tools of the Trade

Dave Campbell, a technical fellow at Microsoft, has a degree in robotics and has worked with large data sets his entire career. About five years ago he started investigating big data issues in earnest.

Why the buzz now, he asked himself recently. What has fundamentally changed?

His conclusion is that big data, and its potential applications in business and beyond, have reached a tipping point. There are several reasons for this, he said. First, more data than ever is available in digital form. Data storage is now inexpensive and plentiful. And, finally, more advanced computers and software are handling the new deluge of data with gymnastic algorithms that can help spot never-before-seen trends and provide new insight.

What’s more, the technology needed to harness big data is available in the cloud, which makes it even more accessible for businesses – no up-front costs or infrastructure are needed to get going.

Armed with an understanding of the worth of big data, Campbell said, businesses should think of their data in two different dimensions of value – refined data, and combined data. Microsoft can help with both, he said.

Dave Campbell, technical fellow at Microsoft.

Businesses can store large amounts of data with Windows Server, and once they have all of their data, they can manage and refine their structured databases with Microsoft SQL Server 2012.

Using their structured data, most universally found in databases that use Structured Query Language (SQL), businesses can select exact pieces of that data using columns and rows – perhaps the rows with a certain zip code or the columns with a specific product type.

However, one of the biggest areas of growth and opportunity in big data is around unstructured data. This data, which includes everything from email and Tweets to photos on Flickr and likes on Facebook, doesn’t have the architecture of structured data but can be just as valuable to a business.

Microsoft is working with communities around Hadoop, an open-source data platform for managing unstructured data, to help customers work will all types of data, both structured and unstructured.

“Being able to reach out and combine the data I have and to work with other groups and organization to bring in the world’s data provides a tremendous amount of value,” Campbell said. “Our approach is to put tools in the hands of businesses and other users that will allow them to derive their own insights.”

Microsoft is also working to integrate Hadoop with SQL Server and Windows Azure to ensure customers can combine all their data sources. The Windows Azure Marketplace can help businesses find trustworthy third-party data to combine with their own.

“Say I have a hotel on the beach in Florida, and I want to bring in weather data to improve my business planning for my hotel’s occupancy, and to better forecast room rates and demand. Now I can take my data, and combine it with information from an outside organization without making a lot of investment,” Kelly said.

Ashvini Sharma, group program manager in Microsoft’s Office Business Intelligence team.

With a firm handle on their data, and by incorporating outside data that can be combined with in-house data to help it pack an insightful punch, businesses can then turn to Microsoft’s business intelligence capabilities within Office, for example. Applications such as Excel, PowerPivot and SharePoint can help them find insight, analyze, and visualize that data.

“We know users, we know what they’re looking for, and we can provide them highly accessible ways of making a decision,” said Ashvini Sharma, group program manager in Microsoft’s Office Business Intelligence team.

Sharma said the main people using big data today are data scientists and others who are highly proficient at using technological tools. But that is all changing, and Microsoft will provide a familiar and intuitive platform for accessing big data, he said.

Sharma spoke to a business customer last year who was trying to incorporate more big data insight. Because Hadoop runs on Linux, the company would remotely connect its PCs to a Linux machine; open a text editor where they would type a query to Hadoop using the data warehouse system Hive to execute a search in that system; and then wait for minutes, hours or days for the search to return the search results. The company would FTP the search results back to PCs to open them.

“This is what people are doing today to get insights out of the big data world,” Sharma said.

What if businesses could run the query with their existing tools such as Office and save all of that effort? This is the big data value Microsoft is bringing to businesses.

”You’ll just take what you already know today and extend it to a new set of technology to get insights and wisdom you used to have to wait or ask others to get for you,” Sharma said.

Microsoft even uses its own big data tools, including Bing, to deliver more than 100 petabytes of data in the form of search results; at Microsoft Advertising to target 14 billion ads per month; at Kinect for Xbox 360, where machine learning and sensors have revolutionized hands-free controlling; and at Exchange, where Microsoft uses machine learning to detect spam in up to four billion e-mails a day.