Five Reasons Massachusetts To Lead Big Data Market

Map of colleges and universities in the Boston, Massachusetts area (Photo credit: Wikipedia)

Big Data is joining the cavalcade of high tech buzzwords. Having been anointed, that means Big Data now has a study trumpeting its gigantic size and growth rate, a raft of companies going after the opportunity, and now a state initiative to capitalize on a public/private partnership. But can Massachusetts take the lead in this new industry?

Before getting into this question, let's explore what Big Data means. Based on interviews with several industry players, there is not one definition. But, as the Mass Technology Leadership Council (MassTLC) President, Tom Hopcroft, told me in a June 7th interview, all of them seem united under the broad banner of the Three Vs:

Volume. Big data involves the extraction of useful insight from volumes of data so large that traditional database tools cannot handle the workload. Hopcroft points out that in the last year, the amount of new data created is sufficient to fill 60,000 Libraries of Congress;

Velocity. Data analysis used to be a so-called batch processing -- managers got reports the morning after the day's transactions. Now, managers need instant analysis so they can use the insights to make decisions when they can still influence business outcomes; and

Big Data companies target a $64 billion database and analytics market, according to the MassTLC's BigData and Analytics report. Within Massachusetts, there are 120 Big Data companies.

These include hardware companies, like EMC (EMC), companies that analyze social media to help marketers pick the right messages and focus new product development, and others that analyze information driving securities prices for traders and operators of trading platforms.

And by 2018, MassTLC estimates that Massachusetts could employ 120,000 people who work in Big Data. This would represent a doubling of the current number -- 12,000 of whom work for the 120 start-ups and the balance who analyze data for big companies in health care, pharmaceuticals, and consumer products, according to the MassTLC.

One Massachusetts big data company, Crimson Hexagon (CH), is based in Boston's Seaport District. CH was co-founded by a Harvard professor and its software helps big companies to analyze social media -- using a 136 billion post database collected since 2008, according to my June 7th interview with CH's marketing VP, Wayne St. Amand.

CH helps an automobile manufacturer, for example, to mirror in its advertising copy the social media conversation about its car. So, for example, if people were talking more about a Toyota (TM) Camry's fuel efficiency on Twitter and Facebook, CH would identify this trend and Toyota might include a reference to fuel efficiency in its advertising.

St. Amand declined to provide revenue figures but he exulted that CH is "busier than ever and there is a lot of demand for what we do." He believes that CH is very popular with big companies because it does deeper analysis than do competing products.

For example, instead of merely telling you trends in specific keywords, it helps clients to identify the factors that are driving those trends. And this makes CH clients more comfortable taking action based on the findings because of the credibility that comes from understanding the factors driving the trends.

Lexington, Mass.-based StreamBase Systems helps financial institutions manage a fire hose of data to make better decisions. Mark Palmer, StreamBase CEO, told me in a June 7th interview -- citing Aite Group research -- that from 2001 and 2012, the volume of electronic trading has risen from under 30% to over 90% -- and this boosts both the volume and velocity of data that traders must analyze.

StreamBase helps retail foreign exchange broker, Gain Capital, to give customers the best prices from among 15 different markets where currencies are traded. Palmer declined to disclose its fee -- but claimed that it's less than the $250,000 a year Gain would pay to do this itself.