Karmasphere: Helping to Tame A Tool Called Hadoop

Hadoop is one of those buzzwords that pop up frequently in Silicon Valley, baffling all but the people who toil deep in data centers. But that’s likely to change, and a startup called Karmasphere hopes to be one of the reasons.

The software, more formally known as Apache Hadoop, springs from the desire of people to gain insight from large computing jobs that produce staggering amounts of data. This is an area where Google has had huge influence, coming up with ways to spread complex tasks over thousands of small, inexpensive server systems and analyze the information they generate. The technology Google developed to handle these tasks is called MapReduce.

Hadoop is an attempt to emulate the benefits of MapReduce. It was developed by a former developer at Yahoo named Doug Cutting–who named the program after his child’s stuffed toy elephant–and is now at a startup called Cloudera that is a distributor of the software.

Hadoop is already being used by a surprising array of companies, helping them more effectively distribute their computing jobs among servers in their own data centers or public computing services operated by companies such as Amazon.com.

Software companies in the past developed specialized products called data warehouses that helped analyze business data, frequently some time after it was generated. Think about rolling up the tally from a retail chains’ cash registers once a week, or overnight, and running queries against that information to try to glean customers’ actions and preferences.

But the Web is generating even more data, and companies want to analyze it much more quickly so they can change what visitors see on their Web sites. “This is somebody figuring out this morning what they are going to change this afternoon,” says David Liddle, a Silicon Valley venture capitalist who is a veteran of data-analysis ventures.

The appeal of Hadoop is one reason that Liddle, general partner at U.S. Venture Partners, and Ann Winblad, another experienced investor in the field who co-founded Hummer Winblad Venture Partners, are jointly plunking $5 million into Karmasphere, a startup that has developed tools to make Hadoop more accessible.

The closely held company, based in Los Gatos, Calif., is focusing on software for personal computers that it says helps developers create, deploy, debug and monitor applications created with Hadoop that run on servers. While Hadoop is powerful, it is is also “pretty technical and has a tedious learning curve,” says Martin Hall, Karmashere’s chief executive and co-founder.

Karmasphere has already been distributing a free version of its software, Hall says. With the initial funding round, to be formally announced Thursday, the company will work on introducing a commercial version, he says.

What companies ultimately want to do, Winblad says, is to find value in information from business operations that they now don’t have the power to mine, which is sometimes called data exhaust. “It’s not just exhaust anymore,” she says. “It’s truly how you optimize your business.”