It's the same approach used by Google, Twitter, Facebook and other high-volume sites to analyze their Web log data. It opens up that overwhelming world of machine-created data for new analysis, and organizations are putting it to use in intriguing ways. For instance, the Tennessee Valley Authority has sensors on power-transmission lines and facilities across the country that create data at an approximate rate of 50 to 100 times a second, says Cutting. Hadoop makes it affordable to save and analyze that data, which is helping the TVA identify issues, patterns and grid problems. That, in turn, is helping the agency predict-and avoid-power failures.

While Hadoop can handle the storage and processing, integrating that much data is still something of a challenge. Third-party companies are stepping in to help with that task, including Cloudera, where Cutting is now the chief architect. Cloudera mostly partners with other companies, who use its tool, Sqoop, to pull Hadoop data into relational databases. Pentaho and Talend offer solutions that simplify the integration process with more traditional data integration interfaces and tools, and IBM plans to. Previously, you needed Java programmers to custom write this sort of integration, according to Pentaho's founder and CEO Richard Daley.

More organizations will be looking for these types of solutions if Karmasphere's survey of Hadoop developers is correct. Karmasphere is another tech company focused on making Hadoop more accessible, though it focuses on developers skilled in traditional programming languages. Karmasphere CEO Martin Hall says companies are uncomfortable making business decisions based on samples of data. Hadoop lets them analyze more data (way more data, actually), and that's driving its adoption.