YARN: Answer To Hadoop's Shortcomings?

YARN framework may not be ready for prime time, but it could bring Hadoop to the next level, says Pervasive Software's chief technologist.

Apache Hadoop is certainly not the only open source distributed file system solution to emerge in recent years -- competitors such as HPCC are out there as well -- but it's certainly the best-known and most popular platform in the burgeoning big data space.

Despite its adeptness at bringing data processing and analysis to raw storage, however, Hadoop has its shortcomings, such as batch processing delays and its reliance on MapReduce for data processing.

YARN, which stands for Yet Another Resource Negotiator, is a new framework that Cloudera calls "more generic than the earlier MapReduce implementation," in that it runs programs that don't follow the MapReduce model.

"In a nutshell, YARN is our attempt to take Hadoop beyond just MapReduce for data processing. What YARN allows you to do is do much more than MapReduce in the same Hadoop cluster," Arun Murthy, chair of Apache Hadoop Project Management Committee and one of Hortonworks' founders, recently told InformationWeek.

Pervasive Software chief technologist Jim Falgout uses this analogy: Think of YARN as a distributed operating system -- "which it is," he told InformationWeek in a phone interview. "It has a distributed scheduler, a distributed file system. And it allows you to run jobs that are distributed."

But pre-YARN Hadoop is "basically an operating system that allows you to run one application only," Falgout chuckled. "That's MapReduce." He added, "YARN is a really important part of Hadoop going forward. It shows a lot of maturity in the Hadoop community to say that MapReduce is great, but it doesn't solve all the problems. It's not for every solution that you want to build."

Falgout has addressed this topic on other forums as well. "MapReduce is great at solving problems such as indexing the world's web sites. However, it's not the most flexible or the most efficient platform for every computation problem out there," he wrote in a recent blog post on the Data Science Central site.

YARN opens up Hadoop, allowing developers to build different types of applications that take advantage of the platform's attributes. "If you want to work with Hadoop, pre-YARN, you have to write all of your applications on this nice, distributed operating system in MapReduce," Falgout said. "If Linux was built that way, it wouldn't have gotten very far."

Given Falgout's thoughts on the YARN, it's little surprise that Pervasive Software, an Austin, Texas-based provider of data management and analytics products, is deeply involved in Hadoop and big data. The company's software offerings include DataRush, which is designed to boost parallel performance of data preparation and analytics tasks, and RushAnalyzer, a visual workflow tool for data access, preparation, analysis and reporting.

Pervasive plans to port its DataRush framework to YARN, Falgout said, but has yet to announce a specific timetable. "We don't see YARN being used in a lot of customer production systems yet," he added. "We know it's coming, so we're going to start working on YARN probably within the next month."

Cloudera, meanwhile, has implemented an early version of YARN in its CDH4 distribution of Hadoop, but the software is considered an alpha release. "It's out there. You can look at it," said Falgout. "You can play with it, but it's not ready for prime time."

YARN's implementation wouldn't directly impact end users, of course, but it could offer indirectly benefits that make the notoriously difficult Hadoop platform easier to use. "That was kind of a big theme at Strata Hadoop World recently: Making Hadoop more adoptable, making it easier to use," Falgout said. "Getting it out beyond the early adopters who were willing to write MapReduce code, and go through all that pain because they were getting a big benefit out of it."

Predictive analysis is getting faster, more accurate and more accessible. Combined with big data, it's driving a new age of experiments. Also in the new, all-digital Advanced Analytics issue of InformationWeek: Are project management offices a waste of money? (Free registration required.)

There's no doubt Google has made headway into businesses: Just 28 percent discourage or ban use of its productivity ­products, and 69 percent cite Google Apps' good or excellent ­mobility. But progress could still stall: 59 percent of nonusers ­distrust the security of Google's cloud. Its data privacy is an open question, and 37 percent worry about integration.

IT’s tried for years to simplify business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.