Build YARN Apps on Hadoop with Apache Slider: Technical Preview Now Available

A significant reason for the increased adoption of the Hortonworks Data Platform by customers and partners has been Apache Hadoop YARN. This major advance, introduced last October in HDP2, allows Hadoop to move from many single-purpose clusters to a versatile, integrated data platform that hosts multiple business applications.

YARN has become the architectural center of Hadoop. We intend to make it easier for applications to work in a YARN environment, and benefit from the integrated capabilities and technologies that form the blueprint for enterprise Hadoop. For that reason, we’re very excited about Apache Slider.

Introducing Apache Slider

In April, Hortonworks submitted Slider to the Apache Software Foundation as an incubator project. Two months later, we’re happy to make Slider available as a Technical Preview.

Slider provides a set of platform services that enables other applications to quickly run in the YARN environment. It provides four key benefits:

Simplified on-boarding of existing apps to Hadoop YARNWith Slider, distributed applications that aren’t YARN-aware can now “slide into YARN” to run on Hadoop – usually with no code changes.

Full capabilities of a YARN applicationSlider allows applications to be provisioned, started, stopped, restarted on failure, monitored, expanded & shrunk, all while the application is running. It allows users to create and run multiple instances of applications, even with different application versions if needed.

Integrated participation in enterprise HadoopSlider allows applications to participate in a Hadoop ecosystem in an integrated and cooperative way, leveraging Hadoop’s data and processing resources, as well as benefiting from the security, governance, and operations capabilities of enterprise Hadoop.

Slider allows long-running applications, real-time services and online applications to easily integrate into a Hadoop environment. It complements Apache Tez, which is quickly gaining adoption as the batch and interactive engine of Hadoop. These engines, depicted in the diagram below, provide targeted mechanisms for developers to create YARN-based applications.

Running HBase, Accumulo and Storm under YARN

Along with the Slider technical preview, we are including three Apache projects as samples for Slider:

Apache HBase

Apache Accumulo

Apache Storm

Customers who have been eager to run these projects within their Hadoop cluster are encouraged to try them out. For developers, we’ve included the setup and configuration information for each of these projects to assist in understanding how to “slide” existing distributed applications onto YARN.

Managing Slider Applications

As Slider brings new applications to the Hadoop ecosystem, the applications will need to be managed – not just by YARN, but also by Hadoop operators and administrators. The answer, of course, is to leverage the open source Apache Ambari project. Slider automatically makes the application manageable by Ambari without further work on the application’s part. Today we’re also releasing a Slider View for Ambari Technical Preview that can manage Slider applications.

Innovating the Core of Enterprise Hadoop

As YARN took Hadoop 2 beyond batch, Slider will further accelerate that promise by bringing new applications to Hadoop, and lowering the barrier to run in a YARN environment. It’ll do so in a way that treats the applications as an integrated part of the Hadoop ecosystem.

Slider continues Hortonworks’ commitment to innovating at the core of 100% open source Enterprise Hadoop, while enabling partners to expand the range of solutions that make up the modern data architecture. We think Slider will change the face of Hadoop, and we’re excited for you to try it today.

Hadoop YARNPart of the core Hadoop project, YARN is the architectural center of Hadoop that allows multiple data processing engines such as interactive SQL, real-time streaming, data science and batch processing to handle data stored in a single platform, unlocking an entirely new approach to analytics.… Read More