Application Development on Apache YARN Simplifies with Release of Weave

Developers love the latest and greatest development tooling. And next up it’s Kafka, a real-time framework for managing data streams or YARN (or Yet Another Resource Manager) aka MapReduce 2.0, the next generation processing framework for Apache Hadoop.

Continuuity, the provider of scale-out application server for Apache Hadoop, announced the release of Weave, a new open source project for Java developers to build, deploy, scale and manage scalable, distributed applications on YARN.

Weave is a simple set of libraries for easy management of distributed applications on Apache YARN. The tool allows you to build a much wider class of applications on top of YARN. The continued popularity of the JVM (Java Virtual Machine) has breathed new life into languages. Weave is similar to a local JVM as it makes developing Java-based YARN applications even easier.

“We developed Apache Hadoop YARN to support a wide set of use cases in a generic manner, and it is very important to make it easy for developers to exploit the features and capabilities of YARN. We are excited to see the new types of applications that can be built on YARN using Weave,” said Arun Murthy, co-founder of Hortonworks. “We are thrilled that Continuuity has decided to help make YARN even easier to use, and that they are sharing their efforts with the community.”

YARN is used to run and manage BigFlow applications and is extremely powerful, but it is difficult to get test and debug, and complex to build new kinds of non-MapReduce applications and frameworks on it. Weave, under the Apache 2.0 license, extends the benefit and power of Apache YARN and use YARN’s distributed capabilities with a programming model that is similar to running threads.

“Continuuity’s mission is to make building Big Data applications easier for developers. With the release of Weave, we’re supporting an increasingly diverse set of real-time application patterns to spur innovation in the Hadoop ecosystem,” said Jonathan Gray, CTO and co-founder of Continuuity. “Our goal is to continue collaborating with the community and improving Weave and YARN.”

Weave features simple API, simplified archive management, improved control over application logs, metrics and errors, better communicate with an application and easy communication between containers and the Application Master.

Premium Research

Wikibon argues strongly against Revolution towards a 3rd platform. The conclusion from this analysis is that applications will evolve; conversion should be avoided like the plague. The greatest opportunity is to continuously adapt today's operational applications by the addition of real-time or near real-time analytics applied directly to the current organizational processes and applications that support these processes. This is likely to translate to the greatest value to most organizations, and where possible avoid the risks of converting systems. The study of organizations that have applied real-time analytics to their current operational systems have shown incredible improvements in lower costs and greater adaptability. Business and IT executives should understand the enormous potential for adding decision automation through real-time analytics to current operational applications in their organizations. New technologies should be judged by their ability to support real-time analytics applied to operational systems, and supporting incremental improvement over time.

In a recent web-based survey conducted by Wikibon, 300 North American enterprises whom had either been utilizing, or considering the adoption of public cloud, answered questions regarding IaaS (Infrastructure as a Service) perceptions and usages. These questions varied in topic but were centered around an examination of which workloads were best suited for usage in the public cloud. This research examines a few additional key insights that shed some light on the growing IaaS world.

Today's Technology infrastructure management is largely non-differentiated and wasteful. Technology executives must re-think the strategic role of human capital and begin to implement new ways to consume IT as a service. This post draws on the learnings of senior executive Alan Nance from Royal Philips who is dogmatic in its approach to transforming its infrastructure to a service model.

There have only been two successful volume introductions into the marketplace in the last 50 years - DRAM and NAND flash. There has to be a clear volume case with good economics for 3D XP to be able to gain a foothold in consumer products. Without volume in the consumer space, there is unlikely to be much volume traction in the enterprise space. CIOs, CTOs and enterprise professionals should take a wait and see stance, and monitor the adoption of 3D XP in the consumer and military spaces. If and when there is volume production for 3D XP, enterprise adoption should start about two years later.

The use of open source software continues to accelerate and expand in the marketplace, especially in areas where technology is significantly disrupting established business models. IT organizations should be actively seeking to understand how open communities operate, how different licensing models work, and how they can be more actively engaged with both the vendors and communities that are shaping open source software.

CIOs understand that a clear cloud strategy is critical for IT today. Wikibon believes the biggest mistake organizations can make is converting major applications into the public cloud (including SaaS) without thinking about the implications to their existing business process workflows. Wikibon recommends that IT develop and implement a hybrid cloud strategy using the existing management workflows and compliance processes for both the public and private cloud components in the hybrid cloud.

In 2014, Wikibon defined a new category "Server SAN" that sits at the intersection of software-defined storage, hyperscale methodologies and converged infrastructure. This article is the executive summary of primary research that gives the status of the market, examines the vendor ecosystem, lays forth the revenue and 10 year forecast and gives direction for expansion beyond simple "hyperconverged infrastructure". This information is available for public consumption, the full research is available to Wikibon clients.

In this research paper, Wikibon looks back at the introductory Server SAN research, adjusts the Server SAN definition to include System Drag, and increases the speed of adoption of Server SAN based on very fast adoption from 2012 to 2014. The overall growth of Server SAN is projected to be about a 23% CAGR from 2014 to 2026, with a faster growth from 2014 to 2020 of 38%. The total Server SAN market is projected to grow to over $48 billion by 2026. The traditional enterprise storage market is projected to decline by -16% CAGR, leading to an overall growth in storage spend of 3% CAGR through 2026. Traditional enterprise storage is being squeezed in a vice between a superior, lower cost and more flexible storage model with Enterprise Server SAN, and the migration of IT towards cloud computing and Hyperscale Server SAN deployments. Wikibon strongly recommends that CTOs & CIOs initiate Server SAN pilot projects in 2015, particularly for applications where either low cost or high performance is required.

If containers are at the center of a shift in how applications are developers and delivered, and their pace of growth and change is unprecedented in IT history, this could have a massive ripple effect on both suppliers and consumers of the ecosystem of IT technologies.