Elasticsearch on YARN and SSL Support in Elasticsearch Hadoop

I am happy to announce Elasticsearch for Apache Hadoop 2.1.Beta3 has just been released. We are introducing two new features: SSL connectivity and enhanced HTTP authentication and dedicated support for running Elasticsearch on YARN.

Elasticsearch on YARN

With 2.1.Beta3, we introduce the Elasticsearch-on-YARN (aka es-yarn) project for running an Elasticsearch cluster within a YARN environment. Similar to the repository-hdfs plugin, es-yarn is distributed as part of the Elasticsearch for Apache Hadoop (aka es-hadoop) project, but is independent and has no dependencies outside YARN itself.

With es-yarn, one can now provision, start and stop Elasticsearch directly on a YARN cluster. In YARN lingo, es-yarn bootstraps a client that deploys a dedicated ApplicationManager in YARN which, on its behalf, creates one container for each Elasticsearch node required.

For the user, es-yarn is a straight-forward CLI (Command-Line Interface) for deploying and managing the life cycle of the Elasticsearch cluster within YARN.

Stop the cluster

SSL and HTTP authentication

es-hadoop uses REST over HTTP to communicate with Elastisearch. Release 2.1.Beta3 introduces official support for basic HTTP authentication allowing Hadoop jobs running against a restricted Elasticsearch cluster to identify themselves accordingly. While es-hadoop has supported authentication through its proxy options, with 2.1.Beta3 it is graduated to an individual component and thus can be used within or outside the context of a proxy configuration.

Further more, the new 2.1 Beta release introduces SSL/TLS support for cryptographic connections between Elasticsearch and your Hadoop cluster. Thus data-sensitive environments can transparently encrypt the data at transport level to prevent snooping and preserve data confidentiality.

Note that while self-signed certificates are supported (though are disabled by default) for development, for production environments we strongly recommend using a proper authority to create your certificates.

Strata Barcelona

If you happen to be in Barcelona next week and are interested in Elasticsearch, we'd love to talk to you!