How-to: Use Vagrant to Set Up a Virtual Hadoop Cluster (For CDH 4)

This guest post comes to us from David Greco, CTO of Eligotech. For a how-to on this subject for CDH 5, see this post.

Vagrant is a very nice tool for programmatically managing many virtual machines (VMs) on a single physical machine. It natively supports VirtualBox and also provides plugins for VMware Fusion and Amazon EC2, supporting the management of VMs in those environments as well.

Vagrant provides a very easy-to-use, Ruby-based internal DSL that allows the user to define one or more virtual machines together with their configuration parameters. Furthermore, it offers different mechanisms for automatic provisioning: You can use Puppet, Chef, or shell scripts for automating software installation and configuration on the machines defined in the Vagrant configuration file.

A typical use case for Vagrant is to build working/development environments in a simple and consistent way. At my company, Eligotech, we are developing a product aimed to simplify the usage of Apache Hadoop, and CDH, Cloudera’s open source distribution, is our reference Hadoop distribution. We often need to set up a Hadoop environment on our machine for testing purposes, and we found Vagrant to be a very handy tool for that purpose.

I put together an example of a Vagrant configuration file that you can test for yourself. You’ll need to download and install Vagrant (instructions) and VirtualBox. Once everything has been installed, just copy-and-paste the text below to a file named Vagrantfile and put it in a directory named, for example, VagrantHadoop. This configuration file assumes you have at least 32GB of memory on your box; if that’s not the case, you can edit the file to suit your environment (to run fewer slaves, for example, by commenting out some of the slave configurations).

This file defines six machines to be assigned the following CDH 4 roles:

vm-cluster-node1: This is the master; besides running the CM master, it should run the namenode, secondary namenode, and jobtracker.

vm-cluster-node2: This is a slave, it should run a datanode and a tasktracker.

vm-cluster-node3: This is a slave, it should run a datanode and a tasktracker.

vm-cluster-node4: This is a slave, it should run a datanode and a tasktracker.

vm-cluster-node5: This is a slave, it should run a datanode and a tasktracker.

vm-cluster-client: This machine plays the role of gateway for the cluster.

Click here to learn the meaning of the different items in the configuration file. In particular, you can see that depending on the particular provider, either VirtualBox or VMware Fusion, the memory size is changed in a different way. Observe how simple it is to switch between providers for customizing environment-specific things!

This Vagrant file does another very important thing: It installs Cloudera Manager automatically on the master node, vm-cluster-node1.

To create the virtual cluster, open a shell and just go to the directory holding the Vagrant file, i.e. VagrantHadoop. Under that directory, run:

1

&gt;vagrant up--provider=virtualbox

After a while, depending on how fast your machine is, Vagrant will return control — meaning that all the VMs are up and running.

At this point you are ready to configure your cluster through CM’s web UI via http://vm-cluster-node1:7180.

Filed under:

This is fantastic. Thank you very much for the Vagrantfile config – I’m sure I’m not the only one out there that had been trying to hack this out myself for a while.

One thing to note is that the box the configuration depends on (assuming its Vagrant’s official “precise64″) needed to be installed manually with this command before I could vagrant up successfully:
vagrant box add precise64 http://files.vagrantup.com/precise64.box