Sunday, April 27, 2014

Installing Hadoop is a hassle; it involves a variety of steps, some proficiency on Linux commands and writing to various files. If you have tried manual installation, you know what I'm talking about.So, here is a simple Linux shell script. Save the following script as bulletinstall.sh and on your Ubuntu-ready machine, run it using:$ sudo sh bulletinstall.shThis script has been tested on Ubuntu 14.04 LTS; if you experience any issues, feel free to drop a comment. Here is the script:#!/bin/bash# This document is free to share and/or modify, and comes with ABSOLUTELY NO WARRANTIES. I will not be responsible for any damage or corruption caused to your Computer. Do know your stuff before you run this and backup your important files before trying out.# Author: owaishussain@outlook.com# LINUX SCRIPT TO INSTALL HADOOP 1.2.1 ON A MACHINE# If you already have this file, then put it in /tmp directory and comment out "wget"HADOOP_URL=http://www.us.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gzecho "************************************************************"cd /tmpecho "Downloading Apache Hadoop from $HADOOP_URL (you may change the version to any other, but this one has been tested)"wget "$HADOOP_URL"echo "STEP 1/10 COMPLETE..."echo "************************************************************"# Hadoop will be deployed on "/usr/local" directory, if you want to change it, then modify the path under this sectionecho "************************************************************"echo "Extracting files..."tar -xzf hadoop-1.2.1.tar.gzecho "Copying to \"/usr/local/\""cp -R hadoop-1.2.1 /usr/local/echo "STEP 2/10 COMPLETE..."echo "************************************************************"# Oracle JDK7 will be downloaded from a 3rd party repository (ppa:webupd8team/java. Credit to them). In case, it is unavailable, you may use an alternative in the first command under this sectionecho "************************************************************"echo "Installing Oracle JDK7. Please accept the Oracle license when asked"add-apt-repository ppa:webupd8team/javaapt-get update# If you are skeptik about JDK7 (and I won't blame you for that), you may switch comments respectively, in the two lines below to install JDK6apt-get install oracle-java7-installer#apt-get install sun-java6-jdkecho "STEP 3/10 COMPLETE..."echo "************************************************************"# This script assumes that you will run Hadoop on new user \"hadoop\". If you wish to choose different user, you're most welcome to; just comment whole section below, but be sure to replace the username everywhere in the script with what you wantecho "************************************************************"echo "Creating user and group named\" hadoop\""adduser hadoopadduser hadoop sudoecho "Creating home directory for user"mkdir -p /home/hadoop/tmpecho "Assigning rights on home directory"chown -R hadoop:hadoop /home/hadoop/tmpchmod 755 /home/hadoop/tmpecho "Changing ownership of Hadoop's installation directory"chown -R hadoop:hadoop /usr/local/hadoop-1.2.1echo "STEP 4/10 COMPLETE..."echo "************************************************************"# This is to test and initialize the newly created userecho "************************************************************"echo "Logging in. ***** PLEASE PROVIDE PASSWORD AND RUN 'exit' (without quotes) TO LOG OUT *****"su - hadoopecho "STEP 5/10 COMPLETE..."echo "************************************************************"echo "************************************************************"echo "Setting environment variables"cd /usr/local/hadoop-1.2.1/confecho "export JAVA_HOME=/usr/lib/jvm/java-7-oracle" >> hadoop-env.shecho "STEP 6/10 COMPLETE..."echo "************************************************************"echo "************************************************************"echo "Configuring Hadoop properties in *site.xml"mv core-site.xml core-site.xml.bcktouch core-site.xmlecho "<?xml version=\"1.0\"?><?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?><configuration><property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/tmp</value></property><property> <name>fs.default.name</name> <value>hdfs://localhost:54310</value></property></configuration>" > core-site.xmlmv mapred-site.xml mapred-site.xml.bcktouch mapred-site.xmlecho "<?xml version=\"1.0\"?><?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?><configuration><property> <name>mapred.job.tracker</name> <value>localhost:54311</value></property></configuration>" > mapred-site.xmlmv hdfs-site.xml hdfs-site.xml.bcktouch hdfs-site.xmlecho "<?xml version=\"1.0\"?><?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?><configuration><property> <name>dfs.replication</name> <value>1</value></property></configuration>" > hdfs-site.xmlchown hadoop:hadoop /usr/local/hadoop-1.2.1/conf/*-site.xmlecho "STEP 7/10 COMPLETE..."echo "************************************************************"echo "************************************************************"echo "export JAVA_HOME=/usr/lib/jvm/java-7-oracle" >> /home/hadoop/.bashrcecho "export HADOOP_HOME=/usr/local/hadoop-1.2.1" >> /home/hadoop/.bashrcecho "export PATH=$PATH:/usr/local/hadoop-1.2.1/bin" >> /home/hadoop/.bashrcecho "STEP 8/10 COMPLETE..."echo "************************************************************"echo "************************************************************"echo "Installing OpenSSH Server"apt-get install openssh-serverssh-keygen -t rsa -P ""cat /home/hadoop/.ssh/id_rsa.pub >> /home/hadoop/.ssh/authorized_keysecho "STEP 9/10 COMPLETE..."echo "************************************************************"echo "************************************************************"echo "Removing temporary files"# If you want to preserve the downloaded Hadoop application, you may comment out the first command under this sectionrm hadoop-1.2.1.tar.gzrm -R hadoop-1.2.1echo "STEP 10/10 COMPLETE..."echo "************************************************************"echo "Congratulations! Your Hadoop setup is complete. Please log into hadoop user and start hadoop services."