Hadoop 3.X Installation Guide

Hadoop 3.x is the latest release of Hadoop which is still in alpha phase. Developers who are interested in Hadoop can install the product and report to Apache if they found any issues or bugs. There are many new features that are introduced in Hadoop 3.x.

In this blog, we will be discussing about how to install Hadoop 3.x in a pseudo distributed mode and exploring HDFS new features.

100% Free Course On Big Data Essentials

Subscribe to our blog and get access to this course ABSOLUTELY FREE.

Here are the list of changes and features that are introduced in Hadoop 3.x

We have created a folder called Hadoop3_data and inside we have created 2 directories with names NameNode & DataNode

The same you can see in the below screenshot.

Configuring ssh & pdsh

Install and setup ssh

If you are using Debian OS, install ssh with the below command

sudo apt-get install ssh

If you are using a Non-debian OS, install ssh with the below command

yum install openssh-server

After the installation, generate the ssh key with the below commands

Generate ssh key for hadoop user using the command:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Copy the public key from .ssh directory to authorized_keys folder.
Change the directory to .ssh and then type the below command to copy the files
into the authorized _keys folder.
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
To ensure whether the keys have been copied, type the command:
cat authorized_keys
Change the permission of the .ssh directory.
chmod 600 ~/.ssh/authorized_keys

After setting, save and close the file and now update the bashrc file using the command source .bashrc

NameNode Format

That’s it your Hadoop 3.x is ready. Let’s now format the name node.

Use the command ./hdfs namenode -format in the $HADOOP_HOME/bin directory

After successful format, you will get the message as shown in the below screen shot

We have successfully formatted the namenode, let’s now start the Hadoop daemons one by one. Move into the $HADOOP_HOME/sbin directory and type the below commands

Starting Hadoop daemons

Starting HDFS daemons

Starting name node

./hadoop-daemon.sh start namenode

Starting datanode

./hadoop-daemon.sh start datanode

Starting secondary namenode

./hadoop-daemon.sh start secondarynamenode

Starting YARN daemons

Starting Resource Manager

./yarn-daemon.sh start resourcemanager

Starting Node Manager

./yarn-daemon.sh start nodemanager

We have successfully started all the Hadoop daemons. You can check their status using the jps command.

You can also start all these commands using one command i.e., start-all.sh as shown in the below screenshot.

Exploring HDFS

In Hadoop 3.x, HDFS come up with some new features to deal with the files, you can do all kinds of storage operations from web UI itself. Let’s see how to do that.

In Hadoop 2.x, web UI port is 50070 but in Hadoop3.x, it is moved to 9870. You can access HDFS web UI from localhost:9870 as shown in the below screenshot

You can see all the HDFS configurations in this page, to access webHDFS, click on Utilities–>Browse the file system

You can see few options added in it i.e., Creation of New folder, Upload files, Cut and Paste files from one directory to another directory.

Before creating the folder, make sure that the user has correct permissions to perform operations on that directories. If not, you can change the permissions using the command

hadoop fs -chmod -R 777 /

Let’s create a new folder and upload some data into it. To create a new folder, click on the folder icon and give some name to the directory as shown below.

You can see that the folder has been created successfully in the below screenshot

To upload files, click on upload symbol and browse your file system to select the file that you need to upload.

You can also delete the files by clicking on the delete symbol beside the directory or file as shown in the below screenshot.

You can also cut and paste the files from one directory to another directory.

Select the files which you want to cut and click on cut option and then click on Ok as shown below.

Now move into the folder where ever you want to paste this file and just click on Paste option. After clicking on Paste, you can see that file has been pasted in that directory as shown in the below screenshot.

This is how you can perform operations on files using HDFS web UI in Hadoop 3.x.

We hope this blog helped you in understanding how to install Hadoop 3.x in a single node cluster and how to perform operation on HDFS files using HDFS web UI.

Enroll for Hadoop Training conducted by Acadgild and become a successful big data developer.

3 Comments

Hi, Mr.Kiran Krishna:
May i know at mapred-site.xml need to be added two lines xml code {mapreduce.admin.user.env} and {yarn.app.mapreduce.am.evn}, and what theirs meaning?
Because if without this two lines xml codes, then will pop out error with “could not find org.apache.hadoop.mapreduce.v2.app.MRAppMaster ” when running wordcount application on hadoop.
Thanks.