{"id":4347,"date":"2020-08-13T17:28:43","date_gmt":"2020-08-13T11:58:43","guid":{"rendered":"https:\/\/www.h2kinfosys.com\/blog\/?p=4347"},"modified":"2020-12-14T14:54:51","modified_gmt":"2020-12-14T09:24:51","slug":"install-hadoop-with-step-by-step-configuration-on-ubuntu","status":"publish","type":"post","link":"https:\/\/www.h2kinfosys.com\/blog\/install-hadoop-with-step-by-step-configuration-on-ubuntu\/","title":{"rendered":"How to Install Hadoop with Step by Step Configuration on Ubuntu?"},"content":{"rendered":"\n

In this tutorial, we are going to install Hadoop Apache on a ubuntu system followed by the configuration of install Hadoop.<\/p>\n\n\n\n

Open the terminal in Ubuntu. Hit Ctrl<\/strong>–Shift<\/strong>–T<\/strong><\/p>\n\n\n\n

Step 1) Add a new user.<\/strong><\/p>\n\n\n\n

To keep things clean we will create a new user with the name \u201chadoop\u201d.<\/p>\n\n\n\n

sudo addgroup hadoop<\/code><\/pre>\n\n\n\n
You will be asked for a password. Enter the password and press enter.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Then execute the following command to create user.<\/p>\n\n\n\n
sudo adduser --ingroup hadoop supper_user<\/code><\/pre>\n\n\n\n
Then enter the information.<\/p>\n\n\n\n
Note: Remember the password which you entered.<\/strong><\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Step 2) Now Configure SSH<\/strong><\/p>\n\n\n\n
If the SSH is not installed in your system enter the given below command in the terminal.<\/p>\n\n\n\n
sudo apt-get install openssh-server<\/code><\/pre>\n\n\n\n
Now add the supper_user to the sudo group.<\/p>\n\n\n\n
sudo adduser supper_user sudo<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
First, convert to the supper_user that you created above using the command given below.<\/p>\n\n\n\n
su supper_user<\/code><\/pre>\n\n\n\n
Enter the password which you set above while creating the user.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Now for creating the authentication key pairs for SSH <\/strong>execute the command below.<\/p>\n\n\n\n
ssh-keygen -t rsa -P \"\"<\/code><\/pre>\n\n\n\n
Remember to execute this command from supper_user that we created.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Now activate the SSH access using the command given below.<\/p>\n\n\n\n
cat $HOME\/.ssh\/id_rsa.pub >> $HOME\/.ssh\/authorized_keys<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Let\u2019s now test our SSH using the following command.<\/p>\n\n\n\n
ssh localhost<\/code><\/pre>\n\n\n\n
If you see the below-given output. Congratulations you are on your way.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Step 3) Download the Hadoop from the given website<\/strong><\/p>\n\n\n\n
 <\/strong>Download Hadoop<\/strong><\/a><\/p>\n\n\n\n
\"Download<\/figure>\n\n\n\n
I downloaded the Hadoop <\/a>in the download folder. Now move to download folder using the command given below.<\/p>\n\n\n\n
cd \/home\/ahmed\/Downloads\/<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
sudo tar xzf hadoop-3.3.0.tar.gz<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
sudo chown -R supper_user:hadoop hadoop-3.3.0<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Step 4) Configuration of Hadoop<\/strong><\/p>\n\n\n\n
We are going to modify the following files.<\/p>\n\n\n\n
  1. bashrc<\/li>
  2. hadoop-env.sh<\/li>
  3. core-site.xml<\/li>
  4. hdfs-site.xml<\/li>
  5. mapred-site-xml<\/li>
  6. Yarn-site.xml<\/li><\/ol>\n\n\n\n
    File 1:<\/p>\n\n\n\n
    sudo vi .bashrc\nexport HADOOP_HOME=\/home\/hadoop\/hadoop\nexport HADOOP_INSTALL=$HADOOP_HOME\nexport HADOOP_MAPRED_HOME=$HADOOP_HOME\nexport HADOOP_COMMON_HOME=$HADOOP_HOME\nexport HADOOP_HDFS_HOME=$HADOOP_HOME\nexport YARN_HOME=$HADOOP_HOME\nexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME\/lib\/native\nexport PATH=$PATH:$HADOOP_HOME\/sbin:$HADOOP_HOME\/bin<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Once you added the above-given variable in the file, exit from the editor.<\/p>\n\n\n\n
    Now apply changes using the given command below.<\/p>\n\n\n\n
    source ~\/.bashrc<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    File 2:<\/p>\n\n\n\n
    Edit hadoop-env.sh File using the command given below.<\/em><\/strong><\/p>\n\n\n\n
    sudo vi \/home\/ahmed\/Downloads\/hadoop-3.3.0\/etc\/hadoop\/hadoop-env.sh<\/code><\/pre>\n\n\n\n
    Uncomment the export statement add the path of Java.<\/p>\n\n\n\n
    export JAVA_HOME=\/usr\/lib\/jvm\/java-8-openjdk-amd64<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    File 3:<\/p>\n\n\n\n
    Now lets edit core-site.xml file.<\/em><\/strong><\/p>\n\n\n\n
    sudo vi $HADOOP_HOME\/etc\/hadoop\/core-site.xml<\/code><\/pre>\n\n\n\n
    Add the below-given data in the core-site.xml<\/p>\n\n\n\n
    <property>\n  <name>hadoop.tmp.dir<\/name>\n  <value>\/home\/hdoop\/tmpdata<\/value>\n<\/property>\n<property>\n  <name>fs.default.name<\/name>\n  <value>hdfs:\/\/127.0.0.1:9000<\/value>\n<\/property><\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Now move to the given below directory<\/p>\n\n\n\n
    cd $HADOOP_HOME\/etc\/hadoop<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    sudo mkdir -p \/app\/hadoop\/tmp<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    sudo chown -R supper_user:hadoop \/app\/hadoop\/tmp<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    sudo chmod 750 \/app\/hadoop\/tmp<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    File 4:<\/p>\n\n\n\n
    Now let\u2019s edit hdfs-site.xml File<\/p>\n\n\n\n
    sudo vi $HADOOP_HOME\/etc\/hadoop\/hdfs-site.xml<\/code><\/pre>\n\n\n\n
    Add the following lines in the configuration tag.<\/p>\n\n\n\n
    <configuration>\n<property>\n<name>dfs.replication<\/name>\n<value>1<\/value>\n<description>Default block replication.<\/description>\n<\/property>\n<property>\n<name>dfs.datanode.data.dir<\/name>\n<value>\/home\/supper_user\/hdfs<\/value>\n<\/property>\n<\/configuration><\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Save and exit.<\/p>\n\n\n\n
    File 5:<\/p>\n\n\n\n
    Now edit mapred-site.xml file.<\/p>\n\n\n\n
    sudo vi $HADOOP_HOME\/etc\/hadoop\/mapred-site.xml<\/code><\/pre>\n\n\n\n
    Add the following data in the mapred-site.xml file.<\/p>\n\n\n\n
    <configuration>\n<property>\n<name>mapreduce.jobtracker.address<\/name>\n<value>localhost:54311<\/value>\n<description>MapReduce job tracker runs at this host and port.\n<\/description>\n<\/property>\n<\/configuration><\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    File 6:<\/strong><\/h3>\n\n\n\n
    Now we are going to edit yarn-site.xml file.<\/p>\n\n\n\n
    sudo vi $HADOOP_HOME\/etc\/hadoop\/yarn-site.xml<\/code><\/pre>\n\n\n\n
    <property>\n  <name>yarn.nodemanager.aux-services<\/name>\n  <value>mapreduce_shuffle<\/value>\n<\/property>\n<property>\n  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class<\/name>\n  <value>org.apache.hadoop.mapred.ShuffleHandler<\/value>\n<\/property>\n<property>\n  <name>yarn.resourcemanager.hostname<\/name>\n  <value>127.0.0.1<\/value>\n<\/property>\n<property>\n  <name>yarn.acl.enable<\/name>\n  <value>0<\/value>\n<\/property>\n<property>\n  <name>yarn.nodemanager.env-whitelist<\/name>   \n  <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME<\/value>\n<\/property><\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Save and exit.<\/p>\n\n\n\n
    Now let\u2019s format HDFS NameNode using the command below.<\/p>\n\n\n\n
    $HADOOP_HOME\/bin\/hdfs namenode -format<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    $HADOOP_HOME\/sbin\/start-yarn.sh<\/code><\/pre>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Type \u201cjps\u201d to check the Hadoop processes are running.\"\"<\/p>\n","protected":false},"excerpt":{"rendered":"
    In this tutorial, we are going to install Hadoop Apache on a ubuntu system followed by the configuration of install Hadoop. Open the terminal in Ubuntu. Hit Ctrl–Shift–T Step 1) Add a new user. To keep things clean we will create a new user with the name \u201chadoop\u201d. You will be asked for a password. […]<\/p>\n","protected":false},"author":1,"featured_media":4364,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[138],"tags":[1216,1215,1217],"class_list":["post-4347","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-hadoop-tutorials","tag-configuration-on-ubuntu","tag-how-to-install-hadoop","tag-step-by-step-configuration"],"_links":{"self":[{"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/posts\/4347","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/comments?post=4347"}],"version-history":[{"count":0,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/posts\/4347\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/media\/4364"}],"wp:attachment":[{"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/media?parent=4347"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/categories?post=4347"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/tags?post=4347"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}