{"id":4738,"date":"2020-09-04T14:52:46","date_gmt":"2020-09-04T09:22:46","guid":{"rendered":"https:\/\/www.h2kinfosys.com\/blog\/?p=4738"},"modified":"2023-05-17T13:40:09","modified_gmt":"2023-05-17T08:10:09","slug":"hadoop-mapreduce-examples","status":"publish","type":"post","link":"https:\/\/www.h2kinfosys.com\/blog\/hadoop-mapreduce-examples\/","title":{"rendered":"Hadoop & Mapreduce Examples"},"content":{"rendered":"\n

Using Hadoop Mapreduce, First of all, start the Hadoop<\/a> Cluster using the commands given below.<\/p>\n\n\n\n

$HADOOP_HOME\/sbin\/start-dfs.sh<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
$HADOOP_HOME\/sbin\/start-yarn.sh<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Check by typing jps in the terminal if all the Nodes are running.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Do you remember in the last article we looked at how a word counter works?<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Using Hadoop Mapreduce Let\u2019s implement the above.<\/p>\n\n\n\n

You need to create three files.<\/p>\n\n\n\n

    \n
  • Reduce.java<\/li>\n\n\n\n
  • Map.java<\/li>\n\n\n\n
  • WordCount.java<\/li>\n<\/ul>\n\n\n\n

    Reduce.java<\/em><\/strong><\/p>\n\n\n\n

    package com.impetus.code.examples.hadoop.mapred.wordcount;\n\nimport java.io.IOException;\nimport java.util.Iterator;\n\nimport org.apache.hadoop.io.IntWritable;\nimport org.apache.hadoop.io.Text;\nimport org.apache.hadoop.mapred.MapReduceBase;\nimport org.apache.hadoop.mapred.OutputCollector;\nimport org.apache.hadoop.mapred.Reducer;\nimport org.apache.hadoop.mapred.Reporter;\n\npublic class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable>\n{\npublic void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output,\nReporter reporter) throws IOException\n{\nint sum = 0;\nwhile (values.hasNext())\n{\nsum += values.next().get();\n}\noutput.collect(key, new IntWritable(sum));\n}\n}<\/pre>\n\n\n\n
    Map.java<\/em><\/strong><\/p>\n\n\n\n
    package com.impetus.code.examples.hadoop.mapred.wordcount;\n\nimport java.io.IOException;\nimport java.util.StringTokenizer;\n\nimport org.apache.hadoop.io.IntWritable;\nimport org.apache.hadoop.io.LongWritable;\nimport org.apache.hadoop.io.Text;\nimport org.apache.hadoop.mapred.MapReduceBase;\nimport org.apache.hadoop.mapred.Mapper;\nimport org.apache.hadoop.mapred.OutputCollector;\nimport org.apache.hadoop.mapred.Reporter;\npublic class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable>\n{\nprivate final static IntWritable one = new IntWritable(1);\n\nprivate Text word = new Text();\n\npublic void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter)\nthrows IOException\n{\nString line = value.toString();\nStringTokenizer tokenizer = new StringTokenizer(line);\nwhile (tokenizer.hasMoreTokens())\n{\nword.set(tokenizer.nextToken());\noutput.collect(word, one);\n}\n}\n}<\/pre>\n\n\n\n
    WordCount.java<\/em><\/strong><\/p>\n\n\n\n
    package com.impetus.code.examples.hadoop.mapred.wordcount;\n\nimport org.apache.hadoop.fs.Path;\nimport org.apache.hadoop.io.IntWritable;\nimport org.apache.hadoop.io.Text;\nimport org.apache.hadoop.mapred.FileInputFormat;\nimport org.apache.hadoop.mapred.FileOutputFormat;\nimport org.apache.hadoop.mapred.JobClient;\nimport org.apache.hadoop.mapred.JobConf;\nimport org.apache.hadoop.mapred.TextInputFormat;\nimport org.apache.hadoop.mapred.TextOutputFormat;\n\npublic class WordCount\n{\npublic static void main(String[] args) throws Exception\n{\nJobConf conf = new JobConf(WordCount.class);\nconf.setJobName(\"wordcount\");\n\nconf.setOutputKeyClass(Text.class);\nconf.setOutputValueClass(IntWritable.class);\n\nconf.setMapperClass(Map.class);\nconf.setCombinerClass(Reduce.class);\nconf.setReducerClass(Reduce.class);\n\nconf.setInputFormat(TextInputFormat.class);\nconf.setOutputFormat(TextOutputFormat.class);\n\nFileInputFormat.setInputPaths(conf, new Path(args[0]));\nFileOutputFormat.setOutputPath(conf, new Path(args[1]));\n\nJobClient.runJob(conf);\n\n}\n}<\/pre>\n\n\n\n
    Now you need to compile java files.<\/p>\n\n\n\n
    There are two ways to compile java files.<\/p>\n\n\n\n
    mvn clean install<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Or run the following command.<\/p>\n\n\n\n
    javac -d . Map.java Reduce.java WordCount.java<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    If you used javac -d command then run the following command too.<\/p>\n\n\n\n
    jar cfm wordcounter.jar Manifest.txt com\/impetus\/code\/examples\/hadoop\/mapred\/wordcount\/*.class<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Now let\u2019s create an input folder in HDFS.<\/p>\n\n\n\n
    Hdfs dfs -mkdir ~\/wordcount\/input<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Now we are going to create two input files. <\/p>\n\n\n\n
    sudo vi input_one<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    And put the following content inside it.<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    And another file.<\/p>\n\n\n\n
    sudo vi input_two<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Using the command below move the file to HDFS file system<\/a><\/p>\n\n\n\n
    hdfs dfs -copyFromLocal input_one ~\/wordcount\/input\/<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Do the above for both input files.<\/p>\n\n\n\n
    Now check if both files have been moved.<\/p>\n\n\n\n
    hdfs dfs -ls ~\/wordcount\/input\/<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Using Hadoop Mapreduce Now run the map-reduce using the command given below.<\/p>\n\n\n\n
    $HADOOP_HOME\/bin\/hadoop jar wordcounter.jar \/input \/output<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    By running the below-given command you will be able to see the output.<\/p>\n\n\n\n
    bin\/hadoop dfs -cat ~\/wordcount\/output\/part-00000<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"
    Using Hadoop Mapreduce, First of all, start the Hadoop Cluster using the commands given below. $HADOOP_HOME\/sbin\/start-dfs.sh $HADOOP_HOME\/sbin\/start-yarn.sh Check by typing jps in the terminal if all the Nodes are running. Do you remember in the last article we looked at how a word counter works? Using Hadoop Mapreduce Let\u2019s implement the above. You need to […]<\/p>\n","protected":false},"author":1,"featured_media":4754,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[67],"tags":[1329,1332],"class_list":["post-4738","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hadoop-big-data-skill-test","tag-hadoop-mapreduce","tag-mapreduce-examples"],"_links":{"self":[{"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/posts\/4738","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/comments?post=4738"}],"version-history":[{"count":0,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/posts\/4738\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/media\/4754"}],"wp:attachment":[{"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/media?parent=4738"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/categories?post=4738"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.h2kinfosys.com\/blog\/wp-json\/wp\/v2\/tags?post=4738"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}