Creating and Running your Java (JDK6) Code on Hadoop 1.2.1

1. Import hadoop-core-1.2.1.jar to your project as external jar
2. And others Import commons-logging-1.1.3.jar (optional)
3. Write Mapreduce job code, compile and create the class files
7. Export the project as JAR file – filename.jar
8. Start the hadoop cluster

hduser@neo:/usr/local/hadoop$ bin/start-all.sh

9.Copy data to HDFS
hduser@neo:~$ bin/hadoop dfs -copyFromLocal <local-dir> <hdfs-input-dir>
example:-

hduser@neo:~$ bin/hadoop dfs -copyFromLocal /tmp/ncdc /user/hduser/ncdc

10. Run the MapReduce job
hduser@neo:~$ bin/hadoop jar filename.jar <classname-containing-main-function> <hdfs-input-dir> <hdfs-output-dir>
example:-

hduser@neo:~$ bin/hadoop jar /home/naved/workspace/MaxTemperature/bin/MaxTemperature.jar MaxTemperatureWithCombiner /user/hduser/ncdc/ /user/hduser/ncdc-output

11. Retrieve the job result from HDFS
hduser@neo:~$ bin/hadoop dfs -getmerge <hdfs-output-dir> <local-dir>
example:-

hduser@neo:~$ bin/hadoop dfs -getmerge /user/hduser/ncdc-output /tmp/ncdc-output

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s