Memunic Chronicles in Atomic Arena

After setting up your hadoop cluster which we learnt in our earlier blogs( Pseudo Distributed Cluster , Fully Distributed Cluster ), we now set up eclipse on ubuntu to develop Map Reduce programs for our Hadoop Cluster. Below are the steps to do so: Install Eclipse on ubuntu, by using $ sudo apt-get eclipse. Next step is to download the Hadoop Eclipse Plugin. Many tutorials suggest to build the plugin jar file Yourself using "ant". Trust me, this method is a quite messy and You may screwup Your hadoop install changing the various build.xml and build-properties.xml files. The best way is to directly download this jar file from here by clicking this line. Next copy the jar file to eclipse/plugins directory. Its going to be /usr/lib/eclipse/plugins in my case. Start eclipse, go to "Window >> Open Perspective >> Other". From perspectives window, you should see “Map/Reduce”, Select it and click "OK". You will see “Map/Reduce” perspec...

For setting up a multi-node cluster, You first need to learn and understand setting up a pseudo distributed cluster. You can go thru my earlier blog by clicking in here to learn this process.   Now that You know how to set up a pseudo distributed Hadoop cluster, You can do the below steps to build your multimode cluster. I will mention the process of setting up a two node cluster here with a "master" and a "slave" machine. The master is where the name node and job tracker runs and is the single point of interaction and failure in the cluster. The slaves run data node and task tracker and act as per the direction of the hdfs and map reduce master. You can use this process to scale up to as many nodes as You want. 1. Pick Your 2nd computer which You want to make as "the slave" in the cluster. We will call it "slave". Find out its ip address issuing the ifconfig command in the terminal. If you are in a wifi LAN to set up your cluster a...

Briefly what is psuedo-distributed single node Hadoop cluster: It is a Hadoop distributed cluster having Single Node. All the five daemons (NN,DN,JT,TT,SNN) are running as individual JVM (java process) instances. We will get all distributed features of Hadoop. This mode is used to develop, test, and staging (before production) Map Reduce programs quickly. It writes all the system, daemon and user logs to different files on the cluster To install Hadoop in this mode, we have to modify the hadoop-env.sh, core-site.xml, hdfs-site.xml, mapred-site.xml. hadoop-env.sh contains all the Environment variables which are required for running Hadoop. All the default values are fine, Only we have to modify the JAVA_HOME=“The location of Java Home Directory” Core-site.xml contains all the cluster specific information like io, temp directory, sorting, file system uri etc. Hdfs-site.xml contains all the parameters which are specific to HDFS like name node directory, data node directory...

Very Frequently You would need files to be shared between Your local OS and Your ubuntu VM running using Oracle VirtualBox. Here are the steps to set up Your Shared Folder on Ubuntu Virtual Box VM. Start Your Ubuntu VM in Oracle Virtual Box. Go to Devices >> Install Guest Additions. This will install the Guest additions in a terminal session. Next reboot Your VM. Create a Folder in Your native OS which You want to share. I have named it as “Shared”. Now in Your VM, Go to Devices >> Shared Folders and then hit the “Add” button. Browse for the folder in Your native OS that You recently created to be shared. Now In Your VM system, create a folcer to be Shared in Your home directory. Get the full path of this folder. The one that I have created is : /home/atom/Shared . Check the option Make Permanent while doing so. Now open Terminal in Your VM and issue the command : sudo mount -t vboxsf Shared /home/atom/Shared . Note that You would have to run this command to m...