> Hi,> Would much appreciate a pointer to a mapreduce tutorial which explains> how i can run a simulated cluster of mapreduce nodes on a single PC and> write a Java program with the MapReduce Paradigm.> Thanks very much.> Sitaraman>

Hi all, Thanks much for the pointers. I ran word count it appears,successfully. Currently in inspected the output like so,/usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat /user/sitaraman/output/*Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library/usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 whichmight have disabled stack guard. The VM will try to fix the stackguard now.It's highly recommended that you fix the library with 'execstack -c<libfile>', or link it with '-z noexecstack'.13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to loadnative-hadoop library for your platform... using builtin-java classeswhere applicableas 2do 1do: 1i 2not 1say 1

What is the name of the file in the output directory? Can i read itsay using emacs or vi where will the directory /user/sitaraman/outputbe in the linux fs, that is how to locate where it is....Thanks again for all the help.Sitaraman

On 12/28/13, Hardik Pandya <[EMAIL PROTECTED]> wrote:> I recently blogged about it - hope it helps>> http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/>> Regards,> Hardik>> On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <> [EMAIL PROTECTED]> wrote:>>> Hi,>> Would much appreciate a pointer to a mapreduce tutorial which explains>> how i can run a simulated cluster of mapreduce nodes on a single PC and>> write a Java program with the MapReduce Paradigm.>> Thanks very much.>> Sitaraman>>>

For the first question, the name of the file containing the response of MRprocess is like that: part-r-00000. This file is inside the outputdirectory.

The second question, to locate the file in your linux file system you haveto go to your datanode directory. This directory path is in yourhdfs-site.xml file in {HADOOP_INSTALLATION}/etc/hadoop. In my case the fullpath of the MR response file is:/home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-1387647958374/current/finalized.

> Hi all,> Thanks much for the pointers. I ran word count it appears,> successfully. Currently in inspected the output like so,> /usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat> /user/sitaraman/output/*> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library> /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which> might have disabled stack guard. The VM will try to fix the stack> guard now.> It's highly recommended that you fix the library with 'execstack -c> <libfile>', or link it with '-z noexecstack'.> 13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load> native-hadoop library for your platform... using builtin-java classes> where applicable> as 2> do 1> do: 1> i 2> not 1> say 1>> What is the name of the file in the output directory? Can i read it> say using emacs or vi where will the directory /user/sitaraman/output> be in the linux fs, that is how to locate where it is....> Thanks again for all the help.> Sitaraman>> On 12/28/13, Hardik Pandya <[EMAIL PROTECTED]> wrote:> > I recently blogged about it - hope it helps> >> >> http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/> >> > Regards,> > Hardik> >> > On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <> > [EMAIL PROTECTED]> wrote:> >> >> Hi,> >> Would much appreciate a pointer to a mapreduce tutorial which explains> >> how i can run a simulated cluster of mapreduce nodes on a single PC and> >> write a Java program with the MapReduce Paradigm.> >> Thanks very much.> >> Sitaraman> >>> >>

On 12/29/13, Diego Gutierrez <[EMAIL PROTECTED]> wrote:> Hi,>> For the first question, the name of the file containing the response of MR> process is like that: part-r-00000. This file is inside the output> directory.>> The second question, to locate the file in your linux file system you have> to go to your datanode directory. This directory path is in your> hdfs-site.xml file in {HADOOP_INSTALLATION}/etc/hadoop. In my case the full> path of the MR response file is:> /home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-1387647958374/current/finalized.>> where my datanode directory is: /home/user/hadoop/data2.2/hdfs/namenode.>> In the /finalized directory, there are files named like that:> blk_1073741836. In one of those files is the MR response file. This aproach> is not good if you want to open the response with vi for example, instead> you can copy the response file into your linux file system using> *copyToLocal* command(> https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/FileSystemShell.html#copyToLocal).> The syntax could be like that: bin/hadoop dfs -copyToLocal> /output/part-r-00000 /home/user/>> Hope this help>>> 2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>>>> Hi all,>> Thanks much for the pointers. I ran word count it appears,>> successfully. Currently in inspected the output like so,>> /usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat>> /user/sitaraman/output/*>> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library>> /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which>> might have disabled stack guard. The VM will try to fix the stack>> guard now.>> It's highly recommended that you fix the library with 'execstack -c>> <libfile>', or link it with '-z noexecstack'.>> 13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load>> native-hadoop library for your platform... using builtin-java classes>> where applicable>> as 2>> do 1>> do: 1>> i 2>> not 1>> say 1>>>> What is the name of the file in the output directory? Can i read it>> say using emacs or vi where will the directory /user/sitaraman/output>> be in the linux fs, that is how to locate where it is....>> Thanks again for all the help.>> Sitaraman>>>> On 12/28/13, Hardik Pandya <[EMAIL PROTECTED]> wrote:>> > I recently blogged about it - hope it helps>> >>> >>> http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/>> >>> > Regards,>> > Hardik>> >>> > On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <>> > [EMAIL PROTECTED]> wrote:>> >>> >> Hi,>> >> Would much appreciate a pointer to a mapreduce tutorial which>> >> explains>> >> how i can run a simulated cluster of mapreduce nodes on a single PC>> >> and>> >> write a Java program with the MapReduce Paradigm.>> >> Thanks very much.>> >> Sitaraman>> >>>> >>>>

Hi Diego, You have mentioned where my datanode directory is:/home/user/hadoop/data2.2/hdfs/namenode.it should be datanode ?Sitaraman

On 12/29/13, Sitaraman Vilayannur <[EMAIL PROTECTED]> wrote:> Thanks Diego, will use the copyToLocal.> Sitaraman>> On 12/29/13, Diego Gutierrez <[EMAIL PROTECTED]> wrote:>> Hi,>>>> For the first question, the name of the file containing the response of>> MR>> process is like that: part-r-00000. This file is inside the output>> directory.>>>> The second question, to locate the file in your linux file system you>> have>> to go to your datanode directory. This directory path is in your>> hdfs-site.xml file in {HADOOP_INSTALLATION}/etc/hadoop. In my case the>> full>> path of the MR response file is:>> /home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-1387647958374/current/finalized.>>>> where my datanode directory is: /home/user/hadoop/data2.2/hdfs/namenode.>>>> In the /finalized directory, there are files named like that:>> blk_1073741836. In one of those files is the MR response file. This>> aproach>> is not good if you want to open the response with vi for example, instead>> you can copy the response file into your linux file system using>> *copyToLocal* command(>> https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/FileSystemShell.html#copyToLocal).>> The syntax could be like that: bin/hadoop dfs -copyToLocal>> /output/part-r-00000 /home/user/>>>> Hope this help>>>>>> 2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>>>>>> Hi all,>>> Thanks much for the pointers. I ran word count it appears,>>> successfully. Currently in inspected the output like so,>>> /usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat>>> /user/sitaraman/output/*>>> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library>>> /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which>>> might have disabled stack guard. The VM will try to fix the stack>>> guard now.>>> It's highly recommended that you fix the library with 'execstack -c>>> <libfile>', or link it with '-z noexecstack'.>>> 13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load>>> native-hadoop library for your platform... using builtin-java classes>>> where applicable>>> as 2>>> do 1>>> do: 1>>> i 2>>> not 1>>> say 1>>>>>> What is the name of the file in the output directory? Can i read it>>> say using emacs or vi where will the directory /user/sitaraman/output>>> be in the linux fs, that is how to locate where it is....>>> Thanks again for all the help.>>> Sitaraman>>>>>> On 12/28/13, Hardik Pandya <[EMAIL PROTECTED]> wrote:>>> > I recently blogged about it - hope it helps>>> >>>> >>>> http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/>>> >>>> > Regards,>>> > Hardik>>> >>>> > On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <>>> > [EMAIL PROTECTED]> wrote:>>> >>>> >> Hi,>>> >> Would much appreciate a pointer to a mapreduce tutorial which>>> >> explains>>> >> how i can run a simulated cluster of mapreduce nodes on a single PC>>> >> and>>> >> write a Java program with the MapReduce Paradigm.>>> >> Thanks very much.>>> >> Sitaraman>>> >>>>> >>>>>>>

That was my mistake, you are right it should be "datanode" at the end ofthe path. The full path should be:/home/user/hadoop/data2.2/hdfs/datanode. Sorry about that.2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>

> Hi Diego,> You have mentioned where my datanode directory is:> /home/user/hadoop/data2.2/hdfs/namenode.> it should be datanode ?> Sitaraman>> On 12/29/13, Sitaraman Vilayannur <[EMAIL PROTECTED]> wrote:> > Thanks Diego, will use the copyToLocal.> > Sitaraman> >> > On 12/29/13, Diego Gutierrez <[EMAIL PROTECTED]> wrote:> >> Hi,> >>> >> For the first question, the name of the file containing the response of> >> MR> >> process is like that: part-r-00000. This file is inside the output> >> directory.> >>> >> The second question, to locate the file in your linux file system you> >> have> >> to go to your datanode directory. This directory path is in your> >> hdfs-site.xml file in {HADOOP_INSTALLATION}/etc/hadoop. In my case the> >> full> >> path of the MR response file is:> >>> /home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-1387647958374/current/finalized.> >>> >> where my datanode directory is: /home/user/hadoop/data2.2/hdfs/namenode.> >>> >> In the /finalized directory, there are files named like that:> >> blk_1073741836. In one of those files is the MR response file. This> >> aproach> >> is not good if you want to open the response with vi for example,> instead> >> you can copy the response file into your linux file system using> >> *copyToLocal* command(> >>> https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/FileSystemShell.html#copyToLocal> ).> >> The syntax could be like that: bin/hadoop dfs -copyToLocal> >> /output/part-r-00000 /home/user/> >>> >> Hope this help> >>> >>> >> 2013/12/28 Sitaraman Vilayannur <[EMAIL PROTECTED]>> >>> >>> Hi all,> >>> Thanks much for the pointers. I ran word count it appears,> >>> successfully. Currently in inspected the output like so,> >>> /usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat> >>> /user/sitaraman/output/*> >>> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library> >>> /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which> >>> might have disabled stack guard. The VM will try to fix the stack> >>> guard now.> >>> It's highly recommended that you fix the library with 'execstack -c> >>> <libfile>', or link it with '-z noexecstack'.> >>> 13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load> >>> native-hadoop library for your platform... using builtin-java classes> >>> where applicable> >>> as 2> >>> do 1> >>> do: 1> >>> i 2> >>> not 1> >>> say 1> >>>> >>> What is the name of the file in the output directory? Can i read it> >>> say using emacs or vi where will the directory /user/sitaraman/output> >>> be in the linux fs, that is how to locate where it is....> >>> Thanks again for all the help.> >>> Sitaraman> >>>> >>> On 12/28/13, Hardik Pandya <[EMAIL PROTECTED]> wrote:> >>> > I recently blogged about it - hope it helps> >>> >> >>> >> >>>> http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/> >>> >> >>> > Regards,> >>> > Hardik> >>> >> >>> > On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <> >>> > [EMAIL PROTECTED]> wrote:> >>> >> >>> >> Hi,> >>> >> Would much appreciate a pointer to a mapreduce tutorial which> >>> >> explains> >>> >> how i can run a simulated cluster of mapreduce nodes on a single PC> >>> >> and> >>> >> write a Java program with the MapReduce Paradigm.> >>> >> Thanks very much.> >>> >> Sitaraman> >>> >>> >>> >> >>>> >>> >>

I suggest to read the chapter 2 of the book "Hadoop: the definitive guide".In that chapter mapReduce is explained with a nice example, also there arelines of code for map and reduce functions. Every phase of MapReduce isexplained in detail and there are examples of how mapReduce can be runusing ruby, phyton or c++ with Hadoop.2013/12/27 Sitaraman Vilayannur <[EMAIL PROTECTED]>

> Hi,> Would much appreciate a pointer to a mapreduce tutorial which explains> how i can run a simulated cluster of mapreduce nodes on a single PC and> write a Java program with the MapReduce Paradigm.> Thanks very much.> Sitaraman>

> May be you can reference <<Hadoop in action>>>>> 2013/12/27 Sitaraman Vilayannur <[EMAIL PROTECTED]>>>> Hi,>> Would much appreciate a pointer to a mapreduce tutorial which explains>> how i can run a simulated cluster of mapreduce nodes on a single PC and>> write a Java program with the MapReduce Paradigm.>> Thanks very much.>> Sitaraman>>>>

+

Sitaraman Vilayannur 2013-12-27, 14:57

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext