hbase-user mailing list archives

Re: Hadoop not working after replacing hadoop-core.jar with hadoop-core-append.jar

Date

Sat, 11 Jun 2011 09:20:18 GMT

Hi all,
I have updated my Hadoop+HBase article with the filenaming-related
information of the Hadoop core jar file (thanks for your notification,
St.Ack).
FYI, Hadoop 0.20.2 and 0.20.203.0 differ in how they find their hadoop-* JAR
files.
Hadoop 0.20.2 (bin/hadoop):
for f in $HADOOP_HOME/hadoop-*-core.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
Hadoop 0.20.203.0 (bin/hadoop):
for f in $HADOOP_HOME/hadoop-core-*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
Hence I made a note to the article that you only need to rename the custom
built JAR files if you are running the older _0.20.2_ release.
Follow-up question:
Since both Hadoop 0.20.203.0 and HBase 0.90.3 were recently released (the
article is about 0.20.2 and 0.90.2), is there anything else I can update in
the article? For instance the section with the matrix showing version
compatibilities between Hadoop and HBase? Here at work we are still testing
our slightly customized Hadoop 0.20.203.0 version (we have integrated some
patches which are not in the stock release) and I haven't had the chance to
look at HBase 0.90.3 at all yet. Just shoot me a message and I'll integrate
the missing information in the article.
Andrew Purtell wrote:
> We cover this issue in the online book, but as this thread suggests,
> we might make that better by pulling in some of Michael Noll's material
> either directly with his permission or by reference. We don't "own" the
> Hadoop wiki so can't do anything there. Our own wiki (and website) needs
> a refresh. When that happens we can cover this issue, perhaps with a
> compatibility matrix (with links to build or distro instructions),
somewhere up front.
Oh, almost forgot to reply to that: Of course feel free to integrate my
article back into the HBase docs. I had already offered St.Ack to help out
with the writing but back in April the decision was made to just link to my
blog (i.e. a reference rather than a book/website/wiki update). Personally,
I'd prefer this information to be available in the official docs rather than
on an individual's website. Just let me know what you guys prefer.
Best,
Michael
PS: It's getting increasingly harder to keep track of all the Michael's on
this list. ;-)
On Fri, Jun 10, 2011 at 21:19, stack <saint.ack@gmail.com> wrote:
> Thank you Mike. I made your suggested change and pushed to the site
> (takes an hour or two to be visible). I also wrote Michael Noll to
> make stronger note on jar naming, that append jar does not conform
> (Michael is usually responsive so should get fixed soon).
>
> Thanks for the input.
>
> St.Ack
>
> On Fri, Jun 10, 2011 at 11:43 AM, Mike Spreitzer <mspreitz@us.ibm.com>
> wrote:
> > Thanks for the clarification. How about the following update to the
> text?
> > Where it currently says
> >
> > <para>Because HBase depends on Hadoop, it bundles an instance of the 240
> > Hadoop jar under its <filename>lib</filename> directory. The bundled
241
> > Hadoop was made ...
> >
> > we clarify that the bundled JAR is ONLY for use in standalone mode,
> perhaps
> > like this:
> >
> > <para>Because HBase depends on Hadoop, it bundles an instance of the 240
> > Hadoop jar under its <filename>lib</filename> directory --- but this
is
> > <emphasis>only</emphasis> for use in standalone mode. The bundled 241
> Hadoop
> > was made ...
> >
> > Also, why does this text say 240 at first and then 241? And why not
> > explicitly give the name of the JAR file in question, which is
> > hadoop-core-0.20-append-r1056497?
> >
> > Thanks,
> > Mike Spreitzer
> >
> >
> >
> >
> > From: Stack <stack@duboce.net>
> > To: user@hbase.apache.org
> > Date: 06/10/2011 02:33 PM
> > Subject: Re: Hadoop not working after replacing hadoop-core.jar
> with
> > hadoop-core-append.jar
> > Sent by: saint.ack@gmail.com
> > ________________________________
> >
> >
> > On Fri, Jun 10, 2011 at 10:06 AM, Mike Spreitzer <mspreitz@us.ibm.com>
> > wrote:
> >> stack <saint.ack@gmail.com> wrote on 06/06/2011 10:57:50 PM:
> >>> From: stack <saint.ack@gmail.com>
> >> Let me see if I have got this straight. Hadoop branch-0.20-append is
> not
> >> an immutable thing, it has evolved a little over time.
> >
> > Yes.
> >
> >> The hadoop-core.jar that is included in the HBase distribution was built
> >> from
> >> some version of branch-0.20-append. If my own Hadoop cluster is EXACTLY
> >> the same version of branch-0.20-append then I do not need to replace any
> >> files anywhere. However, since nobody is telling me the version of
> >> branch-0.20-append from which HBase's hadoop-core.jar was built,
> >
> > It says so in the jar name. The jar is called >0.20-append-r1056497.
> > The latter is the svn revision we built the jar from.
> >
> >
> >> I can not
> >> in any case or way be confident that my cluster is running EXACTLY the
> >> same version even if it is branch-0.20-append.
> >
> > Not true.
> >
> >> So the net result is that
> >> in all distributed cases (except when I import pre-built Hadoop+HBase
> from
> >> Cloudera or elsewhere) I have to build branch-0.20-append and copy it's
> >> core JAR into my HBase lib. Have I got this right? The book still does
> >> not say that clearly. In fact, the book still points to my old email
> >> saying I did it the other way around. Your reply above clearly seems to
> >> imply that I need to replace HBase's hadoop core JAR only in some
> >> distributed cases. Yet the rest of the email conversation on this point
> >> seems to have settled that HBase's hadoop core JAR needs to be replaced
> in
> >> all distributed cases.
> >>
> >
> > My reading of the text is that the jar should always be replaced. I
> > added the callout that cited your old mail thread because I thought it
> > might be of interest. It seems to have only made confusion so in the
> > edit I posted above (for review), I'd completely removed it.
> >
> > The edit has not been pushed to the website. I thought I'd get a bit
> > of input on it first. Here is the link again:
> > http://svn.apache.org/viewvc?view=revision&revision=1134129
> >
> > St.Ack
> >
> >
>