I'm attempting to build a custom LoadFunc for pig and I'm running into arather silly issue.

My project has several dependencies and I've been trying to create a singlejar that contains all of these dependencies (I'm using the maven assemblyplugin).

Now if I manually place all dependencies in my pig script using a list of"REGISTER foo.jar" statements it all works.What I want is to have a single jar with everything in it so that I onlyneed a single jar in the pig code.

After some playing around I found that it I unpack all the dependenciesinside my jar and unpack them in the root of the jar it all seems to work... and turns into a big mess with warnings about "[INFO]META-INF/MANIFEST.MF already added, skipping".

What is the clean way of doing this and still let PIG find the dependenciesinside the jar?Can anyone point me towards a working example?

> Hi,>> I'm attempting to build a custom LoadFunc for pig and I'm running into a> rather silly issue.>> My project has several dependencies and I've been trying to create a single> jar that contains all of these dependencies (I'm using the maven assembly> plugin).>> Now if I manually place all dependencies in my pig script using a list of> "REGISTER foo.jar" statements it all works.> What I want is to have a single jar with everything in it so that I only> need a single jar in the pig code.>> After some playing around I found that it I unpack all the dependencies> inside my jar and unpack them in the root of the jar it all seems to work> ... and turns into a big mess with warnings about "[INFO]> META-INF/MANIFEST.MF already added, skipping".>> What is the clean way of doing this and still let PIG find the dependencies> inside the jar?> Can anyone point me towards a working example?>> Thanks.>> --> Best regards / Met vriendelijke groeten,>> Niels Basjes>

This is not a Pig question, it is more of a Java packaging question. Whatexactly went wrong with the maven assembly plugin? Maybe the maven shadeplugin would work better? (though I've never tried it myself)For me - the simplest way is to just register all the needed dependenciesand I've seen such approach in some hadoop docs. Also note that some of thedependencies can already be present in pig/hadoop environment, for exampleJodaTime. So you don't have to register them. But make sure you arecompliant with the versions of those dependencies.Hope that helps, let me know if you have questions.

> Hi Niels,>> Pig supports globbing in REGISTER statements. You could place all the jars> in a certain location and have them all be picked up.>> REGISTER *.jar>>> On Tue, Apr 16, 2013 at 1:44 PM, Niels Basjes <[EMAIL PROTECTED]> wrote:>> > Hi,> >> > I'm attempting to build a custom LoadFunc for pig and I'm running into a> > rather silly issue.> >> > My project has several dependencies and I've been trying to create a> single> > jar that contains all of these dependencies (I'm using the maven assembly> > plugin).> >> > Now if I manually place all dependencies in my pig script using a list of> > "REGISTER foo.jar" statements it all works.> > What I want is to have a single jar with everything in it so that I only> > need a single jar in the pig code.> >> > After some playing around I found that it I unpack all the dependencies> > inside my jar and unpack them in the root of the jar it all seems to work> > ... and turns into a big mess with warnings about "[INFO]> > META-INF/MANIFEST.MF already added, skipping".> >> > What is the clean way of doing this and still let PIG find the> dependencies> > inside the jar?> > Can anyone point me towards a working example?> >> > Thanks.> >> > --> > Best regards / Met vriendelijke groeten,> >> > Niels Basjes> >>

NEW: Monitor These Apps!

All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext