Support for the Facebook Hadoop branch

Details

Description

I've been working with Joe Xie on support to get Giraph running on the Facebook Hadoop branch. He verified today that the examples worked on their cluster. I need to clean up my changes a little, but otherwise, will submit a cleaned up diff. As a side note, does anyone know how we can get Hudson support for Giraph?

Avery Ching
added a comment - 30/Aug/11 17:44 It's good to hear that you can run it on your cluster. As far as the unittests, that is strange. I was able to repeat the same issues and will look into a fix.

Avery Ching
added a comment - 30/Aug/11 23:51 Looks like I needed to change the groupId so that the right dependencies are pulled in for hadoop. Please try this one out. The unittests all passed for me.
(i.e. mvn -Dhadoop=facebook -Dhadoop.jar.path=/Users/aching/Desktop/hadoop-0.20.1-core.jar package)

I'm not up to date on FB's distribution. It's available to the public? If we're going to support this, should instructions be given in the README, as there are for the non-secure build option? What's the long term story for these API support #ifdefs? At the moment it's a clever solution to a vexing problem, but longer term it would be good to have a solution that doesn't leave code as comments.

Jakob Homan
added a comment - 31/Aug/11 04:00 I'm not up to date on FB's distribution. It's available to the public? If we're going to support this, should instructions be given in the README, as there are for the non-secure build option? What's the long term story for these API support #ifdefs? At the moment it's a clever solution to a vexing problem, but longer term it would be good to have a solution that doesn't leave code as comments.

In theory, I believe that Facebook's distro is online (https://github.com/facebook/hadoop-20-warehouse). The long term story is to factor out the parts into modules and then compile them based on the user profile. Then we don't have to "munge" anything anymore. At least that's what I've thought of for now. I'm open to better solutions. Pre-processing will get unmaintainable if we have to support every version of Hadoop. That being said, we should support the big customers of Giraph and that likely includes Facebook as well.

Avery Ching
added a comment - 31/Aug/11 05:58 In theory, I believe that Facebook's distro is online ( https://github.com/facebook/hadoop-20-warehouse ). The long term story is to factor out the parts into modules and then compile them based on the user profile. Then we don't have to "munge" anything anymore. At least that's what I've thought of for now. I'm open to better solutions. Pre-processing will get unmaintainable if we have to support every version of Hadoop. That being said, we should support the big customers of Giraph and that likely includes Facebook as well.
I'll add instructions to the README and submit a new patch.