In case the producer / consumer don't require sorting to happen, take a look
at ChainMapper:
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapred/lib/ChainMapper.html
If you do want the stuff to happen after sorting, take a look at:
http://hadoop.apache.org/mapreduce/docs/current/api/org/apache/hadoop/mapreduce/lib/chain/ChainReducer.html
More esoteric cases will require you to do separate map reduces, at least
with the current hadoop framework.
On Thu, Jan 6, 2011 at 2:27 PM, W.P. McNeill <billmcn@gmail.com> wrote:
> Say I have two MapReduce processes, A and B. The two are algorithmically
> dissimilar, so they have to be implemented as separate MapReduce processes.
> The output of A is used as the input of B, so A has to run first.
> However,
> B doesn't need to take all of A's output as input, only a partition of it.
> So in theory A and B could run at the same time in a producer/consumer
> arrangement, where B would start to work as soon as A had produced some
> output but before A had completed. Obviously, this could be a big
> parallelization win.
>
> Is this possible in MapReduce? I know at the most basic level it is
> not–there is no synchronization mechanism that allows the same HDFS
> directory to be used for both input and output–but is there some
> abstraction
> layer on top that allows it? I've been digging around, and I think the
> answer is "No" but I want to be sure.
>
> More specifically, the only abstraction layer I'm aware of that chains
> together MapReduce processes is Cascade, and I think it requires the reduce
> steps to be serialized, but again I'm not sure because I've only read the
> documentation and haven't actually played with it.
>