Title

Authors

Abstract

MapReduce is a programming model from Google forcluster-based computing in domains such as searchengines, machine learning, and data mining. MapReduceprovides automatic data management and fault tolerance toimprove programmability of clusters. MapReduce’s executionmodel includes an all-map-to-all-reduce communication,called the shuffle, across the network bisection. SomeMapReductions move large amounts of data (e.g., as muchas the input data), stressing the bisection bandwidth andintroducing significant runtime overhead. Optimizing suchshuffle-heavy MapReductions is important because (1)they include key applications (e.g., inverted indexing forsearch engines and data clustering for machine learning)and (2) they run longer than shuffle-light MapReductions(e.g., 5x longer). In MapReduce, the asynchronous natureof the shuffle results in some overlap between the shuffleand map. Unfortunately, this overlap is insufficient in shuffle-heavy MapReductions. We propose MapReduce withcommunication overlap (MaRCO) to achieve nearly fulloverlap via the novel idea of including the reduce in theoverlap. While MapReduce lazily performs reduce computationonly after receiving all the map data, MaRCOemploys eager reduce to process partial data from somemap tasks while overlapping with other map tasks’ communication.MaRCO’s approach of hiding the latency ofthe inevitably high shuffle volume of shuffle-heavyMapReductions is fundamental for achieving performance.We implement MaRCO in Hadoop’s MapReduce and showthat on a 128-node Amazon EC2 cluster, MaRCO achieves23% average speedup over Hadoop for shuffle-heavyMapReductions.