Details

Improved MultiFileInputFormat so that multiple blocks from the same node or same rack can be combined into a single split.

Description

The MultiFileInputFormat takes a set of paths and creates splits based on file sizes. Each splits contains a few files an each split are roughly equal in size. It would be efficient if we can extend this InputFormat to create splits such each all the blocks in one split and either node-local or rack-local.