Computer Science > Distributed, Parallel, and Cluster Computing

Title:
DotDFS: A Grid-based high-throughput file transfer system

Abstract: DotGrid platform is a Grid infrastructure integrated with a set of open and
standard protocols recently implemented on the top of Microsoft .NET in Windows
and MONO .NET in UNIX/Linux. DotGrid infrastructure along with its proposed
protocols provides a right and solid approach to targeting other platforms,
e.g., the native C/C++ runtime. In this paper, we propose a new file transfer
protocol called DotDFS as a high-throughput distributed file transfer component
for DotGrid. DotDFS introduces some open binary protocols for efficient file
transfers on current Grid infrastructures. DotDFS protocol also provides
mechanisms for multiple file streams to gain high-throughput file transfer
similar to GridFTP protocol, but by proposing and implementing a new parallel
TCP connection-oriented paradigm. In our LAN tests, we have achieved better
results than Globus GridFTP implementation particularly in multiple TCP streams
and directory tree transfers. Our LAN experiences in memory-to-memory tests
show that DotDFS accesses to the 94% bottleneck bandwidth while GridFTP is
accessing 91%. In LAN disk-to-disk tests, comparing DotDFS protocol with
GridFTP protocol unveils a set of interesting and technical problems in GridFTP
for both the nature of the protocol and its implementation by Globus. In the
WAN experimental studies, we propose a new idea for analytical modeling of file
transfer protocols like DotDFS inspired by sampling, experimentation and
mathematical interpolation approaches. The cross-platform and open
standard-based features of DotDFS provide a substantial framework for unifying
data access and resource sharing in real heterogeneous Grid environments.