bbFTP

bbFTP can be used to efficiently transfer large amounts of data to/from HECToR. This
document provides a brief introduction to using bbFTP on HECToR. More detailed information
and documentation is available on the bbFTP website.

bbFTP is one of the tools we recommend if you have large amounts of data to transfer to or from HECToR.

bbFTP is an open-source secure multi-streamed file transfer utility, released under the
GNU General Public License. It was written by Gilles Farrache at IN2P3 Computing Center
in Lyon, France. bbFTP has been optimised for the transfer of large files: it securely
authenticates, but then transfers file data in multiple unencrypted streams, thereby
avoiding the computational overhead of encryption and decryption. bbFTP also allows large
windows as defined in RFC1323, and therefore is well suited to the problem. Previous
studies of grid-ftp have shown that there is good scope for improvement over scp
performance.
grid-ftp can be difficult to install and by default, requires a non-trivial certificate based authentication
mechanism that is often too large a barrier for user uptake. bbFTP is a simpler alternative that uses password or
public key based authentication in the same manner as scp, so is immediately accessible
to users with minimal setup costs. However if you already have access to grid-ftp this is an equally good alternative.

Limitations

bbFTP can only transfer files, not directory trees. If you want to transfer a recursive
directory tree then you first need to archive your directory structure into a single file
(using, for example tar) and then transfer this archive. Once transferred, the archive
can be extracted.

How to use bbFTP

There are a number of ways to initiating a bbftp transfer so for simplicity we concentrate
on transferring data between HECToR and YOURMACHINE.

To "get" data from YOURMACHINE onto HECToR, issuing the command from HECToR itself:

The remote filename is relative to the user's home directory on your machine unless it begins
with a "/" in which case it is interpreted as an absolute filename.

The local filename is relative to the directory in which the bbFTP command was
issued unless it begins with "/" in which case it is interpreted as an absolute filename.

When run using this single command, the bbFTP executable is launched at the local and
remote sites simultaneously, and the transfer is initiated over 10 parallel streams (as
indicated by the -m and -p flags). We do not recommend or anticipate the need for the use
of more than 10 parallel streams.

Note that the -E flag specifies the location to the bbFTP executable on the remote site.

To "get" all files in a subdirectory on YOURMACHINE onto HECToR (issuing the command from HECToR):

To "put" a file from HECToR onto YOURMACHINE (issuing the command from HECToR):

Once again, all the rules for relative and absolute filenames apply as in the examples above.
There is also the command "mput" for transferring more than one file.

If you are using a remote bbftp client to connector to hector you should connect to login11b.hector.ac.uk and specify /usr/local/packages/bbftp/bbftpd as the server location.

Note - you will be unable to use bbftp to transfer data to/from the UK RDF filesystem via node login11b.hector.ac.uk.
The RDF filesystems are not mounted on login11b.hector.ac.uk.

If you wish to transfer information to and from your
home site, you should ensure the bbFTP software is also installed there. bbFTP, along with
a full explanation of its usage and modes of operation can be found at the bbFTP website.

Current Service Status

This is an archived website, preserved and hosted by the School of Physics and Astronomy at the University of Edinburgh. The School takes no responsibility for the content, accuracy or freshness of this website. Please email epcc-support@epcc.ed.ac.uk for enquiries.