The IPSendfile Project

This paper describes a message oriented file transfer
service for the Internet. Conventional methods for sending a
data file to another user involve the use of mail or ftp.
Using mail involves splitting and encoding files while use
of ftp is inconvenient and insecure. We have developed a
protocol for the transport of messages consisting of an
arbitrary number of 8 bit bytes that is used as the basis
for a sender intitiated file transfer service.

The internet is used to transport data from machine to
machine, user to user. A very common use is for one user to
send a file to another user. Unfortunately, and this is
very surprising, the Internet family of services does not
include a convenient method for doing this.

The most natural approach would be simply to ask, using the
user interface, for the file to be sent to a destination
user address consisting of the username and domain. This
would treat the file as a message containing arbitrary
binary data of any length and deliver the file to some part
of the user file space.

The first candidate for providing this type of service is
the File Transfer Protocol or FTP [4]. FTP is an interactive
service that requires an end-to-end connection with the
destination host. The source user `logs in' with a username
and password and then instructs the FTP program to move a
file from their file space to the directory of the
destination username.

The first problem with this is that the service is
interactive and requires an end-to-end connection. Most
users would rather send the file and get on with their work
while the file is delivered to the destination. If there is
a network problem, the file should be queued, and delivered
when the problem is resolved. With FTP the user has to try
again later manually.

The second problem is that the user must have a username and
password on the destination machine. This is either
inconvenient, in that the user then must have an account on
every machine that a file is to be sent to, or is insecure
if the destination user is to give out their username and
password for the transfer.

There is a convention adopted by most systems that the user
name `anonymous' can be used by anyone to retrieve files.
This has been extended to the case of sending files by
providing a writable directory, often called `incoming' (or
was that `in.coming'?), where files can be left for a
destination user. Once a file has been deposited in
`incoming' you then have to send email to the destination
user telling them there is a file waiting. This has the
serious drawback that while the file is in `incoming' it can
be viewed or deleted by other users.

FTP doesn't seem to be the answer to the file sending
problem.

The messaging service that is almost universal on the
Internet is electronic mail. It consists of a combination of
SMTP [3] for message transport and RFC822 [1] for message
format. Unfortunately, this service is strongly oriented
towards reletively short electronic mail messages containing
7-bit US-ASCII characters and so is unsuitable as it stands
for transporting binary files of any length. The RFC822
message format standard has recently been extended (MIME
[5]) to handle arbitrary message types in different
encodings, including arbitrary binary. However, this is an
unnatural layering with a binary transport protocol (TCP)
carrying a 7-bit US-ASCII electronic mail service which is
being used to carry binary messages.

It is far more natural to implement a messaging service that
carries arbitrary length binary messages and use it to carry
higher level message services such as a sender oriented file
transfer service.

The `Sydney Unix Network' software and its commercial
successor `MHSnet' that formed the basis for the ACSnet
network does have a suitable architecture. It also has a
sender initiated file transfer system. To send a file to
another person a command like:

netfile bob@cs.su.oz.au myfile

is used. The file is transported across the network by the
MHSnet software and kept in a holding area on the
destination machine. A message is then sent (via email) to
the destination user notifying them that a file has arrived.
The destination user then invokes the command:

netget

to retrieve the file.

MHSnet provides many more facilities than are required to
build an Internet message service. It has a complete
routing table management system, support for any virtual
circuit with a range of delay and error characteristics and
a session layer protocol for establishing connections. It
is also a proprietary package and is not widely available on
the Internet.

To provide a sender initiated file transfer service for the
internet it was decided to build a basic message transfer
service.

The Message Transfer Service carries messages across a TCP
connection. Messages may contain an arbitrary number of
8-bit bytes and no interpretation is made of the message by
the Message Transfer Service. Carried with each message are
the addresses of the source and destination users and a
message type and associated parameters. The Message Transfer
Service will process the received message in a way
appropriate to the message type. This is expected to be
implemented by the Message Transfer Service passing the
message to another process.

The Message Transfer Service has the ability to start a
transfer at an arbitrary point in the message. This would
normally occur when a connection has been re-established
after an unexpected disconnect and is in response to a
receiver asking that the message transfer begin at a
nominated point.

The service consists of three programs:

sendmsg

places message on outgoing queue

sender

processes outgoing queue, sends messages

tcpsmd

receives messages, invokes receiving program for message type

A user invokes sendmsg with a destination user address, a
message type and associated parameters and the data to send.
The message is placed on a queue which is periodically
scanned by sender which attempts to open a connection to
tcpsmd on the destination machine and then, using the
protocol described below, transfers the message. At the
destination machine, tcpsmd receives the file and then
invokes a program to process the particular message type
involved.

The message transfer protocol is designed to be extremely
simple and easy to implement. It may be possible to
implement this service using a language like Perl or with
the Bourne shell with a small number of support programs.
The description of the protocol that follows is informal and
designed to convey the flavour rather than the exact detail.
For a more detailed description see the draft RFC.

A message transfer is initiated by a sender opening a full
duplex TCP connection [2] to a receiver. All commands are
transmitted as 7-bit US-ASCII characters in 8-bit bytes with
the high-order bit set to zero. Each command is terminated
by an ASCII newline character (012). All data is
transmitted as 8-bit bytes.

The receiver begins the session by sending an ``ACCEPT''
string with zero or more options.

The unique identifier is a 14 character string of printable
characters. The `name' is a message type, in the case of a
file transfer it is ``file''. The `param' and `flag' are
character strings consisting of fields separated by spaces.

The receiver responds by sending either:

START address

or

NAK reason

The `address' is a number giving the point in the file that
the transmission begins. This would normally be zero but in
that case of a restart after a connection is broken would be
the address of the last received data. If the receiver
cannot continue with the transfer, it sends NAK and a
`reason' string.

If the receiver has responded with START, the sender
continues with the line:

DATA start

In this command, `start' refers to the start address chosen
by the sender which must be less than of equal to that
chosen by the receiver.

This is followed by the data, exactly size-start bytes, and
then

DEND crc

The `crc' field is a 32 bit CRC, calculated according to the
Ethernet polynomial, over the size-start data bytes.

This transfer is now concluded by the receiver responding
with either

ACK
or
NAK reason

At this point the receiver can close the connection to
indicate it is unwilling to accept any more messages or if
the connection remains open the process can repeat with the
sender sending the initial sequence.

Using the message transfer service we can easily provide a
sender oriented file transfer service.

This consists of three programs:

sendfile

send a file to a destination user

filer

process a file message when received

getfile

retrieve a received file

The user invokes sendfile with the destination user address
and the name of the file or files to be sent. This uses the
sendmsg function to send a message of type file containing
the files to the destination user. At the destination
machine, TCPsmd receives the message and invokes filer to
process it. Filer moves the file into a holding directory
and notifies the destination user (via email) that a file
has arrived. At some later time the destination user invokes
getfile to retrieve the file.