In this article, an improved version of a new networking protocol for distributed or parallel computations is presented. In common, it is suitable just for fast, reliable and featureful interchange of small messages. The protocol's implementation and demo project are provided.

Introduction

First of all, what is meant here as "computational cluster". Cluster is a union of workstations, which is formed for some definite purposes. Computational cluster is a cluster, which is built for heavy computations. It is a specific system that asserts special requirements for network functionality. Main properties of the networking mechanism for a good quality cluster are:

Fast data interchange.

Reliable data transfer.

Broadcasting support. As usual, all workstations inside some net take part in the computational experiment, so broadcasting makes controlling much easier.

Huge data blocks interchange support. Sometimes, for example, initial conditions of experiment can be represented by such a block.

Peer-to-peer networking. Any workstation can be the data source and the data destination, so they all are clients and servers simultaneously.

In fact, majority of parallel computing software toolkits are represented as libraries, which use standard networking protocol TCP/IP [1]. There are a lot of disadvantages of using this protocol:

Low speed of data interchange. The "reliability" and "universality" of TCP has a lot of overhead charges. This protocol is a general-purpose one, so it is suitable for working in such unstable matter as Internet, but in a constant (or quite constant) system, which was developed for computations, it is possible to get more benefits.

TCP does not support broadcasting. UDP does, but it is not reliable and the size of UDP datagram is limited by 65467 bytes [1].

Ideology of logical channel creation before data interchange is redundant for cluster computations. Firstly because cluster, as usual, is a well tuned, good working net. Secondly because, some strategies of cluster computing lead to disordered interchange between workstations.

TCP is a stream-based protocol, but, for determined tasks, bounded blocks interchange is preferable, because it allows to say definitely, when all data, necessary for further operations, have arrived.

Of course, a specialized networking protocol can be adapted for special requirements, which arise for cluster computations. So, CTP is a protocol that is to satisfy needs of arbitrary tasks, which, need support of rapid messages interchange and which, can start heavy computations,s as a reaction for message receipt. Despite the fact that the letter "P" from its name means "protocol", it is not just a specification. CTP is an ideology and toolkit, which allows to use it. So, it is able to replace such products as MPI implementations, PVM, and so on.

Ideology

The majority of existing toolkits for parallel computations use, the so named "message", as a basic abstraction. The basic abstraction used in CTP is "command". Command is an order from somebody to someone to do something (in most cases, workstations in clusters are communicating exactly in this way) or the response for such an order. From the last sentence, it is possible to conclude that a command is characterized by the following parameters:

"somebody" - sender.

"someone" - recipient.

"something" - command's description.

So, first of all, it is needed to define the sender and the recipient somehow. For this purpose, IP addresses will be used. The reason is that IP is used extremely widely and it fully satisfies the requirements (gives unique identifiers to all workstations). Commands will be identified by integer numbers.

In terms of the discussed protocol words, "command" and "message" are, actually, synonyms. "Command" is "message", but not always vice versa.

CTP needs to satisfy the cluster networking requirements, listed above. The way in which this will be achieved follows (in the same order as in the introduction):

For incrementing the speed of interchange, UDP will be used as the basis of the protocol. Moreover, the usage of UDP, without going down to raw networking, will save the user in future from additional problems with the protocol support toolkit's installation.

Reliability of data interchange is to be implemented. Each sent packet will be stored until the recipient has not confirmed the receipt of the data. To maintain this mechanism, packets are to be provided with identifiers. Identification will be performed by assigning integer numbers on the sender-side. These IDs cannot be unique in general, but are to be unique for each sender.

Broadcasting support is one more argument to use UDP as the basic protocol.

Huge data interchange support is to be implemented. If a message that is greater than some limit (65400 bytes, by default) is going to be sent, then it is to be divided into smaller parts. These parts will be enumerated and sent to the recipient separately, one by one. On the recipient's side, they will be united to arrange the initial command. An important note is that the recipient application will get information about the command's arrival only after all its parts have been received. Such commands will be named as "large commands", but on practice, the majority of commands are "normal" (need a single packet for its transfer).

For peer-to-peer interchange, CTP's implementations are to include both client and server functionality, as a solid unity.

CTP/IP's relationship with the OSI-model [2] and UDP/IP ideology is shown on fig. 1.

The fact that CTP covers a number of layers, from transport layer to application layer, proves that the area of its responsibility starts from relatively low level and goes to a high one.

Internal World

The following discussion is carried out in the form of chaotic descriptions of CTP basic concepts, their properties, and functionality. Reading of all this is to form a solid idea on what is going on inside. Such a strange form of statement was chosen, because systematic discussion is to be started from several points simultaneously, but this is, unfortunately, impossible.

Debug abilities of CTP may help to better understand the sequence of operations. There is a feature to place descriptions of all actions and events to a given output stream.

Header

Each CTP-packet is represented, as usual, as header plus body (data). The structure of the header is shown in table 1 (in the order of appearance).

Name

Size (in bits)

Comment

Packet size

16

Unsigned integer. Size of the packet (including header).

Command's number

16

Unsigned integer. Command's number (from 0 to 32767, highest bit is not set). If highest bit is set, then packet represents a confirmation for the message with the corresponding command's number.

Packet's number inside the large message

32

Unsigned integer. For large commands - packets number, from zero to amount of packets (given by next field of the header) minus unity. For normal commands - zero.

Amount of packets for the message

32

Unsigned integer. For large commands - amount of packets needed for its sending. For normal commands - zero or unity.

ID

32

Unsigned integer. Identifier of packet. Must be unique for each sender.

Message size

64 (only 48 are used)

Unsigned integer. Size of the whole command's data (without any headers). So, size of the biggest command is the maximal amount of packets multiplied by the maximal size of packet, namely 232*65400. This equals 280890861158400 bytes, or more than 255 terabytes. Last value allows to consider the size of message as unlimited. It is obvious that 48 bits are enough to store size value, but 64 bits were apportioned for alignment.

Options

8

Set of bits. Each bit determines if corresponding option is set or not. Options will be discussed later.

Table 1. CTP packet's header

It is possible to calculate that the total size of the header is 25 bytes. All important transfer parameters, as IP addresses and ports of sender and recipient, are stored in the UDP header.

Each packet can be fully identified by its sender, its receiver, and ID. Sender provides uniqueness of ID for each recipient in the following way: initial value of ID for next packet that will be sent is taken as pseudorandom number. After sending each packet, it is to be incremented. The very first packet sent to the recipient, is to be marked with special option (see below) to allow recipient to learn the value of the starting ID.

Storages

The flowchart, which illustrates the sequence of operations CTP does for packet interchange, is shown on fig. 2:

Fig. 2. Flowchart of CTP's implementation.

In the flowchart, so named, "storages" are mentioned. There are four storages for data, cumulated during lifetime, which provide functionality.

Session information storage. It stores description of each workstation the current one communicates with. Among description, next packet ID, interchange timeout, and description of packets received from this recipient, are meant here.

Interchange timeout is used to determine when the sent packets need to be resend if they have not been confirmed. This timeout is adoptive (because a cluster can be rather heterogeneous and can involve workstations via both intranet and Internet). Initially, default timeout is taken (100 milliseconds, by default). After the first interchange, its value is taken as time, needed for it, multiplied by coefficient (3, by default).

If confirmation of packet's arrival will not be received during timeout, then the packet is to be resent. The period between resending will grow exponentially. If packet is not be confirmed after 8 re-sendings (255 timeouts will pass), then an error message "Command is not confirmed too long" will be generated. If timeout is set to zero, then this feature is switched off.

Messages can be resent. So, it is necessary to protect the user from receiving one message several times. That is why, descriptions of received packets are stored for each addressee. It is implemented as an ordered list. First element contains the maximal ID of the packet, received in sequence. After this element, there can be more IDs, corresponding to packets, which have been received, but which are greater than the first element. After insertion of each new ID in this list, the sequence, which begins from the first element, is to be truncated. For example, let's assume that this storage contains {7, 9, 10, 11, 13, 14}. This means that all packets with ID less or equal to 7 and equal to 9, 10, 11, 13 and 14 already have been received. After receiving the packet with ID 8, the list will take the form {11, 13, 14}. If all packets arrive in sequence, then the list always contains a single element.

Values of IDs are to be in the endless loop (after 232-1 goes 0). Determining of starting ID, which was generated by the sender, is very important in this stage.

A new entry is added to session information storage when the first message is going to be sent or was received from workstation, or is unknown yet. There is to be a special entry for broadcasted messages.

Sent commands storage. To send the command, packets are to be arranged. Some memory is to be allocated and filled with packet headers and data. The fact is that it will not be freed and unallocated just after sending, but stored to the sent commands storage. A record can be removed from sent commands storage only after all its packets arrival have been confirmed.

This ideology can be implemented not for "each command", but for "each packet" (like in CTP 1.0), but first variant is preferable. In this case, so named "smart buffers" can keep from redundant memory allocations, by reserving and guarding memory needed for headers, while doing packets data arrangement.

Large commands storage is used to arrange the whole large message, when receiving it part by part. It stores the total amount, the vector of parts receiving status, and a buffer for compiling. Each part of the message, except, probably, the last one, is of maximal data size, so parts can easily find their places in the buffer, knowing their numbers. When all parts have been received, the message is considered to be arranged and the server informs the application about data arrival.

Deliveries storage. The whole received message or error description is, so named, "delivery". After generating, they will be added into deliveries list. Then special deliverer threads will take them from the list and pass them to the application.

Speaking in terms of object-oriented programming: objects of classes, which implement the recipient application, can subscribe to get deliveries for given command. In this case, the corresponding object will receive information about the command's arrival, and about errors which are related to this command.

Also, there is to be a default receiver that gets information about the command, which has no subscribers, and about common errors (like error while sockets creation) which have no related commands.

Confirmations

Confirmation is a packet with empty body (header only), which has only three differences from headers of the packet having been confirmed. In confirmations header:

Packet's size is set to header's size.

In command's number, highest bit is set.

Message size is set to zero.

It can be considered as an inefficient solution - to confirm each packet with separate confirmation, but it was done to provide more features by using options. Do not forget that CTP is not only a protocol, but also a toolkit.

After The Packet Have Been Sent

First thing the recipient does when it receives a packet is check if the same packet has been already received. If such a packet was already received, that means that the sender failed to get the confirmation, so confirmation has to be sent again and this packet's receiving procedure can be skipped.

Confirmation has to be, exactly, "sent again", not "resent". Confirmations are not stored in the sent packets storage, but are generated when needed.

If such a packet has not been received earlier, then information about its arrival needs to be stored.

If the got packet represent the normal command, then the server informs the application about data arrival (creates delivery and puts it to the deliveries storage). If it is a part of a large command, then the server stores it to the large command storage. If it is the last remainder part of the message, then delivery also is to be generated.

After the packet has found its place, confirmation has to be sent, and the recipient begins to wait for the next packet.

When the sender receives any confirmation, it is to delete the corresponding record from the sent packets storage. The mechanism, like a physical system, aspires to minimize its potential energy, to free all storages as soon as possible.

Threads

Implementation of protocol functionality is to be multithreaded. There are three types of threads:

Server threads receive packets, implement confirmation support, large commands arrangement and so on. If data arrives or error information appears, the thread adds it to the deliveries storage. Once per some period (100 milliseconds, by default), this thread checks the existence of packets that need resending, and resends them if necessary. There can be an arbitrary amount of server threads, depending on the task.

Deliverer threads check deliveries list, and if it is not empty, then delivers the first delivery to the corresponding subscriber or to the default receiver. If the thread does nothing for a long time (20 seconds, by default) then it will be terminated. On idle loop, a thread can fall asleep for some period (20 milliseconds, by default).

Each command is an order (or response). It can order to do something difficult and enduring. So implementation of deliverers as separate threads allows to compute something "by order", just in place and moment, when it has been requested. Nevertheless, it is strongly recommended not to waste this feature. Do not use, for example, modal dialogs in command receiving handlers, because it will keep the deliverer busy uselessly.

Delivery manager thread creates additional deliverer threads if all existing deliverers are busy and deliveries list is not empty. Of course, the maximum amount of deliverers is limited by some value (50, by default). The protocol's mechanism aspires to reduce loading. On idle loop, a thread can fall asleep for some time (10 milliseconds, by default).

Options

Options allow to add an interesting functionality to the networking. There are five possible options:

DelAfterError - if set, then information about sending this packet will be removed for sent packets storage after the receiver has been informed that its arrival was not confirmed (after error description has been generated). So, this error will be delivered only once, and if data has not arrived, it will be lost.

NoResend - if set, this packet will not be resent even if it was not confirmed.

UniqueCommand - if set, then confirmation of this packet's command will confirm all packets with the same command's number that was sent to the given recipient. It is not allowed to use this option for large messages to protect from integrity corruption.

Broadcast - if set, then this packet is to be broadcasted. Option itself does not have any influence on the networking. As usual, for broadcasting, the user has to specify the recipient's IP address, like: 255.255.255.255. But this option provides two important things which are necessary for broadcasting with CTP. First of all, sent message has to get its ID and has to be stored in session information storage in entry, which corresponds to "broadcasting". Secondly, this message will get the ability to be confirmed by an arbitrary workstation, because it is impossible to know beforehand, who will get it. The command is considered to be confirmed if at least one workstation confirms it. It is not recommended (but possible) to use it for large commands.

StartSession - if set, then this packet is the first one which is sent by the sender to this recipient. So, ID, brought by it, can be taken as the minimal ID for the session with this sender. This is taken into account while checking, if packet has been received earlier or not.

Note that the concept of "session" means one-way channel. If two workstations are exchanging some data, then both have entries for each other in the session information storage.

These options can be used in any combination: separately, all jointly, and so on.

For example, options set ErrorOnce, NoResend and UniqueCommand, ORed together, can be useful for commands like: "answer me if you are alive" (it is also called "ping"). For commands that are sent often, which are small and which does not bring information, but does response or confirmation - the recipient is working.

Implementation for Windows

Initially, CTP was implemented for the Windows operating system to become the basis of the networking mechanism used in (Cellular Automata Modeling Environment & Library) project [3]. See project's home site. Of course, it also could be used for arbitrary applications, which needed rapid messages interchange and heavy computations "by order".

The protocol's implementation is represented by a set of classes. The class which implements the main functionality of CTP has the name CCTPNet. The description of all classes which are involved in the CTP's implementation follows.

IPAddr Class

Objects of IPAddr class represent the IP-address of the workstation. This class does not need any explanations except the source.

SmartBuffer Class

Objects of the class SmartBuffer represent "smart buffers" that save CTP's implementation from redundant memory allocations. It reserves the place for the packet's header once per definite size of the data (maximum amount of data in a single packet) on the fly. So, the user just puts data to the smart buffer. Then sending function inserts headers, and packets are ready to go out. This class' definition follows:

class SmartBuffer
{
public:
// Constructor. Parameters:
// +) datasize - amount of data to be used;
// +) autodel - if true then this buffer will be freed automatically by
// protocol's implementation after it will be sent and confirmed if
// needed. So, working with such buffer, you will have to create it with
// operator new, but do not to delete it. If this parameter equals false
// you will have to do new and delete manually
// +) headsize - size of header of each packet;
// +) maxdatasize - maximum size of data in single packet (without header)
SmartBuffer(unsigned int datasize=0, bool autodel=true,
unsigned int headsize=25, unsigned int maxdatasize=65400);
// Constructor. Parameters:
// +) fname - name of file to be stored in internal buffer as data;
// +) datasize - amount of data to be stroed just before the file;
// +) autodel - if true then this buffer will be freed automatically by
// protocol's implementation after it will be sent and confirmed if
// needed. So, working with such buffer, you will have to create it with
// operator new, but do not to delete it. If this parameter equals false
// you will have to do new and delete manually
// +) headsize - size of header of each packet;
// +) maxdatasize - maximum size of data in single packet (without header)
SmartBuffer(LPCTSTR fname, unsigned int datasize=0, bool autodel=true,
unsigned int headsize=25, unsigned int maxdatasize=65400);
// Destructor
virtual ~SmartBuffer() {delete[] m_pBuffer;};
// Access to key values
// Header size
inline unsigned int GetHeadSize() {return m_uHeadSize;};
// Data size
inline unsigned int GetDataSize() {return m_uDataSize;};
void SetDataSize(unsigned int datasize);
// Maximum data size for single packet
inline unsigned int GetMaxDataSize() {return m_uMaxDataSize;};
// Allocated buffer size
inline unsigned int GetBufferSize() {return m_uBufferSize;};
// Auto deleting
inline bool GetAutoDel() {return m_bAutoDel;}
inline void SetAutoDel(bool autodel) {m_bAutoDel=autodel;}
// Access to key pointers
// Returns begining of the buffer
inline char* GetBufferBegin() {return m_pBuffer;};
// Returns current pointer
inline void* GetCurPtr() {return m_pCurPtr;};
// Sets current pointer to pointer to data of i-th packet (i from zero to
// GetPacketsCount()-1). Result false if index i is out of bounds
inline bool CurPtrToDataPtr(unsigned int i)
{char* res=GetDataPtr(i);
if (res) {m_pCurPtr=res; returntrue;} elsereturnfalse;};
// Sets current pointer to the begining of data
inline void CurPtrToDataBegin() {m_pCurPtr=m_pBuffer+m_uHeadSize;};
// Access to packets
// Returns amount of packets
inline unsigned int GetPacketsCount()
{return m_uBufferSize/(m_uHeadSize+m_uMaxDataSize)+
((m_uBufferSize%(m_uHeadSize+m_uMaxDataSize))?1:0);};
// Returns pointer to header of i-th packet (i from zero to
// GetPacketsCount()-1). Result is zero if index i is out of bounds
inline char* GetHeadPtr(unsigned int i)
{char* res=i*(m_uHeadSize+m_uMaxDataSize)+m_pBuffer;
if (res>m_pBuffer+m_uBufferSize) return NULL; elsereturn res;};
// Returns pointer to data of i-th packet (i from zero to
// GetPacketsCount()-1). Result is zero if index i is out of bounds
inline char* GetDataPtr(unsigned int i)
{char* res=GetHeadPtr(i);
if (res) return res+m_uHeadSize; elsereturn NULL;};
// Returns size of i-th packet (i from zero to GetPacketsCount()-1),
// including header. Only last packet's size can differ from header's size
// plus maximum data size. Result is zero if index i is out of bounds
inline unsigned int GetPacketSize(unsigned int i)
{if (i<(m_uBufferSize)/(m_uHeadSize+m_uMaxDataSize))
return m_uHeadSize+m_uMaxDataSize;
elseif (i==(m_uBufferSize)/(m_uHeadSize+m_uMaxDataSize))
return (m_uBufferSize)%(m_uHeadSize+m_uMaxDataSize);
elsereturn0;};
// Data and header access routines
// Put data of size GetHeadSize() from src to the are of i-th header.
// Returns true if data was copied successfully to the existing header etc
// and false otherwise
bool PutHead(void* src, unsigned int i);
// Put byte of data bt to internal buffer from current pointer (if dest is
// negative) or from dest-th byte.. Current pointer will be moved to the
// end of put data (skipping headers) if movecur equals true. Returns true
// if data was copied successfully
bool PutDataByte(unsigned char bt, bool movecur=true, int dest=-1);
// Put data of size size from src to internal buffer from current pointer
// (if dest is negative) or from dest-th byte. Current pointer will be
// moved to the end of put data (skipping headers) if movecur equals true.
// Returns true if data was copied successfully, without truncation etc and
// false otherwise
bool PutData(void* src, unsigned int size, bool movecur=true, int dest=-1);
// Put string to internal buffer from current pointer (if dest is negative)
// or from dest-th byte. Current pointer will be moved to the end of put
// data (skipping headers) if movecur equals true. Returns true if data was
// copied successfully, without truncation etc and false otherwise
inline bool PutDataString(char* str, bool movecur=true, int dest=-1)
{return PutData(str,strlen(str)+1,movecur,dest);};
// Put data from file fname to internal buffer from current pointer
// (if dest is negative) or from dest-th byte. Current pointer will be
// moved to the end of put data (skipping headers) if movecur equals true.
// Returns true if data was copied successfully, without truncation etc and
// false otherwise
bool PutDataFile(LPCTSTR fname, bool movecur=true, int dest=-1);
// Trim the buffer, by cutting the content, excluding the part of buffer
// from current pointer to the end. It is strongly to perform this
// operation only after all buffer's modifications
void Trim();
protected:
// Calculates needed buffer size
inline unsigned int GetNeededBufferSize(unsigned int datasize,
unsigned int headsize, unsigned int maxdatasize)
{return datasize?(datasize/maxdatasize*(headsize+maxdatasize)+
((datasize%maxdatasize>0)?(datasize%maxdatasize+
headsize):0)):headsize;};
// Calculate pointer by dest (if negative - then by m_pCurPtr). Result
// pointer will be put to ptr. If prtnsize is not NULL then portion size
// will be also calculated and put to variable, pointed by prtnsize
inline void DestToPtr(int dest, char*& ptr, unsigned int* prtnsize);
// Data members
bool m_bAutoDel; // Detele it automatically or not
unsigned int m_uHeadSize; // Size of header
unsigned int m_uDataSize; // Size of data
unsigned int m_uMaxDataSize; // Maximum size of data in single packet
unsigned int m_uBufferSize; // Size of allocated internal buffer
char* m_pBuffer; // Pointer to internal buffer
char* m_pCurPtr; // Current pointer
};

NetSender Class

Class NetSender is a base class for CCTPNet, that implements the main functionality of CTP. NetSender is used only to describe the interface of the common network sending class for an arbitrary protocol.

NetReceiver Class

If there is NetSender class, then there must be NetReceiver class also. NetReceiver is used to describe the interface of objects that can subscribe for the delivery of information about data arrival and errors.

class NetReceiver
{
public:
// Is called when have received data pointed by data
virtualvoid OnReceive(void* data)=0;
// Is called when there was an error, described with data pointed by data
virtualvoid OnError(void* data)=0;
};

If an object is to subscribe for deliveries, it is necessary for its class to be NetReceiver's descendant (support of multiple inheritance in C++ allows to add an additional ancestor to any class). Member-functions OnReceive and OnError will be called on message arrival and on error, respectively. In the first case, pointer data will point to the description of arrived data; in the second case, data will point to some error description generated by NetSender's descendant.

Parameters of these member-functions are pointers to void, not to some concrete class, because NetReceiver class is also oriented to arbitrary protocols. When working with CTP, OnReceive's parameter will point to an object of CCTPReceivedData class, and OnError's parameter will point to an object of CCTPErrorInfo class.

CCTPReceivedData Class

Objects of this class describe and give access to the received data. It needs no explanation.

Everything must be clear above, except one circumstance: why timestamp is needed as a field of this class? It is the fact that the moment, when a network error occurred, may be very important, for example, for building log files. But the time when the error's description has been delivered may differ greatly from the time when it had taken place. To avoid such mistakes, timestamp was decided to be included as a field of CCTPErrorInfo class, and will be filled just during the object's construction. Timestamp for the current moment can be always retrieved with the help of the static member-function GetTimeStamp with the accuracy of thousandth of a second.

CCTPNet Class

This class implements CTP's main functions (client and server simultaneously). The following definition describes the members:

CCTPStatusDlg Class

This class allows to display a dialog which shows CTP loading: the amount of elements in storages and the amount of deliverer threads. It also gives the ability to suspend servers. The class has a constructor, which takes a reference to an object of CCTPNet class, to keep an eye on. CCTPStatusDlg class definition follows:

This code is provided in the file NetIncludes.h. It is also necessary to link WinSockets library ws2_32.lib to your project (choose "Project" | "Settings" | "Link", then type "ws2_32.lib" in "Object/library modules" edit field).

Then, start Winsock up. For this purpose, for example, put the following code in the project's main window's initialization function:

WSADATA wsaData;
WSAStartup(MAKEWORD(2,2),&wsaData);

After this, place the following code to start the CTP server up:

m_pCTP = new CCTPNet(m_pCTPReceiver,1515);
// Server created suspended so it needs to be started manually
m_pCTP->SetSuspended(false);

The last code is correct in the assumption that m_pCTPReceiver is NetReceiver's descendant. For example, you can add NetReceiver to your main window's parents.

Measurements

The demo application which is provided with this article allows to try the described CTP implementation. It also includes implementations of TCP and UDP in the same framework, so all these protocols can be used together and, obviously, can be compared (fig. 3).

For this experiment, system clocks of two workstations were synchronized via SNTP with the same time server, using NetTime 2.0. Extreme values of interchange time were taken for the diagram.

A similar result of CTP and UDP shows that CTP's implementation doesn't use a critical amount of resources. Its overhead expenses are small enough to be ignored.

CTP is twice faster than TCP while working with normal commands and not very large commands that can be brought by several packets. That is great result, because the overwhelming majority of interactions in clusters are performed using normal messages. Order to start computations, query of some values, and response for such queries are small.

TCP is better for large commands. Nevertheless, huge data blocks appear in cluster computations rarely, for example, on the stage of task separation (and even here, not always). An important note is that CTP is not critically slow for large blocks, so it can be used as networking mechanism in clusters, paying attention to the previous paragraph.

Besides, a test has been performed on two nodes, because this is more interesting here: pure protocol's implementation throughput. Results of comparison for rapid interchange between dozens of nodes are to be more pleasant for CTP, because its activities will stay the same, but TCP will loose a lot on channels creating and recreating. For CTP, it does not matter who the recipient is.

Reason why it is impossible to overcome TCP is because it is on kernel level, but CTP is implemented by the application. From one side - this is a disadvantage of the last one, but from another, it is absolutely independent and complete.

session information is taken into account now; so now we have adaptive timeouts, intelligent resending, and so on;

now there is no need to store anything in the system registry;

confirmations interchange ameliorated;

received packets storage concept was substituted and improved;

full-scale broadcasting support;

support of multiple servers;

more features for "smart buffers";

featurefull debug and logging interface;

all timeouts and delays become tunable;

timing performance becomes significantly better;

a lot of bugs fixed.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

Share

About the Author

Lev Naumov.
MSc in Computer Science. Graduated from Computer Technologies Department of Saint-Petersburg State University of Information Technologies, Mechanics and Optics.
Worked as C/C++ and Java programmer. Now - the research worker in "CAMEL Laboratory" and PhD student in Computer Technologies Department of Saint-Petersburg State University of Information Technologies, Mechanics and Optics.
Has scientific achievements in field of physics, automata theory, cellular automata theory, cluster computing. There are some publications.

I haven't tried it with VS2008, but as I may see it all is related to changes in compiler. Try to add #pragma once in headers which are claimed to redefine something. There are lots of discussions of these movement from VC 6 to VS 2008 over the Internet, for sure.

I think it would be worth mentioning that a UDP packet cannot be greater than 1500 bytes over Ethernet. The API might except packets up to 65K but then it breaks them up before sending them. If you are sending packets greater than 1500 then you will start to see network degradation. This is very important because this could be the begining of several network problems especially if you are using several multicast groups with a high frequency of data. You should always if possible try to send non-fragmented UDP packets.

A UDP header is 8 bytes
An IP header is 20 bytes
An Ethernet header is 18 bytes
And the trailing checksum is 2 bytes

Therefore yields 1488 bytes for data in a non-fragmented UDP packet.

The same thing goes with sending to many small packets. Make sure you run tests etc...

My name is Teenu Tyagi. I am currently working with Infosys,Pune.
I am writing a document on Command Transfer protocol and planning to post it on our Knowledge Management intranet, so that other employees in my company can also benefit from reading it.

I have referred the data available on the site : http://www.codeproject.com/KB/IP/ctp.aspx

I would like to include parts of this in my document. I request you to kindly grant me permission to do so. I assure you that I would give appropriate credits and references (in-line citations) in my document.

I would be highly obliged if you could mail me back granting permission for the same.

Hi Dum,
On my account( teenu.tyagi@gmail.com ) also i am getting the same message that i am getting on message board - "I've answered you via e-mail".
I couldnot find your reply on my mail Id.
Could you please send your reply on my above mentioned e-mail account.
I will be very thankful to you

hello
i am working on cluster computation from last 4-5 months, i need to know how this protocol is useful for cluster computation and better than the message passing.
i am using lam mpi for my programming.

I was testing the demo application with (compiled with VC 8.0)
When i send a file in CTP it is OK the frist time, the second time i got always an assertion error \vc8\include\list line 213: expression: list iterator is not derefercable

I just have a quick look to the source code of CUDPNet, I found that, whatever what port the socket class bind to, it will send to that port as the code of "send" and "createsocket" both reference to "m_uPort".

So, my problem is, under this circumstance, user is not able to bind a port but send to different port, is that the design or what?

Coz under NAT consideration, we always want to detect where the connection from (NAT address, NAT port) and want to specify where (NAT address, NAT port) to send to. However, by using your class, I can't do that as no way to me to specify which port to send to. So, what can I do if two nodes A and B, under NAT, want to communicate, by using your class?

Class CUDPNet (and CTCPNet also) were developed as trivial samples with only purpose: compare the throughput of networking protocols' implementations. So, they are not bound to solve compicated requirements of specialized tasks. If, in any case, you want to use these classes for your problem, you can (if I understood you right):
1) develop their descendants, which will support needed functionality;
2) use the array of CUDPNet class objects.

"As is", if it could be named a licence. Actually, I haven't thought about it, when I've started. Further works (if any) will be made under some licence, but this... You are free to use it with reference to me. Moreover, could you be so kind to send me notifications, that you have implemented something, using CTP. I'm just collecting the solutions.

Hi! I've modified the demo project adding 2 more lines. On my OnReceive method in CCTPNetReceiver class I've comment the lines where the the save file dialog appears and I've added a response to the sender in which I'm sending him another file. While I get the first file, I don't get the response file on the other side. Can u explain what is happening? CTP Status show that a large command is being delivered but there is no response on the other side.
I've also seen that when I'm using TCP for sending files I don't get the hole file. Sometimes I get some bytes from the file and sometimes I get the hole file. Is this normal?

Hi dum!!!The files are bellow 1Mb and are not special in any way. Here is a snippet of the code. This is what I've modified in OnReceive from the CTP receiver:
...
else if (rd->command==NET_FILE) {
// Handle NET_FILE command

SmartBuffer* sm=new SmartBuffer("C:\\Winter.bmp");
bool b = (dlg->m_pCTP)->Send(*sm,NET_FILE,rd->from);
if(b == true)
TRACE0("Sent file!!");
else
TRACE0("Can't send file!!");
// Ask for filename
//CFileDialog fd(FALSE,NULL,NULL,OFN_HIDEREADONLY|OFN_PATHMUSTEXIST,"All files (*.*)|*.*||");
//if (fd.DoModal()==IDOK) {
// // Copy recieved data to file
// CStdioFile file;
// file.Open(fd.GetPathName(),CFile::typeBinary|CFile::modeCreate|CFile::modeWrite);
// file.Write(rd->pBuf,(UINT)rd->size);
// file.Close();
//}
}
...
In this case I don't always receive the winter.bmp file(it has 700k).
I've also tried usign the TCP server to transfer data. While transfering some files with your demo app I've seen that on the receiver side I don't always receive the whole file. Let's say that the file to send has 700kb. On the receiver side I get a file with a smaller size. This is arbitrary. With CTP I always get the whole file(if I get it )...Did you encounter this problem with TCP or CTP?

Hi!
While interchanging smaller files I receive on the other end the file. I'm using your protocol to build a distributed system for image processing. I'm sending over the network images. A client receives an image with a command, does some operations on it then sends it back to the source. As I said, I've tried to modify your demo to respond to a received file with a file(I thought I was doing something wrong in my app).

sorry dum...i'm new with multithreading and syncronization objects...i don't have a solution...i tried using TCP to send big files and CTP to send small messages...while using TCP I've seen that not all the times I get all the bytes in the sent file...this test was done using your demo app from which I've removed the save dialog file from TCPReceiver...if you manage to resolve this problem please tell me...

Obviously I will. But problems with TCP are really very strange and unbelivable. I can imagine that there is bug in CTP, but in TCP...
Thank you very much for the information. Please keep me informed in progress and state of ypur project.

Wow ur application is really cool. the CTP is something i m looking for which is really fast for file transfer. One thing is though, can this be implemented using single threaded?As in will the performance be very much compensated?

It cannon be absolutely singlethreaded. In any case you'll need one server thread, one deliverer thread. Delivery manager thread always has single instance. Why do you need singlethreaded implementation Please explain it to me and I'll think about including it into next version.

mambohoe81 wrote:

As in will the performance be very much compensated?

I didn't get, what do you mean? When you'll have hard traffic with many small messages you'll loose a lot with singlethreaded implementation. After delivering of one big file will wait until it will be handled by your single deliverer thread.

Well what i meant was will singlethreaded version of this application experience alot of performance reduction as compared to the multi threaded one.
Reason why i asked whether if there is a single threaded version is because i am rather new to programming and was nt so sure of the concepts of multithreading. Cos my application involves other functions like image processing and i was afraid multi-threading might cause my whole app to hang if the threads are not synchronised properly. But someone told me that the image processing and user interface need not be multi-threaded, so i guess everything will work even though the network communications part is multi-threaded. Is it true?

This article is really excellent. I tried the code and its really good and fast!!!
One question:
i tried the SendFiel function in the sampl eapplication. It works fine on files up to 1 MB. When i try to send a file i.e 10MB, nothing happens. the file is never received.
You have any idea?
Thanks a lot
Thomas

This article is really excellent. I tried the code and its really good and fast!!!

Thanx

RSE Thomas wrote:

i tried the SendFiel function in the sampl eapplication. It works fine on files up to 1 MB. When i try to send a file i.e 10MB, nothing happens. the file is never received.
You have any idea?

I have tried 15 Mb now - it works. But there can be a problem if you have bad connection and CTP failed to deliver you file during timeout. Actually I do not recommend to use this protocol for regular huge files interchange. As I've seen it can be unstable. In CTP 2 this will be solved. Now, for stabe interchange of hudge files it would be great to tune timeouts under file sizes.

Pic 3 shows that CTP takes less time in 3 cases
when the data sizes are empty, 50 bytes, and 60KB.
For the last 3 cases, CTP takes more time to send the messages. Could you explain why CTP is faster by 70%.
The first 3 cases seem to be faster by 10 - 20%.

Please, pay attention that ordinate axis has logarithmic scale. For example, for empty message we have 165 ms for TCP and 95 ms for CTP. Lets calculate: 165/95 is near 1.74. This means that CTP is more than 70% faster.;PAnnouncement: I hope, in the nearest future you'll see CTP 2.0, which will differ greatly and which is really novel.

I think you need include MFC. CTP doesnot uses much from this library (CWinThread only as I remember), but you will need it for interface. I do not think that it will be a problem to build CTP under Borland C++;).