Introduction

This article is the second of a multi-part series that will cover the architecture and implementation of components needed to create and maintain a robust, scalable, high performance, massive multiplayer online game server and game engine.

The first article of the series focused on building a Scheduling Engine to drive organized, real-time change in a virtual world. The present article focuses on the design and implementation of a TCP/IP communication component, designed to efficiently handle communications between the game server and remote game clients (players).

Background: BBS Games to MUDs

Back in the 80's, I ran a modem-based bulletin board system (BBS) that let users dial in and leave messages for other users, share files, and play simple multi-player games. BBS-hosted games (anyone remember TradeWars?) were multiplayer in the sense of having a common, persistent virtual world with which multiple players could interact. Each day, players would dial in to the bulletin board system to use up their quota of daily strategic moves and to see what other players had done in/to the virtual world. Because only one person could play the game at a time, real-time player interaction was not possible.

When I headed off for college in the early 90's, I had to shut down the BBS. Just when I thought my days of online games were over, I found out my dorm room had an Ethernet outlet...

A friend from high school, now living hundreds of miles way, introduced me to MUDs (online multiplayer games) as a way to meet up online and have fun. Players connected to MUDs using Telnet to connect to a server via TCP/IP, and then Telnet sent the player's typed commands to the game server, allowing interaction with the game. Best of all, my friend and I could both play and interact in the same virtual world at the same time.. Whereas BBS games involved multiple players taking turns, Internet MUDs involved multiple players all playing simultaneously. With Ethernet access in my room and his encouragement, it didn't take me long to get hooked. I started as a player, and when I became bored of playing, I moved on to world creation. When I became bored of world creation, I set up and hosted my own MUD from my dorm room Ethernet, and began making modifications and enhancements to standard MUD code.

While working on MUDs, I spent a lot of time learning the ins and outs of the Berkeley sockets API which handled client communication with the game engine. I'd like to think that my experience with socket communication back in the 90's was not wasted, and in fact, when I started snooping around the modern .NET TCP/IP client- and server-related objects, I found the Berkeley sockets API skeleton hiding in the closet.

So, without further ado, I present to you my own, modernized implementation of a high-performance TCP/IP server, complete with discussion of (and solutions to) various socket-related quirks I encountered along the way.

Terminology - Sockets, and Outgoing / Incoming Connections

"An Internet socket (or commonly, a socket or network socket), is an end-point of a bidirectional communication link that is mapped to a computer process communicating on an Internet Protocol-based network, such as the Internet." (Wikipedia).

In my own words, a network socket is an object that, when connected, can send and receive data to/from another computer. A socket can be in a connected or disconnected state, and can send and receive data. Pretty straightforward...

A socket object is created when your computer initiates a connection to a remote host. A connection request is sent to a specific port on a remote host. If the remote host accepts the connection, the socket you created enters a connected state, allowing you to transmit and receive data to/from the remote host. In this article, an Outgoing Connection refers to a socket object, created locally, used to initiate contact and communicate with a remote host.

A networked computer also has the capability of listening for incoming connection requests. To listen for incoming connections, a socket is created and then bound (assigned) to a specific port / address. That socket is then instructed to listen for incoming connections. When a remote host requests a connection to the specified port on your computer (and your computer accepts the connection request), an additional, dedicated socket object is created on your computer to handle data transmission with that specific remote host. In this article, an Incoming Connection refers to a connected socket, created in response to fulfilling a remote host's request for a connection with your computer.

Peer-to-Peer vs. Client/Server

In peer-to-peer communication, a single machine acts as both a server (receiving incoming connections) and as a client (initiating outgoing connections to remote hosts). There are just enough differences between incoming and outgoing connections to make implementing peer-to-peer software annoying. Examples: Firewalls may prevent incoming connections from a remote host, but still allow connection to the same remote host via the use of an outgoing connection request. And, there are other small differences between outgoing and incoming connections: outgoing connections point to a remote host name and port, and if connectivity breaks, you may be able to use the same host name and port information to reestablish the outgoing connection. However, the sockets used to handle incoming connections are assigned their own port number when a listener/server socket accepts an incoming connection. Therefore, if connectivity with an incoming connection is lost, you cannot simply reconnect to the remote machine using the host name and port associated with the (now-broken) incoming connection.

Okay, enough confusion. For my sake and yours, this is not a peer-to-peer project. It's a straightforward client-server TCP/IP implementation. Here is how it works:

The client is responsible for initiating a connection to the server. Therefore, the client only needs to process and take care of a single outgoing connection.

The communications server listens for connection requests from clients. The communications server must process and take care of multiple incoming connection requests and incoming connections.

Example Use

Before delving into the design and implementation details, I'd like to present several snippets of code that demonstrate how the communications library can be used:

Design and Implementation

For the sake of article length, I'm going to jump right into the design and implementation. The class OutgoingConnection is used by the game client to establish an outgoing connection to the game server. When the connection is accepted, the game server creates an instance of the class IncomingConnection to communicate with the connected client.

Both OutgoingConnection and IncomingConnection inherit from the common base class, ConnectionBase. This base class provides communication-related functions (i.e., sending and receiving data) that are common to both incoming and outgoing connections.

The interface ICommunicationsBase is implemented by both incoming and outgoing connections. Elements common to both incoming and outgoing connections include:

a socket (the connection or pipeline used to send and receive messages),

an event that will be raised whenever a new message is received from the pipeline,

a method that can transmit data across the pipeline to the remote host, and

a way of closing the socket if/when the connection needs to be terminated.

(The interface also inherits from ITrackBytes, which tracks bytes sent and received for diagnostic purposes.)

Outgoing Client Connection to the Server

The client's connectivity with the server is defined by the OutgoingConnection class. In addition to inheriting the basic communication capabilities from class ConnectionBase, it also contains the following:

A method, public bool Connect(string hostName, int portNumber), that is used to initiate a connection with the communications server.

Three threads, one dedicated to receiving incoming data, another dedicated to sending outgoing messages, and the third dedicated to extracting messages from the incoming data stream.

Incoming Client Connection on the Server

On the server side of things, an instance of the IncomingConnection class is used to track and communicate with each connected client. In addition to inheriting the basic communication capabilities from ConnectionBase, it also contains the following:

A pointer to its parent communications server object.

An ID which can be used to associate the connection with an Entity in the game.

A pointer to an incoming message handler, IIncomingMessageHandler, that can be used to transmit incoming commands from the client to the associated in-game Entity.

A visual representation of the relationship between OutgoingConnection, IncomingConnection, and ConnectionBase is presented below:

Transmitting Messages

At the heart of the client-server communication is the transmission and receipt of messages. A message is a complete parcel of information - when packaged up, sent, received, and then unwrapped, a message contains the same (complete) information as sent by the sender.

The Send/Receive Buffer

The thread-safe SendReceiveBuffer is used to buffer incoming and outgoing message data. Within the class, bytes waiting to be sent and incoming bytes waiting to be processed are stored in ThreadSafeQueue<byte[]> _outgoingByteChunks and ThreadSafeQueue<byte[]> _incomingByteChunks, respectively.

When data is received from a remote client, it is enqueued into _incomingByteChunks, where it sits, waiting to be combined and processed. When data is ready to be sent, bytes are dequeued from _outgoingByteChunks for transmission. Because it's inefficient to transmit small chunks of data multiple times, small byte chunks are combined prior to transmission. In this way, multiple messages may be sent via a single TCP/IP data packet:

Messages and Fragmentation

Blocking vs. Non-blocking Sockets

In blocking mode, sending data to a remote client halts the current thread's execution until all data is sent. Similarly, attempting to receive data from a remote client halts the current thread's execution until data is received.

When sending data to a remote client in non-blocking mode, part or all of the data is transmitted, and the send operation immediately returns. When receiving data, if the client has not sent any data, receive() will return a value indicating that no data has yet been received. If the client has sent data, part or all of the transmitted data will be read.

We don't want the communications server waiting around for send and receive operations to complete; we want to read available data that has been sent from a client and then immediately move on to read available data from the next connected client. Similarly, we want to send as much waiting data as possible to a client, and then immediately move on to send waiting data to the next client. Therefore, the current communications library implementation uses non-blocking sockets.

Dealing with Partial Data Transmission

A message is defined as a complete parcel of information - when packaged up, sent, received, and then unwrapped, a message contains the same (complete) information as sent by the sender. When data is transmitted via TCP/IP in non-blocking mode, complete messages may be broken apart (fragmented) into multiple packets of data for transmission. Therefore, messages may need to be reassembled into complete parcels of information when received by the recipient.

The technique most frequently used to ensure complete receipt of a message (and to help identify the type of message being transmitted) is to prefix the message with a header. In the presented solution, a message header consists of two integers (each 4 bytes). The first integer specifies the size of the complete, transmitted message, and the second integer specifies the transmitted message type:

4 bytes 4 bytes n bytes
[Msg Size][Msg Type][Compressed Msg]

For flexibility, it would be nice to transmit any instance of any class across the wire. Although using binary serialization to serialize and then compress objects may not be the most efficient method of transmitting data between client and server, its flexibility and ease-of-use made it the method of choice for data transmission for this sample project. To maximize efficiency and speed in a real-world scenario, custom serialization should be implemented and used. The following method is used to convert a message (from any serializable object) into a byte array suitable for transmission.

Receiving Message Parts: MessageBuffer

Because messages may be received in multiple pieces, the message header (the first four bytes of which contain the total size of the message) is used to determine when a full message has been received. The MessageBuffer class is used to determine if a full message has been received and, if so, read the message from the incoming data stream. The class MessageBuffer uses a MemoryStream to store incoming data (which may or may not contain a complete message). In this way, the MessageBuffer class extracts concrete messages from a continuous stream of data.

The method bool ReadMessage(out object o, out int transmissionType) is used to read the next message from the MessageBuffer, if a full message exists in the buffer. The method returns false if no full message is yet available (as defined by the message size in the header); otherwise, it deserializes the message into object o and populates the message's transmission type.

Full use of the MessageBuffer class is demonstrated in the method bool GetNextFullMessage(). The method either reads new incoming messages or returns false if no messages (or only a partial message) are available:

///<spanclass="code-SummaryComment"><summary></span>/// Checks to determine if a full message has been received. If so, raises the
/// MessageReceived event.
///<spanclass="code-SummaryComment"></summary></span>internalbool GetNextFullMessage()
{
bool fullMessageReceived = false;
bool error = false;
// No new data received, so no need to check for a new messageif (_incomingByteChunks.Count <= 0) returnfalse;
// Append the new data received to the incoming message buffer
_incomingMessageBuffer.AppendNewIncomingBytePackets(_incomingByteChunks);
// Records the number of bytes we have available// and sets up the message buffer for reading
_incomingMessageBuffer.RewindAndCreateBinaryReader();
while (_incomingMessageBuffer.ReadHeader(out error))
{
object o;
int transmissionType;
// Try to read the complete message. If the message is incomplete,// ReadMessage returns false and we exit the loopif (!_incomingMessageBuffer.ReadMessage(out o, out transmissionType))
break;
fullMessageReceived = true;
// Raise an event to signal a full message was receivedif (o != null && MessageReceived != null)
{ MessageReceived(this, new ObjectEventArgs(o, transmissionType)); }
}
if (error == true)
{
// Clear the incoming byte chunks - see if// we can start over with a fresh slate
_incomingByteChunks.Clear();
}
_incomingMessageBuffer.RemoveReceivedMessagesFromBuffer();
return fullMessageReceived;
}

Receiving Messages

When a complete, incoming message is received by the server, the server uses an implementation of interface IIncomingMessageHandler to interpret the message. In the demo project, the ServerMessageHandler class handles the server's receipt of incoming messages:

TCP/IP Tricks and Tips

Detecting Dropped Connections

When writing a TCP/IP client and server, a frequently encountered problem is the issue of clients dropping their connections. Specifically, when a client drops its connection to the server in an abnormal manner, the server can still use the partially closed socket to "send data to the client". In other words, after a client drops his/her connection, the server may blissfully continue sending data into a black hole, seemingly unaware that the client is no longer fully connected. This actually isn't a bug; TCP/IP is a full-duplex network protocol, which means the connection can be closed "half way". Therefore, additional work is required to determine if the client's connection is fully connected (capable of both sending and receiving data to/from the server). In the presented communications library, the following (quick and dirty) code executes occasionally to determine which connections have been "dropped":

Demo Project

The demo/test project, TestCommunicationsWinforms, is a Windows Forms application that tests both the server and the client functionality of the communications library. The application performs the following:

Sets up a server socket to start listening for incoming connections on port 4006.

I am trying to find a way to uniquely identify a client in the list of connections.

I found the 'AssociatedID' which you have described as:

/// On the server side, the AssociatedID links an incoming connection
/// to an Entity in the game (by Entity ID)

On the server side I am noticing that all values for AssociateID are all '0' (i.e. for each client connection). I tried setting it myself from the client side before creating the connection and the server still reports AssociatedID as the value 0. How do I determine which clients the messages are coming from?

The server is registring the clients but when I send a large objekt(like a bitmap) to client from the server nothing happens.
When I debug(stepping) through the code then the client is receiving the object !!

From the server I use: ICommIncomingConnection.sendmessage(object o, int type)

Well, I didn't make this article, but sending something big like a bitmap isn't very smart. If it is a small bitmap (like 32 pixels by 32 pixels or so) then it is alright. If your sending something very big, then it might take more time to load then you think. It also might be that the library doesn't support looping through bytes until they are all sent.

Two or three people have emailed me regarding bumping up the number of "simulated clients" in the test project, and that bumping up the number above 100 causes a failure.

Each "simulated client connection" is fairly costly - these are very responsive, multi-thread-each connections that are meant to simulate a *remote* client connection to the server. Trying to simulate too many remote clients causes a slowdown and a large number of threads to be created. To accurately test how the server handles thousands of client connections, you need to set up multiple machines or multiple processes, each simulating 50-or-so clients, all connecting to the same server instance. The server performance remains robust under these conditions.

In summary, the failures seen when bumping up the number of simulated clients occurs due to the cost and performance simulating that many remote clients - not due to failure of the server. Hope this helps clarify the issue and helps with testing. I'm thrilled that there is some interest in the library! Pretty cool. Anyway, feel free to email me if you have additional questions. Thanks,

Hard to debug for you given what you've posted, but I can tell you that there is a port of MiniLZO that doesn't use unsafe / pointers. Basically, I ported the C# version of MiniLZO to work with Silverlight (where unsafe is not allowed). Perhaps it would solve your issue. If not, please send more details. I've been successfully using the compression routines for quite a while now, without error. Thanks,

Basically when you use IOCP you are letting the OS manage your IO requests. For both in OS and application IO requests management is a complicated thing. If you let your application wait till the IO is complete, you loose performance. Now if you are not waiting (doing it asynchronously) you should have some sort of mechanism to know when your IO is complete. If you poll continuously, you loose efficiency, because much of the CPU cycle is wasted looking if the IO is over or not. Now if you poll slowly you loose/compromise response time. There are many ways of managing IO and one is using completion port.

So what you basically do is you create a completion port and associate handles to it (for instance SOCKET handles). Now for each asynchronous IO (usually using overlapped IO) that you do on that handle, you get notification on that port when it is over. So now your problem reduces to retrieving the notification as quickly as possible. Completion port is nothing but a FIFO queue maintained by the OS.

The IOCP model is very efficient. It is scalable and do have multiprocessor support. IOCP handles the thread pool, no. of concurrent running threads etc. In the context of writing a server, you usually use this model when you have to maintain tens and thousands of connections. It is not worth the time if you are writing a complex server code that uses IOCP model just for a few (maybe hundred) connections.

You can look up in the MSDN for APIs such as CreateIoCompletionPort(), GetQueuedCompletionStatus() etc..

I m trying to do a client - server standalone console applications.
I use your same code and split it into two console apps that run alone and when client send messages to server, nothing happens.
I try to debug your code and i find that in SendReceiveBuffer class when u are ready to rise an event to indicate that a message is received the variable that should contain the message is null.
Can anybody help me?.
Thanks in advance.

Make sure you are not trying to run two servers (one in each console app). You should have one console app running as the server and the other console app running as a client, connecting to the server. If you want to post source code or email it to me, I'd be happy to take a quick look,