"Serial tunneling" is a name for an exchange of serial
data between two computers, typically using the TCP/IP protocol. In network
terminology, a "tunnel" is a path that passes data without the sender and
receiver having to know or care about the details of the transfer mechanism.
"Serial tunneling" was originally coined as a term to describe a method for
allowing instruments and computers that were designed to communicate via RS-232
to now communicate via TCP/IP. Its meaning has broadened to encompass a
variety of serial data exchanges between instruments via Ethernet. The
EtherSmart Wildcard can implement "Serial Tunneling" by initiating or accepting
TCP/IP Ethernet connections and exchanging binary and/or ASCII text data with
other Ethernet-enabled devices on the local network.

A comprehensive suite of pre-coded driver functions is
available to simplify the implementation of serial tunneling in your
application program. These can be classified as buffer management, data
transmission and reception, connection control, and inter-task service
management functions. Each of these is discussed in turn in the context of the
Tunnel_Test
function from the demo program.

Listing 1‑6 presents the interactively callable Tunnel_Test
function from the demo program. Let’s take a look at how this function works.
At your QEDTerm terminal, type:

main

Tunnel_Test( )

As usual, interactive function calls from the terminal
must be typed with no spaces between the function name and the (
character, and must include at least one space after the ( character. After typing
this command, Tunnel_Test
is running.

Tunnel_Test
prints a welcoming statement telling us that it is waiting for us to open a
connection using Putty in "Raw" mode. For information about how to download
and use the Putty terminal, see the section above titled "Using the Free Putty
Ethernet Terminal for Testing and Development". Briefly, after a simple Google
search for "Putty", download the small executable, and double click its icon to
start it. Simply type the EtherSmart Wildcard’s IP address (such as 10.0.1.22)
and port 80 into the boxes in the Putty configuration screen, select the "Raw"
mode, and click the "Open" button to establish the connection. Type some
(optional) text followed by a carriage return in the Putty terminal window to
send the first line to the EtherSmart.

Tunnel_Test
waits for an incoming Ethernet connection on this module’s IP address on port
80 (the default local port). The Ether_Connection_Manager
routine that is running in the Ethernet task automatically stores the
linefeed-delimited first line into the HTTP_Inbuf
counted buffer, and examines the line to see if it starts with "GET ". If the
first line starts with GET (which is the request keyword issued by a web browser),
the connection manager invokes the web server as explained in the next
section. If the first line does not start with GET (as is the case in this
example), the Ether_Passive_Non_Web_Connection
returns a true value and the wait loop ends. The function then prints the
following message to the QEDTerm serial terminal:

An incoming connection has been
accepted;

each incoming line will be printed to
the serial terminal, and a

response prompt will be sent to the
ethernet terminal.

To exit this routine, wait 20 seconds
without typing a line.

The first line has already been loaded into the HTTP_Inbuf;
an explanation for this behavior is presented in the Web Server section below.
Tunnel_Test
prints the contents of this line to the QEDTerm serial terminal by fetching the
16-bit count that is stored in the first 2 bytes of HTTP_Inbuf,
and using Emit
to print each subsequent character that is stored in the buffer. Note that the
characters are stored starting at the HTTP_Inbuf
address + 2, as the buffer count occupies the first two bytes. Because the
buffers are in paged memory, the page-smart fetch routines FetchInt
and FetchChar
are used to get the buffer count contents; standard C assignments don’t work in
paged memory.

The main do…while loop calls Ether_Get_Line
to input a carriage-return delimited line of text into the Ether_Inbuf,
and invokes a for
loop to print the buffer contents to the QEDTerm terminal screen. For each
incoming line, the "Line was received>" prompt is printed to the Putty
terminal window by the Ether_Send_Buffer
function. Ether_Send_Buffer
sends a message to the Ethernet task that was set up by Ether_Task_Setup_Default
called by main,
and the Ethernet task responds by sending a message in the ether_response
mailbox. The call to Ether_Await_Response
fetches the contents of and clears the ether_response
mailbox. The least significant word returned by Ether_Await_Response
is the number of bytes actually sent by Ether_Send_Buffer;
in this simple example we discard this value. Note that a more sophisticated
program would typically use the more efficient non-blocking function Ether_Check_Response
to monitor the and clear ether_response
mailbox.

The Tunnel_Test
function terminates when Ether_Get_Line
does not receive any characters within the specified 20 second timeout.

As described in the "EtherSmart Driver Data Structures:
The LBuffer, a Counted Buffer" section above, functions that manipulate buffers
need to know the number of valid bytes stored in the buffer. A convenient
place to store this information is in the buffer itself. For this reason, this
driver code uses a data structure called an "LBuffer", defined as a buffer with
a 16-bit count stored in the first two bytes of the buffer, with the data
following.

In general, the buffers are in paged RAM. C assignment
operators do not work in paged memory, so the operating system’s store and
fetch routines as declared in the memory.h file in the Mosaic include directory
must be used to access data stored in the buffers. These functions include FetchChar,
FetchInt,
StoreChar,
and StoreInt.

The input and output buffers for serial tunneling (and
email) are called Ether_Inbuf
and Ether_Outbuf,
respectively. These functions each return the base 32-bit extended address
(xaddress) of the LBuffer. Each of these buffers has a maximum size that is
set at initialization time. The default size of Ether_Inbuf
and Ether_Outbuf
is given by the constant ETHER_BUFSIZE_DEFAULT.
The functions that write to the buffers such as Cat and Ether_Outbuf_Cat
accept the maximum buffer size as an input parameter, and ensure that they
never write beyond the allowed buffer size. If you need specify a new buffer
base xaddress or size that is different from the values set at initialization
time, use the functions Ether_Set_Inbuf
and Ether_Set_Outbuf.
For complete descriptions of each of these functions, consult the EtherSmart
Wildcard Glossary document.

For example, a function like Ether_Send_LBuffer
expects the 32-bit extended address (xaddress) of the LBuffer as an input
parameter, and sends the contents on the active Ethernet connection. This
function automatically fetches the number of data bytes in the buffer (its
"count") from the buffer’s first 2 bytes, and sends data starting at the
specified xaddress+2. Receiving functions such as Ether_Get_Data, Ether_Get_Chars
and Ether_Get_Line
expect the 32-bit extended address (xaddress) of an LBuffer as an input
parameter, and receive incoming data to the buffer, storing the count at the
specified xaddress, and storing the data starting at xaddress+2. The glossary
entry for each buffer handling function tells whether it is dealing with an
uncounted buffer or an LBuffer.

Rule for
Accessing Data in EtherSmart Buffers and Data Structures

Because the
EtherSmart buffers and data structures are in paged memory, the page-smart
routines FetchInt,FetchChar,StoreInt
and StoreChar
are used to access them; standard C assignments don’t work in paged memory.

The data transmission function Ether_Send_LBuffer
sends a counted LBuffer out on the active Ethernet connection. It accepts the
32-bit base xaddress of an LBuffer, a timeout count in milliseconds (msec), and
a modulenum as input parameters. Recall that an LBuffer stores the number of
valid bytes in the first 2 bytes of the LBuffer, with the data following. Ether_Send_Buffer
accepts the 32-bit base xaddress of an uncounted buffer, the buffer count
(number of bytes to be sent), a timeout in msec, and the modulenum. It sends
the specified buffer contents out on the active Ethernet connection. Ether_Send_2Buffers
sends the contents of two uncounted buffers sequentially, each with a specified
base xaddress and count. See the glossary entries of these functions for
additional details.

Each of these transmission functions requires that the
Ethernet task running the Ether_Service_Loop
is active, as set up by Ether_Task_Setup
or Ether_Task_Setup_Default.
The transmission functions each send a message via the ether_command
mailbox to the Ethernet task, which then performs the requested action and
writes a response to the ether_response
mailbox. The contents of this mailbox must be checked and cleared by the
application program using either the non-blocking Ether_Check_Response
function, or the blocking Ether_Await_Response
function. The least significant 16-bit word of the result that is returned by
any of these functions is the number of bytes actually sent by the transmission
function.

The actions performed by the data reception functions are
easy to understand once the naming terminology is defined. These functions
either "Add" bytes, appending them to a buffer that may already contain data,
or "Get" bytes, storing them starting at the beginning of the buffer. The
reception functions can receive "Data" which is a binary stream of bytes, or
line-oriented ASCII "Chars". A "Line" is a set of ASCII "Chars" ending with a
specified "end of line" (eol) delimiter such as a linefeed (ASCII 0x0A) or
carriage return (ASCII 0x0D).

Note that it is permissible to use the Ether_Add_Data
or Ether_Get_Data
functions for either binary or ASCII data, as long as no line-oriented decision
making is required during the reception process. However, the ASCII character
oriented routines should not be used for binary data, as binary characters that
happen to equal the specified "eol" character would get special treatment that
could lead to unexpected results. The recommended course of action is to use
functions that contain the sub-words "Chars" or "Line" for ASCII streams, and
use the functions that contain the sub-word "Data" for binary streams.

The Ether_Add_Data
function receives a binary stream of bytes into a counted LBuffer; as usual, it
requires that the Ethernet task is running. Recall that an LBuffer stores the
count in its first 2 bytes, with the data following. Ether_Add_Data
accepts as input parameters the 32-bit base xaddress of an LBuffer, the maximum
number of bytes to be added to the buffer, a timeout parameter in units of
milliseconds, and the modulenum of the specified Wildcard. It fetches the
starting count from the LBuffer, adds it to the specified buffer base
xaddress+2, and uses the resulting xaddress as the location where the first
byte of incoming data will be stored. The data input operation stops if the
amount of data in the specified buffer (including any prior data, but not
including the 2-byte count) exceeds the specified maximum-number-of-bytes
parameter. The function terminates when either the specified number of bytes
has been received, or the specified timeout is reached, whichever comes first.
Before exiting, the function writes the updated count into the first 2 bytes of
the LBuffer.

The non-appending Ether_Get_Data
function accepts the same input parameters as Ether_Add_Data. Ether_Get_Data
simply writes a 16-bit zero to the count in the first 2 bytes of the specified
LBuffer, and calls Ether_Add_Data.
The result is that the first byte of data is stored at the specified LBuffer
xaddress+2.

The most configurable reception function for line-oriented
ASCII streams is Ether_Add_Chars.
Its function prototype and glossary entry are as follows:

This function sends a message to the Ethernet task which
requests and stores incoming ASCII data from the specified EtherSmart
modulenum. The data is appended to xlbuffer, a counted buffer
whose byte count is stored in the first 2 bytes of the buffer, and the count is
incremented by the number of appended bytes. The xlbuffer parameter is a
32-bit extended address that holds the 16-bit buffer count followed by the
buffer data. The appended data is stored starting at xlbuffer+2+prior_count,
where prior_count
is the 16-bit contents at xlbuffer upon entry into this routine. The data
input operation stops if the amount of data in the specified buffer (including
any prior data, but not including the 2-byte count) exceeds the specified maxbytes
parameter. A maximum of maxlines are accepted, where a "line" is a data
sequence ending in the specified eol (end of line) character. If the maxlines
input parameter = -1, then the line limit is ignored. If maxlines
= 0, then all except the last incoming line are discarded, and only the last
line is added to the buffer, excluding the final eol character which is
discarded and not added to the buffer. The eol parameter is a single
character that specifies end of line. Typical values are ‘CR’ = 0x0D or ‘LF’ =
0x0A. Dual-character eol sequences such as CRLF are not allowed. If eol =
‘LF’, we define the "alternate eol" is a ‘CR’. For all other eol chars
(including a ‘CR’), the "alternate eol" is a ‘LF’. If the discard_alt_eol
flag parameter is true, the alternate to the specified eol character is
discarded/ignored by this routine. If the flag is false, the alternate eol
char does not get special treatment, and is stored in the buffer like any other
character. If the no_msbitset flag parameter is
true, then any characters having its most significant (ms) bit set (bit7,
bitmask = 0x80) is discarded and is not stored in the buffer. This is useful,
for example, if the incoming data is sent by a Telnet application; some
configuration data is transmitted that can be filtered out by discarding
characters with their ms-bits set. This function exits within the specified timeout_msec
whether or not the maximum number of bytes or lines have been accepted. When
the action function dispatched by the Ethernet task has completed, a response
comprising the command byte in the most significant byte, module number in the
next byte, and numbytes_appended
in the remaining 2 bytes is placed in the ether_response
mailbox. To test for a buffer overrun, fetch the 2-byte count from xlbuffer
and test whether it is greater than or equal to the allowed maxbytes.
After calling this routine the application must clear the ether_response
mailbox using Ether_Check_Response
or Ether_Await_Response.

The non-appending Ether_Get_Line
function simply zeros the count at the specified LBuffer and calls Ether_Add_Line.
As a result, Ether_Get_Line
stores characters starting at the beginning of the buffer instead of appending
it after any prior buffer characters.

While this may sound complicated, using these functions is
actually quite straightforward. A simple example extracted from Listing 1‑6 illustrates how to use Ether_Get_Line:

// get 1 line into counted
ether_inbuf (count is stored in first 2 bytes)

num_received = (int) Ether_Await_Response(E_MODULENUM);
// get lsword of result

In this code snippet, we first initialize some variables
that are passed as parameters to Ether_Get_Line.
One of the rules of calling operating system or kernel extension functions
is that no function nesting is allowed: it is illegal to invoke one of these
functions within the parameter list of another. By capturing the results
returned by the Ether_Inbuf
and Ether_Inbufsize
functions into variables, we avoid the nesting prohibition when calling Ether_Get_Line.
The Ether_Get_Line
function is invoked to place the received characters into the counted LBuffer Ether_Inbuf,
with a maximum number of received characters limited to Ether_Inbufsize
(default value = 510 set at initialization time). The end of line character is
specified as the ASCII carriage return (0x0D). Note that the standard line
ending sequence for most standard TCP/IP text transfers is carriage return
followed by a linefeed (0x0D0A). We pass a true (nonzero) flag as the discard_alt_eol
parameter to specify that the alternate end of line character (the linefeed)
should be not placed into the buffer. We pass a true flag as the no_msbitset
parameter to specify that any characters with their most significant bit set
should not be stored in the buffer. The 20,000 msec timeout and modulenum
complete the input parameter list. Ether_Get_Line
stores the input parameters in the Ether_Info
structure and sends a message in the ether_command
mailbox to the Ethernet task which performs the requested action and places the
response in the ether_response
mailbox. We then invoke Ether_Await_Response
to poll and clear the ether_response
mailbox. The number of bytes received is returned in the least significant 16
bits of the result, and we load this into the num_received
variable for use in the program as shown in Listing 1‑6.

Rule for
Calling Operating System Functions

When calling
operating system or kernel extension functions, no function nesting is allowed:
it is illegal to invoke one of these functions within the parameter list of
another.

The pre-coded EtherSmart Wildcard software provides
routines to connect to a remote computer, monitor the status of a connection,
disconnect, and optionally flush bytes that are in the input buffers. Table 1‑6
summarizes these functions.

A serial tunneling connection may be established either at
the request of a remote computer (an "incoming connection request"), or by the
EtherSmart Wildcard itself (an "outgoing connection request"). Only one
connection may be active at a time; this limitation is set by the Lantronix
XPort. If there is no active connection, any incoming connection request
directed at the proper local IP address and local port (port 80 by default)
will be accepted by the XPort hardware. There is no way to change the local
port number for incoming connections "on the fly"; see the section titled
"Configuring the XPort Device" for more details.

When an incoming connection is accepted by the XPort, the Ether_Connecion_Manager
called by the Ether_Service_Loop
in the Ethernet task updates the connection status. Your application program
can monitor the status of the connection by calling Ether_Connect_Status.
Table 1‑7 summarizes the meaning of the status parameter returned by this function. A zero value means that there is no active connection. Odd values
indicate transient conditions as summarized in the table. A value of 2 means
that we have successfully initiated an outgoing connection to a remote
computer. A value of 4 means that we have accepted an incoming connection, and
the connection manager has identified it as non-web, while a value of 6 means
that an incoming connection has been accepted and has been identified as coming
from a web browser. To identify the type of the incoming connection, the
connection manager looks to see if the first characters of the first line are
"GET " (without the quotes); a web browser starts its request with the
uppercase GET substring. If GET is found, the connection is identified as
HTTP/web; otherwise, it is identified as non-web.

Passive HTTP connection interrupted by a remote disconnect
during a send

8

Passive HTTP connection with web service completed,
awaiting connection close

The good news is that, in nearly all cases, you don’t have
to worry about the exact numerical values in Table 1‑7. A simple
zero-versus-nonzero check of the Ether_Connect_Status
return value tells the application program whether a connection is present or
not. The application program does not have to test for a web connection because
web connections are typically handled "in the background" without the need for
intervention by the application program. The useful function Ether_Passive_Non_Web_Connection
returns a true (nonzero) flag when the Ether_Connect_Status
is 4. Thus, calling Ether_Passive_Non_Web_Connection
is a good way to monitor for an incoming serial tunneling request from a remote
computer.

The Ether_Disconnect_During_Send
function returns a true value when the remote computer disconnected while we
were sending data. This low-level function is typically not used in
application programs.

To establish an outgoing connection, call Ether_Connect,
passing it the destination IP address and remote port, a timeout in
milliseconds, and the modulenum. Its function prototype is:

void Ether_Connect(
char ip1, char ip2, char ip3, char ip4, int port,

int timeout_msec,
int modulenum )

For example, to connect EtherSmart Wildcard modulenum 3 to
a remote host with IP address 10.0.1.145 on port 23, allowing up to 15 seconds
for the connection to take place, execute:

Ether_Connect( 10, 0, 1,
145, 23, 15000, 3);

If there is no current connection (remember that the XPort
can only implement one connection at a time), this function will try to connect
to the specified remote for up to the specified 15 seconds. If there is
already an existing connection when this function is called, no connection
attempt will be made. You can invoke Ether_Connect_Status
to check whether there is an active connection. As with nearly all of the
serial tunneling functions, the Ether_Connect
function dispatches a message in the ether_command
mailbox to the Ethernet task which performs the requested action and returns a
response in the ether_response
mailbox. The application program must monitor and clear the response mailbox
using either Ether_Check_Response
or Ether_Await_Response.
The least significant 16 bits returned by one of these functions after calling Ether_Connect
contains an error flag. The error flag is zero if the connection attempt
succeeded, and nonzero if it failed. See the glossary entry for Ether_Error
for a description of the error code values.

To perform a simple disconnect, pass the specified
modulenum parameter to Ether_Disconnect.
To disconnect and "flush" (discard) any incoming bytes from the input buffers,
execute Ether_Disconnect_Flush.
Just as with Ether_Connect,
the least significant 16 bits returned by the next invocation of Ether_Check_Response
or Ether_Await_Response
yields the error result for the operation.

You can also flush the input buffers at any time using the
Ether_Flush
and Ether_Flush_NBytes
functions. Ether_Flush
is the most thorough flush; it waits up to 0.25 seconds since the last
discarded byte and, if no additional byte becomes available in that time,
exits. The maximum execution time of this routine is 8 seconds, even if data
is being continually flushed during this time; this prevents an indefinite
"hung" state. Ether_Flush_NBytes
accepts a number of bytes and a timeout parameter, and flushes up to the
specified number of bytes from the buffer.

The serial tunneling services are provided by means of the
Ethernet task that is set up at initialization time by Ether_Task_Setup
or Ether_Task_Setup_Default.
Your application program invokes the driver functions, most of which work by
sending a message in the ether_command
mailbox to the Ethernet task which performs the requested action. The presence
of a dedicated Ethernet task provides a means to rapidly service communications
events such as incoming connection requests, requests from a web browser, and
communications to and from the application task. Table 1‑8 lists the Ethernet driver functions that manage the inter-task communications.

As explained in the "EtherSmart Driver Data Structures"
section above, a "task" is an environment that is capable of running a
program. The task contains a user area, stacks, and buffers that enable a
program to run independently of other tasks. The multitasking operating system
switches between tasks, providing timely provision of a variety of services. A
"mailbox" is a 32-bit variable in common memory that conveys a message from one
task to another. A mailbox must be cleared before a new message may be sent;
this helps to synchronize the tasks and ensures that no messages are discarded
without being read by the receiving task. A mailbox may be read and cleared by
a "blocking" function that waits until the mailbox contains a message (that is,
until it contains a non-zero value) and then reads out, reports, and zeros the
mailbox. It may also be checked by a "non-blocking" function that reads and
reports the contents of the mailbox and, if it contains a non-zero value,
clears it. Non-blocking mailbox reads are often preferred because they avoid
tying up one task while waiting for another task to send a message.

The EtherSmart driver code uses mailboxes to coordinate
the provision of communications services by the Ethernet task to the
programmer’s main application task. This is all done transparently, so you
don’t have to be an expert on inter-task communications to use the EtherSmart
Wildcard. Three 32-bit mailboxes named ether_command,
ether_response,
and ether_gui_message
are allocated in common RAM when the initialization routine executes. For
example, when the application program invokes a command such as Ether_Get_Data,
a message is dispatched via the ether_command
mailbox to the Ethernet task. Consequently, the Ethernet task gets the
incoming data and places it in the specified buffer. The Ethernet task then
sends the number of bytes received as the least significant 16 bits in the ether_response
mailbox. The most significant 16 bit word of the mailbox contains the command
byte and the modulenum. This mailbox must be cleared by the application
program using either the blocking function Ether_Await_Response,
or the non-blocking function Ether_Check_Response.
The main loop of your program’s application task should periodically call one
of these two functions to manage the interactions with the EtherSmart task.

If an error occurs during the execution of a function by
the Ethernet task, a 16-bit error variable in the Ether_Info
structure is set. The application task can read this variable using the Ether_Error
function, and can clear it using the Ether_Error_Clear
function. Some functions also return the error value as the least significant
16 bits in the ether_response
mailbox.

The Ether_Service_Loop
is a simple infinite loop. In the loop body are calls to Ether_Connection_Manager,
Ether_Command_Manager,
and Pause.
The default version of Ether_Service_Loop
specifies the contents of the ether_service_module
variable as the parameter passed to Ether_Connection_Manager
and Ether_Command_Manager
in the infinite loop body. The ether_service_module
variable is set by the initialization functions Ether_Init
and its calling functions (Ether_Setup,
Ether_Setup_Default,
and Ether_Task_Setup).
It specifies which Wildcard module is accessed by the Ether_Service_Loop
routine running in the Ethernet control task. Because each of these variables
is automatically initialized by higher level functions, you typically don’t
have to worry about them. You can even have two EtherSmart Wildcards
installed, one for revectored serial, and the other providing communications
services (serial tunneling, email, and web service).

When an incoming connection is accepted by the XPort, the Ether_Connecion_Manager
called by the Ether_Service_Loop
in the Ethernet task updates the connection status. Your application program
can monitor the status of the connection by calling Ether_Connect_Status
as described in the prior section. Incoming connection requests are accepted
by the XPort if the connection is directed to the correct local IP address and
local port. The default local port is (decimal) 80, which is the standard port
used by web browsers to request a web connection. The XPort uses port 80 as
the local port so that web connections can be accepted, but in many
applications we also want to be able to detect and service incoming serial
tunneling connections.

If the variable pointed to by HTTP_Enable_Ptr is
true, the Ether_Connection_Manager
automatically stores the linefeed-delimited first line into the HTTP_Inbuf
counted buffer, and examines the line to see if it starts with "GET ". If the
first line starts with GET (which is the request keyword issued by a web
browser), the connection manager invokes the HTTP_Server
to serve out the requested web page as explained in the next section. If the
first line does not start with GET and if the variable pointed to by Ether_Tunnel_Enable_Ptr is
true, then Ether_Passive_Non_Web_Connection
returns a true value.

The application task can use the control variables
accessed pointed to by Ether_Tunnel_Enable_Ptr
and HTTP_Enable_Ptr to
direct this process of accepting incoming connections. After execution of any
of the initialization functions, both of these variables are true by default.

If your application will never use the dynamic webserver
function, you can simplify the handling of incoming serial tunneling
connections by setting the HTTP_Enable_Ptr
variable to zero. This disables the automatic loading of the first
linefeed-delimited line of an incoming connection into HTTP_Inbuf,
allowing the application program to manage the entire reception process.

If your application will use the dynamic webserver but
will never need to accept incoming serial tunneling connections, you can zero
the variable pointed to by Ether_Tunnel_Enable_Ptr.
In most cases this will not affect performance, but in some extreme cases it
may increase the robustness of error recovery in handling web requests.