GSM Gateway modem, service, interface and emulator

Objective

Since I own Netduino Plus and SIM900 GPRS Shield, I decided to find a use for these two components
and build a GSM gateway. The aim of this project was to provide basic functionality for remote clients
to send SMS messages through a web interface along with the ability of reception of incoming SMS
messages and phone calls. To put this all together I had to use C#, .Net Micro Framework, C++, Boost, MySQL,
PHP and JavaScript.

Functional layout

Because this was not my first attempt to create a GSM gateway (two previous attempts failed because of complexity),
I decided to analyze the problem before I started doing any programming. One can say that sending an SMS message
through a GSM modem takes a few lines of code to do it. It is a simple task unless you want your gateway
to do more than send SMS messages. In my case, I needed to receive SMS and phone calls as well and then save these
into a database. To accomplish this, you need to divide the solution into multiple individual parts.

Netduino + GPRS Shield - is the project's cornerstone. Netduino communicates with the
GPRS shield throught a serial port and also acts like server to receive and process client requests, such
as "Send SMS" or a broadcast "Hello" message. It has to be a fast and asynchronous service.

Server service - the purpose of this service is to select enqueued requests from the
database, send them to a lower layer (Netduino) and update their status according to the completion result.
Internal mechanism ensures that requests are popped from the database in a given interval or when a remote
client sends a PUSH command with a notification await handler. This service exposes a simple HTTP server
to receive commands from either a web service or a remote client.

Web interface - enables the end user to manage the database and send commands to
the service. This can either be a GUI frontend which directly communicates with the service or an
interchange server such as REST, JSON or SOAP.

Netduino

As some of you may know, GSM modems use serial communication to talk to a terminal. In my project,
Netduino represents the terminal. For example, to send an SMS message, Netduino issues AT+CMGS command,
then sends the message body and then waits for acknowledgement as seen below:

AT+CMGS="00421944176952"> Hello world
+CMGS:<NUM>OK

The main issue with such a terminal is that you want to be notified of incoming calls or SMS
messages while being able to issue client requests to the modem. But you cannot have two threads
receiving input from the modem, because it gets intercepted by one of them, not all of them.
And you cannot have one thread doing all the work, requests must be processed asynchronously
at the time of reception, the same goes for notifications issued by the modem which have highest priority.

I managed to resolve this issue by dividing the terminal firmware into four threads:

Main thread - when the firmware loads, it first enables ethernet interface
and assigns it a static IP address. Then it opens serial port to talk to the modem and initializes it.
For example, it tells modem to send +CRING instead of RING notifications, or not to send CLIP info.
After the initialization sequence, main thread enters "reading-only" loop, which blocks thread
until a new line is received from the modem. This has a big impact on how the rest of the terminal
works, see below.

Output channel - sends messages such as received call, received SMS or discovery
messages "Hello" and "Goodbye" to the service. This works independently from the other threads, so that
the service can process messages immediately.

Requests scheduler - when a request is completed, a new response is generated
based on the completion code. This response is scheduled within the requests scheduler which then
sends these responses to the clients who originated the request.

Server - accepts new requests from clients. This thread does not block while
a request is being executed, however only one request is allowed to be enqueued at a time. Sending
a new request while previous one is still being processed results in an error. Server accepts two kinds
of requests, those that need to be processed by modem and those which are processed directly by
the server (such as request to obtain product information).

Lets examine two scenarios.

First, a remote client wants to send an SMS messages

He issues a request
which gets intercepted by the server thread. Server ensures there is no request being processed.
Then it creates a new command to be issued to the modem (see image below for all commands supported
by the terminal), sets it as the only one active command and executes first sequence, which results
in sending modem this line:

AT+CMGS="<RECIPIENT>"

Server thread then resumes its execution. Once the modem is ready to accept the message body, it issues:

>

As soon as this sequence is received, main thread unblocks and starts processing. First of all it checks
whether there is an active command being processed. If yes then it forwards this response to the command
ProcessResponse handler which decides whether the command has been completed, failed or needs
more processing. Once the command completes, it generates a response and schedules it for completion.
Requests scheduler dequeues this response and sends it back to the remote client.

// Receive next line from modemwhile((line = Receive(modem))!=null){// Main thread must lock IssuedCommand, because server may have issued// command to the modem and did not set the active command yet// but is holding mutexlock(IssuedCommand.SyncRoot){// Here, the issued command is always the last executed command
issuedCommand = IssuedCommand.ActiveCommand;// If there is an issued command awaiting to be completedif(issuedCommand !=null){try{// Process responseif(issuedCommand.ProcessResponse(modem, line)){// Complete issued command
issuedCommand.Complete(outputChannel, requestsScheduler);// Create next command in the chain and set it as active command
IssuedCommand.SetActiveCommand(issuedCommand.Next(modem));}continue;}catch(UnrecognizedCommandException){// Complete issued command
issuedCommand.Complete(outputChannel, requestsScheduler);// Reset command
IssuedCommand.SetActiveCommand(null);}}// ... default response handlers, see example #2

Second, an SMS message has been received by the modem

Modem sends a notification message to the terminal:

+CMTI:"SM",<NUM>

Main thread checks whether there is an active command being processed. If not, the message is
caught by the default response handlers. One of them, dedicated to the +CMTI, simply creates a new
command to read all unread SMS messages and sets it as the active command.

Next cycle, once modem sends response to the AT+CMGL command, this response will be processed
by ProcessResponse handler, which reads message header and content. Once there is no message
to be read, ProcessResponse returns true, main thread will then call Next method which
simply creates another command to delete all read messages. After that, a response is generated and sent
through the output channel to the service which intercepts it and saves into the database.

PC service

PC service (or server service), is the most complex piece of the puzzle. It consists of multiple worker threads,
asynchronous queues and fault protections, just to make life easier for the Netduino firmware and the web service.

When the service starts, it first connects to the database. Without a valid connection, no request can be made
or on message received. After the connection is established, program creates modem repository, an auto-updating
list of active modems. This initialization triggers the first discovery request, which sends a broadcast message
requesting all modems to send back their product informations. If no modem within a given time interval responds,
the request will be resent until there is at least one active modem in the repository. This operation runs in a separate
thread, so the main thread continues initialization by creating input channel (reception of incoming messages),
requests pool (all requests that need an active modem are processed here), requests scheduler (keeps track of active
requests), server (HTTP server to control the service) and finally it creates a snapshot.

If any error occurs at any stage, already initialized resources are freed and the whole initialization process starts again.
Once the service is up and running, only a remote client can stop it. To demonstrate how does the service work,
we have to use an example.

Sending an SMS message through the service

When a client wants to send an SMS message, it has to first insert the message into the database. Then it optionally sends
a PUSH message to the service, requesting the scheduler to select all not processed messages from database. If no PUSH
message is issued, the scheduler will process the request in the next cycle.

Once the message is about to be executed, a new worker thread is created for the request, taking the message
as a parameter. The thread is started and request begins its execution by requesting the modem repository to create
a modem request. Unlike server request, modem request has its own queue in which it gets executed. Simply described
as a packet sent to the terminal awaiting to be completed. Once the modem request is assigned, the message can be delivered
through it to the terminal awaiting an acknowledgement whether or not the execution completed successfuly.

When the request completes (either successfuly or with an error), it updates message state in the database to "Completed"
and optionally notifies the remote client about the completion. The notification is sent only when the client sends
a PUSH command together with a parameter indicating that a notification is required. In that case a new entry is created
in the notifications list and keeps the connection alive until completion or cancellation. If the request execution
takes too long, the notification can be cancelled with a "Timeout" error, however this does not specify whether the
request completed successfuly or not.

Web interface

In order to enable remote clients to send SMS messages, I had to make a web service and a web interface.
Web service simply accepts requests, stores them in the database and notifies the service. The web interface
(control panel) allows me to see sent/received SMS messages and phone calls, service status messages,
modems being present on the network, manage remote accounts and most important, to control the service. Both web service
and control panel are built using PHP and JavaScript.

Testing

The most difficult part was the testing. Take for example Netduino. I would have to rebuild
the firmware and deploy it every time I made the smallest change in the code. Also I could not just
send SMS messages until I was absolutely sure that the code behind is working properly. So I had to come
up with a solution, otherwise it would be a waste of time and money.