A multitouch control for the smartphone is much more similar to traditional remote controlling to an aircraft.
Therefore this was a test to how good a quadrocopter can be controlled with such an App.
In this case two technologies can be distinct. The first is bluetooth as radio link, the second is controlling via a smartphone.
Bluetooth as radio link is really easy to use because it’s buildin within the smartphone and on the aircraft it’s also easy to handle as a serial device. The drawback is a low range. Therefore it’s not perfect for fast aircraft like some sort of planes.
Controlling with a smartphone is nice, because it’s really handy and a lot of people does already have one. The drawback is really bad haptical feedback. It’s harder to handle it without the feeling where the sticks are. But if the thumbs are not lifted, the controlling of a quadrocopter for example, is really good.

Some notes on the multitouch implementation. Like on a simple single touch App it is the same callback method for multitouch
handling, the onTouchEvent method. Within the parameter from the type (class) MotionEvent there is a getAction method which has the information about what touch the event has called, first or second one. It’s not a really a easy to guess api, but it works and within the net are a lot examples for handling it.

A usual Android ™ smartphone includes all necessary technology to use it as a remote control for a quadrocopter.

Accelerometer and magnetometer to measure the attitude

Touch sensitive display

Bluetooth

The app does only have 300 lines of code without any optimizing. For attitude angle measuring the important methods are SensorManager.getRotationMatrix and SensorManager.getOrientation. With these information the pitch and roll input can be controlled. The position of a finger on the touch display can be determined with the method onTouchEvent(MotionEvent event). This is used to control yaw and thrust.

That’s all to control the QC. But most QC flight controls need some special stick position to arm their motors. For this a button or something similar is needed. If this button is pressed, the special stick positions are send for a couple of seconds to arm the motors. After that the control of the QC is as desribed.

The first tests for video transmission an o UAV was made with Skype ™ which has some drawbacks. Therefore the software has been extended to also transmit video from the Android ™ smartphone to a laptop. Technically this is realized through the preview function on the Android API. Currently 15 frames per second are used. Every frame is converted to a jpeg picture and send as UDP package to the ground station software. This approach has several advantages:

Frame rate and video quality can be chooses

Only one app to have UAV control, telemetry data and video downlink

Video can be mirrored in software if it’s necessary to attach a mirror in front of the smartphone cam.

Video can be turned

Should be possible to take pictures or record movies in HD at the same time the preview is transferred.

In the video you see 3 setups. On the first one the WLAN access on the DSL router was used for both, the smartphone and the laptop. The latency was about 0.2 seconds.

On the second one the mobile access (vodafone ™) on the smartphone was used while the laptop still used the WLAN DSL access. This time the latency was about 0.26 seconds. The next mobile tower is only 150 meters away. At this video there a some errors in the pictures which is a bug in the software and has to be fixed.

On the third time both devices used two separate mobile access (vodafone ™) and the latency was about 0.36 seconds which is not too bad for a 3G connection. Especially because of this scenario has to use a kind of “proxy” server for the connection between the two devices.

A couple of improvements was made on the project. One of the hardest to get working was the
mobile to mobile device connection. A mobile internet access usually uses NAT technology which is
nice if you only access servers in the internet and if you start the connection. Things become
harder if you want to initiate a connection to a mobile device. For the “real time” concerns of the
project a UDP connection is preferred over a TCP connection. This is very similar to internet phone
applications like SIP or Skype ™.
Unfortunately there is no ready to use library appropriate to the system. This means such a library
has to developed.
The approach to get a connection is known as “NAT punching” and may end up in two different
types of connection between mobile devices. The first one is a direct connection between both
devices, the second is a connection with a server in between like a proxy. The direct connection is
preferred because of less delay.

The next improvement was the carrier. A bigger quadrocopter with more easy access on the
smartphone box. The box is also fixed on the copter for both, no pendular effect and better judgment
of the copter attitude. The bigger copter also get more flight time. Full loaded with a 2,45 Ah battery
it stays 10+ min. in the air.

The last improvement was to transmit telemetry information like gps position and compass. On the
ground station software the information is used to calculate distance, position and orientation of the
uav relative to the ground station. The raw gps information are also logged into a file.
For the video transmission the telephony software skype ™ is still used. But this time with the two
mobile devices and the cloudy day it was not possible to have a real fpv flight.

Safety concerns

Some notes on security. The complete system have a lot of things that can fail, so it’s important to
take care about these things. Most interesting are all wireless connections. In this system there are
two of them. First the Bluetooth connection between smartphone and flight control. For this there is
a watchdog implemented in the Arduino which switch to a failsafe position if the smartphone does
not send commands anymore. The failsafe position result into an horizontal attitude of the QC and
let em go down. Fast enough to get out of the air but slow enough not to be damaged on ground
contact.
The second wireless connection is between smartphone and ground station and maybe a “proxy”
server in between. If this communication is lost both sites try to reconnect to each other to regain
control. If the smartphone get no commands it does also send no commands on the QC via
Bluetooth which result in the above described behavior.
If only the video transmitting breaks, the groundstation show still position, altitude and direction of
the uav. With these information it could be possible to get the uav back because the aircraft has to
be a self stabilizing one.
Not really a security issue but still very useful is the gps logging on the groundstation which makes
“search and rescue” operations more successful.
Advantages of this approach

What is the advantage of this approach compared to more “traditional” one?

The technology on mobile internet devices improves a lot the last years and do that further on the
next years. With LTE there is a promising, full packet orientated communication with less latency
and bigger bandwidth available soon. Transmitting a lot of data in a short time is possible and the
range of operation is really high. Mobile devices like smartphones have all necessary sensors in
it for a uav. They are also relatively cheap and can be used for a lot of usecases. Imagine that a
“stable” carrier, plane or quadrocopter, can be build up for 100 to 140 Euro. A smartphone may
cost 200 Euro. That’s not very much money for a rc model with uav capabilities, including a cam,
gps and more.

After controlling the FPV car via Internet the next step was to enhance the system to get a quadrocopter with it into the air.
For this goal the number of channel has to be increased from two to four. Also the software needs some logic in case of a transmission lost on both, the bluetooth connection and also the udp internet connection. In both cases there is implemented a failsave regulation to get the copter to the ground on an well-defined manner.

The fist test was not successful, because the MWC stability mode was not stabile enough for the test and the smartphone mounting was too flexible too see the copter movements early enough. Therefore a DJI Naza was ordered but took some time for delivery (more than a month). This flight control is really stable and adequate for the experiment. To handle the flexible mounting the copter is picked up in traditional manner with a direct sight to it. After a touch down it can more easily becontrolled via fpv.

The test was successful so far. Even with the not really great O2 net connection it is full controllable but also with a delay on the video transmission at about 0.1 up to 1/4 sec. This is also the reason to use a really stable copter at stable flight mode.

I worked on my 3G data link based FPV car with the Android Smartphone. Improved the following things

Wide angle lense

“4 channel” RC transmitter for control input generation

Automatic reconnect function on both sides, Smartphone and Laptop

The first thing is an optional lense to get view angel of nearly 90 degres. That helps a lot to navigate and control a vehicle with first person view. Another effect is, that the cam gets more light from the environment and that improves also the speed of dark-bright adaption.

The second thing was to find something with which it is easier to control a vehicle than with the mouse pad. An old remote from an xufo is used for that. To get the stick positions an Arduino reads the potentiometer values and send it via FTDI interface to the laptop.

The last was, to improve the software on Smartphone and Laptop a bit. The main thing was to introduce a detection if connection is lost on both sides. If this is the case, the smartphone software try to reconnect and the laptop software informs the user that connection is lost and wait for a new connection.

Now the car is much more controllable, even with a poor 3G data link as you can see on the following video:

In the last post the connection between the Android smartphone and an RC Car was established. Now it’s also possible to have a connection from the smartphone to the Internet for both, video streaming and remote controlling (RC). Both is needed for a first person view controlling of a vehicle.

But why controlling a FPV car via Internet? Because theoretically there is no limit about the range except the battery capacity. Normal FPV gear and RC components have without the need of a special radio license in Germany a range about half or one kilometer with clear line of sight between sender and receiver. On ground it’s even less, maybe 100 or 200 meter at best. With a 3G data link it depends on the telecommunication net which is big on most places. That’s the reason for doing a little bit “research” on this topic.

The video can be produced through the on board cam of the smartphone. The cam video can be streamed to the net. I use Skype ™ for this purpose, because it’s fast and easy available for my Galaxy S2 and support video telephony.

The tricky part was the car controlling. I decided to establish the connection from the smartphone to my laptop. This worked well even through the home router. Next part was the protocol. TCP is not the perfect option, because it can be very slow with all the checks in it. UDP is much better, because it doesn’t matter if some of the controlling information don’t reach the car. It’s much more important that – new – controlling information coming fast to the car.

For this the smartphone act as the client. On the laptop site there is a small server software which communicate with the smartphone. This software wait for the initial UDP connection of the smartphone and send after that control information via UDP to the smartphone. The smartphone transmit these information via Bluetooth to the Arduino.

A first test show that this all works together but with a lack of accuracy and some delay.

I think the software part of this approach can be improved for faster and better reactions and also more reliability. This approach also use 3G data link and IPv4. You can imagine how this will be improved if IPv6 and 4G data link is available.