Wednesday, April 22, 2015

One of the most evident sensors of Android devices is the camera. An
ordinary smartphone's camera is able to capture a lot of interesting
information but has its limitations too. Most evidently, its viewing
angle depends on the position of the device (so it is not fixed and
hard to measure) and its bandwidth is (mostly) restricted to the
visible light. It is therefore an exciting idea to connect special
cameras to Android devices.

In this post, I will present an integration of FLIR Lepton
Long-wavelength Infrared Camera to an Android application over
Bluetooth Low Energy connection. Long-wavelength IR (LWIR) cameras
are not new. Previously, however, they were priced in the thousands
of dollars range (if not higher). Lepton is still pricey (currently
about 300 USD) but its price is low enough so that mere mortals can
play with it. FLIR sells a smartphone integration product (called
FLIR One) but it is currently only available for iPhone and locks
the camera to one device. Our prototype allows any device with BLE
connection to access this very special camera.

The prototype system presented here needs a relatively long list of
external hardware components and it is also not trivial to prepare
these components. This list is the following:

An Android phone with Bluetooth 4.0 capability. I used Nexus 5
for these experiments.

An FLIR Lepton module. My recommendation is the FLIR Dev Kit
from Sparkfun that has the camera module mounted on a
breakout panel that is much easier to handle than the original
FLIR socket.

Fortunately the BBB's SPI interface is completely compatible with
the Lepton's so the "hardware" just needs a couple of wires. Do
the following connections (P9 refers to the BBB's P9 extension
port).

FLIR

BBB

CS

P9/28 (SPI1_CS0)

MOSI

P9/30 (SPI1_D1)

MISO

P9/29 (SPI1_D0)

CLK

P9/31 (SPI1_SCLK)

GND

P9/1 (GND)

VIN

P9/4 (DC, 3.3V)

2. Prepare the BBB environment

I use Snappy Ubuntu. Grab the SD card and download the image as documented here.
Before flashing the SD card, we have to update the device tree in the
image so that the SPI port is correctly enabled. Unpack bt_ircamera.zip
that you have just downloaded and go to the dt subdirectory. There you
find a device tree file that I used for this project. Beside the SPI1
port, it also enables some serial ports. These are not necessary for
this project but may come handy.

Compile the device tree:

dtc -O dtb -o am335x-boneblack.dtb am335x-boneblack.dts

The output is the binary device tree (am335x-boneblack.dtb) that needs
to be put into the kernel image file. Let's suppose that the downloaded
image file is ubuntu-15.04-snappy-armhf-bbb.img and you have an empty directory at /mnt/img.
Then do the following:

fdisk -l ubuntu-15.04-snappy-armhf-bbb.img

Look for the first partition and note the partition image name and the offset:

Now you have an SD card that you can insert into the BBB and boot from
it. Once you reached the Ubuntu prompt and logged in (ubuntu/ubuntu),
there's one thing more: the Snappy prototype application depends on the
libpng package which is not part of the default Snappy image. But before
you do it, check whether the SPI device was enabled correctly:
root@localhost:~# ls /dev/spidev1.0
/dev/spidev1.0

Now about the png library. Download the armhf image from this location:

The BLED112 stores the GATT tree in its firmware, hence in order to
provide the GATT services that connect the BBB with the Android device, a
new firmware needs to be generated and installed in the dongle. The
config files are located in the config subdirectory in the
bt_ircamera.zip archive. Follow the steps in this post to generate and install the new firmware. Once you are done, you can simply plug the dongle into the USB port of the BBB.4. Install the prototype applications

The prototype system has two parts. The application running on the BBB
acts as BLE server, fetches images from the FLIR camera and transmits
them over BLE. The Android application acts as BLE client, fetches
images from the BLE server and displays them. The BBB part is located in
bt_ircamera.zip and the Android part is in IRCamera.zip. The latter is
just the source part of the Android Studio project tree - I omitted all
the garbage that Android Studio generates into the project folders. For
the BBB installation, follow the instructions in this blog post. Launch the BBB application like this as root:
/apps/ircamera.sideload/1.0.0/bin/ircamera /dev/ttyACM0
and you are ready to go. On the Android side, select the BLE node with
the name "test", connect, click the "Take picture" button, wait for the
image to download and there you are. Note that the images are saved on
the SD card, which means that they also appear in the stock "Photos"
application.

Now at last we can get to the technical issues with this prototype. One
interesting aspect is that there is no standard BLE service that
provides the functionalities - image capture triggering, image fetching -
our system needs. That's not a problem, we defined our own BLE service.
It is easiest to follow this service in irc_gattBLED112.xml
(bt_ircamera.zip, config subdirectory).

The interaction goes like the following. The BLE client connects and
reads the irc_len characteristic. This characteristic is tagged as
"user" on the BLE server side meaning that the BLE application must
generate the value on the fly, when the attribute is read. In our case,
reading this attribute fetches an image from the FLIR camera, converts
it into PNG format and stores it in the apps' data folder, returning
only the PNG file size. The Android application now can fetch the image
piece by piece. First the Android application writes the irc_offs
characteristic to inform the BLE server, what is the starting location
of the fragment it wants to fetch. Then it reads the irc_pic
characteristic which returns a maximum of 20 bytes of image data. This
makes the image download very slow (takes about 10-20 second to download
a general 5-6 Kbyte image to the Android application) but the
restriction comes from a BLE protocol layer. Maybe the old RFCOMM from
Bluetooth Classic would have been actually a better option for this
application.

Update: I updated the client/server application to make the download faster (it is still quite slow). In order to speed up, I removed the explicit setting of the file offset (so the irc_offs characteristic is not used anymore). This made the download faster but there's still room for improvement.

Other than the issue with fragment size, both applications are pretty
straighforward. Maybe the colors of the IR image are worth discussing a
bit. The FLIR camera returns a matrix of 80x60 pixels, each pixel has a
depth of 12 bit. Grayscale presentation is the most evident option but
most displays have only 256 gray colors. In order to make the IR shades
more visible, I used fake coloring. The algorithm is very simple: after
the image is fetched, the maximum and the minimum IR intensity is
calculated and the range between the two are mapped into a rainbow
gradient of 400 colors.

About the blog

This blog is a personal diary about my adventures with the Google Android platform. I write it in the hope that others may find my experiences useful but please, beware. The blog is created as I gain experience about the platform myself so errors, omissions, etc. may be found in the entries.