Thursday, July 4, 2013

Many good tutorials can be found about the Parcelable protocol in
Android. If the Android framework has to serialize a set of values
(either for AIDL-based IPC or for Intent-based message passing), it can
serialize basic types or objects that implement the Parcelable protocol.
In short, Parcelable objects are able to serialize their states into a
Parcel object and they can retrieve their state from the serialized
form.
But what happens if the two classes (the one which is serializing and
the one which is retrieving its state from serialized form) are not the
same? If you remember my post about classloading in Android, there is no
way loading a class from another APK. If you pass a serialized object
to another application whose codebase comes from a different APK, the
target application has to rely on its own version of the class with the
same name. Android framework does not verify that the sender and
receiving classes are the same, they just have to have the same name
(including package name).

There are two apps in the package. Actually, it is one of my first
example programs, here is the description from a 2007 post of this blog.
There are two applications, intentsender and intentreceiver.
Intentsender has an internal intentreceiver plus it can talk to the
external intentreceiver app. Both applications have a version of
aexp.data.TwoNumbers class. In this example they are just slightly
different - they log different messages.

What happens when you click on the "exp / int" (explicit Intent with
internal intentreceiver) button. Both of these activities are residing
in the intentsender APK therefore the class which will be used to
serialize and deserialize the object are the same.

But if you click on the "exp / ext" (explicit Intent with external
intentreceiver) button, the message has to pass the APK boundary (it is
processed in the intentreceiver app). Intentreceiver has nothing else
but its own version of aexp.data.TwoNumbers. The log demonstrates it:

Android framework did not care that the two TwoNumbers classes were not
the same, it just called them blindly to serialize and deserialize. Actually, if you look into the Parcel implementation, you will see that the serialized version of the Parcelable only stores the fully qualified class name so it is not surprising that intentreceiver loaded whatever version of the class it could find under the given name. Fortunately, the two versions were compatible (they produced and
consumed the same serialized stream) so there were no problems.

Why is this useful other than misleading yourself with classes that look
the same but they aren't? If you want to expose just part of your
serialized object to the receiver application, this trick comes handy.
Otherwise it is best to avoid it.

Thursday, June 20, 2013

Once upon a time I made a Londroid presentation that briefly mentioned
the possibility of detecting taps on the phone's body with an
accelerometer (it is in slide 31). You may rightly ask, why do you want
to detect taps with the accelerometer when there is the touchscreen?
First, the touchscreen may be switched off by the screensaver or the
user may have deactivated it because he or she is in a call or put the
phone into a position where the touchscreen may be touched
unintentionally. Then the user may want to tap the back of the phone.
Lastly, the user may be interested in the strength of the tap, not only
its location or the fact that the tap happened.

Let's recall that graph in that old Londroid presentation.

The graph shows the absolute value of the acceleration vector while the
phone's body was tapped twice. The base rate is the gravity acceleration
(about 10 m/sec^2) and the acceleration caused by the taps was added
(in the actual configuration, subtracted) from the gravity acceleration,
resulting two characteristic spikes.

But where was the phone's body tapped? By looking solely on the
accelerometer's data, I don't think there is a way to find it out. But
fortunately we have other sensors and the gyroscope is particularly
relevant. Here is the well-known image of the sensor axes in Android.

In case of the gyroscope, the sensor measures rotation around the axes.
If you tap the device in the right lower corner, we can expect a short
rotation around the x and y axes if the phone supported by a reasonably
flexible base (e.g. a human hand and not a block of granite).

But can this be really detected? I made a series of measurements to
prove it. Note here that I was just interested in the theoretical
possibility and did not want to do a product-grade implementation. So I
tested only on my Galaxy Nexus.

First here is the absolute acceleration value for 3 consecutive taps.

As
expected, the time of the tap can be safely detected. These taps were
all in the lower right corner so let's see what the gyroscope measured
along the x and y axes.

X axis, just for two taps:

Y axis, just for two taps:

As you can see, the taps are detected by the gyroscope too as short
rotations. In case of both axes, there was a spike of positive rotation
velocity first.

Now let's see what happens, if we tap the lower left corner.

X axis, just for two taps:

Y axis, just for two taps:

As you see, the X graph has the same character but the form of the Y
graph has changed. Instead of a marked positive spike, there is a small
negative spike, followed by a positive spike.

Let's see if the experience is repeated in the upper left corner.

X axis, just one tap:

Y axis, just one tap:

Now the character of the spikes on both axes changed. A small positive spike is followed by a large negative spike.

This is not a kind of movement the gyroscope was designed for so it is
premature to state that this waveform pattern will be exactly the same
for all the Android devices sporting gyroscope out there. But I found
these measurements convincing enough to support the statement that it is
possible to implement rudimentary tap location using only the
accelerometer and the gyroscope in the device.

Sunday, March 10, 2013

In my previous post I wrote about my enthusiasm of integrating car
sensors with the mobile device's own sensors. The first step was to
implement a data capture application that saves sensor data into a file.
This time I decided to save the gyro, accelerometer and compass sensor
data from the phone and the speed as measured by the car's speedometer. The most interesting aspect here is how we obtain the
speed data.

There are two levels of interfaces playing role here. OBD2 interface and
the protocols accessible over it is the first level offered by the
car's on-board electronics. This interface is not trivial to talk to but
fortunately all the complexity of OBD2-related protocols is implemented
by the super-popular ELM327 microcontroller from ELM Electronics (or
its compatible copies). On one side the ELM327 chip talks to the car's
ECU using the OBD2 interface. On the other side, it offers a modem-like
serial interface. Popular connection option is USB but for our Android
client, Bluetooth is a better option. ELM327-compatible adapters are
available at a low cost, for example I bought the very basic version
below.

ELM327 has a very simple, yet powerful interface. In the first phase the
chip is set up by issuing a series of AT commands reminiscent of the
Hayes modem commands. The best place for reference is the ELM327 specification.
The most important command is the ATS0 command. This instructs the chip
to auto-detect the protocol used on the OBD2 interface. Once this
auto-detection is successful, the car sensors can be queried by sending the PID code of the sensor in question. For example for the car's speed one sends:

010D

The ELM327 chip then talks to the car's ECU, obtains the sensor data and returns a response like this:

410D0C

Where "4" is the ID for response, 10D is the PID of the sensor and 0C is
the value. Some OBD2 PIDs have complex coding but fortunately for the
speed sensor 0C simply means 12 km/h.

Before you start using this test application, make the Bluetooth pairing
of your device with the sensor from the Android Bluetooth settings.
Either you can use the real thing (ELM327 interface connected to a real
car) or you can use Ryan C. Gordon's great OBDSim application
that can run on a Linux box and can simulate a Bluetooth-connected
ELM327. In theory, OBDSim can also run on Windows, I tested only on
Linux. Setting it up is not trivial but it saves a great deal of time if you develop for the ELM327 interface.

Then you can start the test application. First connect to your ELM327
device from the menu ("Connect a device - secure"). If the connection
works properly, you should see a series of AT commands and responses
flashing on the screen then the speed appears (e.g. 0 km/h if you do it
on a real car). This time you can start the sampling ("Sampling" check
box) and the application will save all the sensor information into a
file in the SDCARD in CSV format. The name of the file is
gyrocarcapture_<date>.csv and its format is very simple. One
warning here: sadly the timestamp format used for the speed sensor and
the device's own sensor are different: the sensor timestamps are coming from the SensorEvent object and the time stamp for the speed data comes from System.currentTimeMillis(). As device sensor events are very
frequent, I use the last device sensor timestamp for the speed sensor
too when processing these files. The speed sensor sampling frequency is
approximately 1 Hz.

The application can be extended easily to sample other car sensors too.
However, in its present form it is good enough to detect if the driver
slows down in front of a speed bump (accelerometer-car speed sensor
fusion).

Friday, March 1, 2013

Last year I had a presentation at Droidcon Tunis. The conference was inspiring with a clever and enthusiastic audience and the presentation became a success with over 16000 views since the end of the conference. Not bad for a sensor processing topic with some scary math in it. Plus my name became a landmark in Tunis which was pretty much a new sensation for me.

Call for papers for Droidcon Tunis 2013 is now open but sadly, I can't submit anything. My presentation material is not ready because some topics I could speak about are not public, others are not mature enough. But you out there, you should consider submitting papers to this conference. Last year's conference was fun - also my first time in Africa. I am sure that this year will be just better.

Friday, February 22, 2013

My conclusion from the speed bump post was that the built-in accelerometer in the Android phone is probably not enough to detect the obstacle. Alexander Kosenkov commented that maybe the accelerometer vector can be processed in a more sophisticated way but my experience is that the car generates such an amount of vibrations that makes extracting relatively faint signals from the accelerometer rather hard if not impossible.

While I was contemplating on this issue, I got a mail from Edward Pultar, president of Valarm, a start-up based in Los Angeles. These guys created an Android application and associated web services that started as a theft-prevention tool and is being repositioned as a general platform for all sorts of sensor data acquisition and processing. Read their press kit about the exact features of their products.

The most fascinating idea in their product is the use of OBD2 interface to connect to the car's own sensors. OBD2 with a Bluetooth plug is widely used by car fanatics to obtain diagnostic data including speedometer, tachometer, engine temperature, on-board GPS (if the car is equipped with it) and many other sensors. Valarm's idea was to use this wealth of data for use cases other than car diagnostics, e.g. monitoring whether the car is driven recklessly. They do sensor fusion with the car sensors and with the phone's own sensors. For example cars don't normally have accelerometers that can detect excessive vehicle vibration caused by bumpy roads or faulty tires but a smartphone has. The phone is also equipped with a wealth of communication options, large memory and is an application platform. It is therefore an ideal device to aggregate data from different sensors (internal and external) and to send the data to web services.

This may sound like a futuristic research paper but to my utter surprise, even my own car which is well in its teenager years has the OBD2 interface. All I need is a commercially available Bluetooth plug (connection by cable is also possible but is less practical) and the car will talk to my Android phone. This makes Alexander's idea about incorporating the speed profile for the speed bump detection pretty easy to implement.

Tuesday, February 19, 2013

Somebody came in from the internet some months ago and asked whether speed bumps (also called sleeping policemen) can be detected with an Android phone. You know, speed bumps like this:

I was cautious because there's a lot of traps here. Obviously the detected signal will differ depending on the form factor of the speed bump, the type of the car and even the way the phone is fixed to the car (or the lack of fixing). But I was also curious and as car-related phone applications are very fashionable today, I took the challenge. Meanwhile the guy who brought up the problem disappeared so I feel free to share what I achieved during the last one and half months.

The solution that I produced is far from perfect. First and foremost, I did not make any effort to target other phones than my trusty Nexus S. Phones which have sampling frequency of around 50 Hz in SENSOR_DELAY_FASTEST mode will work fine, significantly different sampling frequencies are not supported. It is more important, however, that I was not able to create a foolproof algorithm for the problem.

Initially I thought that some sort of rotation could be measured when the front of the car is lifted by the speed bump but I was not able to detect this rotation from the sensor signal. The car generates all sorts of vibrations and it is not easy to extract this relatively small rotation from the noisy background. So I went back to the acceleration spike which is generated when the car hits the bump and I used just the acceleration's absolute value as described in this presentation. The trouble is that this spike is pretty reminiscent of a class of other spikes like knocks, etc. For example here is a beautiful double street bump somewhere in Budapest.

One would think that signals like these could be easily detected. Now comes, however, a bumpy road with potholes, also typical in Budapest.

There are no street bumps in the second signal but the potholes generate "knocks" that are hard to distinguish from the "knocks" caused by street bumps.

There is no perfect solution to this problem, at least I could not find any. Eventually I obtained the best result by calculating the cross-correlation between the acceleration signal and a pre-recorded speed bump signal. I re-sampled this representative signal for +/- 10% of its initial frequency and calculated the cross-correlation with all the three variants (initial, +10% frequency, -10% frequency). Then I used a sliding window to calculate the power of the cross-correlation signal. This is the pre-recorded sample I used for cross-correlation:

Cross-correlating then calculating the power in a sliding window yields this signal for the double speed bump (different colors show the power of the 3 variants of the re-sampled signal):

And this is the result of the same calculation for the bumpy road:

Using a limit of about 150, the two signals can be separated and no fake bumps are detected even on the bumpy road.

There are a lot of problems with this approach. First, speed bumps come in all shapes and sizes and this method can detect speed bumps that are similar to the one used to generate the reference signal. Then there is the problem of the car itself. I did not have the chance to test with a lot of different cars but I bet the different suspension systems will yield drastically different signals in different car types. Anyway, I thought somebody may find this post and this post can start some thinking.

About the blog

This blog is a personal diary about my adventures with the Google Android platform. I write it in the hope that others may find my experiences useful but please, beware. The blog is created as I gain experience about the platform myself so errors, omissions, etc. may be found in the entries.