The project involves an experiment in implementing a human-computer interface
by tracking finger and wrist motions.

Brief:

"Ever wish you could control a computer just by moving your fingers or your
hand?"

We did.

The primary goal is to design and build a functional prototype implementing an
intuitive interface for a user to interact with a computer (or computing
device). The MCU provides a control module for direct access to the sensors and
currently acts as the computer role of the human-computer interface. To
demonstrate the interface, some sample demo applications include:

A calculator

A monitor

A stopwatch

A rock-paper-scissors game

This project required us to create analog and amplifier circuits in addition to
the software modules. Our goals were to create a streamlined device with a low
profile, low weight, low power consumption, and comfortable to wear.

The project idea initially involved a robot to output a raster on sidewalk.
Then it involved a portable multiplayer gaming device. Then it involved a
virtual keyboard / abacus. Finally, it evolved into the current project.

One of the ECE 476
assigned readings,
"Synthetic Serendipity", features a device which uses gestures and body
motions to interact with a wearable computer - We immediately realized that it
would be very possible create such a Human Controllable Interface (HCI) with
PVDT film.

We focused on the interface aspect because computer interfaces have not changed
drastically in over 30 years (since the mouse and its cousins touchpad,
trackball, and thumbstick). A major obstacle to ubiquitous and portable
computing is the intrusiveness of computer interfaces. Previously, the size and
awkwardness of computers presented the major obstacles to portability. However,
with today's technology, it seems that today's primary obstacle is the lack of
an intuitive and convenient interface.

The orthodox interface for current computation is the keyboard and mouse,
devices that do not lend themselves very well to portability. A significant
amount of research has been made into alternatives such as voice interfaces,
eye focus tracking, and one-handed keypads. We intended to experiment with a
system that uses hand gestures and movements.

There are two main goals to this project.

To create a simple input mechanism

To create example applications that use the input scheme

Since the primary goal is to create an input device, the Gauntlets of MicroComputation
(Gauntlets of uComputation), that
can be used for any application. In addition, the software infrastructure
should allow almost any external application to be integrated into the system.
This means that this project might be able to control or run other projects!

Logical Structure

The project high level design involves several modules:

The sensor module is responsible for reading the fingers and hand. The signals
feed into the signal processing module which cleans up the signal and amplifies
them for the MCU in the control module. Finally, the output pipes to the LCD
inside the display module.

Most people will not be moving a finger more than three times per second, so
the filter does not restrict operation.

The gain of a non-inverting op-amp is equal to:
A = 1 + R1/R2 = 7.1

This gain is necessary to amplify the signal from the piezoresistor for the
ADC.

Tradeoffs

On the hardware, we were unable to obtain a 3-axis accelerometer (which would
allow up-down motion detection) from Kionix. However, since the gesture sensing
is the primary focus, a 2-axis
accelerometer
sampled from Analog Devices provides adequate functionality.

There is a size and weight tradeoff that is made in order to incorporate the
4x20 LCD. If a smaller (4x16 or 2x16) LCD was used, there would be some space
and weight savings. In addition, the backlight is not disconnected, and
contrast level is locked by hardware.

Another issue is using lycra gloves for the foundation of the gesture sensing.
While the double lycra glove allows more reliable and accurate sensing, they
may make wearing the device less than completely transparent.

While accurate sensing is always a goal, reliability and device issues force
the sensing to and processing to rely on reading finger position as a binary
state rather than a continuum of positions.

On the software side, the limited memory on the MCU prevents us from
implementing any long and complicated programs. This requires a systems
programming mindset focusing on specialized code and design simplicity.

Standards, Trademarks, Copyrights, and Patents

The device software is ANSI C compliant as much as the Codevision compiler
allows and does not infringe on any trademarks or copyright...

There are patents for gesture input systems such as the
electric field sensing but none of them use piezoelectrics for sensing
motions.

The device may seem similar to the
Nintendo Power Glove but there are a few differences. First, the Power
Glove uses optics and light intensity to gauge finger bend while this device
uses piezoresistors. In addition, the Power Glove does not sense hand motions.

The heart of this project is the sensor module. There are two main sensing
mechanisms inside the sensor module.

Piezo film strips for the fingers

Accelerometers for the hand

Finger sensing

Piezoelectric materials deform when subjected to an electric field and produce an
electric charge when mechanically deformed. The equivalent electrical circuit
is an AC voltage source in series with a capacitor. Thus, these materials can
be used as mechanical sensors by analyzing the voltage change across the film
to deduce the deformation of the film. The voltage spike is in response to
a changing mechanical stress and not to the absolute stress, essentially
acting like a dynamic strain meter. However, by adding in a capacitor, the
charge can be stored for a longer time and make it so that the charge generated
is proportional to the absolute stress. These properties are the basis of the
finger-sensing scheme.

Piezo films are laid across the fingers and taped down to ensure a proper
response from the piezo. However, the current generated by the piezo is very
small. If the piezo is just connected to a voltmeter, the voltage spike is on
the order of 3 - 5 V. When the piezo is connected to other resistive elements
such as the rest of our circuitry, the voltage spike drops down to 0.3 - 0.5 V
so signal is amplified in the signal processing module.

Hand sensing

A 2-axis accelerometer from Analog Devices is mounted on top of the hand to
tracks left-right and forward-back hand movement. The accelerometer is taped
down to prevent shifting. Unfortunately, we were unable to obtain a 3-axis
accelerometer to allow for up-and-down movement.

The Analog Devices accelerometer, ADXL203CE, has a range of +-1.7 g with a
sensitivity of 1 V/g. This allows the accelerometer to be used directly without
amplification. In addition, the accelerometer has a resting state voltage of
about 2.3 V.

This stage amplifies signals from the piezo strips. The schematic for amplifier
is displayed below.

This amplifier has a gain of approximately 7 and is enough to ensure that the
signal ranges from 0 to +5 V. A low pass filter with a frequency response of 3
Hz filters out noise. The frequency response is fast enough since it is
unlikely that someone will move a finger more than three times a second. Diodes
are placed at the input to the ADC to protect the MCU.

The third non-MCU hardware component is the Optrex DMC20481 4x20 character LCD.
The LCD is similar to the LCD that used in class (uses the Hitachi 44780
interface), but there are two main differences. First, the LCD has a backlight
(which is not used). Second, the LCD needs a -5 to 0 V input to the contrast
pin with -5 V being all black (full contrast) and 0 V being no contrast.

We use a TC7660 charge pump voltage converter to convert the +5 V to -5 V. The
schematic is shown below.

Software development for the project focuses on building a very basic (single-
threaded) operating system to provide a foundation for small applications. It
presents the applications with a small API for interfacing with the gesture
sensors and performs some necessary device functions. It also provides a simple
GUI for displaying a menu for launching the applications.

At the highest level, the operating system is merely a shell program which
calls the application driver functions and also includes some commonly used
functions and variables. While this system can be expanded to provide more
functionality and complexity (e.g. multi-threading, complete hardware
abstraction layer, synchronization, etc...), the project time constraints limit
us to the most basic implementation.

Developers can easily write new applications for the system by following the
template given in the sandbox.c code. Applications can be added to the system
GUI by adding a few lines of code in config.h and os.c. From the operating
system GUI, the user can select to run the application - the operating system
then calls the provided application driver function which transfers the thread
of execution to the application. From here on, the application has full control
of the device (with some caveats). To reduce code redundancy, the operating
system provides some interrupt code and functions for common functionality. A
millisecond-precise timer is provided along with USART interrupt access. A
read_input() function allows applications to interface with the gesture
sensors, and a wait_ms() function provides a method for delaying the
application execution. Global variables include an LCD buffer and variables
indicating input (from the sensors).

Despite the design simplicity, the operating system still presented some issues
- while applications receive full device control, there should be some way for
the user to quickly return to the main GUI or restart the device. In addition,
the operating system may need to regularly execute some maintenance operations.
The solution for this problem is to assume the developer is cooperative and
require occasional calls to a system function exec_system(). This system
function provides a method for the operating system to execute the necessary
operations without having device control. To provide quick restart and exit,
the system function periodically reads user input and resets the watchdog timer
only if a specified input is not detected. If that input is detected, the
watchdog is not reset, and the device eventually "reboots."

Another issue arose from deducing the workings of the Codevision IDE - to
provide modularity, the code expects developer applications to be separate *.c
and *.h files. However, the Codevision IDE is very finicky about the project
settings necessary to allow multiple files (for this particular project, os.c
MUST be added to the project first). In addition, other issues like lack of
pointers or a heap or even the dereferencing operator made coding a bit more
challenging.

However, the main challenge is simplifying application integration. Currently,
adding a new application requires adding the application *.c and *.h to the
project then adding the necessary lines of code to the GUI code. In addition,
the application needs to follow a few rules - it needs to call system_exec()
occasionally. Outside of these operations, the developer and application has
full freedom. We consider this an adequate level of compromise (our conclusion
from writing the sample demo applications).

The device is able to read inputs from the fingers and the hand. We are able to
cycle through the menu options, run different applications, and reset the system
without much problem. The device does not feel excessively bulky or
uncomfortable to wear for short periods of time. The software infrastructure
functions correctly and allows us to add new applications to the system.
Overall, we are satisfied with the project. However, as always, it would have
been nice to integrate greater functionality and streamline the design.

Speed

The MCU runs at 16 Mhz. We experienced no problems with slowdown or artifacts
while running any of the applications. In general, there are no time-critical
programs except for input polling currently in this device so speed is not a
crucial factor. Note, this device should probably not be used for hard real
time systems, at least not in its current form. The code would need to be tweaked
for any real-time application.

The polling speed is easily changed to accommodate people of different reflexes
by simply modifying a #define in the OS. There is an absolute max polling speed
of three inputs per second, but it is unlikely any human will be able to
sustain such a rate with any amount of accuracy (unless they are using
performance enhancers).

Accuracy

Finger and hand sensitivity can be easily changed for each person by modifying
the thresholds for each input inside the OS. We have found that our default
thresholds have been adequate for most people. Most false positive inputs were
due to a twitchy finger or hand. Excessive false negatives is fixed by modifying
the sensitivity. These settings must be calibrated.

Safety

Safety is a huge concern for us since we are mounting an electronic device on
top of the arm. To ensure that the device is safe to wear, we have done the
following things:

Mounted the device on some insulating material such as the glove armguard so
that no electronic component is in direct contact with skin.

Clustered most of the wires in bundles to avoid a "wire maze" and reduce the
probability of a wire being caught in something.

Reduced the profile of the device by mounting them as close to the
armguard or glove as possible. This is to also reduce the probability of a
component being caught in something

After our packaging efforts, we feel that the device is about as dangerous as
other portable electronic devices like an iPod. With no cover. Unsupervised
children should still probably not play with this.

Interdevice operability

Our device does not interfere with the operation of other devices (no RF,
minimal CPU noise). There might be some slight danger to people with electronic
pacemakers or other highly sensitive life-saving equipment.

If anything, our device can be integrated into other Atmel MCU based projects
as an input device. Alternatively, other programs can be integrated into the
device.

Usability

The device is usable by most people. Currently, the device is right-handed
only, but it can become left-handed by flipping the orientation of all the
different boards around and finding a left-handed glove. The large amount of
#defines in the operating system allow for customization to fit most people's
needs. Furthermore, we have tested the device with multiple people and the
defaults were adequate for them, but the option exists for fine-tuning.

Notes

A RS232 interface was originally planned, but we simply ran out of time to
fully integrate it into the OS. The hardware capability is already on the chip,
but the software to integrate it to the OS is not. However, the code should be
trivial for a motivated developer to produce.

We feel that the device was a success. We were able to create a usable interface
and a module system for easy integration of applications. The glove functions
as expected and has a wide range of customizable options. It is unfortunate
that we were unable to fully integrate the RS232 connection, but that was a
minor feature. It would have been nice if we would have been able to
fine/create packaging for the circuit boards with a cover for the LCD, but that
is also beyond the scope of our original intentions.

Our design does not use any standards nor does it reuse someone else's design
or code or intellectual property. We were fortunate to not have to sign any
non-disclosure agreements (NDA) to receive any of our sample parts, nor did we
ask for any parts under false pretenses.

There is a possible patent opportunity in this device. After we had conceived
the initial device and were researching similar devices, we could not find any
piezo-based gesture input devices. Seeing as how there were patents for
specific types of gesture devices and not the general category, and further
seeing no applicable patents, we can conclude that it might be possible to
patent this.

IEEE Ethics

2) There is no possibility of a conflict of interest. We do not work for nor
own stake in any of the companies mentioned. In addition, all of this work is
freely available.
5) By basing this device around piezoelectrics and accelerometers, we encourage
others to look into integrating these components into their projects, whether
they be related to gesture sensing or not.
7) We are open to any suggestions for improvement or bug fixes. In addition, we
are willing to accept contributions from anyone and will credit you.
8) We strived to make this device as customizable as possible so that almost
everyone will be able to use it.
9) We attempted to make this device as reasonably safe as possible. With the
exception of an overall packaging, we avoid contact with the skin by any
electrical component.

The finished project with the second glove covering
and more duct tape

Notes:

The calculator module compiles, but has not actually been tested on the MCU.

There is 1 pin available for PORT A, which should be linked to bit 5 in the input_data_bin variable.
Currently, bits 7-6, and 4-0 are used already in that variable, leaving bit 5 available.

PORT B and D are unused (USART would have used some
of the pins, but that is not currently implemented). One thing is that the headers for PORT D
are partially used up for VDD and GND. However, wires could be directly connected to the MCU pins.

Vertical motion on the wrist can be measured by putting piezos on top or bottom of the hand-wrist joint.
Unfortunately, we did not think of this earlier.

Lessons Learned

Accelerometers can withstand a lot of heat and still work. In addition, they
give really nice voltage responses (at least those from Analog Devices)

Piezos can be iffy if not placed correctly.

Most places give out samples, especially if you stress the fact that you are a
poor college student with a tight budget. Other places do not.

When things are not working (such as the fingers not registering), check if it
is not user error first.

(input_data_bin & 0x10 == 0x10) is not the same as ((input_data_bin &
0x10) == 0x10).

Wires have a tendency to break. Over and over again. At the worst times. At the
worst joints. Especially when they were working the day before.