Programming for EyeTap Systems

An ``EyeTap'' is a camera/processor/display device you can wear. Here, two real-life cyborgs play by reversing X displays.

Over the past few years, we have been
working on a number of programming environments for EyeTap systems.
A common feature of EyeTap systems is the diverter, which renders
image content as left-right reversed or top-bottom reversed.
Accordingly, we have been developing a number of options and
methodologies for dealing with the problem of reversed display.

Although early cathode-ray-tube EyeTap systems relied on
swapping deflection yoke wires (in color EyeTaps one must be
careful to swap only the horizontal yoke wiring, otherwise images
are chopped off because of the critical timing of the filter wheel
as synchronized with the vertical deflection), certain conditions
arise where it is inconvenient to reverse the hardware.

Moreover, because the sensor apparatus (whether it be a
camera or similar sensory system) also sees the world in reverse,
it is often desired that the display actually portray a backward
image to compensate properly, and thus the two will reverse each
other and cancel out. Without reversing X, images would appear to
the viewer as shown in Figure 1.

Figure 1. An Image without Reversing X

One early attempt was a program called caplive, developed by
Mann and extended by Mann, Fung and several others into something
called xcaplive, along with a set of utilities called glinaccess,
meant for use with EyeTap systems.

More recently, a thesis project proposed by Mann and carried
out by Manders involved the reversal of XFree86 to facilitate use
of XFree86 with an EyeTap system.

With the restructuring of the XFree86 code of late, XFree86
4.0.x has provided an easy means of left-right reversing the
Xserver. Specifically, new installations of XFree86 version 4.0.x
have an option to make use of a shadow buffer. In a regular install
of XFree86 4.0, most chipsets will offer the option of rotating the
Xserver by pi/4 radians. This is due to the efforts of Mark
Vojkovich.

The initial code provided in the XFree86 4.0 distribution
performs a harder task than what is needed to left-right reverse
the Xserver. Any one of the drivers provided in XFree86 4.0 will
have essentially the same structure that will be discussed here
later on, but for the purpose of example, the NVIDIA driver (nv)
has been chosen. Note that this manipulation has been performed
successfully on almost all of the XFree86 4.0 drivers.

First, it is advantageous to examine the code to which the
necessary modifications will occur. After expanding the XFree86 4.0
source code (available at
ftp://ftp.xfree86.org/),
we get the file
.../xc/programs/Xserver/hw/xfree86/drivers/nv/nv_shadow.c. Listing
1 comes from this file.

Now, this code is active when the Shadow Buffer option is
selected in the XFree86Config file. All it is doing is writing from
allocated memory to the graphics card. Understanding this code is
not that difficult if we consider what some of the variables
are.

The sections of the hardware memory-mapped regions (pointed
to by pNv->FbStart) that need to be redrawn
because of changes are outlined by pboxes. A pbox is simply a set
of four offsets, two x offsets and two y offsets, and num is the
number of pboxes that need to be redrawn. The width and height of
the pboxes may be calculated from these offsets.

The code in Listing 1 redraws on pbox at some time upon each
iteration of the outer while loop, and src and dst are the memory
addresses of the source and destination addresses in memory. The
real changes that need to be made are that we must write the rgb
information to the destination maintaining the order of the rgb
bits, but one level of abstraction higher. These bit groups must be
written in reverse order. This means we must write the pixels out
in the destination starting from the pbox x coordinate opposite to
the code in Listing 1. This may be accomplished by changing the
following:

This will get us to the other end of the pbox in the representation
in memory.

At this point, the code is beginning to read the source
address from the correct location. For each horizontal line, the
actual groups of rgb bits must be written out in reverse. This
implies that the inner while must change as well.

Unfortunately, we can't write the bits out one ``width'' at a
time (which is what the memcpy is doing in the original code). We
must traverse the sections in finer increments, which leads us to
change

The previous code takes care of the painting of the pixels. Next we
need to correct the positioning of the mouse (which is not yet
reversed). A subrountine at the start of nv_shadow.c, named
NVPointerMoved, allows us to do this. If we look at the
NVPointerMoved subroutine, the code is used to change x coordinates
to y coordinates and similarly to change y to x. This is, of
course, not needed to reverse the Xserver, but the code may be
modified to perform the function needed. Listing 2 has been
commented to outline the necessary modifications.

What has been done in the new section of code is to take the
original y coordinate and leave it unchanged and subtract the
original x coordinate from width of the screen. This is all that is
needed for left-right reversing the mouse. The variable
pNv->Flip is important and will be explained in what
follows.

The changes so far will left-right reverse the Xserver; now
all that is needed is to get the driver to call it. Certainly the
easiest way is to add an option to the XFree86Config file such that
when called, the Xserver would be left-right reversed.

We need to get our new option (which we will call FlipLR)
parsed by the driver. Specifically, when we add option ``FlipLR''
to the device section of the XFree86Config file we want to have our
Xserver reversed.

These are the usual options available to the driver. They
will differ slightly from driver to driver, but some similar
structure exists in each of the drivers. What is needed in the
structure is a line that should appear as:

The subroutine NVPreInit is used to parse the XFree86Config
file and set various structure parameters to the correct values.
The lines that need to be added can occur within a wide range of
the subroutine. In keeping with the overall design, with the NVIDIA
driver, it makes sense to add some code at line 1061 (right after
the Rotate option is parsed):

This will set the required variables in the pNv structure to the
values necessary for left-right reversing the Xserver. At this
point, the field pNv->Flip doesn't exist. This needs to be added
to the header file and will be dealt with later.

If the variable pNv->Flip did in fact exist, we would have
parsed the XFree86Config file properly. Now what is necessary is to
have the function NVRefreshAreaFlip used when needed. This is done
in NVScreenInit. Within NVScreenInit, there is a section dealing
with the shadow buffer. Using a reasonable editor to make the
changes, you can search for the string ``if(pNv->ShadowFB)'' in
the NVIDIA driver; this occurs around line 1,779 of nv_driver.c.
The original piece of code is shown in Listing 6.

All that is left to do is add the appropriate entries to the
header file. The name of this file varies depending on the driver.
Of course it is always a .h file, and that narrows the field
greatly. All drivers will have a similar header, but for the NVIDIA
driver, the file is nv_type.h. If the modifications are on a
different driver, it's probably best to look at the NVIDIA .h file
and grep for some similar entries in the directory of the driver
that is being modified.

The pNv->Flip field must be defined. There is a struct
that will be similar to Listing 8. To this listing we simply want
to add the entry

Dr. Steve Mann
is regarded by many as the inventor of the wearable computer, of
the EyeTap camera (a device that causes the eye itself to function
as if it were a camera) and of the Reality Mediator (wearable image
processor for the visually challenged,
http://wearcam.org/mr.htm).
He is currently a faculty member at the University of Toronto,
Department of Electrical and Computer Engineering.
Corey Manders currently holds
degrees in Computer Science (University of Toronto) and Fine Arts
(York University) and will begin a Master's Degree in Applied
Science at the University of Toronto starting in September 2001
under the supervision of Professor Steve Mann. For the last year he
has been a resident cyborg in Mann's Humanistic Intelligence
Lab.