From Hand to Mouse:
Comments on Elizabeth Keating & Gene Mirus
“American Sign Language in Virtual Space: Interactions Between Deaf Users of
Computer-Mediated Video Communication and the Impact of Technology on Language
Practices”

Justine Cassell, MIT Media Lab

In this fascinating and evocative essay, Keating and Mirus look at the effects
on American Sign Language of video-mediated communication technology. Hearing
populations around the world today take for granted that they can communicate
in their own language with those at a distance. The telephone is pretty much
available everywhere, and operators are no longer necessary to place a call,
so simply picking up the phone and dialing a number puts a speaker in contact
with the chosen interlocutor. Until the last couple of years, the same was
not true for the Deaf. Not until the TTY, introduced widely only in the 1970s,
was communication possible among non-co-located Deaf, and even then only through
the use of a foreign language. In the case of the TTY it is fairly clear that
technology is mediating communication in a significant way. Because there is
no written form for signed languages, the TTY is requiring communication not
only to be printed, but to be translated into English. Of course, independent
of technology, the Deaf community is already more than familiar with mediated
communication. Since the vast majority of American Deaf live and work in hearing
contexts, much day-to-day communication takes place through the mediation of
interpreters.
However, when video is brought into the equation, and the Deaf can communicate
with one another directly, over long distances, are we witnessing a return
to face-to-face communication, a different kind of mediation, or transformation
and creation of a new communicative reality ? This question can really only
be asked now, when videoconferencing has become affordable due to software
such as Microsoft’s NetMeeting. The authors bring up all of these options when
they describe their interests as “the symbolic properties of language as it
is used together with new potentials for communication afforded by the computer,
particularly computer mediated images of signed language (visual language)
as well as the recreation of face-to-face communication in virtual space or
the invention of face-to-computer communication.” The authors’ data are drawn
from a two-year study of four deaf families, and it is clear that fascinating
adaptations to the technology are being made by the participants. However,
in order to untangle what is going on – mediation, transformation, or a return
to “real” face-to-face communication -- I think we need to compare ASL VMC
to other modes of mediated communication: (a) spoken language VMC, or (b) typed
English chat by Deaf interactants, or (c) communication between a signer and
an English speaker by way of an interpreter. Otherwise, we won’t be able to
tell which of the findings relate to the unique properties of signed languages,
which to mediated communication, and which to the role that technology plays
in communication.

First, a clarification for those readers who may not be familiar with web
cam technology. For the most part, the quality of the web cam image is limited
to a resolution of 640x480 pixels, which affects the size of the window in
which the interlocutor is visible, and the crispness of the image. The video
frame rate is under 30 frames per second under the best of conditions, which
means that signers need to slow down their signing in order not to appear jerky. Finally,
the speed of the computer processor and the speed of the Internet connection
also affect how much lag time exists between when one participant signs and
the other receives the signs, which affects whether back channel information
can be read as such. All of this means that, much like any other kind of computer-mediated
communication, only some aspects of face-to-face conversation are reproduced
in CMC. From the point of view of the designer of technology, this leads to
the question of what notion of face-to-face communication has been mapped to
the technological context – what aspects of F2F communication were maintained
at the expense of others? From the point of view of the user, we ask what kinds
of disruptions the technology provokes, and how users overcome them.

A significant literature on video-mediated communication (Aiello, 1972; Sellen,
1995; Bracco, 1996; Finn and Sellen, 1997) has shown in particular that processes
of grounding and turn-taking tend to be disrupted. For example, comparing VMC
to F2F conversation, Cohen (cited in (O'Conaill and Whittaker, 1997)) found
more speaker turns in F2F, and more interruptions. Likewise, (O' Conaill, Whittaker
et al., 1993) comparing LiveNet, ISDN (longer lag times than LN), and F2F,
found that participants in the lowest fidelity system produced fewer backchannels
and interrupted less often. They engaged in longer turns, with fewer turn exchanges,
and floor management being carried out in an explicit manner. O’Conaill, Whittaker
and Wilbur attribute these results, in part, to the fact that VMC systems do
not allow mutual gaze – having it be clear that one’s gaze has met that of
the other participant. Of course, eye gaze is also an essential cue in American
Sign Language and photos of the VMC set-up (e.g. Figure 4) show the extent
to which it’s difficult to gauge eye cues on VMC. But (Whittaker (to appear))
also points out how disruptive the lag between sound and audio on VMC is for
spoken languages. It is possible that VMC for signed languages will not suffer
from this problem. In fact, an early theory of technology and communication
posited that, the closer the set of modes supported by a technology approximate
to those of face to face communication, the greater the efficiency of the communication
using that technology. Thus, they claim, a technology that supports both visual
and linguistic modes should always outperform one supporting only the linguistic
mode. ASL is a neat test-case, since the visual mode is the linguistic mode.

Literature on textual CMC is also relevant here. Much as Keating and Mirus
describe in the case of Ben and Ned negotiating the technological context of
their interaction, text-based computer-mediated communication also demonstrates
a fair amount of metalinguistic stage-setting about the nature of the medium,
and consequently necessary adaptations (Cherny, 1995; Herring in press). The
authors’ lovely example of an interactant picking up the web cam to disambiguate
a word is not unique to ASL participants using VMC. As (Streeck, 1996) reminds
us, physical objects and their representations using iconic and deictic gestures
(including, in this instance, with a web cam) are essential to meaning-making
– to symbolicization -- in conversation.

Modifications to words for the purpose of CMC is a topic that has been studied
by (Cherny, 1995) and others – there exists a whole lexicon of forms shortened
to deal with the comparatively slow speed of typing over speech, and also to
mark habitu}s. Linguistic variation on CMC can have more permanent effects,
as described by (Paolillo, 1999) in his study of channel #India on the EFNet
IRC (see also (Paolillo, 1996)).

One topic, however, seems uniquely relevant to American Sign Language, and
as such uniquely revealing of the effects of VMC on any language. The use of
fingerspelling between friends, who know one another to prefer ASL, and the
use of signed-English-like constructions is a puzzle that doesn’t seem to be
adequately explained by strategies of adjustment to properties of the new communication
technology. However, Keating and Mirus’ previous examples of explicit negotiation
about, and lack of certainty about properties of the medium make it clear that
participants are not entirely comfortable with their own model of the interlocutor’s
reception. In this sense, use of formal forms such as fingerspelling could
indicate difficulty in constructing such a model of the interlocutor’s understanding. Disambiguating
among the different possible explanations might involve looking at pairs who
have used VMC over longer or shorter periods of time vs. pairs who are used
to communicating with one another over longer or shorter periods of time.

Other analyses of distinctly sign-like phenomena such as this one would be
welcome. For example, what happens to body-shift for frame of reference, empathy,
and narrative perspective in VMC (Rose, 1992)? The fixed orientation of the
web cam precludes the signer from shifting his/her body and eye gaze while
still ensuring proper camera framing. How do signers accommodate? These sorts
of more discourse-level phenomena are particularly interesting because of the
cinematic effects associated with them in face-to-face signing (Poulin and
Miller, 1995; Krentz (forthcoming)). How do signers negotiate cinematography
techniques when faced with cinematic inflexibility?

Of course, this is the danger with forays into new areas of research: your
readers want to know about all of the possible topics associated with the topic
that you have actually chosen to research. In that vein, I would add a few
more requests. Keating & Mirus’ manuscript leaves me wanting to know whether
VMC changes the nature of social capital (Bourdieu, 1986; Putnam, 1993) in
the Deaf community -- a topic very discussed among of sociologists of technology,
see for example (Pelto and Mxller-Wille, 1983)? Are more Deaf organizations
formed of non co-located members? Do families feel that bonds are stronger? In
sociolinguistic terms, do fewer ASL dialectal variations arise, as Deaf populations
converse across regional boundaries? Does ASL devolve into one standard form? Or,
perhaps more likely, do ASL dialects become income-based rather than regional,
as a “digital divide” allows only well-off families to use VMC? Finally, do
no users of VMC adapt the technology itself? In the current analysis, technology
appears to be assumed – a locus for change, but not changeable itself.

The fascinating data presented here raise more questions than they answer. And
the answers are relevant to practitioners hoping to serve the Deaf population
(Craft, 1996), as well as to linguistic anthropologists. Luckily, the authors
have clearly done a very careful job in their fieldwork: filming, transcribing
and analyzing an extremely valuable data set. Interestingly, as is the case
with much research on ASL, the current analysis reinforces our belief that
ASL patterns like spoken languages. However, in order to examine the language
practices of particular communities, we need to examine more particularities. Are
the changes described here particular to VMC? To computer technology? To technology?
To signed languages? To ASL?