Legal Implications of Complex Virtual Reality Systems Speech
delivered to DEFCON I (Las Vegas, July 10, 1993). Curtis E.A.
Karnow1
I. INTRODUCTION
A. We know about criminal problems
1. New Crimes--and old fashion crimes--using
computers.2
a. breaking and entering
b. trashing information: viruses
C. changing information
d. Copyright - the new federal felony
e. The scope of the federal government's
interests: "federal interest" Computers: any
communications across state lines triggers the
interest of federal law enforcement.
B. Now, as the industry matures, we face civil litigation
issues.
1 Curtis Karnow is a partner at the San Francisco law
firm of Landels, Ripley & Diamond, and chairs the firm's
Competitive Practices Group. His practice emphasizes
intellectual property litigation and computer law. He is a
faculty member with the American Arbitration Association, a
former federal prosecutor, and serves as temporary judge with
various Bay Area courts. He can be reached through the Internet:
karnow@cup.portal.com. These materials are copyrighted (c) 1993
by Curtis Karnow. The reader's professional courtesy is
appreciated in providing attribution to the further use of these
materials.
2 See generally, "Note, Computer-Related Crimes," 30
Am.Crim.L.Rev. 495 (1993).
2
II. BACKGROUND: WHAT'S NOW AND WHAT'S COMING IN VR
A. What is VR
By 'virtual reality' I mean massively networked, broadband,
interactive immersive computer-mediated experience. The
data, including audio visual displays, feedback and other
sensory input are a function of the operating system,
multiple layers of application software including those
running on a host or hosts, together with remote databases
and programs resident on remote platforms.
New Techniques
1. direct retinal imagery
2. VDT cursors controlled by amplified brainwaves
3. the surround:
a. Battletech
b. SEGA goggles
c. Sega "Activator" infrared beams for the body
movement
d. the CAVE
B. We're moving from single actor to multiuser
1. i.e. from one communicating simply with a CPU and
software to an environment in which many people, from around the
world, are contributing in real time.
2. MUDs
3. Given the choice, people generally chose to interact
with people, not machines. The experience of game publishers
reveals this: think of the attraction of the Head-to-head modes
for flight simulators like Falcon 3.0; like Sierra online.
C. We're moving from an exoskeleton to the elimination of
the interface.
1. From the goggleheaded, bodysuited, multiwired
oddball to the user who sheds his chains, his body, and walks
into the new universe
2. You know about the Zero
insertion force socketZIF? You know GUl? Here's a new three
letter word- ZUI- zero user interface. That's what we're aiming
at. We want to for et about the technology; we don't want to see
it, hear, it, smell, it or
3
face it; we want to think it dissolved, gone, vanished, we want
to take it completely for granted, until we believe the
environment is real.
Computers will dissolve too. When we talk about
totally immersive VR, or chips lodged in the brain, or other
metaphors for complete integration between mind and machine-
that's what we getting at: ZUI [zoo-ee] zero user interface; the
vanished machine.
3. What is it for the environment to be real? You think
this room is real? The street outside? A person from the 14th
century would't be sure, and someone from the Stone age would be
sure- that this was a dream. Things get real, after a while,
after we adjust to them, and after we go onto other things, think
about new issues, and start using the technology for other goals.
Like the zipper, the car and phone: these things disappear on you
as you use them, transparently, for other work. I don't worry
about whether the voice I hear on the phone is "really" my client
or friend- the sound I hear is vaguely connected, vaguely
relates, to the "real" voice. I don't mind. I think it really is
the voice of my friend.
Later, I will discuss our willing suspension of disbelief,
and our extraordinary ability to feel comfortable in initially
extraordinary circumstances. Let me simply note here that a
"real" environment is just one that we know well, that responds
at such a speed and sufficiently predictable that we ignore it,
that we take it for granted.3
3 So I reject attempts to distinguish synthetic from natural
environments. All environments are equally synthetic or natural,
depending on who or what is doing the sensing. See, 25 Siggrahic
Computer Graphics No.3 as an appendix to a report to the 1992 NSF
Invitational Workshop "Research Directions in Virtual
Environments," later expanded by Warren Robinett in the article
"Synthetic Experience: A Proposed Taxonomy." That "taxonomy"
suggests synthetic reality or "Synthetic Experience" as opposed
to a "Natural Experience". The latter requires the direct
perception of some properties or behaviors of something
physically present before the perceiver. A "Synthetic
Experience" is the perception of a representation or simulacrum.
4
D. Interim Conclusion:
Mulituser + ZUI ¯ real interaction with real people and
with real parts of the real world.
Now, you think the world of courts, lawyers and judges will
step in? You bet.
Murphy: If something can go wrong, it will.
Karnow: If something goes wrong, someone will sue.
III. VR AFFECTED BY THE LAW
A. Abstract: the Complex Backdrop
Summary: Error as a function of at least four general
problems. First, the high complexity of computer code; second,
the theoretically inability ever to completely debug code;
thirdly, the fact that VR systems rely on hidden interpolation,
and may utilize fuzzy logic; fourthly, the extremely high speed
of useful projected VR systems.
1. High complexity in computer code
The software to be used in high end VR systems is, and will
increasingly be, complex. This complexity is a function of at
least four factors: the size of the programs, multiple providers
of software and hardware; multiple simultaneous users, and the
intentional programming and anticipation of chaotic behavior.
a. New Operating systems such as Pink: extremely
large code. Microsoft NT is reputed to exceed 12.5 million lines
of code. Add applications, driver software. Contrast:
Lines of
code
Handheld bar code scanners 40000
Electronic four speed transmission 19000
Cellular telephone 30,000
Air traffic controller computer 130,899
Automated Teller network 600,000
Telephone router 2,100,000
New Operating System (Pink) 3,500,000
(could be more:)
5
B-2 bomber 3,500,000
Seawolf Submarine combat Computer 3,600,000
b. Multiple entities: software developers and
publishers, telecommunications nets [fibre optic, satellite,
etc.] , hardware platform vendors and operators. No one is in
charge.
c. Many simultaneous users.
d. High complexity and unpredictability will be
sought out by designers to mimic "real life".
Objects must have many properties. High chaos is engendered
by interacting autonomous computational agents- the most complex
of which are interacting human beings!4
Virtual reality's power lies in its wholly immersive,
kinaesthetic front-end: and importantly, the kinaesthetic
interface will be a function of many users: both the users
currently in the environment and probably prior users. A truly
sophisticated v.r. environment will be extremely complex: it will
be theoretically unpredictable; that is, chaotic, just as real
complex systems exhibit rule-bound, but chaotic behavior. Indeed,
I suggest it is this unpredictability, these open-ended
processes, that will make a crucial contribution to the felt
perception that the experience qualify as a convincing virtual
reality. 5
Current personal [stand-alone] computers are relatively
simple, because they are basically uncoupled from the user and
the rest of the environment; or, more accurately, they are
coupled in a few simple and direct ways: data input and output-
keyboards and printers, screens, and such, and some dependence on
the stability of the electrical/hardware system.
But: as the number of elements of the systems increases, as
the number of the semi-autonomous agents -- humans and computer -
4 See C. Morningstar & F. Randall Farmer, "The Lessons of
Lucasfilm's Habitat," Cyberspace: First Steps (M. Benedikt, ed.
1992), reprinted in Virtual Reality 93 - Special Report (Al
Expert) 23, 25.
5 See C. Morningstar & F. Randall Farmer, "The Lessons of
Lucasfilm's Habitat," Cyberspace: First Steps (M. Benedikt, ed.
1992) , re rinted in Virtual Reality 93 - Special Report (Al
Expert) 23, 25.
6
- increases, chaotic, unpredictable behavior will emerge. Recall
that chaotic evolution can be generated with only three
interacting independent systems.6
2. Theoretically Inability to debug
a. Millions of lines of code interacting with each
other and in the "real world" environment. Studies, including
some published in Scientific American, establish that moderately
complex programs cannot be completely debugged, and that probably
no program can be thoroughly debugged. For the mathematicians
among you, the issues are related to the halting problem: there
is no way to determine when all bugs have been eliminated. There
is no way to simply examine a program and tell if it is buggy;
instead, the program must be actually executed, and run under all
possible circumstances, to detect many bugs. And, of course,
there is not enough time in the life of the universe to do that
for most useful programs!
b. Think of the complexity of developing a single bulletproof
applications package: "handling memory and system kludges, BIOS
incompatibilities, DMA and IRQ conflict madness, funky foreign
motherboards, different DOS and Windows and OS/2 versions,
different motherboards, different address schemes, different
drive adapters, different keyboards, different busses, different
compression schemes and even different mice. . . ."7
3. Interpolation and Fuzziness
The synthetic appearance of virtual worlds, to be
"realistic" must be the result, at least in part, of
interpolation and of fuzzy processing. Blanks are not an
acceptable part of the illusion: the software must fill in the
blanks, and must allow for spectrums of colours, sizes and shapes
and other dimensions of objects. When you recall that (1) all
objects and their properties or dimensions are, in fact, data,
and (2) users are making decisions in a virtual world based on
the properties of the objects presented, then it becomes clear
that inaccuracies of perception, and then inaccuracies in user
conclusions, are possible-- not as a result of error in the
classic sense, i.e. not as a result of bugs or glitches or
unforeseen effect of multiple inputs, but as a direct function of
the calculation at issue, as a direct result of the fact that the
problem is fuzzy, or the software has interpolated likely values
for the data.
6 D. Ruelle, Chance and Chaos 81 (1991).
7 "Dvorak V. Somerson" PC Com utin jan
1993 at p. 108
7
We handle this all the time in real life; but in a virtual
world we may forget, or be unaware, that this is going on. For
example, we examine a chart showing stock performance, and
associated facts making up the historical performance of the
company, the economy, and so on. Looking at a paper chart, IF WE
THINK ABOUT IT, we can tell where the data in incomplete, where
interpolating lines join known data points, where two end points
create a range of possible values. Now consider a virtual world,
where the user flies through mutlicoloured and mutlishaped
terrain, which provides the information to him at a virtual
glance. This application currently exists. Under these
circumstances, it can be difficult to separate out plainly what
is known and what is not.
The goals and techniques of virtual reality development make
this problem of interpolation especially problematic. Recall
that the success of the computer-generated immersive experience
depends on our willing suspension of disbelief: that is, out
ability to pretend, and to believe that pretence deeply. When we
enter a virtual world, our point of view changes dramatically. We
might be looking out the eyes of a robot across the room, looking
back at our own body: We will believe that we really are out-of-
body, looking at ourselves. We might be flying above terrain,
down blood vessels, across coloured charts, or sitting in a room
talking to people whose real bodies are in Iceland.
In a successful VR environment, we will act as if we are in
those places. Experiments have shown that even when one sense-
such as vision- is replaced by tactile feedback such as patters
of tingling on the skin, subjects will react as if they really
see- such as lurching backwards when the tingling sensation is
artificially amplified.8
In short, a shift in the felt location of consciousness can
be induced by a variety of means, as every good designer of
exciting, interactive games intuitively knows. The sense of self
and of consciousness is identified generally with the body, of
course, but only when our attention to drawn to the physical
body, and only in default of some other potential location for
the reception of sensory experience. As Marvin Minsky has noted,
8 Daniel C. Dennett, Consciousness Ex
lained 341 (1991) .
8
We have the sense of actuality when every question
asked of our visual systems is answered so swiftly that
it seems as though those answers were already there.9
Powerful virtual reality routines of necessity will play on our
enormous capacity for belief, to make sense of, and find patterns
in, whatever surrounds us, especially with respect to the basic
propositions of where and what we are. In this context, it may
be impossible to separate fact from fiction, data from
interpolation, and substance from cosmetics.
4. Performance: Very high speed.
The illusion of reality depends on it. We want no
latency: that's how we measure gratifying power in today's
systems. We want, and we will get, immediate results, immediate
effects. Instant Sensual Gratification!
So, the sum of these parts:
High Complexity +
High Speed +
Interpolation and fuzzy data +
Inevitable Error +
¯ Potential civil liability ¯ as ZUI becomes the norm.
B. Forgetting Invisible Technology
To some extent, visible, kludgy technology is its own
warning; it contains its own seat belts. You can't go too fast
in a model T Ford, you don't get too many accidents. Visibly
problematic technology has a built-in safety belt: people are
careful as they manipulate the machine; they don't rely without
question on the product; they watch for the final result, and
have, it follows, an independent but simultaneous sense of when
the technology is working. Later, as the technology gets
integrated, works well, and finally disappears from our
consciousness, the checks and balances evaporate.
At first, we maintain an independent sense of whether the
result is right, that something is working, but that isn't
simultaneous: we check later, and maybe we find the error and
maybe we don;t. Current spreadsheet applications fall into this
Category, as does aircraft manufacturing: we're pretty sure it
9 The Societ Of Mind 257 (1985) .
9
works, but can't check in real time. When the numbers looks bad
in retrospect, or engines fall off the wings of a 747, the damage
is done, but we do go back and fix it.
Later, we loose the ability to cross-check the results,10 or
fix them. Voice mail and phone message machine? This is most
apparent in highly complex computer systems, and in complex
economic and social systems. The reason is, in brief, that the
systems are sufficiently complex that there is no such thing as a
control group; the system cannot be literally duplicated.
For example: when one first begins to use a calculator, one
has a sense of when the result is right; and one might check the
calculator by hand- my wife does! When it works repeatedly, one
considers using the technology for calculations that cannot be
otherwise checked: there is no independent, simultaneous sense of
whether the technology works. We assume it does, like we assume
the phones will work and a meteor will not strike the earth, and
we get on with our ultimate tasks without worrying about the
tools.
In a sophisticated high-speed ZUI VR environment, we may not
know if and when the system crashes - or not until it's too late
to avoid injury and death.
How real a problem is that likely to be?
C. The real world's infrastructure is based on software: it
makes the industrial world go round: There is a reasonable
probability of injury as VR interacts with real world events.
[Indeed there is no VR application that does not interface with
the real world, because it always at least interfaces with a real
human.]
As computers become more advanced and sophisticated-- and VR
represents the Cutting edge of that sophistication -- computers
10 Loosing the ability might also mean not having time to
do so. Manual searches of databases are possible, but not
reasonably feasible. Akins v. District of columbia, 526 A.2d 933
(Dist.Colum. March 31 1987) (criminal justice system computer
failed to generate prior reports of convictions, resulting in
release of dangerous felon; manual retrieval not attempted) .
Loosing the ability may simply mean that a function, which could
be done by a human or otherwise cross-checked, has been
permanently delegated to a machine, such as data retrieval. Brown
v. Unites States, 790 F.2d 199, (1st Cir. 1986) (damaged weather
buoy provided erratic data, ultimately leading to death of
fishermen who relied on data) .
10
become more reliable and more integrated into our infrastructure;
BUT when they fail-- and they will fail-- it is stunningly
spectacular.
1. Failure Examples: :
a. Medical procedures /telepresence
b. recall the last Black Monday on Wall Street,
1987? when computer based trading brought the market down
c. Computer bugs / SRI list11
(1)3 bits of code brought down the east
coast phone service between June 26 July 2, 1991.
(2)1987: lethal doses of x rays from
Computer controlled machine; glitch when commands typed in too
rapidly- buffer problem.
(3)Crashes of Army Blackhawk helicopters in
the mid-1989: EMF radiation caused problems in unshielded
connections between inflight computer and control surfaces.
(4)Confusing interface for the missile
launch system on the Aegis cruiser Vincennes contributed or
caused the launch of the missile that shot down the civilian
Airbus Iran Air flight 655 (290 dead).
(5)Error in modeling programs, or
inappropriate modeling, responsible for collapse of buildings
(I.e. Hartford Civic Center) and other products such as ships.
d. Training: SimRefinery; military; NASA
e. product defects
f. MUDs take a hug amount of Computing power and
communications bandwidth. They can and do cripple computer
networks. They are illegal on many campuses, and banned in the
entire continent of Australia.
2. How do these issues get translated in to the legal
world? HOW do legal rules get implicated, how do the
courts get involved?
If this were a group of lawyers, we could chat about torts,
various specific statutes, and the differences between deign
defect and manufacturing defects, implied warranties, and the
rest of it.
But none of that matters, at heart. The law is enormously
flexible: ancient doctrines are applied in the twinkling of an
11 Peter Neumann, FTP CRVAX.SRI.COM and go to the
appropriate directory with "CD RISKS:"
11
eye to new technologies. The point is that is if anyone thinks
they have been wronged, injured, hurt -- if anything in the real
world goes not accordingly to plan -- to someone's plan-- then
that someone can file a complaint and start a lawsuit. It only
takes about $80 or so to pay the filing fee in state court in
california.
The legal community has, or will have, an interest in the
unforeseeable results of complex virtual reality systems. Two
highly complex systems are merging- virtual systems and the law.
The results of that are -- unpredictable.