CSE 171: User Interface Design: Social and
Technical Issues10. More Notes on Algebraic Semiotics, and Other Topics

There is an important basic duality between designing
signs and interpreting signs: in the first, we know about signs in
the source system, and we map them to a target system that can best preserve
the information of greatest interest, whereas in the second, we know about one
(or more) target sign, and we seek to infer signs in the source system from
which it (they) might have come, which also requires us to do some inference
about the mapping. Examples of design that go beyond the usual computer
science applications of using some mixture of media to present some given
content, include giving a good explanation for something, finding a good
"icon" (in the usual informal sense of computer graphics), choosing a good
file name, and making a good analogy. Examples of the dual situation include
understanding graphics, multimedia texts, verbal explanations, poems, films,
metaphors, equations, or indeed, anything at all. (It is interesting to
notice that people often in fact do not fully understand signs, e.g.,
films, even though they may spend a lot of time with them, and enjoy them.)
Another very important class of examples concerns the evaluation of
signs, as requested in several of the homework problems for this class: here
you must not only figure out what the source sign system and semiotic morphism
might be, but also what some alternative semiotic morphisms, to potentially
different target sign systems, might be.

In each case, there is a transformation (movement, translation,
interpretation or representation) of signs in one system to signs in another
system, and the examples of design go in the opposite direction from
the examples of understanding. Thus algebraic semiotics helps us see the dual
relation between design and understanding, and also suggests ways that each
process can contribute to the other:- In general, it is easier to investigate
understanding than it is to investigate design, which after all is inherently
more creative. But once we understand something about a mode of
understanding, we can apply it to the corresponding design problems; of course
the reverse trajectory also works, but it will be less common. There are many
interesting applications of this approach, one of which is designing
visualizations for scientific information, as discussed earlier in the course.
Another is to developing a better understanding of humor.

Many newspaper cartoons, consisting of 1 to 4 small scenes (i.e., panels),
achieve their effect by setting up some situation, and then
recontextualizing it, i.e., introducing new elements and relations
into the conceptual space, which have the effect of forcing a new organization
for some parts of the conceptual space that was originally set up. The first
space is in general itself some kind of blend, and its reconceptualization is
also a blend, of the old space with the new material; this often has a
humorous effect. Similar phenomena can be found in music, poetry, and
probably in every art form, though the effect is by no means always comic. It
seems to me clear that evolution has provided us with positive feedback for
improvements in understanding in the form of mental pleasure, since this has
an obvious survival value. One familiar example is the so called "Eureka"
experience, when we suddenly see the solution to some problem that we have
been pondering for a long time. Let's call this reblending.

In light of our success with viewing an oxymoron as an inconsistent pair of
blends, often with a "cross space" mapping which imports at least some of the
(generally less conventional) contradictory meaning into the (generally more
conventional) blend, it seems reasonable to believe that some fairly large
areas of humor can be characterized in terms of reblending. Once we have this
understanding, we are in a position to apply it to design. For example, we
might want to make the use of certain difficult interfaces a lighter and more
pleasant task by (carefully and selectively!) introducing some humor. It is
important to notice that, because the psychological impact of
recontextualization depends on its novelty, repeating the same joke again and
again will not be effective interface design, and in fact, will prove
irritating to users. Many designs have had to be redone for this or closely
related reasons. For example, overly cute icons or avatars can quickly become
irritating (or slowly, if they are a bit less cute), and cuteness as a
semiotic phenomenon is closely related to humor, in that it involves partially
conflicting interpretations (e.g., child-like facial features but a serious
message). Two instances from MicroSoft include the barking dog in PowerPoint
and the obsequious paper clip in Word (although this is a non-default option
in the most recent release).

We can also understand (some aspects of) narrative structure in terms of
semiotic morphisms. For example, books on screen writing by Syd Field
prescribe a precise (but naive) dramatic structure for the plots of Hollywood
movies: they should have three acts, for setup, conflict, and resolution, with
"plot points" that move action from one act to the next. We can describe this
"Syd Field structure" using a very simple sign system with a main constructor
that builds a thing of sort "plot" from three things of sort "act", and we can
then check whether a given film has this structure by seeing whether there is
a semiotic morphism from the (structure of the) film to this sign system.

It is clear that as internet bandwidth grows, video will be used more and
more; moreover, complex interactive websites are aready common, many with
story lines. Techniques for drama apply directly to such cases, but are also
much more broadly applicable, in suggesting ways to make displays (i.e.,
"texts" in the very general sense of signs that are to be interpreted by
users) more interesting. My own favorite examples of this come from
mathematics. Probably we have all had the experience of trying to read a
proof and being frustrated, at least initially, by not being able to see where
it was going, or why it was structured as it was. This is often because the
proof author did not describe the difficulties that arose in constructing the
proof, but instead just described the machinery that was constructed to
overcome the difficulties. For example, one often comes upon an assertion in
the middle of a proof whose relation to the main result to be proved is far
from clear. It will help if the proof author has broken this assertion out of
the main text and labelled it a "lemma," but even then, it is often far from
clear why it will be needed later on.

The "Syd Field" structure might suggest first showing how a proof fails
without a lemma (in dramatic terms, this gives rise to a conflict), and then
showing how it goes through with the lemma - the "first act" will of course
set up the proof, including what is to be proved and what is assumed. We used
this structure in proving a
simple property of flags on our Tatami project
website (ignore the proof details, and look at the explanation for why the
first attempt at the main result fails, motivating the second attempt, which
succeeds with a lemma).

It is also interesting to consider the problem of translating from one
language to another. This has been much studied in computer science, and with
today's increased processing and memory power, is finally bearing some fruit
(though sometimes that fruit may seem a bit raw - e.g. if you try the
translation option on some websearch engines, the results may be amusing
and/or depressing). Semiotic morphisms illuminate some of the difficulties.
It will help bring out the issues if we focus on the particularly difficult
case of poetry, noting that less severe versions of the same difficulties can
arise in any form of language. First, notice that poems may have, or fail to
have, many different kinds of structure; for example, they may or may not be
divided into stanzas, have a fixed meter, or rhyme; they are usually divided
into lines, but even this does not always hold. Moreover, some poems have a
geometrical form that is of interest (e.g., poems by e e cummings), and many
different conventions are used for punctuation. All of this is easily
expressed using sign systems, with various kinds of constructors and
relations. For example, iambic pentameter is a particular sign system for
syllabification. Sonata form plays a similar role in music from the classical
period.

Secondly, notice that structures are often more natural for one language
than for another. For example, it is much easier to rhyme in Romance
languages like Spanish and Latin than in English. From this, it follows that
it may be desirable to fail to preserve certain features, such as rhyme, when
translating across languages; and it may even be desirable to add some feature
to the translation that was not present in the original, e.g., adding rhyme
when translating from Chinese to Spanish. Of course, every feature may be
preserved to a certain extent, rather than being either fully preserved or not
preserved.

Many people agree that mathematical proofs are an area where better design
could have a big impact, e.g., in K12 education. The above ideas provide a
way to explore how computer technology might help with this. Of course, there
are many other design areas where making the content more dramatic could help,
such as textbooks, courses, and manuals; private sector media already make
extensive use of drama in their work, e.g., look at TV newscasts, or "real
crime" documentaries, or other "reality" programming. But it should not be
thought that drama is the only issue in narrative; the extensive literatures
on creative writing and the novel raise many other issues. Also, there is the
the structure of oral narratives of personal experience that we studied at The Structure of Narrative, due to the
linguist William Labov and others, and there is a famous discussion of the
semiotic structure of Russian fairy tales due to Vladimir Propp. The main
point of the present discussion is to claim that any instance of such a
structure can be seen as arising through a semiotic morphism to a sign system
that embodies the given structure. I also claim that being aware of this
viewpoint can help designers in their practical work, if they are also aware
of the variety of narrative structures that are potentially applicable.

It is interesting to see how our theory of semiotic morphisms solves a
problem that has been noticed by many theorists of narrative structure, which
is that texts often fail to include some of the features that are supposed to
be part of the generic structure; even important features are sometimes left
out. This corresponds to the fact, with which we are already familiar, that
semiotic morphisms can be partial, i.e., they can fail to preserve
some aspects of source signs. Moreover, the fact the target is a sign
system, not just some fixed given structure, means that many
different structures, perhaps with a variety of substructures, etc., can be
possible, not just some single fixed structure. This allows a great deal of
flexibility. For example, the Labov structure permits arbitrary sequences of
narrative clauses, and allows many different kinds of evaluative material.
Moreover, its opening and closing sections are optional. Artists often play
with structure to create interesting effects; for example, false endings in
classical symphonies. See The Structure of
Narrative for the details of narrative structure.

The Labov structure can be applied in many different ways. For example,
user manuals for computer systems often describe sequences of steps, and these
may be less tedious to read if they are given a more narrative-like structure,
although this can of course be overdone, with unpleasant results, as we
previously saw with humor and cuteness. The point here is to consider
semiotic morphisms to the Labov narrative structure sign system.

A Flame on the Information Scandal

It is said that we live in an "Age of Information," but it is an open
scandal that there is no theory, not even a definition, of information that is
both broad enough and precise enough to give such an assertion much meaning.
An appropriate theory would help us to understand and to design information
systems, in a wide sense that includes computer-based systems as well as
systems that are based on more traditional media such as paper. However, a
major motivating example is Information Systems in the narrow sense
of computer-based systems for storing and retrieving information, e.g.,
database systems. User interface design also provides valuable insights into
the kinds of problem that are important.

The need for a good theory of information is pressing. Society is
demanding ever larger and more complex information systems. Billions, perhaps
trillions, are spent each year on software, but many systems that are built
are never used, and at least one third of systems begun are abandoned before
completion. Moreover, many systems once thought adequate no longer are, while
many others were never adequate. Among many sobering examples are the
disastrous baggage handling system at the Denver International Airport, an IBM
default on an 8 billion dollar contract to build the next generation U.S. air
traffic control system, and a major IBM public relations disaster with its
computer feed of real time sports data to journalists at the Atlanta Olympic
games. Our knowledge of how to build effective information systems is very
far from meeting the needs of society. Errors in requirements, that is, in
understanding what kind of system is needed, have been identified as the most
important problem, and it is also widely agreed that social factors are the
most important source of difficulty in writing good requirements for large and
complex systems. Thus it is very dangerous to ignore the social dimension of
information!

This implies that an adequate theory of information would have to take
account of social context, including how information is produced and used,
rather than merely how it is represented; that is, we need a social
theory of information, not merely a theory of representation. On the other
hand, formal aspects of information are inherent to technical systems:
computers are engines for storing, processing and retrieving formal
representations. Thus the essence of designing such systems successfully is
to reconcile their social and technical aspects. In addition to
these practical problems, another important issue is the intellectual
coherence of offerings within departments devoted to computer science,
information science, etc. The lack of an adequate notion of information may
be even more of a scandal here, due to the historic emphasis on adequate
theoretical foundations in the academic world.

But perhaps it is impossible to find an adequate theory of information.
Bowker has discussed mythologies that support the notion of information,
Haraway has given a daring modern cyborg myth, and Agre has argued that the
notion of information is itself a myth, mobilized to support certain
institutions, such as libraries. Nevertheless, in the paper Towards a Social, Ethical Theory of
Information, I make an attempt to show how a notion of information
can be grounded in the dual aspects of the social and the structural, and in
addition, argue that information has an inherent ethical dimension. This
theory is a social semiotics, and it could be taken as a theoretical
foundation for this course, although I have chosen not to emphasize it here;
however it does seem that the practical considerations developed in this
course lend support to such a theory of information.

Undoubtedly the best known and most popular information theory today is
that of Claude Shannon. Perhaps its most basic concept that of the
bit, which of course is very fundamental in computer science. The
number of bits associated with some information is really just a measure of
its size, and is useful for determining the amount of memory needed
for storing it, or how long it would take to download it. However, the number
of bits in a file tells us nothing about its content. So, although
this is a valuable theory in its proper domain, it is of no use for the
broader challenges that we have faced in this course, such as preserving the
most important information when designing an overview of some information
source (e.g, the homepage of a large website).

Shannon's theory of information is a reductionist, scientific theory, with
many real engineering applications, and so many attempts have been made to
extend it to cover not just the volume, but also the content, of information.
None of these attempts have been successful, and I would claim that no
reductionist theory can possibly succeed, because no such approach can take
account of the concepts, methods, and values of the members of social groups,
which we know from our study of ethnomethodology, are esential to
understanding how real information/social systems actually work.

In fact, semiotics is an information theory of exactly the kind we seek,
provided it is considered to be grounded in social reality, rather than in
Plato's abstract mathematical heaven. Thus, we have been studying information
theory all quarter! Social foundations for semiotics are covered in some
detail in the paper Towards a
Social, Ethical Theory of Information, and are also briefly reviewed
in the webpaper The Ethics of
Databases, which is actually mainly concerned with the reverse
process, of inferring the values of the designers from a structural analysis
of an interface. This is a very interesting kind of exercise, with many
potential applications; I use the name natural ethics for the broad
project of extracting values from artificial objects, regarded as signs. The
first key to this project is that the priorities and levels of sign systems,
and the preservation properties of semiotic morphisms, reflect the importance
that a designer has assigned to sign parts.

Although I think this is a terrifically interesting piece, I am afraid that
parts may be too difficult for this class. Hence I suggest you read it over
lightly, and pause whenever you find something that interests you. Among
other things, this paper discusses the logic of multiple contradictions in
narrative (building on Greimas), the relation between static and dynamic
analyses of narrative, modal logic, fuzzy logic, and more, all in the
dynamical systems paradigm of the paper we read earlier, Multimedia Phase-spaces. If
you get interested in this, you should also look at my Notes on Gradient Logic. We will not follow up on
this material.