ADAPTIVE SOFTWARE

Peter Norvig and David Cohn

Harlequin Incorporated

The Problem With Software

The problem with software is that it takes too much time and money to develop, and is brittle when used in situations for which it
was not explicitly designed. Various software design methodologies address this problem:

1970s: Structured Programming makes it feasible to build larger-scale software systems - provided you have a
specification of the desired results at the start of the project and the specification rarely changed. A typical application is a
database report writing program which reads an input file and produces an output file. We call this an input/output based
application.

1980s: Object-Oriented Programming makes it easier to reorganize when the specification changes, because
functionality is split up into separate classes that are designed to have minimal interaction between them However, each
change to the specification (or to the environment) still requires programmer intervention, with a costly
redesign/reimplement/rebuild/retest cycle A typical application is a desktop publishing system, where user-initiated events
(mouse clicks, menu selections, etc.) trigger computation. We call this a user-initiated event based application.

Today: Adaptive Programming is aimed at the problem of producing applications that can readily adapt in the face of
changing user needs, desires, and environment. Adaptive software explicitly represent the actions that can be taken and the
goals that the user is trying to achieve. This makes it possible for the user to change goals without a need to rewrite the
program. A typical application is an information filter which searches the Internet or company intranet for information of
personal interest to the reader. Note that much of the searching may go on when the user is not even logged in. The
application does more on behalf of the user without constant interaction, and the sophistication comes from a splitting of
responsibilities between the program and the user. We call this an agent based application.

Of course, there have been other proposed methodologies for improving software. Some address the problem of managing
change, but only adaptive programming is about anticipating change and automatically dealing with it within a running program,
without need of a programmer. As a definition, we can say:

"Adaptive software uses available information about
changes in its environment to improve its behavior."

The Challenge of Complex Environments

Software is expected to do more for us today, in more situations, than we ever expected in the past. This is the challenge of
complex environments. The complexity comes from three dimensions. First, there are more users.
Now everyone, not just trained
professionals, uses software. Second, there are more systems and more interactions
among them. A company that once had a
homogeneous mainframe now has a wide variety of desktop, server, and mainframe machines, running a wide variety of protocols.
Third, there are more resources and goals. Programmers are accustomed to trading off time versus space. Now
they also have
to worry about bandwidth, security, money, completeness of results, quality of information, resolution of images and audio, and
other factors, and they have to make the right trade-offs for a wide variety of users.

Together, these three dimensions make the designer's job harder. The designer can't foresee all the circumstances in which an
application will be used, and thus can't always make the right design choices. This can shorten a product's lifetime, because
upgrades will be needed for unforeseen situations, and it undermines the end user's confidence that the product will even perform
as advertised. We can summarize the difficulties posed by complex environments with this table:

Standard Environment

Complex Environment

Certain

Uncertain

All inputs observable

Some inputs hidden

Deterministic

Non-deterministic

Repeatable

Unique

No other players

Other players

Static

Dynamic

Discrete

Continuous

Virtual world

Real world

Five Myths of Traditional Software Development

Traditional software development is based on some principles which are no longer appropriate under this new world of
complex
environments. It is worth looking at five of these myths.

The myth of the specification: Traditional software development assumes that the first task is to
determine the
specification, and all design and development follows after the specification is finalized. More modern approaches stress
spiral rapid development methods in which the specification is constantly being refined. The target is really a combination of
related components, not a single application.

The myth of software maintenance: The word "maintenance" gives the impression that the software has
somehow
degraded, and needs to be refurbished to its original condition. This is misleading. The bits in a program do not degrade.
They remain the same, while the environment around the program changes. So it is really a process of
upgrading or evolving to meet the needs of the changing environment. By viewing it
as a maintenance problem, one risks making the
mistake of trying to preserve the old structure when its time has passed.

The myth of the black box: Abstraction plays a crucial role in the development of reliable software.
However, treating a
process as a black box with an input/output specification ignores the practical problem of resource usage. When a system
built out of these black boxes is too big or too slow, there is no way to improve the performance other than to tear the
boxes apart. Some work has been done on the idea of Open Implementation, in which modules have one interface for
input/output behavior, and another orthogonal interface for performance tweaking. Adaptive software adds to this by
making the orthogonal interface a two way system ÷ there is a feedback loop that provides information on the results of
the performance tweaking.

The myth of design choices: In traditional software development, the designers consider several
alternatives to implement
each component, and then make choices based on the desired product. The alternatives that were not chosen, along with
the rationale for the choice, are then discarded. That means when the environment changes (perhaps a new communications
protocol becomes popular, or the user gets a computer that is twice as powerful), one needs to start all over again to see
what choices are now the best. Often, the original programmers have moved on, and so their design rationale is lost. The
alternative is to capture the design criteria as a formal part of the program, and let the program reconfigure itself, as the
environment changes, to optimally satisfy the criteria.

The myth of the expert programmer: Most programmers have pride in their abilities, and feel they can
always come up
with a good solution, given a suitable specification of the problem. In reality, programmers are only one resource available
to a project, and an expensive resource at that. Furthermore, it is impractical to ship a developer with each program (except
for very large and expensive custom applications). This suggests that we find a tradeoff where the programmers and
designers do what they can do best ÷ formally describe what the program is trying to do. Then the program itself does the
calculating and configuring necessary to achieve these goals in the current environment.

Key Technologies for Adaptive Software

Now we know what adaptive software does: it uses information from the environment to improve its behavior over time, as the
program gains more experience. And we know why it is difficult to do this: because specifications are incomplete and changing,
because the environment is constantly changing, because carefully crafted designs may rely on assumptions that become obsolete,
and because there is never as much programmer time as you really need. In this section we survey some of the tools that help us
overcome these difficulties. Here are five of the most important:

Dynamic Programming Languages provide a robust framework for writing applications that
persist over long lifetimes, and can be updated while they are running.

Agent Technology is a point of view that encompasses the idea of acting in accord to a user's
preferences.

Decision Theory provides the basic terminology to talk about an uncertain world, and
about what the preferred outcomes are.

Reinforcement Learning gives us a way to learn what sequence of actions to perform, given local
results of the worth of individual actions.

Probabilistic Networks provide powerful algorithms for computing optimal actions based
on whatever is known about the current state of the world.

Static languages like C require the programmer to make a lot of decisions that nail down the structure of the program and the data
it manipulates. Dynamic languages like Dylan and CLOS (Common Lisp Object System) allow these decisions to be delayed, and
thus provide a more responsive programming environment in the face of changing specifications. Changes that would be too
pervasive to even try in static languages can be easily explored with dynamic languages. Dynamic languages provide the interfaces
by which a program can change or extend its performance. For example, in Dylan, a running program can add a method to an
existing class without access to the original source code; can define a brand new class or function under program control; can
debug another Dylan program, even one running remotely over the web. All this makes it possible to build, re-build and modify
complex programs using components. Java is currently the most popular language with dynamic features, although it is not as
thoroughly and elegantly dynamic as Dylan and CLOS.

There has been a lot of talk about software agents, and not much agreement on what they are. The Oxford English Dictionary
entry for "Agent" says: from Latin agere ("to do"), 1. One who acts or exerts power.
2. He ... who produces an effect. 3. Any natural force. (e.g. a chemical agent).
4. One who does the actual work - a deputy, substitute, representative.

So we see that essentially, an agent does something and optionally does it for someone. Of course, all programs are intended to
do something for someone. Therefore, in order to make a distinction between "software agent" and just a "program," we stress
that software agents should be able to immediately respond to the preferences of their sponsors, and act according to those
preferences. In addition, we think of an agent as existing in a complex environment. It can perceive part of the environment, and
take some actions, but it does not have complete control over the environment.

In short, we want the agent to do the right thing, where "right thing" is defined by the user or users at
that point. There are
three reasons why this is difficult:

You can't always get what you want: In a complex environment, you generally can't optimize
everything at once. You
can't have a program that is 100% accurate, yet uses no resources and runs in no time. So we need a way for the user to
say which resources and results are most important, and which ones are less so.

You never know what's going to happen: Traditional programs are built with Boolean logic, which treats
everything as
"true" or "false." This is appropriate for the internal working of a digital computer, but out in the real world there are always
going to be events about which we are uncertain. For example, it is "likely" that we will be able to establish a given network
connection, but we won't know for sure until we try.

You're not the only one in the world: There are other programs out there which can change the
environment, and with
which we can cooperate, contract, compete, and communicate.

Decision Theory is the combination of utility theory (a way of formally representing a user's preferences)
and probability theory (a way of representing uncertainty). Decision Theory is the cornerstone for designing proper adaptive software in an uncertain,
changing environment. It is "the" language for:

Describing adaptive software: To respond to user preferences and deal with uncertainty, decision
theory provides the only mathematically sound formalism to describe what it means to "do the right thing."

Building adaptive software: In some simple cases, once you've done an analysis of a problem using
decision theory, it becomes clear how to implement a solution using traditional methods. But more often we need to use
decision-theoretic technology like reinforcement learning or probabilistic. Sometimes, a compilation step can be used, so that the
run-time computational demands are minimal.

Communicating with agents: What should a user say to a software agent? Or what would one software
agent say to another? Most importantly, they should be able talk about what they want to happen, and so they need
the language of utility theory. But they also may want to talk about what they believe, and that requires probability
theory.

Reinforcement Learning is a powerful technique to automatically learn a utility function, given only a representation of the
possible
actions and an immediate reward function - an indication of how well the program is doing on each step. Given
enough practice
(either in the real world or in a simulated environment), reinforcement learning is guaranteed to converge to an optimal policy.
Reinforcement Learning problems could in principle be solved exactly as a system of nonlinear equations, but in practice the
problems are much too large for an exact solution, so we rely on an approximation technique based on dynamic programming.

As an example, consider assigning cellular phone calls to channels. You want to use the available channels wisely so that
there is
no interference and a minimum of dropped calls. Using reinforcement learning, Harlequin invented an algorithm that dynamically
adapts to changes in calling patterns, and performs better than all existing published algorithms. Reinforcement learning has been
applied to other domains, such as flying an airplane, scheduling elevators, and controlling robots.

Probabilistic networks (also known as Bayesian networks, Decision networks, and Influence diagrams) are a way of
representing
a complex joint probability distribution as a graphical model of nodes (representing random variables) and arcs (representing
dependence between variables). There are algorithms for determining the value of any variable, for learning qualitative and
quantitative dependencies from data, for quantifying the sensitivity of an answer to uncertainties in the data, and for computing
value of information: the worth of acquiring a new piece of information.

Side bar 1: Industries and Applications for Adaptive Software

Here are some real-world applications of adaptive software undertaken by the Adaptive Systems Group at Harlequin:

Crime Pattern Analysis and Fraud Detection: Harlequin makes a data visualization tool for fraud and crime
analysis called
Watson. For mid-size databases, trained users can easily navigate through the various displays to find the patterns
they are looking
for. However, for larger databases we needed to add data mining techniques to help spot the patterns. As new patterns in the data
occur, we provide ways of visualizing them.

Financial: A credit card company wanted to be able to track purchase authorization requests, and
decide which requests were
likely to be from a stolen card, or were likely to be a credit risk. Harlequin supplies the software upon which the authorization
system is written. The key for the customer was to have an efficient system capable of processing large volumes of transactions in
real time, but still have a flexible system, where changes could be made in minutes, not days, with no need to bring the system
down to install upgrades.

Manufacturing: A multinational company faced the problem of needing to continually build new versions
of their product to meet
new industry standard specifications. Each test run of a new product variation costs about $10,000. Harlequin is actively
consulting on this project to bring together all the prior test results and accumulated experience of the organization in one online
tool, and to provide sophisticated statistical optimization techniques to recommend an optimal experimental test plan, based on
past results and predicted future contingencies.

Electronic Publishing: Harlequin's raster imaging processing software for high-end commercial printing is
the fastest on the
market. But customers are interested in the speed of the total solution, so we are developing a dynamic workflow management
system that will optimize the tracking and scheduling of the customer's complete set of job tasks. This requires monitoring the jobs
and being able to predict how long each job will take at each processing stage.

Telecommunications: A telecommunications company wanted to build an experimental switching system for delivering video on
demand to the home. This requires strict real-time response (30 frames per second, no matter what), but it also requires flexibility
in the software. The customer required that they be able to add new functionality and even redefine existing functions while the
switch is running, because it would not be acceptable to interrupt service. Harlequin worked with the customer to meet these
requirements using a real-time version of CLOS. The result was a switching system that met all requirements, and was built with
only 25 programmers, as compared to the 250 needed on the system it replaced. A corresponding 10-fold improvement was also
seen in development costs and engineering change turn-around time.

Side bar 2: Groups Working on Adaptive Software

Harlequin's Adaptive Systems Group brings added
value to Harlequin's programming tools, web
tools, and postscript printing software, and also offers consulting services for other companies. Some of the projects are shown in
the accompanying side bar.

Microsoft's
Decision Theory and Adaptive Systems Group was started by
several experts in decision theory (David Heckerman, Jack Breese and Eric Horvitz), and was recently renamed to include
"Adaptive Systems" in the title. The group contributed the Print Troubleshooter in Windows 95 and the Answer Wizard in
Microsoft Office.

The Symposium on Flexible Computation from the AAAI Fall Symposium dealt
with "procedures that allow a graceful tradeoff to be made between the quality of results and allocations of costly resources, such
as time, memory,
or information. Systems employing flexible computation gain the ability to adapt the quality of their response to dynamic changes in
requirements."

The Adaptive Intelligent Systems group run by Barbara
Hayes-Roth at Stanford is investigating "systems that coordinate perception, reasoning, and action
to pursue multiple goals while functioning autonomously in dynamic environments."

The Demeter Group at Northeastern University says that
"adaptations of the
interactions of components can only be defined in terms of accumulated individual component adaptations. New techniques are
starting to provide higher levels of abstractions for adaptability. Software architecting brings system level abstractions. The
Adaptive Software family of languages allows this higher abstraction at the programming level."

The Open Implementation project at Xerox PARC provides the
basis of the black
box metaphor used in this paper. Xerox's Aspect Oriented Programming project
goes one
step further. "AOP works by allowing programmers to first express each of a system's aspects of concern in a separate and
natural form, and then automatically combine those separate descriptions into a final executable form using a tool called an Aspect
Weaver™."

Some people use "adaptive" for software
that helps people with disabilities. Other people use "adaptive" to refer to evolutionary,
neural, or biological computation. See the book The Mind, the Brain, and Complex Adaptive Systems, edited by Harold
Morowitz and Jerome Singer.