The UMAP Journal 14(1

Introduction
The power and usefulness of artificial neural networks have been demonstrated in several applications including speech synthesis, diagnostic problems, medicine, business and finance, robotic control, signal processing, computer vision and many other problems that fall under the category of pattern recognition. For some application areas, neural models show promise in achieving human-like performance over more traditional artificial intelligence techniques. What, then, are neural networks? And what can they be used for? Although vonNeumann-architecture computers are much faster than humans in numerical computation, humans are still far better at carrying out low-level tasks such as speech and image recognition. This is due in part to the massive parallelism employed by the brain, which makes it easier to solve problems with simultaneous constraints. It is with this type of problem that traditional artificial intelligence techniques have had limited success. The field of neural networks, however, looks at a variety of models with a structure roughly analogous to that of a set of neurons in the human brain. The branch of artificial intelligence called neural networks dates back to the 1940s, when McCulloch and Pitts [1943] developed the first neural model. This was followed in 1962 by the perceptron model, devised by Rosenblatt, which generated much interest because of its ability to solve some simple pattern classification problems. This interest started to fade in 1969 when Minsky and Papert [1969] provided mathematical proofs of the limitations of the perceptron and pointed out its weakness in computation. In particular, it is incapable of solving the classic exclusive-or (XOR) problem, which will be discussed later. Such drawbacks led to the temporary decline of the field of neural networks.

has seen renewed interest in neural networks. and mathematics. represent the set of inputs presented to the unit U. and a sampling of applications. both among researchers and in areas of application. Each accepts a weighted set of inputs and responds with an output. and Carpenter and Grossberg’s Adaptive Resonance Theory model [Wasserman 1989. computer science. We describe several of the more important neural models. followed by a discussion of some of the available hardware and software used to implement these models. The field has generated interest from researchers in such diverse areas as engineering.
Definition
Inspired by the structure of the brain. Figure 1 presents a picture of one unit in a neural network.
Let X = ( x1 . neuroscience... physics. xn ) . Kohonen’s network. and improved hardware have all contributed to the revival of the field. U
x1
x2
S = ∑ wi x i
A = f (S )
x3
Figure 1. Fukushima’s model. Hopfield’s network. called nodes or units.. however.The UMAP Journal 14(1) The last decade. the neuron.. Neural-network paradigms in recent years include the Boltzmann machine. Freeman and Skapura 1991]. A single unit in a neural network. Each unit is designed to mimic its biological counterpart. where the xi are real numbers. Each input has an associated weight that represents the strength
. better training algorithms. psychology. Rumelhart’s competitive learning model. a neural network consists of a set of highly interconnected entities. The development of more-powerful networks. x2 .

which we will discuss later. Applied to U. An example of a neural network structure... represent the weight vector corresponding to the input vector X .
Figure 2.
For a simple linear network. these weighted inputs produce a net sum at U given by S = ∑ w i xi = W ⋅ V . will allow the weights to be modified dynamically. w2 .. Figure 2 shows one example of a possible neural network structure. The output at unit U is in turn a function of A. wn ) . so that f(cS)=cf(S). There may also be units with external inputs and/or outputs. The output of one unit typically becomes an input for another. so A = f ( S ) . f is a function of only the net sum.The UMAP Journal 14(1) of that particular connection. A neural network is composed of such units and weighted unidirectional connections between them. An activation function f determines the new activation value of a unit from the net sum to the unit and the current activation value. The state of a unit U is represented by a numerical value A. Learning rules. the activation value of U.. the number of units may be in the thousands. In some neural nets. with wi real. usually taken to be just A. f(S1+S2)=f(S1)+f(S2)
. In the simplest case. Let W = ( w1 . the activation function is a linear function.

Inputs and outputs for a neural net that implements the boolean exclusives (XOR) function. the input vectors are repeatedly presented. in addition to the network topology. for which the output is the boolean exclusive-or (XOR) of the inputs. wmk ) ⎜ 2 ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ xm ⎠
So the output vector Y = ( y1 . is limited in the range of output vectors it can associate with input vectors.
Hence.… . For example. where each xi is either 0 or 1.
Table 1. an important component of most neural networks is a learning rule.The UMAP Journal 14(1)
⎛ x1 ⎞ ⎜ ⎟ x yk = ( w1k . (You can easily show that the two weights wl and w2 would have to satisfy three inconsistent linear equations. During training periods.) Implementing the XOR function is a classic problem in neural networks.y2 .… .x2). with its fixed weights. consider the set of input vectors (x1. yn ) is given by
T
Ynx 1 = Wmx 1 X mx 1
T
Learning Rules
A simple linear network. w2 k . A learning rule allows the network to adjust its connection weights in order to associate given input vectors with corresponding output vectors. No simple linear network can produce outputs as shown in Table 1. until the network learns the desired
. and the weights are adjusted according to the learning rule. as it is a subproblem of other more complicated problems.

the change in weight is
(
)
δ wij = rxi ( t j − y j )
Where
r is the learning rate. In
T
each training iteration.
where e is a constant called the learning rate.The UMAP Journal 14(1) associations. Y ( y1 .
. interference may occur and the network may not be able to learn the associations.… . McClelland and Rumelhart et al. Under the delta rule. tj is the target output. 1986]. [McClelland and Rumelhart et al. A single-layer model usually uses either the Hebb rule or the delta rule. letting the X . yn ) be the input and output vectors that we wish to associate. 1986]. Y pairs vary over the associations to be learned. A disadvantage of the Hebb rule is that if the input vectors are not mutually orthogonal. A network using the Hebb rule is guaranteed (by mathematical proof) to be able to learn associations for which the set of input vectors are orthogonal.
The delta rule changes the weight vector in a way that minimizes the error. the change δ wij in the weights is calculated as follows. In the Hebb rule. usually taken to be the reciprocal of the
number of training vectors presented.e.the initial weight vector toward the optimal one (the one that corresponds to minimum error) [McClelland and Rumelhart et al. It can be shown mathematically that the delta rule provides a very efficient way to modify . xm ) . The delta rule was developed to address the deficiencies of the Hebb rule. It is possible for a network to learn more associations with the delta rule than with the Hebb rule. until Y = W T X . It is this ability to learn that is one of the most attractive features of neural networks. a number of such iterations can be made. During the training period.… . and yj is the actual output at unit Uj.. the weights are adjusted by
δ wij = exi y j . the difference between the target output and the actual output. i. Let
X ( x1 .

Because it uses a threshold function. w12 ) for which the scalar product net sum
S = W ⋅ X = w11 x1 + w21 x2
leads to an output of 1 for input (0. and responding with output 0 otherwise. and 0 otherwise (see Table 2).0).l) or (1. net sum.
Table 2. The impossibility proof is easy. There would have to be a weight vector for which
W = ( w11 . 0). Devised by Rosenblatt. That is. the inputs (1. while those with net sum less than T lie on the other side.l) and (0. a perceptron is a single-layer network with an activation function given by
⎧1 if S > T f (S ) = ⎨ ⎩0 otherwise
where T is some constant. the line with equation wllxl + w21x2 = T divides the xlx2-plane into two regions.
But even though it uses a nonlinear activation function.l) or (1. Input vectors that produce a net sum S greater than T lie on one side of the line. the perceptron still cannot implement the XOR function.
Threshold Networks
Much early work in neural networks involved the perceptron. For the network to represent the XOR function. with sums (wl+w2)
. Inputs. as illustrated in Figure 4. a perceptron is not capable of responding with an output of 1 whenever it is presented with input vectors (0.The UMAP Journal 14(1) prove that a network using the delta rule can learn associations whenever the inputs are linearly independent [1986].0). and desired output for a perceptmn that implements the boolean exclusives (XOR) function. such a network is called a
threshold network.
Now.

in fact.1) (1. But if wl > T and w2 > T. the perceptron laid foundations for much of the later work in neural networks. While such limitations were the cause of a temporary decline of interest in the perceptron and in neural networks in general. 0) and (0. and similarly for <. must produce outputs on one side.
. 1). with sums wl and w2. be overcome by adding more layers.1) (0.The UMAP Journal 14(1)
w11x1 + w21x2 = T
(0. there is a multilayer threshold system that can represent the XOR function. while the inputs (1.
and 0. then wl + w2 > T.0)
(1. there are many other functions that cannot be represented by a single-layer network with fixed weights. as we will see in the following section. The limitations of single-layer networks can. must produce outputs on the other side.0)
Figure 4. In fact. The graph of w11x1+w21x2=T. So a perceptron cannot represent the XOR function.

By induction. Take. The input vector to the first layer is X . The layers with no external output connections are referred to as hidden layers (Figure 5). However.
Hence
Z = W2 W1 X = (W2W1 ) X
(
)
Consequently.
hidden layer
Figure 5. the output Y = W1 X of the first layer is given as input to the second layer. for example. A multilayer network. the case of a two-layer linear system. a linear system with any number n of layers is equivalent to a single-layer linear system whose weight matrix is the product of the n intermediate weight matrices. and the second layer produces output
Z = W2Y . the system is equivalent to a single-layer network with weight matrix W = W2W1.The UMAP Journal 14(1)
Multilayer Networks
A multilayer network has two or more layers of units. any multilayer system with fixed weights that has a linear activation function is equivalent to a single-layer linear system. with the output from one layer serving as input to the next.
.

The idea. no changes are made to the weights of the connections.The UMAP Journal 14(1)
+1 +1 -1
-1 +1
+1
Figure 6. Such networks can learn arbitrary associations by using differentiable activation functions. If the difference is zero. and Williams introduced the backpropagation training algorithm. similar to that of the delta rule. A multilayer system representation of the XOR function. is to adjust the weights to minimize the difference between the real output and the expected output. [1988]. the error is calculated from the delta rule and is propagated back through the network. also referred to as the generalized delta rule [1988].
. A theoretical foundation of backpropagation can be found in McClelland and Rumelhart et al. the output vector is compared to the expected output. At the output layer. If the difference is not zero. [1986] and in Rumelhart et al. In fact. Hinton.
Multilayer networks have proven to be very powerful. any boolean function can be implemented by such a network [McClelland and Rumelhart 1988].
Multilayer Networks with Learning
No learning algorithm had been available for multilayer networks until Rumelhart.

particularly in pattern recognition problems. as well as pricing information and the addresses of suppliers [1992]. A neural network accelerator board. The training of a neural network through software simulation demands intensive mathematical computation. Dewdney [1992]). Such networks include competitive learning. with its 80170 ETANN (Electronically Trainable Artificial Neural Network) chip. making it less than ideal for real-time use. McClelland and Rumelhart et al.. Many programs feature a neural-network development system that supports several different neural types. has developed an artificial retina [1989]. including some for microcomputers. and a number of commercial software packages are available. both analog and digital implementations are available.g. Carver Mead at UCLA. can provide high-speed performance. i. a leading researcher in analog neural-net chips. for which there are four major models [Wasserman 1989.e. Another alternative is a chip that implements neural networks in hardware. Two companies lead in commercialized neural network chip development: Intel. and test networks for different applications. 1986].. to allow the user to build. The network acts as a regularity detector and tries to discover structure in the patterns presented to it. Unsupervised learning implies the absence of a trainer and no knowledge beforehand of what the output should be for any given input. and Neural
. often leading to excessive training times on ordinary generalpurpose processors. backpropagation has been a widely used algorithm. Other neural network models employ unsupervised learning schemes. such as the NeuroBoard developed to support the NeuroShell package. the network is provided the expected output and trained to respond correctly. Freeman and Skapura 1991. e. All the models discussed so far use supervised learning. train. In spite of some drawbacks.The UMAP Journal 14(1) One drawback of backpropagation is its slow rate of learning. Reid and Zeichick provide a description of 50 commercial neural-network products. NeuroBoard’s speed is up to 100 times that of a 20 MHz 30386 chip with a math coprocessor.
Software and Hardware Implementation
It is relatively easy to write a program to simulate one of the networks described in the preceding sections (see.

diagnostic problems.The UMAP Journal 14(1) Semiconductor. however. given a number of different faulty states of an engine such as open plug. Marko et al. medical illnesses. Neural networks have been shown to be particularly useful in solving problems where traditional artificial intelligence techniques involving symbolic methods have failed or proved inefficient. Such networks have shown promise in problems involving low-level tasks that are computationally intensive. in the evaluation of credit card applications. speech recognition. where traditional algorithmic approaches have been ineffective. In both cases. [I990] trained a neural net to diagnose engine malfunction. for example. the chip is interfaced with a software simulation package. robotic control and computer vision. particularly when many layers are used. pattern recognition. and neural-network-based systems will become greater complements to conventional computing systems. however. including vision. The first chips with on-chip training capability should be available soon. it is extremely complicated to design a rule-based expert system to do the same diagnosis. etc. Neural nets have also been used in the banking industry. Researchers at Ford Motor Company are developing a neural-network system that diagnoses engine malfunctions. Such
. which is used for training and adjustment of weights. While an experienced technician can analyze engine malfunction given a set of data. have been concentrated in the area of pattern recognition. broken manifold. A major shortcoming of neural networks lies in the long training times that they require. with its DNNA (Digital Neural Network Architecture) chip. These chips. Neural networks. Hardware advances should diminish these limitations. with their massive parallelism. the adjusted weights are then transferred to the chip [Caudill 1991].
Applications
Neural networks have been applied to a wide variety of different areas including speech synthesis. and many other problems that fall under the category of pattern recognition. can provide the computing power needed for these problems. The trained network had a high rate of correct diagnoses. based on backpropagation. do not have the capabilities of on-chip learning. Most neural network applications.

Stubbs [I990] presents three biomedical applications in which neural networks have been used. involving the development of an Integrated Diagnostic System (IDS). The IDS is a hierarchical multilevel system. Three fault-detection algorithms have been used. Given a new design. Furthermore. The outcome of the experiment is in some cases dependent on a number of variables. Since the 1970s. one of which involves drug design. it may be able to be modified to conform to the new specifications. in efforts to help them with the production of new parts. Systems based on neural networks offer promise for a fast and reliable real-time system to help overcome these difficulties. Traditional techniques include the linear discriminant function and the analysis of covariance. Neural networks in this application allow for better performance and for the diagnosis to be accomplished in real time. as is seen in the work of Dietz et al. depending on the SSME sensor data.The UMAP Journal 14(1) nets have been used for classifying a given input into one of a number of categories and have demonstrated success. These employ statistical methods that have a high computational complexity and a low degree of reliability. Neural networks have also been used in biomedical research. If one is found. with the dependence usually a nonlinear function that is not known. be managed by neural networks. which often involves the analysis and classification of an experiment's outcomes. researchers have been developing a neural network to identify aircraft parts that have already been designed and manufactured. This work involves the development of a fault diagnostic system for the SSME that is based on three-layer backpropagation networks. [1989]. when compared to other more conventional techniques. At Boeing Aircraft Company. which integrates various fault detection algorithms to provide a monitoring system that works for all stages of operation of the SSME. work has been done on monitoring the Space Shuttle Main Engine (SSME). Such problems can. better performance is realized by parallel algorithms running on parallel architectures. the system attempts to identify a previously designed part that resembles the new one. particularly in the presence of noise. thus saving time and money in the manufacturing process. because of the parallel structure of neural networks. even with noisy input. in many cases. Non-steroidal anti-inflammatory
.

causing a temporary setback to research. The rate of adverse reactions (ADR) is about l0%. each representing a particular property of the drugs.. -------. Peter J. M. Using neural nets: Diagnostic expert nets. some overly optimistic hopes for success were not always realized. 1990. Denning.. and D. IEEE Computer 21: 77-88.
References
Carpenter. The predicted rates given by the model matched within 5% the observed rates. Caudill. which in some cases may cause adverse reactions. The science of computing: Neural networks. a solid basis of theory and applications is being formed.
Conclusion
In the early days of neural networks. AI Expert 6 (4) (April 1991): 40-45. American Scientist 80: 426-429. 1991.The UMAP Journal 14(1) drugs (NOSAIDs) are a commonly prescribed class of drugs. A. 1989. as well as to determine the properties that tend to make for "safe" drugs.K. Embedded neural networks. Today. neural networks will never replace conventional methods. and the field has begun to flourish.. 1988. Neural Networks. Algorithm: Recreational Computing 3 (4) (October-December 1992): 11-15. Jet and rocket engine fault diagnosis in real time. though.1% being fatal [Stubbs 1990]. and S. The ART of Adaptive Pattern Recognition by a Self-organizing Neural Network. Ali. G. 1992.
. with 1% of these involving serious cases and 0. Journal of Neural Network Computing (Summer 1989): 5-18. 1991. E. Dewdney. J. Computer recreations: Programming a neural net. Skapura. Freeman. but for a growing list of applications. Dietz. the neural architecture will provide either an alternative or a complement to these other techniques. Grossberg. using four inputs. and M. For some tasks. W. Reading MA: Addison-Wesley. a much better performance than by other techniques. 1992. A1 Expert 5 (9) (September 1990): 43-47. Such a neural network might be used to predict the ADR rate for new drugs. Kiech. A three-layer backpropagation neural network was developed to predict the frequency of serious ADR cases for 17 particular NOSAIDs.