Data Fitting, Clustering, and Pattern Recognition

Like its counterpart in the biological nervous system, a neural network can learn and therefore can be trained to find solutions, recognize patterns, classify data, and forecast future events. The behavior of a neural network is defined by the way its individual computing elements are connected and by the strengths of those connections, or weights. The weights are automatically adjusted by training the network according to a specified learning rule until it performs the desired task correctly.

Neural Network Toolbox includes command-line functions and apps for creating, training, and simulating neural networks. The apps make it easy to develop neural networks for tasks such as data-fitting (including time-series data), pattern recognition, and clustering. After creating your networks in these tools, you can automatically generate MATLAB® code to capture your work and automate tasks.

Network Architectures

Neural Network Toolbox supports a variety of supervised and unsupervised network architectures. With the toolbox’s modular approach to building networks, you can develop custom network architectures for your specific problem. You can view the network architecture including all inputs, layers, outputs, and interconnections.

Dynamic networks use memory and recurrent feedback connections to recognize spatial and temporal patterns in data. They are commonly used for time-series prediction, nonlinear dynamic system modeling, and control systems applications. Prebuilt dynamic networks in the toolbox include focused and distributed time-delay, nonlinear autoregressive (NARX), layer-recurrent, Elman, and Hopfield networks. The toolbox also supports dynamic training of custom networks with arbitrary connections.

Competitive layers recognize and group similar input vectors, enabling them to automatically sort inputs into categories. Competitive layers are commonly used for classification and pattern recognition.

Self-organizing maps learn to classify input vectors according to similarity. Like competitive layers, they are used for classification and pattern recognition tasks; however, they differ from competitive layers because they are able to preserve the topology of the input vectors, assigning nearby inputs to nearby categories.

A self-organizing map consisting of a competitive layer that can classify a data set of vectors with any number of dimensions into as many classes as the layer has neurons.

Training Algorithms

Training and learning functions are mathematical procedures used to automatically adjust the network's weights and biases. The training function dictates a global algorithm that affects all the weights and biases of a given network. The learning function can be applied to individual weights and biases within a network.

Neural Network Toolbox supports a variety of training algorithms, including several gradient descent methods, conjugate gradient methods, the Levenberg-Marquardt algorithm (LM), and the resilient backpropagation algorithm (Rprop). The toolbox’s modular framework lets you quickly develop custom training algorithms that can be integrated with built-in algorithms. While training your neural network, you can use error weights to define the relative importance of desired outputs, which can be prioritized in terms of sample, time step (for time-series problems), output element, or any combination of these. You can access training algorithms from the command line or via apps that show diagrams of the network being trained and provide network performance plots and status information to help you monitor the training process.

A suite of learning functions, including gradient descent, Hebbian learning, LVQ, Widrow-Hoff, and Kohonen, is also provided.

Improved Generalization

Improving the network’s ability to generalize helps prevent overfitting, a common problem in neural network design. Overfitting occurs when a network has memorized the training set but has not learned to generalize to new inputs. Overfitting produces a relatively small error on the training set but a much larger error when new data is presented to the network.

Regularization modifies the network’s performance function (the measure of error that the training process minimizes). By including the sizes of the weights and biases, regularization produces a network that performs well with the training data and exhibits smoother behavior when presented with new data.

Early stopping uses two different data sets: the training set, to update the weights and biases, and the validation set, to stop training when the network begins to overfit the data.

Simulink Blocks and Control Systems Applications

Simulink Support

Neural Network Toolbox provides a set of blocks for building neural networks in Simulink. All blocks are compatible with Simulink Coder™. These blocks are divided into four libraries:

Transfer function blocks, which take a net input vector and generate a corresponding output vector

Net input function blocks, which take any number of weighted input vectors, weight-layer output vectors, and bias vectors, and return a net input vector

Weight function blocks, which apply a neuron's weight vector to an input vector (or a layer output vector) to get a weighted input value for a neuron

Data preprocessing blocks, which map input and output data into the ranges best suited for the neural network to handle directly

Alternatively, you can create and train your networks in the MATLAB environment and automatically generate network simulation blocks for use with Simulink. This approach also enables you to view your networks graphically.

Control Systems Applications

You can apply neural networks to the identification and control of nonlinear systems. The toolbox includes descriptions, examples, and Simulink blocks for three popular control applications:

Model predictive control, which uses a neural network model to predict future plant responses to potential control signals. An optimization algorithm then computes the control signals that optimize future plant performance. The neural network plant model is trained offline and in batch form.

Feedback linearization, which uses a rearrangement of the neural network plant model and is trained offline. This controller requires the least computation of these three architectures; however, the plant must either be in companion form or be capable of approximation by a companion form model.

Model reference adaptive control, which requires that a separate neural network controller be trained offline, in addition to the neural network plant model. While the controller training is computationally expensive, the model reference control applies to a larger class of plant than feedback linearization.

You can incorporate neural network predictive control blocks included in the toolbox into your Simulink models. By changing the parameters of these blocks, you can tailor the network's performance to your application.

Accelerated Training and Large Data Sets

You can speed up neural network training and simulation of large data sets by using Neural Network Toolbox with Parallel Computing Toolbox. Training and simulation involve many parallel computations, which can be accelerated with multicore processors, CUDA-enabled NVIDIA GPUs, and computer clusters with multiple processors and GPUs.

Distributed Computing

Parallel Computing Toolbox lets neural network training and simulation run across multiple processor cores on a single PC, or across multiple processors on multiple computers on a network using MATLAB Distributed Computing Server™. Using multiple cores can speed up calculations. Using multiple computers lets you solve problems using data sets too big to fit within the system memory of any single computer. The only limit to problem size is the total system memory available across all computers.

GPU Computing

Parallel Computing Toolbox enables Neural Network Toolbox simulation and training to be parallelized across the multiprocessors and cores of a general-purpose graphics processing unit (GPU). GPUs are highly efficient on parallel algorithms such as neural networks. You can achieve higher levels of parallelism by using multiple GPUs or by using GPUs and processors together. With MATLAB Distributed Computing Server, you can harness all the processors and GPUs on a network cluster of computers for neural network training and simulation.