A common LSTM unit is composed of a cell, an input gate, an output gate and a forget gate. The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell.

LSTM networks are well-suited to classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series. LSTMs were developed to deal with the exploding and vanishing gradient problems that can be encountered when training traditional RNNs. Relative insensitivity to gap length is an advantage of LSTM over RNNs, hidden Markov models and other sequence learning methods in numerous applications.[citation needed]

LSTM was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber.[1] By introducing Constant Error Carousel (CEC) units, LSTM deals with the exploding and vanishing gradient problems. The initial version of LSTM block included cells, input and output gates.[6]

In 1999, Felix Gers and his advisor Jürgen Schmidhuber and Fred Cummins introduced the forget gate (also called “keep gate”) into LSTM architecture,[7]
enabling the LSTM to reset its own state.[6]

In 2000, Gers & Schmidhuber & Cummins added peephole connections (connections from the cell to the gates) into the architecture.[8] Additionally, the output activation function was omitted.[6]

Among other successes, LSTM achieved record results in natural language text compression,[10] unsegmented connected handwriting recognition[11] and won the ICDAR handwriting competition (2009). LSTM networks were a major component of a network that achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset (2013).[12]

There are several architectures of LSTM units. A common architecture is composed of a cell (the memory part of the LSTM unit) and three "regulators", usually called gates, of the flow of information inside the LSTM unit: an input gate, an output gate and a forget gate. Some variations of the LSTM unit do not have one or more of these gates or maybe have other gates. For example, gated recurrent units (GRUs) do not have an output gate.

Intuitively, the cell is responsible for keeping track of the dependencies between the elements in the input sequence. The input gate controls the extent to which a new value flows into the cell, the forget gate controls the extent to which a value remains in the cell and the output gate controls the extent to which the value in the cell is used to compute the output activation of the LSTM unit. The activation function of the LSTM gates is often the logistic sigmoid function.

There are connections into and out of the LSTM gates, a few of which are recurrent. The weights of these connections, which need to be learned during training, determine how the gates operate.

In the equations below, the lowercase variables represent vectors. Matrices Wq{\displaystyle W_{q}} and Uq{\displaystyle U_{q}} contain, respectively, the weights of the input and recurrent connections, where the subscript q{\displaystyle _{q}} can either be the input gate i{\displaystyle i}, output gate o{\displaystyle o}, the forget gate f{\displaystyle f} or the memory cell c{\displaystyle c}, depending on the activation being calculated. In this section, we are thus using a "vector notation". So, for example, ct∈Rh{\displaystyle c_{t}\in \mathbb {R} ^{h}} is not just one cell of one LSTM unit, but contains h{\displaystyle h} LSTM unit's cells.

where the initial values are c0=0{\displaystyle c_{0}=0} and h0=0{\displaystyle h_{0}=0} and the operator ∘{\displaystyle \circ } denotes the Hadamard product (element-wise product). The subscript t{\displaystyle t} indexes the time step.

A peephole LSTM unit with input (i.e. i{\displaystyle i}), output (i.e. o{\displaystyle o}), and forget (i.e. f{\displaystyle f}) gates. Each of these gates can be thought as a "standard" neuron in a feed-forward (or multi-layer) neural network: that is, they compute an activation (using an activation function) of a weighted sum. it,ot{\displaystyle i_{t},o_{t}} and ft{\displaystyle f_{t}} represent the activations of respectively the input, output and forget gates, at time step t{\displaystyle t}. The 3 exit arrows from the memory cell c{\displaystyle c} to the 3 gates i,o{\displaystyle i,o} and f{\displaystyle f} represent the peephole connections. These peephole connections actually denote the contributions of the activation of the memory cell c{\displaystyle c} at time step t−1{\displaystyle t-1}, i.e. the contribution of ct−1{\displaystyle c_{t-1}} (and not ct{\displaystyle c_{t}}, as the picture may suggest). In other words, the gates i,o{\displaystyle i,o} and f{\displaystyle f} calculate their activations at time step t{\displaystyle t} (i.e., respectively, it,ot{\displaystyle i_{t},o_{t}} and ft{\displaystyle f_{t}}) also considering the activation of the memory cell c{\displaystyle c} at time step t−1{\displaystyle t-1}, i.e. ct−1{\displaystyle c_{t-1}}. The single left-to-right arrow exiting the memory cell is not a peephole connection and denotes ct{\displaystyle c_{t}}. The little circles containing a ×{\displaystyle \times } symbol represent an element-wise multiplication between its inputs. The big circles containing an S-like curve represent the application of a differentiable function (like the sigmoid function) to a weighted sum. There are many other kinds of LSTMs as well.[6]

The figure on the right is a graphical representation of an LSTM unit with peephole connections (i.e. a peephole LSTM).[29][30] Peephole connections allow the gates to access the constant error carousel (CEC), whose activation is the cell state.[31]ht−1{\displaystyle h_{t-1}} is not used, ct−1{\displaystyle c_{t-1}} is used instead in most places.

An RNN using LSTM units can be trained in a supervised fashion, on a set of training sequences, using an optimization algorithm, like gradient descent, combined with backpropagation through time to compute the gradients needed during the optimization process, in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to corresponding weight.

A problem with using gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events. This is due to limn→∞Wn=0{\displaystyle \lim _{n\to \infty }W^{n}=0} if the spectral radius of W{\displaystyle W} is smaller than 1.[33][34]

However, with LSTM units, when error values are back-propagated from the output layer, the error remains in the LSTM unit's cell. This "error carousel" continuously feeds error back to each of the LSTM unit's gates, until they learn to cut off the value.

Many applications use stacks of LSTM RNNs[35] and train them by connectionist temporal classification (CTC)[36] to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given the corresponding input sequences. CTC achieves both alignment and recognition.

There have been several successful stories of training, in a non-supervised fashion, RNNs with LSTM units.

In 2018, Bill Gates called it a “huge milestone in advancing artificial intelligence” when bots developed by OpenAI were able to beat humans in the game of Dota 2.[38] OpenAI Five consists of five independent but coordinated neural networks. Each network is trained by a policy gradient method without supervising teacher and contains a single-layer, 1024-unit Long-Short-Term-Memory that sees the current game state and emits actions through several possible action heads.[38]

In 2018, OpenAI also trained a similar LSTM by policy gradients to control a human-like robot hand that manipulates physical objects with unprecedented dexterity.[39]

In 2019, DeepMind's program AlphaStar used a deep LSTM core to excel at the complex video game Starcraft.[40] This was viewed as significant progress towards Artificial General Intelligence.[40]