esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.37

LSTM

A memory that knows what to keep and what to forget: the cell state, the three gates, and why the early notes survive long sequences.

15 min readFree

A memory that can choose

The last lesson, Recurrent Neural Networks, ended with a promise: next, LSTM, a memory that knows what to keep and what to forget. Here it is.

Remember the problem. A plain recurrent network rewrites its notes at every single step. We ran a tiny one in Python and watched a signal from step 1 shrink to about one in 730,000 by step 20. The network could not hold on to what it had read. Learning had the same problem in reverse, because the blame signal also faded on its way back.

The long short-term memory network, or LSTM, gives a network that same choice. By the end you will know what the cell state is, what the three gates do, how the update works, and you will watch one tiny LSTM remember a number for 20 steps.

Key words

LSTM
Long short-term memory. A kind of recurrent network with a protected memory and gates that control it.
Cell state
The long-term memory line that runs through the whole chain of steps.
Gate
A small control that decides how much of something is let through, from nothing to everything.
Sigmoid
A bend that turns any number into a number between 0 and 1. It is the dial behind every gate.
Tanh
A bend that turns any number into a number between -1 and 1.
Candidate memory
A proposed new note, made from the current input, that might be added to the cell state.

The cell state: a conveyor belt

Christopher Olah, in his well-known essay on LSTMs, says the key to LSTMs is the cell state, the horizontal line running through the top of the diagram. He describes it this way: the cell state is kind of like a conveyor belt. It runs straight down the entire chain, with only some minor linear interactions. It is very easy for information to just flow along it unchanged.

Picture a luggage belt at an airport. Bags ride along without being touched. A worker beside the belt can lift a bag off, or place a new one on. Nothing else happens to the bags. That is the cell state: a belt of memory that carries information from step to step, while the gates act as the workers.

Tip

Hold on to this picture: the belt carries memory, and the gates decide what gets taken off it, put on it, and read from it.

Gates are valves between 0 and 1

Olah says gates are a way to optionally let information through. They are composed out of a sigmoid neural net layer and a pointwise multiplication operation. The sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means let nothing through, while a value of one means let everything through.

Here is the sigmoid next to tanh, for a few input scores. A score is just a number the network works out before the bend. Look at how sigmoid never leaves the range 0 to 1, and tanh never leaves -1 to 1.

Python
import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

for z in (-6, -2, 0, 2, 6):
    print("score", z, "-> sigmoid", round(sigmoid(z), 3), "| tanh", round(math.tanh(z), 3))

Output

score -6 -> sigmoid 0.002 | tanh -1.0
score -2 -> sigmoid 0.119 | tanh -0.964
score 0 -> sigmoid 0.5 | tanh 0.0
score 2 -> sigmoid 0.881 | tanh 0.964
score 6 -> sigmoid 0.998 | tanh 1.0

A big negative score gives a gate close to 0, which is shut. A big positive score gives a gate close to 1, which is open. Zero gives a half-open gate, 0.5. The network learns the scores, so it learns when to open and when to close.

The three gates

Wikipedia says an LSTM unit is typically composed of a cell and three gates: a forget gate, an input gate and an output gate. The cell remembers values over arbitrary time intervals, and the gates regulate the flow of information into and out of the cell.

  • Forget gate: what to throw away

    It looks at the new input and the last output, and gives a number from 0 to 1 for each part of the memory. Wikipedia: a value of 1 signifies retention of the information, and a value of 0 represents discarding.

  • Input gate: what to write

    It decides which pieces of new information to store in the current cell state, using the same 0-to-1 system as the forget gate.

  • Output gate: what to show

    It controls which pieces of the current cell state to output. The memory can hold more than the cell needs to say right now.

Olah gives a daily-life example from language. A language model predicting the next word may keep the gender of the present subject in its cell state, so it can use the right pronouns. When a new subject appears, the forget gate wipes the old gender, and the input gate writes the new one.

One step, start to finish

Here is everything that happens inside one LSTM cell at one time step, in the order Olah walks through it.

  1. 1

    Forget gate

    Look at the new input and the last output. For each number on the belt, decide how much to keep. 1 means keep it all, 0 means drop it.

  2. 2

    Input gate

    A second sigmoid decides which values to update.

  3. 3

    Candidate memory

    A tanh layer makes a list of new candidate values that could be added to the belt. Tanh keeps them between -1 and 1.

  4. 4

    Update the belt

    Multiply the old belt by the forget gate. Then add the candidate values scaled by the input gate.

  5. 5

    Output gate

    A third sigmoid decides which parts of the belt to show. Push the belt through tanh, multiply by that gate, and the result is the output of this step.

Notice that step 4 is the whole trick. The old memory is not rewritten from scratch. It is scaled, then something is added. The next section shows it as a formula.

Quick check

At one step the forget gate is 0 for part of the memory. What happens to that part?

The update, as equations

Wikipedia gives the equations for an LSTM cell with a forget gate. The letter x is the input at step t, h is the output, c is the cell state, and the letters f, i and o are the three gates. Each W and U is a set of weights, each b is a small offset, and the sigma sign means the sigmoid.

ft=σ(Wfxt+Ufht−1+bf)it=σ(Wixt+Uiht−1+bi)ot=σ(Woxt+Uoht−1+bo)c~t=tanh⁡(Wcxt+Ucht−1+bc)ct=ft⊙ct−1+it⊙c~tht=ot⊙tanh⁡(ct)\begin{aligned} f_t &= \sigma\left(W_f x_t + U_f h_{t-1} + b_f\right) \\ i_t &= \sigma\left(W_i x_t + U_i h_{t-1} + b_i\right) \\ o_t &= \sigma\left(W_o x_t + U_o h_{t-1} + b_o\right) \\ \tilde{c}_t &= \tanh\left(W_c x_t + U_c h_{t-1} + b_c\right) \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\ h_t &= o_t \odot \tanh\left(c_t\right) \end{aligned}

In words, line by line. The first three lines say each gate is a sigmoid of the new input plus the old output, so each gate is a number between 0 and 1. The fourth line says the candidate memory is a tanh of the same ingredients, so it is a number between -1 and 1.

The fifth line is the belt update: the new cell state equals the forget gate times the old cell state, plus the input gate times the candidate. The little circle with a dot means multiply, number by number. The last line says the output is the output gate times tanh of the new cell state.

Why the memory survives

Look again at the fifth line. The new memory is the old memory times a gate, plus something new. A plain RNN takes its old memory and pushes it through a bend, mixed with the input, every step. In an LSTM the old memory rides the belt, and a gate with a value near 1 lets almost all of it pass.

Wikipedia states the effect, with a limit: RNNs using LSTM units partially solve the vanishing gradient problem, because LSTM units allow gradients to also flow with little to no attenuation. Attenuation means getting weaker. LSTM networks can still suffer from the exploding gradient problem.

It describes the same thing during training. When error values are back-propagated from the output layer, the error remains in the LSTM unit's cell. This "error carousel" continuously feeds error back to each of the LSTM unit's gates, until they learn to cut off the value. In plain terms, the learning signal can ride the belt backward too, instead of being squeezed at every step.

What does this buy? Wikipedia says LSTM aims to provide a short-term memory for an RNN that can last thousands of timesteps, which is where the name "long short-term memory" comes from. It also says LSTM has a relative insensitivity to gap length, which is its advantage over other RNNs.

One LSTM cell in plain Python

Time to watch it work. The code below is one LSTM cell with a single memory number. Every weight is picked by hand, not learned, so you can see why each gate behaves as it does.

Each step gets two inputs: a value, and a write flag that is 1 only on the first step. The weights are set so that the forget gate sits near 1 (its offset is +4), the input gate opens only when the write flag is on (offset -4, and a strong weight of +8 on the flag), and the output gate is mostly open. The candidate memory is tanh of twice the value.

Step 1 carries the number to remember, 0.9, with the flag on. Steps 2 to 20 carry other numbers (0.5, -0.7, 0.3, 0.8, -0.4, repeating) with the flag off. The table shows each gate and the cell state at steps 1, 2, 3, 5, 10 and 20.

Python
import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

# Hand-picked weights: (weight on value, weight on write flag, weight on old h, bias)
W_FORGET = (0.0, 0.0, 0.0, 4.0)     # bias 4 -> the forget gate sits near 1
W_INPUT  = (0.0, 8.0, 0.0, -4.0)    # opens only when the write flag is 1
W_OUTPUT = (0.0, 0.0, 0.0, 4.0)     # output gate mostly open
W_CAND   = (2.0, 0.0, 0.0, 0.0)     # candidate memory = tanh(2 * value)

def pre(w, value, flag, h):
    return w[0] * value + w[1] * flag + w[2] * h + w[3]

def lstm_step(c, h, value, flag):
    f = sigmoid(pre(W_FORGET, value, flag, h))
    i = sigmoid(pre(W_INPUT, value, flag, h))
    o = sigmoid(pre(W_OUTPUT, value, flag, h))
    cand = math.tanh(pre(W_CAND, value, flag, h))
    c = f * c + i * cand
    h = o * math.tanh(c)
    return f, i, o, cand, c, h

# Step 1 carries the number to remember (0.9) with the write flag on.
# Steps 2-20 carry other numbers and the write flag off.
values = [0.9] + [0.5, -0.7, 0.3, 0.8, -0.4] * 4
values = values[:20]
flags = [1] + [0] * 19

c, h = 0.0, 0.0
print("step  forget  input  output  cand    cell")
for t, (v, fl) in enumerate(zip(values, flags), start=1):
    f, i, o, cand, c, h = lstm_step(c, h, v, fl)
    if t in (1, 2, 3, 5, 10, 20):
        print(f"{t:>4}  {f:.3f}   {i:.3f}  {o:.3f}   {cand:+.3f}  {c:+.4f}")

Output

step  forget  input  output  cand    cell
   1  0.982   0.982  0.982   +0.947  +0.9298
   2  0.982   0.018  0.982   +0.762  +0.9268
   3  0.982   0.018  0.982   -0.885  +0.8942
   5  0.982   0.018  0.982   +0.922  +0.8883
  10  0.982   0.018  0.982   +0.922  +0.8239
  20  0.982   0.018  0.982   +0.922  +0.7112

Read the table. At step 1 the input gate is open, 0.982, and the candidate is 0.947, so the cell state jumps to 0.9298. That is the number being written onto the belt.

From step 2 on, the input gate drops to 0.018, almost shut. The candidates keep changing, +0.762, -0.885, +0.922, but they barely get in. The forget gate stays at 0.982, almost fully open for the old memory, so the belt keeps nearly all of what it holds.

The memory fades a little each step, by the factor 0.982, plus a small trickle from the new inputs. At step 20 the cell state is still 0.7112, against 0.9298 at the start. The first number is still there.

LSTM against the plain RNN

Run the sequence twice, once with 0.9 as the first number and once with 0, all later steps the same (0.5). The difference is the lasting effect of the first number. We do this for the LSTM cell above and for a plain RNN with one memory number, one weight of 1.0 on the input, and 0.5 on the old memory, the same style as before.

Python
import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

def lstm_cell_after(first, steps):
    # Same hand-picked weights as before, but only the cell state is tracked.
    c = 0.0
    for t in range(steps):
        value = first if t == 0 else 0.5
        flag = 1 if t == 0 else 0
        f = sigmoid(4.0)
        i = sigmoid(8.0 * flag - 4.0)
        c = f * c + i * math.tanh(2.0 * value)
    return c

def rnn_after(first, steps):
    h = 0.0
    for t in range(steps):
        value = first if t == 0 else 0.5
        h = math.tanh(1.0 * value + 0.5 * h)
    return h

print("steps  LSTM cell effect  plain RNN effect")
for n in (1, 5, 10, 20):
    lstm = lstm_cell_after(0.9, n) - lstm_cell_after(0.0, n)
    rnn = rnn_after(0.9, n) - rnn_after(0.0, n)
    print(f"{n:>5}  {lstm:>16.4f}  {rnn:>16.2e}")
print("forget gate:", round(sigmoid(4.0), 4), "| after 19 forget steps it keeps:", round(sigmoid(4.0) ** 19, 4))

Output

steps  LSTM cell effect  plain RNN effect
    1            0.9298          7.16e-01
    5            0.8647          4.71e-03
   10            0.7897          5.99e-06
   20            0.6586          9.63e-12
forget gate: 0.982 | after 19 forget steps it keeps: 0.7083

The last printed line adds one more number: with a forget gate of 0.982, nineteen steps of forgetting keep 0.7083 of the memory.

Compare the two columns. The plain RNN's memory of the first number is 0.716 at step 1, about 0.0047 at step 5, and 9.63e-12 at step 20, which is 0.00000000000963. It is gone. The LSTM cell still carries 0.6586 at step 20. About two thirds of the first number survives, because the forget gate multiplies it by 0.982 each time instead of squashing it.

Tip

These weights were set by hand to show the idea. A real LSTM learns its gate settings from data. The first number is also easy to track here because only step 1 opens the input gate. Real memories are messier.

Quick check

Which gate setting best keeps an old memory safe over many steps?

Who made it, and when

Wikipedia traces the story. Sepp Hochreiter's 1991 German diploma thesis analyzed the vanishing gradient problem and developed principles of the method. His supervisor, Jurgen Schmidhuber, considered the thesis highly significant.

An early version of LSTM was published in 1995 in a technical report by Hochreiter and Schmidhuber, then published in the NIPS 1996 conference. The most commonly used reference point was published in 1997 in the journal Neural Computation. Olah agrees that LSTMs were introduced by Hochreiter and Schmidhuber (1997), and were refined and popularized by many people in following work.

By introducing Constant Error Carousel (CEC) units, Wikipedia says, LSTM deals with the vanishing gradient problem. That is the "error carousel" from earlier.

One surprise for such a famous idea: the first version had no forget gate. Wikipedia says the initial version of the LSTM block included cells, input and output gates. Felix Gers, Jurgen Schmidhuber and Fred Cummins introduced the forget gate in 1999, enabling the LSTM to reset its own state. This is the most commonly used version of LSTM nowadays, and it is the one in this lesson.

  • 1991

    Hochreiter's thesis studies the vanishing gradient problem and develops principles of the method.

  • 1997

    Hochreiter and Schmidhuber publish the most commonly used reference for LSTM, with cells, input gates and output gates.

  • 1999

    Gers, Schmidhuber and Cummins add the forget gate. This is the version most used today.

Where LSTMs were used

Wikipedia says LSTM has wide applications, including time series analysis, speech recognition and machine translation. Its list of applications also includes handwriting recognition.

It also names two products. In 2015, Google started using an LSTM trained by CTC for speech recognition on Google Voice. In 2016, Google started using an LSTM to suggest messages in the Allo conversation app. CTC, short for Connectionist Temporal Classification, is a way of training that lines up a sequence of sounds with a sequence of letters.

Quick check

In the update c = f x old c + i x candidate, what does a forget gate near 1 do?

Next: GRU, a lighter cousin of the LSTM.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.