Machine Learning Foundations / Advanced / 03.36
Recurrent Neural Networks
Models with a memory: how a network reads data in order, carries notes from step to step, and why its memory fades over long stretches.
14 min readFree
Models with a memory
The last lesson, Image Segmentation, ended with a promise: next, recurrent neural networks, models with a memory. Here they are.
Why would a network need a memory? Look at the networks you have met so far. A convolutional network looks at one photo and gives an answer. The next photo starts fresh. The network does not care what it saw a second ago. For photos, that is fine.
Now read this: "The dog bit the man." And this: "The man bit the dog." Same words, very different news. The order is the meaning. To understand a sentence, you have to remember what came before the word you are reading right now.
Recurrent neural networks, or RNNs, were made for that job. By the end of this lesson you will know what the memory is, how it is passed along, how the network learns, and why the memory is weaker than you might hope.
Key words
- Sequence
- A list of things where the order matters, such as words in a sentence, daily prices or readings from a sensor.
- Time step
- One position in a sequence. Reading one word is one step. Reading the next word is the next step.
- Hidden state
- The network's memory. A small list of numbers that it updates at every step and carries to the next.
- Recurrent
- Running back on itself. The output of one step is fed back in at the next step.
- Unrolling
- Drawing the loop as a chain, with one copy of the network for each time step.
- Gradient
- How much the loss changes if you nudge one weight. It is that weight's share of the blame.
Data that comes in order
Wikipedia says RNNs are designed for processing sequential data, such as text, speech and time series, where the order of elements is important. Here are three everyday kinds:
Sentences
Words arrive one after another. The meaning of "it" in a sentence depends on a noun you read three words ago.
Sensor readings
A watch reports your heart rate every second. One reading means little. The pattern over a minute tells you whether you are resting or running.
Prices
A share price today is read against the days before it. Going from 10 to 12 is not the same as going from 20 to 12.
Tip
A quick test for "is this a job for an RNN?": shuffle the items. If shuffling destroys the meaning (words, music, prices over time), the data is a sequence.
The memory: a hidden state
Wikipedia calls the building block of an RNN the recurrent unit. It maintains a hidden state, a form of memory that is updated at each time step based on the current input and the previous hidden state.
Think of a student taking notes in a long lecture. At each moment, two things decide what goes on the page: what the teacher just said, and what is already written. The student does not rewrite everything. They update the notes. The hidden state is those notes.
The Wikipedia article puts it in one line: the hidden vector plays the role of "memory", a partial record of all previous input-output pairs. The word partial matters. The notes are short. They cannot hold everything.
Here is the update as a formula. A vector is just a list of numbers. The letter h is the hidden state, x is the input, and t is the step number.
In words: the new memory at step t comes from two ingredients. The first is the new input, scaled by a weight. The second is the old memory, scaled by another weight. Add them, add a small offset b, then squash the result with tanh. Tanh is a bend that keeps every number between -1 and 1, so the notes never grow without limit.
The same weights at every step
This is the idea that makes RNNs work, and it is easy to miss. The weights W_x and W_h do not change from step to step. Step 1 and step 100 use exactly the same ones.
A teacher grading a stack of essays uses one marking guide for every essay. She does not invent a new guide for essay 57. In the same way, an RNN has one small set of weights and applies it again and again.
Karpathy gives the reason this matters: there are no pre-specified limits on the length of the sequences, because the recurrent transformation is fixed and can be applied as many times as we like. A sentence of 5 words and a sentence of 500 words use the same machine.
Unrolling the loop
An RNN is often drawn as a box with an arrow that loops back into itself. The loop is the memory. That picture is neat, but it hides what happens over time.
To see it, unroll the loop. Draw the same box once for every time step, left to right, and pass the memory along from each box to the next. Wikipedia's article shows both pictures side by side: a compressed network on the left and an unfolded one on the right.
- 1
Start with empty notes
Before the first step, the hidden state is all zeros.
- 2
Read the first item
The first box takes the first input and the empty notes, and makes the first memory.
- 3
Pass the notes on
That memory goes into the second box, together with the second input.
- 4
Repeat to the end
Each box is the same box with the same weights. Only the input and the incoming notes are different.
Wikipedia adds a warning about the usual drawing. It may be misleading, because practical networks are often organised in layers, and the drawing gives that appearance. What looks like layers are in fact different steps in time, unfolded to produce the appearance of layers.
So an RNN that reads 20 words behaves like a 20-layer network in which every layer shares the same weights. Keep that picture in mind. It comes back when we talk about training.
Shapes of tasks
Because the loop can run for any number of steps, an RNN can be wired to different kinds of jobs. Karpathy lists the main ones. Here they are with names and daily-life examples.
One to many: one input, a sequence out
Karpathy's example is image captioning. The network takes one image and outputs a sentence of words.
Many to one: a sequence in, one answer out
Karpathy's example is sentiment analysis. A sentence goes in, and the network says whether it sounds positive or negative.
Many to many: a sequence in, a sequence out
Machine translation is the classic case. The network reads a sentence in English, then outputs a sentence in French. A second type labels every step as it goes, such as tagging each frame of a video.
Quick check
A network reads a whole movie review and outputs "positive" or "negative". Which shape is that?
A tiny RNN in plain Python
Time to watch the memory move. The code below is a tiny RNN with one memory number and two fixed weights. They were picked by hand, not learned. The weight w_x says how much the new input counts. The weight w_h says how much the old memory counts.
It reads the sequence 1, 0, 0, 2, 0 and prints the memory after each step. It uses only Python's built-in math module.
import math
w_x = 1.0 # how strongly the new input counts
w_h = 0.5 # how strongly the old memory counts
def step(h, x):
return math.tanh(w_x * x + w_h * h)
h = 0.0 # empty memory before the first step
for t, x in enumerate([1.0, 0.0, 0.0, 2.0, 0.0], start=1):
h = step(h, x)
print("step", t, "input", x, "memory", round(h, 4))Output
step 1 input 1.0 memory 0.7616 step 2 input 0.0 memory 0.3634 step 3 input 0.0 memory 0.1797 step 4 input 2.0 memory 0.9699 step 5 input 0.0 memory 0.4502
Read it line by line. At step 1 the input is 1 and the memory jumps to 0.7616. At steps 2 and 3 the input is 0, so nothing new arrives, yet the memory is not zero. It is 0.3634, then 0.1797. The network still remembers the 1, but the memory is getting weaker.
Tip
Notice that the function step is the same at every step. That is the shared-weights idea, written in four lines.
How an RNN learns
You met backpropagation in an earlier lesson: the network makes a guess, the loss says how wrong it is, and the blame is passed backward, so every weight learns its share. An RNN learns the same way, with one twist. The twist is the unrolled chain.
The method is called backpropagation through time, or BPTT. Wikipedia describes it as a gradient-based technique for training certain types of recurrent neural networks. It unfolds the network through time. The unfolded network contains one copy for each input and output, but every copy of the network shares the same parameters. Then the usual backpropagation finds the gradient of the loss with respect to all the network's parameters.
There is one more detail. Since every copy shares the same weights, the weight updates in each copy are summed together. The weight W_h helped at step 1, step 2 and step 3, so it gets the combined blame from all of them. Think of three people on a shared project. The one shared task gets feedback from all three.
- 1
Unroll
Copy the network once for every step in the stretch you are training on.
- 2
Go forward
Read the sequence left to right, passing the memory along, and make guesses.
- 3
Measure
Work out the loss from the guesses. Wikipedia says the total cost is the average of the costs of each time step.
- 4
Go back
Pass the blame from the last step back to the first, through every copy.
- 5
Add and update
Sum the blame for each shared weight, then nudge the weights once.
Long sequences make this costly, so Wikipedia's pseudocode shows a truncated version, where the network is unfolded for only k time steps at a time. Truncated means cut short: you look back only so far.
Why the memory fades
Here is the catch. In the Backpropagation lesson you saw that passing blame back through many layers means multiplying many small numbers, and the blame shrinks. An unrolled RNN is exactly a very deep network.
Wikipedia's article on the vanishing gradient problem says that Hochreiter's 1991 thesis formally identified the problem, and that it affects recurrent networks too. Such networks are trained by unfolding them into very deep feedforward networks, where a new layer is created for each time step of an input sequence.
Wikipedia explains the general effect: the gradients of earlier weights are calculated with increasingly many multiplications, and these multiplications shrink the gradient. Consequently the gradients of earlier weights will be exponentially smaller than the gradients of later weights.
Let us see it in our tiny RNN. We give it a signal at step 1 only, and zeros after that. Then we compare the memory with that signal to the memory without it. The difference is the lasting effect of step 1.
import math
w_x = 1.0
w_h = 0.5
def run(first, steps):
h = 0.0
for t in range(steps):
x = first if t == 0 else 0.0 # only step 1 gets a signal
h = math.tanh(w_x * x + w_h * h)
return h
# The effect of step 1 = memory with the signal minus memory without it.
for n in (1, 2, 3, 5, 10, 20):
effect = run(1.0, n) - run(0.0, n)
print("after", n, "steps: effect of step 1 =", effect)Output
after 1 steps: effect of step 1 = 0.7615941559557649 after 2 steps: effect of step 1 = 0.36339948438905245 after 3 steps: effect of step 1 = 0.1797262071203191 after 5 steps: effect of step 1 = 0.044781024972252656 after 10 steps: effect of step 1 = 0.0013990956166748465 after 20 steps: effect of step 1 = 1.3663040159929856e-06
After 1 step the signal is strong, 0.76. After 5 steps it is about 0.045. After 10 steps it is about 0.0014. After 20 steps it is 1.37e-06, which means 0.00000137, about one in 730,000 of where it began. The network has all but forgotten step 1.
Learning has the same problem in reverse. To learn that a word at step 1 matters for the answer at step 20, the blame has to travel back 19 steps. By the time it arrives, almost none is left, so the network cannot learn the link. This is what Wikipedia means when it says traditional RNNs suffer from the vanishing gradient problem, which limits their ability to learn long-range dependencies.
The opposite problem: exploding
Repeated multiplication can go the other way. Wikipedia says the inverse problem, when weight gradients at earlier layers get exponentially larger, is called the exploding gradient problem. The size of the factor decides which way it goes:
# Walking the blame back through the steps multiplies by about the same number each time.
for factor in (0.5, 1.0, 1.5):
print("factor", factor, "-> after 20 steps:", factor ** 20)Output
factor 0.5 -> after 20 steps: 9.5367431640625e-07 factor 1.0 -> after 20 steps: 1.0 factor 1.5 -> after 20 steps: 3325.256730079651
Twenty multiplications by 0.5 leave almost nothing. Twenty by 1.5 give more than 3,300. Only a factor of exactly 1 keeps things steady, and real training will not stay there on its own. This is a simplified picture: in a real network the factor changes at every step. But the shrink-or-blow-up pattern is the same.
For the exploding case, Wikipedia says Pascanu and colleagues in 2012 recommended gradient clipping: if the gradient vector is longer than a chosen limit, scale it back down so it stays inside that limit. It is like a speed limiter on a car.
Quick check
Why is a long sequence harder for an RNN than a short one?
Why LSTM and GRU were made
Wikipedia says the vanishing gradient problem was addressed by the development of the long short-term memory (LSTM) architecture in 1997, making it the standard RNN variant for handling long-term dependencies. It also says LSTM can learn tasks that require memories of events that happened thousands or even millions of discrete time steps earlier. Later, gated recurrent units (GRUs) were introduced as a more computationally efficient alternative.
The shared idea is a gate, a small control that decides what to keep and what to drop, instead of rewriting the whole memory at every step. That is the next lesson.
The plain RNN
Rewrites its notes at every step. Simple, but the early notes fade fast.
LSTM
Adds gates, including a forget gate. Made in 1997 so that learning signals do not vanish or explode as they travel back.
GRU
A lighter gated design, introduced later as a more efficient alternative to the LSTM.
Where RNNs stand today
RNNs have done real work. Wikipedia lists tasks they have been applied to: unsegmented, connected handwriting recognition, speech recognition, natural language processing and neural machine translation.
But the field has moved. Wikipedia says that in recent years, transformers, which rely on self-attention mechanisms instead of recurrence, have become the dominant architecture for many sequence-processing tasks, particularly in natural language processing, due to their superior handling of long-range dependencies and greater parallelizability. Parallelizable means many steps can be worked out at once, not one after another.
The same article adds that RNNs remain relevant for applications where computational efficiency, real-time processing, or the inherent sequential nature of data is crucial. Learning them is still worth it. The ideas of a hidden state, shared weights, and learning through an unrolled chain show up all over deep learning.
Quick check
What does an RNN reuse at every time step?
Next: LSTM, a memory that knows what to keep and what to forget.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.