Machine Learning Foundations / Advanced / 03.38
GRU
A lighter cousin of the LSTM: two gates instead of three, no separate memory cell, and fewer weights to learn.
13 min readFree
A lighter cousin
The last lesson, LSTM, ended with a promise: next, GRU, a lighter cousin of the LSTM. Here it is.
Quick reminder of the LSTM. Wikipedia says an LSTM unit is typically made of a cell and three gates: a forget gate, an input gate and an output gate. The cell remembers values over time, and the gates control what goes in and what comes out. It works well, but it has a lot of moving parts.
The GRU asks a simple question: can we keep the gating idea and drop some of the parts? Wikipedia describes it as like an LSTM, with a gating mechanism to input or forget certain features, but it lacks a context vector or output gate. Fewer parts means fewer numbers to learn. By the end you will know what the two gates do and how many weights you save.
Key words
- Gate
- A small control that outputs a number between 0 and 1 for each item in the memory. Near 0 means block it. Near 1 means let it through.
- Hidden state
- The network's memory: a short list of numbers updated at every step and carried to the next.
- Update gate
- Decides how much of the old notes to keep and how much to swap for new ones.
- Reset gate
- Decides how much of the old notes to look at while drafting new ones.
- Candidate
- A draft of the new notes, written before the update gate decides how much of it to accept.
- Parameters
- All the numbers a model learns: its weights and biases. This lesson also calls them weights for short.
Two gates and one memory
Picture a student keeping notes in a long lecture. At every moment there is a page of notes. When the teacher says something new, the student has two choices to make.
Update gate
How much of the page do I replace? Replace nothing and the notes stay as they were. Replace everything and the page is fresh. Most of the time it is a mix.
Reset gate
When I draft the new line, how much of the old page do I look at? Look at all of it and the draft leans on old notes. Look at none and the draft comes only from what was just said.
One memory
There is only one thing to carry from step to step: the hidden state. There is no second memory cell next to it, and no output gate deciding what to show.
That is the whole GRU. The LSTM has a cell, a hidden state and three gates. The GRU folds them into one list of numbers with two gates.
Tip
The word "gated" in the name means exactly this: the unit uses gates, small controls that open and close, to manage its memory. A gated recurrent unit is a recurrent unit with gates.
Who made it, and when
Wikipedia says the GRU was introduced in 2014 by Kyunghyun Cho and colleagues. It cites the paper "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation", by Cho and co-authors including Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio. It was presented at EMNLP 2014, a conference on language processing.
The abstract of that paper says it proposes an RNN Encoder-Decoder: one recurrent network reads a sequence of symbols and squeezes it into a fixed-length vector, and a second network decodes that vector into another sequence. The authors tested it as an extra helper inside a statistical machine translation system, and found that it improved the translation performance.
One step, in four moves
Here is what a GRU does each time a new input arrives. It has the input for this step (call it x) and the notes from the last step (call them h, for hidden state).
- 1
Compute the update gate
Mix the new input and the old notes, then squash the result into the range 0 to 1. Each number in the gate says how much of that slot to replace.
- 2
Compute the reset gate
Do the same with its own weights. Each number says how much of that slot of the old notes to look at.
- 3
Draft a candidate
Multiply the old notes by the reset gate, mix them with the new input, and squash the result into the range -1 to 1. This is the draft of the new notes.
- 4
Blend
The new notes are a mix of the old notes and the draft. The update gate sets the mix, slot by slot.
Wikipedia gives the equations of the fully gated unit. Here they are. Here x is the input, h is the hidden state, W and U are weight tables, b is a bias (a small constant the model also learns), and t counts the steps.
In words, line by line. The update gate z is the sigmoid (a curve that squashes any number into 0 to 1) of a weighted mix of the input and the old notes. The reset gate r is built the same way, with different weights. The candidate is the tanh (a similar curve that squashes into -1 to 1) of a mix of the input and the old notes after the reset gate has dimmed them. The symbol with a dot in a circle means multiply slot by slot. The last line says the new notes are (1 minus z) times the old notes, plus z times the candidate.
The blend line, up close
The last equation is the heart of the GRU, so let us try it with numbers. Say the old note in one slot is 0.8 and the candidate wants to write -0.5. Watch what different update gate values do.
old = 0.80 # what the notes say now
new = -0.50 # what the candidate would write
for z in (0.0, 0.25, 0.75, 1.0):
mixed = (1 - z) * old + z * new
print(f"update gate {z:.2f} -> notes become {mixed:6.3f}")Output
update gate 0.00 -> notes become 0.800 update gate 0.25 -> notes become 0.475 update gate 0.75 -> notes become -0.175 update gate 1.00 -> notes become -0.500
With the gate at 0, the old note survives untouched. With the gate at 1, the candidate wins completely. In between, the result slides from one to the other. Notice the two shares always add to 1: (1 - z) for the old and z for the new. The GRU does not have separate forget and input gates. One gate, z, does both jobs.
Run one cell
Let us run a GRU cell by hand, in plain Python with no libraries. The weights are small fixed numbers, chosen only so the example is easy to follow. A real network learns its weights from data. The input has 2 numbers per step and the hidden state has 2 numbers. We feed in a sequence of 4 steps and print the two gates and the hidden state after each.
import math
def sigmoid(x):
return 1 / (1 + math.exp(-x))
def dot(row, vec):
return sum(w * v for w, v in zip(row, vec))
# Small fixed weights: input of 2 numbers, hidden state of 2 numbers.
Wz, Uz, bz = [[0.5, -0.3], [0.2, 0.4]], [[0.1, 0.0], [0.0, 0.1]], [0.0, 0.0]
Wr, Ur, br = [[-0.4, 0.6], [0.3, 0.1]], [[0.2, 0.1], [0.1, 0.2]], [0.0, 0.0]
Wh, Uh, bh = [[0.8, -0.5], [0.4, 0.7]], [[0.3, 0.2], [0.2, 0.3]], [0.0, 0.0]
def gru_step(x, h):
z = [sigmoid(dot(Wz[i], x) + dot(Uz[i], h) + bz[i]) for i in range(2)]
r = [sigmoid(dot(Wr[i], x) + dot(Ur[i], h) + br[i]) for i in range(2)]
rh = [r[i] * h[i] for i in range(2)]
cand = [math.tanh(dot(Wh[i], x) + dot(Uh[i], rh) + bh[i]) for i in range(2)]
new = [(1 - z[i]) * h[i] + z[i] * cand[i] for i in range(2)]
return z, r, new
def show(v):
return "[" + ", ".join(f"{a:6.3f}" for a in v) + "]"
sequence = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0], [0.0, 0.0]]
h = [0.0, 0.0]
for t, x in enumerate(sequence, start=1):
z, r, h = gru_step(x, h)
print(f"step {t} input {x}")
print(" update gate", show(z))
print(" reset gate ", show(r))
print(" hidden ", show(h))Output
step 1 input [1.0, 0.0] update gate [ 0.622, 0.550] reset gate [ 0.401, 0.574] hidden [ 0.413, 0.209] step 2 input [0.0, 1.0] update gate [ 0.436, 0.604] reset gate [ 0.669, 0.546] hidden [ 0.070, 0.480] step 3 input [1.0, 1.0] update gate [ 0.552, 0.657] reset gate [ 0.565, 0.623] hidden [ 0.227, 0.712] step 4 input [0.0, 0.0] update gate [ 0.506, 0.518] reset gate [ 0.529, 0.541] hidden [ 0.169, 0.415]
Read the output. At step 1 the hidden state started at zeros, and after the first input it is [0.413, 0.209]. The update gate is around 0.6, so a good part of the draft went in. At step 2 the reset gates are higher than at step 1 for the first slot (0.669 against 0.401), which means that slot looked more at the old notes while drafting.
At step 4 the input is all zeros, yet the hidden state does not drop to zero. It goes from [0.227, 0.712] to [0.169, 0.415]. The notes still carry traces of the earlier steps. That is the memory at work.
Every number in the gates lies between 0 and 1, never exactly 0 or 1. Gates here are soft dials, not on and off switches.
Quick check
In the GRU, what does the update gate decide?
Counting the weights
Wikipedia says the GRU lacks a context vector or output gate, resulting in fewer parameters than the LSTM. Let us count them for the same sizes. Take an input of 10 numbers and a hidden state of 20 numbers.
Look at the equations again. A plain RNN has one block: a table W that maps the input (20 rows by 10 columns), a table U that maps the old notes (20 by 20) and a bias b (20 numbers). The GRU has three such blocks: one for the update gate, one for the reset gate and one for the candidate. In the usual layout, the LSTM has four: one for each of its three gates, plus one for the new information it writes into its cell.
def rnn_params(n_in, n_hid):
return n_hid * n_in + n_hid * n_hid + n_hid
n_in, n_hid = 10, 20
one = rnn_params(n_in, n_hid)
print("one block (W, U, b):", one)
print("RNN (1 block): ", 1 * one)
print("GRU (3 blocks):", 3 * one)
print("LSTM (4 blocks):", 4 * one)
print("GRU / LSTM:", round(3 * one / (4 * one), 2))Output
one block (W, U, b): 620 RNN (1 block): 620 GRU (3 blocks): 1860 LSTM (4 blocks): 2480 GRU / LSTM: 0.75
The GRU has 1,860 numbers to learn and the LSTM has 2,480, so the GRU is 25 per cent smaller at these sizes. The ratio is always three to four in this layout, whatever the sizes.
What does fewer weights buy you? Less memory, and usually less computing per step. That is the reason the RNN lesson called the GRU a more efficient alternative to the LSTM.
Tip
Treat 1,860 and 2,480 as the counts for the equations in this lesson. Another tool may count its biases a little differently.
GRU or LSTM?
Which one should you pick? The honest answer from the research is: it is hard to tell. Here is what the sources say.
Gates help
Chung and colleagues compared gated units against traditional tanh units on polyphonic music modeling and speech signal modeling. Their abstract says the advanced units are indeed better than the more traditional ones.
GRU matches LSTM
The same abstract says they found the GRU to be comparable to the LSTM. Wikipedia adds that on these kinds of tasks, plus natural language processing, the GRU's performance was found to be similar to the LSTM's.
No clear winner
Wikipedia says the GRU showed that gating is indeed helpful in general, and that Bengio's team came to no concrete conclusion on which of the two gating units was better.
So a fair rule of thumb: gated units beat plain ones in those tests, and the two gated units often score alike. If scores are alike, the smaller one costs less to run. The sources do not name a winner, so if it matters, try both on your own data.
Quick check
A teammate says GRU always beats LSTM. What do the sources say?
Cousins of the GRU
Wikipedia notes that there are several variations on the full gated unit. Two are worth a mention, because they go further down the same road of fewer parts.
Minimal gated unit
Merges the update and reset gates into a single forget gate. One gate instead of two.
Light GRU
Removes the reset gate altogether, swaps tanh for ReLU (a simple curve that outputs zero for negative numbers and passes positive numbers through) and adds batch normalization, a step that keeps numbers in a steady range during training.
Simpler gate inputs
Wikipedia also lists forms where each gate depends only on the previous hidden state, or only on the bias.
Reading both ways
One more idea fits any recurrent unit, a GRU included: reading in both directions. Everything so far reads left to right, so at each step the network knows only the past. But sometimes the words after a spot tell you what it means.
Wikipedia says bidirectional recurrent neural networks connect two hidden layers of opposite directions to the same output. One layer reads forward, one reads backward, and the output can use information from the past and the future at once. They were invented in 1997 by Schuster and Paliwal. The article's example is handwriting recognition, where knowing the letters before and after the current letter can improve performance. Its list of uses also includes speech recognition, translation, protein structure prediction and part-of-speech tagging.
Think of reading a sentence with a smudge on one word. You read to the end of the line, then use both sides to guess the smudged word. The catch is plain common sense: to read backward, the whole sequence must already exist. That suits tasks like labeling a finished sentence, and does not suit predicting the next word as someone types.
Wikipedia also says the two directions do not connect to each other, so they can be trained with similar methods to a standard RNN.
What to keep
Two gates, one memory
The update gate sets how much to replace, the reset gate sets how much of the old notes to look at. One hidden state carries everything.
Lighter
No separate cell and no output gate, so 1,860 weights against the LSTM's 2,480 at input 10 and hidden 20.
Similar results
Studies found it comparable to the LSTM, with no concrete winner.
Quick check
Which is true of a GRU compared with an LSTM?
Next: attention, letting a model look back at what matters.
Sources
- Wikipedia: Gated recurrent unit
- Cho et al. (2014): Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (arXiv 1406.1078)
- Chung et al. (2014): Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (arXiv 1412.3555)
- Wikipedia: Long short-term memory
- Wikipedia: Bidirectional recurrent neural networks
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.