esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.38

GRU

A lighter cousin of the LSTM: two gates instead of three, no separate memory cell, and fewer weights to learn.

13 min readFree

A lighter cousin

The last lesson, LSTM, ended with a promise: next, GRU, a lighter cousin of the LSTM. Here it is.

Quick reminder of the LSTM. Wikipedia says an LSTM unit is typically made of a cell and three gates: a forget gate, an input gate and an output gate. The cell remembers values over time, and the gates control what goes in and what comes out. It works well, but it has a lot of moving parts.

The GRU asks a simple question: can we keep the gating idea and drop some of the parts? Wikipedia describes it as like an LSTM, with a gating mechanism to input or forget certain features, but it lacks a context vector or output gate. Fewer parts means fewer numbers to learn. By the end you will know what the two gates do and how many weights you save.

Key words

Gate
A small control that outputs a number between 0 and 1 for each item in the memory. Near 0 means block it. Near 1 means let it through.
Hidden state
The network's memory: a short list of numbers updated at every step and carried to the next.
Update gate
Decides how much of the old notes to keep and how much to swap for new ones.
Reset gate
Decides how much of the old notes to look at while drafting new ones.
Candidate
A draft of the new notes, written before the update gate decides how much of it to accept.
Parameters
All the numbers a model learns: its weights and biases. This lesson also calls them weights for short.

Two gates and one memory

Picture a student keeping notes in a long lecture. At every moment there is a page of notes. When the teacher says something new, the student has two choices to make.

  • Update gate

    How much of the page do I replace? Replace nothing and the notes stay as they were. Replace everything and the page is fresh. Most of the time it is a mix.

  • Reset gate

    When I draft the new line, how much of the old page do I look at? Look at all of it and the draft leans on old notes. Look at none and the draft comes only from what was just said.

  • One memory

    There is only one thing to carry from step to step: the hidden state. There is no second memory cell next to it, and no output gate deciding what to show.

That is the whole GRU. The LSTM has a cell, a hidden state and three gates. The GRU folds them into one list of numbers with two gates.

Tip

The word "gated" in the name means exactly this: the unit uses gates, small controls that open and close, to manage its memory. A gated recurrent unit is a recurrent unit with gates.

Who made it, and when

Wikipedia says the GRU was introduced in 2014 by Kyunghyun Cho and colleagues. It cites the paper "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation", by Cho and co-authors including Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio. It was presented at EMNLP 2014, a conference on language processing.

The abstract of that paper says it proposes an RNN Encoder-Decoder: one recurrent network reads a sequence of symbols and squeezes it into a fixed-length vector, and a second network decodes that vector into another sequence. The authors tested it as an extra helper inside a statistical machine translation system, and found that it improved the translation performance.

One step, in four moves

Here is what a GRU does each time a new input arrives. It has the input for this step (call it x) and the notes from the last step (call them h, for hidden state).

  1. 1

    Compute the update gate

    Mix the new input and the old notes, then squash the result into the range 0 to 1. Each number in the gate says how much of that slot to replace.

  2. 2

    Compute the reset gate

    Do the same with its own weights. Each number says how much of that slot of the old notes to look at.

  3. 3

    Draft a candidate

    Multiply the old notes by the reset gate, mix them with the new input, and squash the result into the range -1 to 1. This is the draft of the new notes.

  4. 4

    Blend

    The new notes are a mix of the old notes and the draft. The update gate sets the mix, slot by slot.

Wikipedia gives the equations of the fully gated unit. Here they are. Here x is the input, h is the hidden state, W and U are weight tables, b is a bias (a small constant the model also learns), and t counts the steps.

zt=σ(Wzxt+Uzht−1+bz)rt=σ(Wrxt+Urht−1+br)h^t=tanh⁡(Whxt+Uh(rt⊙ht−1)+bh)ht=(1−zt)⊙ht−1+zt⊙h^t\begin{aligned} z_t &= \sigma(W_z x_t + U_z h_{t-1} + b_z) \\ r_t &= \sigma(W_r x_t + U_r h_{t-1} + b_r) \\ \hat{h}_t &= \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h) \\ h_t &= (1 - z_t) \odot h_{t-1} + z_t \odot \hat{h}_t \end{aligned}

In words, line by line. The update gate z is the sigmoid (a curve that squashes any number into 0 to 1) of a weighted mix of the input and the old notes. The reset gate r is built the same way, with different weights. The candidate is the tanh (a similar curve that squashes into -1 to 1) of a mix of the input and the old notes after the reset gate has dimmed them. The symbol with a dot in a circle means multiply slot by slot. The last line says the new notes are (1 minus z) times the old notes, plus z times the candidate.

The blend line, up close

The last equation is the heart of the GRU, so let us try it with numbers. Say the old note in one slot is 0.8 and the candidate wants to write -0.5. Watch what different update gate values do.

Python
old = 0.80   # what the notes say now
new = -0.50  # what the candidate would write

for z in (0.0, 0.25, 0.75, 1.0):
    mixed = (1 - z) * old + z * new
    print(f"update gate {z:.2f} -> notes become {mixed:6.3f}")

Output

update gate 0.00 -> notes become  0.800
update gate 0.25 -> notes become  0.475
update gate 0.75 -> notes become -0.175
update gate 1.00 -> notes become -0.500

With the gate at 0, the old note survives untouched. With the gate at 1, the candidate wins completely. In between, the result slides from one to the other. Notice the two shares always add to 1: (1 - z) for the old and z for the new. The GRU does not have separate forget and input gates. One gate, z, does both jobs.

Run one cell

Let us run a GRU cell by hand, in plain Python with no libraries. The weights are small fixed numbers, chosen only so the example is easy to follow. A real network learns its weights from data. The input has 2 numbers per step and the hidden state has 2 numbers. We feed in a sequence of 4 steps and print the two gates and the hidden state after each.

Python
import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def dot(row, vec):
    return sum(w * v for w, v in zip(row, vec))

# Small fixed weights: input of 2 numbers, hidden state of 2 numbers.
Wz, Uz, bz = [[0.5, -0.3], [0.2, 0.4]], [[0.1, 0.0], [0.0, 0.1]], [0.0, 0.0]
Wr, Ur, br = [[-0.4, 0.6], [0.3, 0.1]], [[0.2, 0.1], [0.1, 0.2]], [0.0, 0.0]
Wh, Uh, bh = [[0.8, -0.5], [0.4, 0.7]], [[0.3, 0.2], [0.2, 0.3]], [0.0, 0.0]

def gru_step(x, h):
    z = [sigmoid(dot(Wz[i], x) + dot(Uz[i], h) + bz[i]) for i in range(2)]
    r = [sigmoid(dot(Wr[i], x) + dot(Ur[i], h) + br[i]) for i in range(2)]
    rh = [r[i] * h[i] for i in range(2)]
    cand = [math.tanh(dot(Wh[i], x) + dot(Uh[i], rh) + bh[i]) for i in range(2)]
    new = [(1 - z[i]) * h[i] + z[i] * cand[i] for i in range(2)]
    return z, r, new

def show(v):
    return "[" + ", ".join(f"{a:6.3f}" for a in v) + "]"

sequence = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0], [0.0, 0.0]]
h = [0.0, 0.0]
for t, x in enumerate(sequence, start=1):
    z, r, h = gru_step(x, h)
    print(f"step {t}  input {x}")
    print("   update gate", show(z))
    print("   reset gate ", show(r))
    print("   hidden     ", show(h))

Output

step 1  input [1.0, 0.0]
   update gate [ 0.622,  0.550]
   reset gate  [ 0.401,  0.574]
   hidden      [ 0.413,  0.209]
step 2  input [0.0, 1.0]
   update gate [ 0.436,  0.604]
   reset gate  [ 0.669,  0.546]
   hidden      [ 0.070,  0.480]
step 3  input [1.0, 1.0]
   update gate [ 0.552,  0.657]
   reset gate  [ 0.565,  0.623]
   hidden      [ 0.227,  0.712]
step 4  input [0.0, 0.0]
   update gate [ 0.506,  0.518]
   reset gate  [ 0.529,  0.541]
   hidden      [ 0.169,  0.415]

Read the output. At step 1 the hidden state started at zeros, and after the first input it is [0.413, 0.209]. The update gate is around 0.6, so a good part of the draft went in. At step 2 the reset gates are higher than at step 1 for the first slot (0.669 against 0.401), which means that slot looked more at the old notes while drafting.

At step 4 the input is all zeros, yet the hidden state does not drop to zero. It goes from [0.227, 0.712] to [0.169, 0.415]. The notes still carry traces of the earlier steps. That is the memory at work.

Every number in the gates lies between 0 and 1, never exactly 0 or 1. Gates here are soft dials, not on and off switches.

Quick check

In the GRU, what does the update gate decide?

Counting the weights

Wikipedia says the GRU lacks a context vector or output gate, resulting in fewer parameters than the LSTM. Let us count them for the same sizes. Take an input of 10 numbers and a hidden state of 20 numbers.

Look at the equations again. A plain RNN has one block: a table W that maps the input (20 rows by 10 columns), a table U that maps the old notes (20 by 20) and a bias b (20 numbers). The GRU has three such blocks: one for the update gate, one for the reset gate and one for the candidate. In the usual layout, the LSTM has four: one for each of its three gates, plus one for the new information it writes into its cell.

Python
def rnn_params(n_in, n_hid):
    return n_hid * n_in + n_hid * n_hid + n_hid

n_in, n_hid = 10, 20
one = rnn_params(n_in, n_hid)
print("one block (W, U, b):", one)
print("RNN  (1 block): ", 1 * one)
print("GRU  (3 blocks):", 3 * one)
print("LSTM (4 blocks):", 4 * one)
print("GRU / LSTM:", round(3 * one / (4 * one), 2))

Output

one block (W, U, b): 620
RNN  (1 block):  620
GRU  (3 blocks): 1860
LSTM (4 blocks): 2480
GRU / LSTM: 0.75

The GRU has 1,860 numbers to learn and the LSTM has 2,480, so the GRU is 25 per cent smaller at these sizes. The ratio is always three to four in this layout, whatever the sizes.

What does fewer weights buy you? Less memory, and usually less computing per step. That is the reason the RNN lesson called the GRU a more efficient alternative to the LSTM.

Tip

Treat 1,860 and 2,480 as the counts for the equations in this lesson. Another tool may count its biases a little differently.

GRU or LSTM?

Which one should you pick? The honest answer from the research is: it is hard to tell. Here is what the sources say.

  • Gates help

    Chung and colleagues compared gated units against traditional tanh units on polyphonic music modeling and speech signal modeling. Their abstract says the advanced units are indeed better than the more traditional ones.

  • GRU matches LSTM

    The same abstract says they found the GRU to be comparable to the LSTM. Wikipedia adds that on these kinds of tasks, plus natural language processing, the GRU's performance was found to be similar to the LSTM's.

  • No clear winner

    Wikipedia says the GRU showed that gating is indeed helpful in general, and that Bengio's team came to no concrete conclusion on which of the two gating units was better.

So a fair rule of thumb: gated units beat plain ones in those tests, and the two gated units often score alike. If scores are alike, the smaller one costs less to run. The sources do not name a winner, so if it matters, try both on your own data.

Quick check

A teammate says GRU always beats LSTM. What do the sources say?

Cousins of the GRU

Wikipedia notes that there are several variations on the full gated unit. Two are worth a mention, because they go further down the same road of fewer parts.

  • Minimal gated unit

    Merges the update and reset gates into a single forget gate. One gate instead of two.

  • Light GRU

    Removes the reset gate altogether, swaps tanh for ReLU (a simple curve that outputs zero for negative numbers and passes positive numbers through) and adds batch normalization, a step that keeps numbers in a steady range during training.

  • Simpler gate inputs

    Wikipedia also lists forms where each gate depends only on the previous hidden state, or only on the bias.

Reading both ways

One more idea fits any recurrent unit, a GRU included: reading in both directions. Everything so far reads left to right, so at each step the network knows only the past. But sometimes the words after a spot tell you what it means.

Wikipedia says bidirectional recurrent neural networks connect two hidden layers of opposite directions to the same output. One layer reads forward, one reads backward, and the output can use information from the past and the future at once. They were invented in 1997 by Schuster and Paliwal. The article's example is handwriting recognition, where knowing the letters before and after the current letter can improve performance. Its list of uses also includes speech recognition, translation, protein structure prediction and part-of-speech tagging.

Think of reading a sentence with a smudge on one word. You read to the end of the line, then use both sides to guess the smudged word. The catch is plain common sense: to read backward, the whole sequence must already exist. That suits tasks like labeling a finished sentence, and does not suit predicting the next word as someone types.

Wikipedia also says the two directions do not connect to each other, so they can be trained with similar methods to a standard RNN.

What to keep

  • Two gates, one memory

    The update gate sets how much to replace, the reset gate sets how much of the old notes to look at. One hidden state carries everything.

  • Lighter

    No separate cell and no output gate, so 1,860 weights against the LSTM's 2,480 at input 10 and hidden 20.

  • Similar results

    Studies found it comparable to the LSTM, with no concrete winner.

Quick check

Which is true of a GRU compared with an LSTM?

Next: attention, letting a model look back at what matters.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.