Machine Learning Foundations / Advanced / 03.13
Deep Q-networks: a network for the table
When the Q-table is too big to write down, a small network guesses the values instead. See the two tricks that keep it steady, then train one in plain Python.
13 min readFree
Where the table runs out
The last lesson, Q-Learning, ended with a promise: deep Q-networks, for when the Q-table is too big to write down. This is that lesson.
A quick recap of two things you already know. In Reinforcement Learning, an agent acts, the world hands back a reward, and the agent learns which actions pay off. In Q-Learning, the agent keeps a table. Each row is a situation, each column is an action, and each cell holds a Q-value: a guess of how much total reward that action will bring from that situation.
A table works while the list of situations is short. The Hugging Face course notes that its two Q-learning games were 'relatively simple because the state space was discrete and small', with 16 states in one and 500 in the other. Then it adds that the state space in Atari games can contain 10^9 to 10^11 states, which is a billion to a hundred billion. The course's verdict is that 'producing and updating a Q-table can become ineffective in large state space environments.'
Key words
- State
- What the situation looks like right now. For a video game, that is the picture on the screen.
- Q-value
- A number for one action in one state: how much reward the agent expects to collect from there on.
- Q-table
- A grid with one row per state and one column per action, holding every Q-value.
- Replay buffer
- A memory that stores past moves so the agent can learn from them again, in random order.
- Target network
- A slowly updated copy of the network, used to set the goal the live network is aiming at.
Why a screen breaks the table
Think of a screen as a grid of tiny dots. Each dot can be a different colour. Every different pattern of dots is a different state, and a table needs one row for each.
Let us count for a very small screen: 20 by 20 dots, each only black or white. The next script counts the possible screens.
cells = 20 * 20 # a tiny 20 by 20 screen, each cell black or white
screens = 2 ** cells
print("cells on screen:", cells)
print("possible screens:", len(str(screens)), "digits long")Output
cells on screen: 400 possible screens: 121 digits long
That is 400 dots and a count with 121 digits. A real game screen has many more dots and many more colours, so no table could ever hold a row per screen.
The swap: a network that guesses
The fix is to stop storing the answers and start computing them. Instead of looking up a row, the agent feeds the state into a small model and reads off a guess.
Here is the Hugging Face description: 'Instead of using a Q-table, Deep Q-Learning uses a Neural Network that takes a state and approximates Q-values for each action based on that state.'
What is a neural network? You will meet it properly in the lesson called Neural Network Basics, later in this course. For now, here is all you need. A network takes some numbers in. It multiplies each by a weight, which is a number it can change, and adds the results up. Real networks stack many of these sums in layers. The agent learns by nudging the weights until the guesses get better.
Table
One stored number per state and action. Exact, but it grows with every new state and knows nothing about states it has not seen.
Network
A few weights that work for every state. It is a guess, but it carries over from familiar states to similar new ones.
The change to the loop
Everything else stays. The agent still acts, collects rewards and aims for the biggest total. Only the place where Q-values live changes.
The Atari result
DeepMind's group showed this working on old Atari 2600 video games. Here is what the papers themselves say.
- 1
2013: seven games
The paper 'Playing Atari with Deep Reinforcement Learning' presents what it calls 'the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning.' The input was 'raw pixels' and the output 'a value function estimating future rewards'. On seven Atari 2600 games, it 'outperforms all previous approaches on six of the games and surpasses a human expert on three of them.'
- 2
2015: forty-nine games
The Nature paper 'Human-level control through deep reinforcement learning' reports that the agent, 'receiving only the pixels and the game score as inputs', surpassed 'all previous algorithms' and reached a level 'comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters.'
Why it was hard: the wobble
Swapping the table for a network sounds easy. It was not. Wikipedia says: 'Reinforcement learning is unstable or divergent when a nonlinear function approximator such as a neural network is used to represent Q.' Divergent means the numbers drift off and never settle.
It lists three causes. The instability 'comes from the correlations present in the sequence of observations, the fact that small updates to Q may significantly change the policy of the agent and the data distribution, and the correlations between Q and the target values.'
In plain words, two things go wrong. First, the agent's experiences arrive in a row, and each one looks like the one before. Second, the network is chasing a goal that moves every time the network changes. Two tricks fix these, one for each.
Trick 1: experience replay
Imagine revising for an exam by reading only the last page you looked at, over and over. You would forget the rest. A better way is a stack of flash cards, shuffled before every session.
That is experience replay. After each move, the agent writes down what happened on one card: the state it was in, the action it took, the reward, and the state that followed. The cards go into a memory called the replay buffer. When it is time to learn, the agent pulls a small random handful of cards, not the latest ones.
Wikipedia describes it as 'a biologically inspired mechanism that uses a random sample of prior actions instead of the most recent action to proceed', and says it 'removes correlations in the observation sequence and smooths changes in the data distribution.'
The Hugging Face course lists two gains. One is efficiency: without a buffer, the agent 'learns from them (updates the neural network), and discards them', which 'is not efficient', while a buffer 'allows the agent to learn from the same experiences multiple times'. The other is memory: given only sequential samples, a network 'tends to forget the previous experiences as it gets new experiences'. The course calls this catastrophic forgetting.
Trick 2: the target network
Learning needs a goal to aim at. In Q-learning, the goal for a move is the reward you got plus the discounted best Q-value of the next state. The catch is that the same network produces both the guess and the goal. So each time the network improves, the goal moves too.
The Hugging Face course puts it with a picture: 'It’s like if you were a cowboy (the Q estimation) and you wanted to catch a cow (the Q-target).' Both run around at every step, which leads to 'a bizarre path of chasing (a significant oscillation in training).'
The fix is a second, slower network. The course says to 'use a separate network with fixed parameters for estimating the TD Target' and to 'copy the parameters from our Deep Q-Network every C steps to update the target network.' (TD target is the course's name for the goal.) Wikipedia says the same thing in one line: updates adjust Q 'towards target values that are only periodically updated, further reducing correlations with the target.'
Tip
Two networks, two jobs. The live network learns on every step. The target network is a photo of the live one, retaken every C steps, and it only supplies the goals.
The loss, in plain words
A loss is a number that says how wrong the guess was. Training means making it small. For Deep Q-Learning, the Hugging Face course says to 'create a loss function that compares our Q-value prediction and the Q-target and uses gradient descent to update the weights.' Gradient descent is the standard way a network nudges its weights downhill, toward a smaller loss.
Here is one card, worked by hand. The agent stepped and got a reward of -0.05. The target network guesses the next state's two actions are worth 0.59 and 0.72. The live network had guessed 0.40 for the move it took.
reward = -0.05
gamma = 0.9
target_q_next = [0.59, 0.72] # copy's guesses for the next state: left, right
current_guess = 0.40 # the live network's guess for the move we took
target = reward + gamma * max(target_q_next)
error = target - current_guess
print(f"target = {reward} + {gamma} * {max(target_q_next)} = {target:.3f}")
print(f"error = {target:.3f} - {current_guess} = {error:.3f}")
print(f"loss = error squared = {error ** 2:.4f}")Output
target = -0.05 + 0.9 * 0.72 = 0.598 error = 0.598 - 0.4 = 0.198 loss = error squared = 0.0392
In words: the goal is the reward plus the discount times the best next guess. The error is goal minus guess. The loss is the error squared, so big misses count much more than small ones.
Read it left to right. r is the reward. The Greek letter gamma is the discount from the Reinforcement Learning lesson. The max picks the best next action according to the target network. Q(s, a) is the live network's guess. Subtract, square, done.
Worked example: a corridor with a tiny network
Time to build one with only Python's built-in tools. The world is a corridor of 8 cells, numbered 0 to 7. The agent starts at cell 0 and can step left or right. Reaching cell 7 ends the episode with a reward of +1. Every other step costs -0.05, so wandering hurts. The discount is 0.9.
A true DQN has many layers. Ours is the smallest honest version: for each action, the guess is a weighted sum of three features of the cell. The features are 1, the position x (cell divided by 7), and x squared. That makes six weights in all, against 16 numbers a table would need.
It has the real parts: a replay buffer that keeps the last 500 cards, random batches of 16, and a target copy refreshed every 50 steps. Exploring starts at 100 percent and falls to 5 percent. It prints the average reward per block of 50 episodes, then the learned values for three cells.
import random
random.seed(3)
N = 8 # corridor cells 0..7, goal at cell 7
ACTIONS = (-1, +1) # 0 = step left, 1 = step right
GAMMA, ALPHA = 0.9, 0.05
BUFFER_SIZE, BATCH, COPY_EVERY = 500, 16, 50
def features(s):
x = s / (N - 1)
return [1.0, x, x * x]
def q(w, s, a):
return sum(wi * fi for wi, fi in zip(w[a], features(s)))
def step(s, a):
s2 = min(max(s + ACTIONS[a], 0), N - 1)
done = s2 == N - 1
return s2, (1.0 if done else -0.05), done
w = [[0.0] * 3 for _ in ACTIONS] # the "network"
w_target = [row[:] for row in w] # slow copy
buffer, steps_done, block = [], 0, []
for episode in range(1, 301):
s, total, done = 0, 0.0, False
eps = max(0.05, 1.0 - episode / 150)
for _ in range(40):
if random.random() < eps:
a = random.randrange(2)
else:
a = 0 if q(w, s, 0) > q(w, s, 1) else 1
s2, r, done = step(s, a)
buffer.append((s, a, r, s2, done))
buffer = buffer[-BUFFER_SIZE:]
s, total, steps_done = s2, total + r, steps_done + 1
if len(buffer) >= BATCH:
for bs, ba, br, bs2, bd in random.sample(buffer, BATCH):
target = br if bd else br + GAMMA * max(q(w_target, bs2, 0), q(w_target, bs2, 1))
error = target - q(w, bs, ba)
for i, f in enumerate(features(bs)):
w[ba][i] += ALPHA * error * f
if steps_done % COPY_EVERY == 0:
w_target = [row[:] for row in w]
if done:
break
block.append(total)
if episode % 50 == 0:
print(f"episodes {episode - 49:3d}-{episode:3d}: average reward {sum(block) / 50:.2f}")
block = []
print()
for s in (0, 3, 6):
print(f"cell {s}: left {q(w, s, 0):.2f} right {q(w, s, 1):.2f}")Output
episodes 1- 50: average reward -0.57 episodes 51-100: average reward 0.44 episodes 101-150: average reward 0.66 episodes 151-200: average reward 0.69 episodes 201-250: average reward 0.69 episodes 251-300: average reward 0.68 cell 0: left 0.20 right 0.30 cell 3: left 0.37 right 0.59 cell 6: left 0.72 right 1.00
The marked lines are the three ideas: store a card (35), sample a random batch (39), take the goal from the target copy (40), and refresh the copy every 50 steps (44, 45). Now read the first block. Early on, the agent mostly moves at random, so it wanders and pays the step cost: an average of -0.57. By episodes 51 to 100 it is at 0.44, and it settles around 0.68 to 0.69.
Is that good? The best possible run goes straight right: six steps at -0.05, then +1 on the seventh. That is 0.70. Our agent reaches 0.69 in the 151 to 200 block and 0.68 in the last one, so it is within a hair of perfect. The small gap is the odd random move it still makes.
Now the values. At cell 6, one step from the goal, going right is worth 1.00 and going left only 0.72, so the agent knows to go right. At cell 3, right is 0.59 against 0.37 for left. At cell 0, right is 0.30 against 0.20. Further from the goal, the same move is worth less, because the reward is further away and discounted. Worked out by hand, the three 'right' values are exactly 0.30, 0.59 and 1.00.
Quick check
In the corridor code, what does the target copy do?
One more fix: overestimation and Double DQN
There is a third problem. Taking the best of several noisy guesses tends to pick a lucky, too-high one. The paper 'Deep Reinforcement Learning with Double Q-learning' says Q-learning 'is known to overestimate action values under certain conditions', and shows that the DQN algorithm 'suffers from substantial overestimations in some games in the Atari 2600 domain.'
The Hugging Face course explains the repair: use two networks, 'to decouple the action selection from the target Q-value generation'. The live network chooses the best next action, and the target network scores it. The same paper reports that its change 'not only reduces the observed overestimations' but also 'leads to much better performance on several games.' Wikipedia says that Double DQN 'outperforms the original DQN algorithm.'
Recap
The problem
A Q-table needs a row for every state. A game screen has far too many possible pictures for any table.
The swap
A network takes the state and guesses the Q-value for each action, so similar states get similar guesses.
The two tricks
Experience replay learns from random old moves. A target network, copied every so often, keeps the goal from moving each step. Double DQN trims overestimation.
Tip
When you read about a DQN, ask three questions. What does the network take in? How big is the replay buffer? How often is the target network copied?
Next: policy gradients, learning which moves to make directly.
Sources
- Hugging Face Deep RL course: Deep Q-Learning (introduction)
- Hugging Face Deep RL course: The Deep Q-Learning Algorithm
- Wikipedia: Q-learning
- arXiv 1312.5602: Playing Atari with Deep Reinforcement Learning
- Nature: Human-level control through deep reinforcement learning
- arXiv 1509.06461: Deep Reinforcement Learning with Double Q-learning
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.