Machine Learning Foundations / Advanced / 03.15
Actor-critic: a player and a coach
One part picks the moves, another judges them, and both learn at once. Build a tiny actor-critic in plain Python.
13 min readFree
A player and a coach
Imagine learning basketball alone. You shoot, you miss, and you only find out how the whole game went when the final whistle blows. Then you try to work out which of your hundred moves caused the loss. That is slow and confusing.
Now imagine a coach standing beside you. After every shot, the coach says: 'That was better than I expected' or 'That was worse.' You do not wait for the final whistle. You adjust right away. And the coach gets better at judging, too, because every shot teaches them what a good position looks like.
That is actor-critic learning, the idea the lesson on policy gradients ended on: a player and a coach learning together. The player is called the actor. The coach is called the critic.
Key words
- Actor
- The part that picks moves. It holds the policy: for each situation, the chance of choosing each move.
- Critic
- The part that judges. It holds a value: a guess of how much total reward to expect from a situation.
- Advantage
- How much better a move turned out than the critic expected. Positive means a pleasant surprise. Negative means a disappointment.
- TD error
- TD stands for temporal difference. It is the gap between what the critic expected and what it sees one step later. It is the quick, cheap version of the advantage.
- Variance
- How much a number jumps around from one try to the next. High variance means noisy. It is hard to learn from a noisy signal.
Two lines to catch up
An agent acts in a world, gets rewards, and wants the most total reward over time. The policy is its habit for choosing moves, and the value says how good a situation is, counting rewards still to come. The lesson on reinforcement learning has the full story, and we will not repeat it here.
Two helpers, one team
Wikipedia defines it this way: the actor-critic algorithm 'is a family of reinforcement learning (RL) algorithms that combine policy-based RL algorithms such as policy gradient methods, and value-based RL algorithms such as value iteration, Q-learning, SARSA, and TD learning.'
It splits the two jobs. In Wikipedia's words, there is 'an "actor" that determines which actions to take according to a policy function, and a "critic" that evaluates those actions according to a value function.' So the actor is the policy side, which you met in policy gradients. The critic is the value side, which you met in Q-learning and DQN.
The Hugging Face course uses the same picture: 'imagine you’re playing a video game. You can play with a friend that will provide you with some feedback. You’re the Actor and your friend is the Critic.' It adds that the critic learns too, so that it can give better feedback next time.
Wikipedia explains why the critic cannot be trained once and left alone: 'Since these functions all depend on the actor, the critic must learn alongside the actor.' When the player changes, what counts as a good situation changes with them. So both keep learning, side by side.
Tip
Do not picture two enemies. The actor never argues with the critic. It simply listens to the critic's score and moves its habits in that direction.
The advantage: better than expected?
Say you take a shortcut to work and the trip takes 20 minutes. Is that good? It depends. If your trips usually take 30 minutes, the shortcut saved you 10. If they usually take 12, it cost you 8. The number alone does not tell you. You need to compare it to what you expected.
That comparison is the advantage. The Hugging Face course says the advantage function 'calculates the extra reward we get if we take this action at that state compared to the mean reward we get at that state.' It also explains what happens next: if the advantage is above zero, the policy is pushed in that direction. If it is below zero, 'our action does worse than the average value of that state' and the push goes the other way.
Wikipedia describes the same thing as an improvement: 'The actor-critic methods can be understood as an improvement over pure policy gradient methods like REINFORCE via introducing a baseline.' A baseline is the usual score you compare against. The critic's value is that baseline.
The TD error: a quick estimate of the advantage
Computing the advantage exactly needs two value tables. The Hugging Face course points to a shortcut: 'we can use the TD error as a good estimator of the advantage function.'
Here is the TD error in words. Take the reward you just got. Add the critic's guess for the place you landed in, shrunk a little by the discount factor, which says sooner rewards count a bit more. Then subtract the critic's guess for the place you were in. A positive result means better than expected.
In that formula, r is the reward for the step, s is where you were, s' is where you landed, V is the critic's guess, and the Greek letter gamma is the discount factor. The critic is trained to make this error smaller.
A tiny example with made-up numbers, run in plain Python. The critic thinks my cell is worth 0.40 and the next cell 0.59. The step paid -0.1. The discount is 0.9.
gamma = 0.9
value_here = 0.40 # the critic's guess for the cell I am in
value_next = 0.59 # the critic's guess for the cell I land in
reward = -0.1 # what the step paid
delta = reward + gamma * value_next - value_here
print(f"TD error = {delta:.3f}")
if delta > 0:
print("better than expected: make that move more likely")
else:
print("worse than expected: make that move less likely")Output
TD error = 0.031 better than expected: make that move more likely
The step paid a small penalty, yet the move landed me somewhere the critic likes. So the surprise is slightly positive: 0.031. The actor will make that move a little more likely.
Notice what is missing. We did not wait for the game to end. One step was enough for one lesson.
Why it is less noisy
Plain policy gradient, which the Hugging Face course calls Reinforce, has a noise problem. The course explains that it collects a whole trajectory, calculates the discounted return, and uses this score to raise or lower the probability of every action in that trajectory. Because of random events in the world and in the policy, 'the same starting state can lead to very different returns.' That is high variance.
Think of grading every homework task by one final exam score. A lucky guess on the exam would praise them all. Actor-critic gives feedback task by task instead.
Less noise
The course says the solution to reducing the variance of Reinforce, and training the agent faster and better, is the actor-critic combination of policy-based and value-based methods. The critic's guess steadies the signal.
Step by step
The TD error only needs the next state. So the actor and the critic can both learn after every step, with no waiting for the episode to end.
The price
The critic's guess can be wrong, especially early on. A wrong coach gives wrong feedback. Wikipedia notes the trade-off: a higher n in its n-step estimate means lower bias, at the price of higher variance.
Bias means a steady lean in one direction. Using only real returns, as Reinforce does, has no lean, which the course calls an advantage of that method. A guessing critic adds a small lean in exchange for much less noise.
Quick check
Why can actor-critic learn after every step, while plain policy gradient waits until the episode ends?
One step of learning
Here is one full round, as numbered steps. Both parts change after every move.
- 1
1. The actor picks
It looks at the state and chooses a move at random, using its current chances.
- 2
2. The world answers
You get a reward and land in a new state.
- 3
3. The critic judges
It computes the TD error: reward, plus its guess for the new state, minus its guess for the old one.
- 4
4. The critic improves
It moves its guess for the old state a little toward what just happened.
- 5
5. The actor improves
If the error is positive, the move it just made becomes more likely. If negative, less likely.
The names you will meet
Three names come up again and again. Each gets one line, from the sources.
A2C
Advantage Actor-Critic. The Hugging Face course says it stabilises learning further by using the advantage function as the critic instead of the action-value function.
A3C
Asynchronous Advantage Actor-Critic. Wikipedia lists it as a 'Parallel and asynchronous version of A2C.'
PPO
Proximal policy optimization. Wikipedia says it 'is a policy gradient method, often used for deep RL when the policy network is very large.'
Where these methods are used
Careful here: two of these claims are about PPO, a policy gradient method, not actor-critic itself. They are what the sources support.
Games
Wikipedia says PPO 'has been applied to many areas, such as controlling a robotic arm, beating professional players at Dota 2 (OpenAI Five), and playing Atari games.'
Robots
The same line mentions controlling a robotic arm. The Hugging Face course also has a unit called 'Actor Critic methods with Robotics environments'.
Chat models
The Hugging Face blog says reinforcement learning from human feedback, or RLHF, uses 'methods from reinforcement learning to directly optimize a language model with human feedback' and that its most recent success was its use in ChatGPT. It adds that what multiple organisations have made work is fine-tuning a copy of the language model 'with a policy-gradient RL algorithm, Proximal Policy Optimization (PPO).'
Wikipedia also says that since 2018, PPO was the default RL algorithm at OpenAI.
Worked example: a five-cell corridor
Time to build one. The world is a corridor with five cells, numbered 0 to 4. You start in cell 0. Cell 4 is the goal. You can move left or right. Each step costs -0.1, and reaching the goal pays +1. An episode also stops after 30 steps.
The shortest trip is four moves right: three steps at -0.1, then +1. So the best possible reward is 0.7 per episode.
- 1
1. The actor is a table
For each cell, two numbers, one for left and one for right. A softmax turns them into chances that add up to 1. Softmax just makes bigger numbers into bigger chances.
- 2
2. The critic is a table too
One number per cell: how much reward it expects from there. All start at zero.
- 3
3. After every move, compute the TD error
Reward, plus 0.9 times the critic's guess for the new cell, minus its guess for the old cell. At the goal, there is no next cell to guess.
- 4
4. Update both
The critic moves its guess a tenth of the way toward the target. The actor adds a tenth of the error to the move it chose, and takes a little from the other move.
- 5
5. Repeat for 500 episodes
The random seed is fixed, so you get the same numbers I did. We print the average reward for every block of 100 episodes.
import math
import random
random.seed(1)
N = 5 # corridor cells 0..4, goal at cell 4
GAMMA = 0.9
ACTOR_STEP = 0.1
CRITIC_STEP = 0.1
MAX_STEPS = 30
prefs = [[0.0, 0.0] for _ in range(N)] # actor table: action 0 = left, 1 = right
value = [0.0] * N # critic table
def probs(state):
a, b = (math.exp(x) for x in prefs[state])
return [a / (a + b), b / (a + b)]
def pick(state):
return 1 if random.random() < probs(state)[1] else 0
def step(state, action):
nxt = max(0, state - 1) if action == 0 else state + 1
if nxt == N - 1:
return nxt, 1.0, True
return nxt, -0.1, False
block, total = 0.0, 0.0
for episode in range(1, 501):
state, done, steps = 0, False, 0
while not done and steps < MAX_STEPS:
action = pick(state)
nxt, reward, done = step(state, action)
target = reward + (0.0 if done else GAMMA * value[nxt])
delta = target - value[state] # TD error
value[state] += CRITIC_STEP * delta # critic learns
p = probs(state)
for a in (0, 1): # actor learns
grad = (1.0 if a == action else 0.0) - p[a]
prefs[state][a] += ACTOR_STEP * delta * grad
total += reward
state, steps = nxt, steps + 1
block += total
total = 0.0
if episode % 100 == 0:
print(f"episodes {episode - 99:3d}-{episode:3d}: average reward {block / 100:.2f}")
block = 0.0
for s in range(N - 1):
left, right = probs(s)
print(f"cell {s}: left {left:.2f}, right {right:.2f}, value {value[s]:.2f}")Output
episodes 1-100: average reward 0.41 episodes 101-200: average reward 0.63 episodes 201-300: average reward 0.65 episodes 301-400: average reward 0.68 episodes 401-500: average reward 0.67 cell 0: left 0.05, right 0.95, value 0.40 cell 1: left 0.03, right 0.97, value 0.59 cell 2: left 0.02, right 0.98, value 0.79 cell 3: left 0.02, right 0.98, value 1.00
Read the first column of results. In the first 100 episodes the average reward was 0.41. By the second block it was 0.63, and later blocks reached 0.65, 0.68 and 0.67. The best possible is 0.7, so the pair learned a near-perfect route. The first block was lower because the actor was still mostly guessing.
Now the tables. In cell 0, the actor goes right with chance 0.95. In cells 1, 2 and 3 it goes right with 0.97, 0.98 and 0.98. The critic's values climb as you get nearer the goal: 0.40, 0.59, 0.79 and 1.00. Cell 3 is one step from the goal, so its true value is 1.00. If you always move right, the true values for the other cells are 0.46, 0.62 and 0.80. The critic's numbers are close, and still catching up. Nobody told it these numbers. It learned them from the TD error.
Tip
Try changing the two step sizes, or the seed. A bigger step size learns faster but can jump around. A smaller one is calmer and slower. That is a hyperparameter, like the dials in the lesson on tuning.
Quick check
In the corridor, what does a positive TD error make the actor do?
The reinforcement learning part, in five steps
This lesson closes the reinforcement learning part of the course. Here is the path in one list.
- 1
1. Bandits
One situation, a few choices, a reward each time. The first taste of explore versus exploit.
- 2
2. Q-learning
A table that scores each move in each situation, and a rule to improve it.
- 3
3. Deep Q-networks (DQN)
Swap the table for a neural network, so it can handle huge worlds.
- 4
4. Policy gradient
Skip the scores and learn the policy directly, by making good moves more likely.
- 5
5. Actor-critic
Keep both: an actor that holds the policy and a critic that holds the values, learning together, step by step.
Each step fixed a weakness of the one before it. Tables do not fit big worlds. Pure value methods choose moves only through their scores. Plain policy gradient is noisy. Actor-critic takes a piece from each.
Next: neural network basics, the building block of deep learning.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.