esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.14

Learning the moves directly

Policy gradient methods skip the move scores and learn the chance of each move. Build one from scratch in plain Python and see why a baseline helps.

14 min readFree

Learn the moves, not the scores

The last lesson, Deep-Q Networks, ended with a promise: policy gradients, learning which moves to make directly.

Quick recap from the Reinforcement Learning lesson: an agent acts, gets a reward, and learns a policy, its habit for picking moves. So far our agents learned a score for each move and picked the highest. The Hugging Face Deep RL course calls that a value-based method: 'In value-based methods, we learn a value function.' The policy is only a side effect: 'We have a policy, but it’s implicit since it is generated directly from the value function.'

Policy-based methods flip it around. The same course says: 'in policy-based methods, we directly learn to approximate π* without having to learn a value function.' The agent keeps no score card. It keeps one thing only: how likely it is to make each move.

Think of getting to class. You could score every route and always take the top one. Or you could keep a habit, 'bike most days, bus when it rains', and nudge it after every trip: a bit more bike after a good ride, a bit less after a bad one. The second is this lesson.

Key words

Policy
The agent's habit for picking moves. Here it is a set of chances, one per move.
Stochastic
Random, with known odds: 'bike 70 percent, bus 30 percent', not 'always bike'.
Return
The reward added up from now until the episode ends. One return per try.
REINFORCE
The first and simplest policy gradient method. Make moves that led to high returns more likely.
Baseline
A typical score you compare each return against. It makes the signal less noisy.
Variance
How much a number jumps around from one try to the next. High variance means a noisy signal.
On-policy
Learning only from tries made by the current policy.

A policy is a set of chances

Wikipedia says the policy 'takes as argument the state of the environment s and produces a probability distribution'. That is just a list of chances that add up to 1.

How do we turn raw numbers into chances? In Logistic regression, a squashing step turned a score into a chance. For several options at once the version is called softmax. Each move gets a preference number, and softmax turns the preferences into chances that are never negative and add up to 1.

Lilian Weng's post mentions the common choice to 'parameterize policy action space by continuous Gaussian or discrete softmax distribution'. Our example has three separate moves, so we use the discrete softmax.

softmax.py
import math

def softmax(prefs):
    top = max(prefs)                      # subtract the top for safety
    exps = [math.exp(x - top) for x in prefs]
    total = sum(exps)
    return [x / total for x in exps]

for prefs in ([0, 0, 0], [0, 1, 2], [0, 1, 5]):
    probs = softmax(prefs)
    print(prefs, "->", [round(p, 2) for p in probs], "sum =", round(sum(probs), 2))

Output

[0, 0, 0] -> [0.33, 0.33, 0.33] sum = 1.0
[0, 1, 2] -> [0.09, 0.24, 0.67] sum = 1.0
[0, 1, 5] -> [0.01, 0.02, 0.98] sum = 1.0

With equal preferences, each move gets a third. Raise one and its chance climbs, but the others never reach exactly zero, so a rare move can still be tried.

Learning now means one thing: nudge the preference numbers. Up makes a move likelier, down makes it rarer. They are the policy's dials.

REINFORCE in plain words

Wikipedia says: 'The REINFORCE algorithm, introduced by Ronald J. Williams in 1992, was the first policy gradient method.' The idea fits in a sentence. Hugging Face says the goal is 'tuning the policy such that good actions (that maximize the return) are sampled more frequently in the future.'

  1. 1

    1. Play one episode

    Use the current chances to pick each move, and record what happened.

  2. 2

    2. Add up the return

    How much total reward did that try earn?

  3. 3

    3. Judge the moves

    A high return means the moves in that try were probably good. A low return means they were probably not.

  4. 4

    4. Nudge the dials

    Raise the chance of the moves you made if the return was high. Lower it if the return was low. The bigger the return, the bigger the nudge.

  5. 5

    5. Repeat

    Play another episode with the new chances.

Hugging Face describes the return as the 'scoring function': 'If the return is high, it will push up the probabilities of the (state, action) combinations. Otherwise, if the return is low, it will push down the probabilities'.

That is the whole method. The maths just says it exactly. Hugging Face writes the loop as: collect an episode, estimate the gradient, update the weights. Here is the update with a baseline b, which we meet next.

θ←θ+α ∇θln⁡πθ(a∣s) (G−b)\theta \leftarrow \theta + \alpha \, \nabla_\theta \ln \pi_\theta(a \mid s) \, (G - b)

In words: take the dial settings (theta) and add a small step (alpha) in the direction that makes the move you made (a) likelier in that situation (s). The step is bigger when the return (G) beats the baseline (b), and turns backwards when it falls short. The 'ln' part is a logarithm, and its gradient is the direction that raises that move's chance. A gradient is the direction of steepest climb.

Hugging Face states the rule bluntly: 'if we win the episode, we consider that each action taken was good and must be more sampled in the future.' In a long game, a lucky win lifts every move in it, good or bad. Over many episodes the good moves win more often, so they gain more. That is also where the noise comes from.

Tip

Think of the policy's score as a hill and the dial settings as where you stand. The gradient points uphill. Each update is one small step up.

Returns versus a baseline

Here is the catch with raw returns. Say every trip scores between 5 and 8 out of 10. Every score is positive, so after every trip the rule says 'make that move likelier'. A route that scored 5 still gets pushed up, just less than one that scored 7. Good and bad both get praise.

Wikipedia names the problem: because REINFORCE is on-policy, 'This can lead to high variance in the updates, as the returns R(τ) can vary significantly between trajectories.' A trajectory is the list of states and moves in one episode.

The fix is a baseline. Lilian Weng writes: 'A widely used variation of REINFORCE is to subtract a baseline value from the return G_t to reduce the variance of gradient estimation while keeping the bias unchanged'. In plain words: instead of asking 'was this score high?', ask 'was this score higher than usual?'. Above usual, push the move up. Below usual, push it down.

Wikipedia agrees: 'A common way for reducing variance is the REINFORCE with baseline algorithm'. In our example the baseline is the simplest thing possible: the average score so far.

  • Without a baseline

    Every positive score raises the move you made. Early luck can run away with the policy.

  • With a baseline

    Only scores better than usual raise the move. Scores worse than usual lower it. The nudges point the right way more often.

  • What stays the same

    On average the update still climbs toward better moves. The baseline cuts the jitter, not the direction.

Worked example: three ways to get to class

Let us run it. Three ways to get to class: bus, bike and tram. Each trip gets a comfort score. The hidden averages are 5, 6 and 7, so the tram is best, but every score wobbles by about 2 points. We made these numbers up. The agent sees only the scores it earns.

Each try is one trip, so the return is just that trip's score. The preferences start at 0, so each route starts at a third. After each trip we nudge every preference by the learning rate times (score minus baseline) times (1 if it was the route taken, else 0, minus its chance). That last part is 'the direction that makes this move likelier', written out for softmax. We run 1,000 episodes and print the chances every 200, once without and once with a baseline.

reinforce.py
import math
import random

MEANS = [5.0, 6.0, 7.0]    # hidden average score of bus, bike, tram
LR = 0.05                  # how big each nudge is

def softmax(prefs):
    top = max(prefs)
    exps = [math.exp(x - top) for x in prefs]
    total = sum(exps)
    return [x / total for x in exps]

def reinforce(seed, episodes, use_baseline, show_every=0):
    rng = random.Random(seed)
    prefs = [0.0, 0.0, 0.0]
    baseline = 0.0                         # average score so far
    for ep in range(1, episodes + 1):
        probs = softmax(prefs)
        move = rng.choices(range(3), weights=probs)[0]
        score = rng.gauss(MEANS[move], 2.0)
        signal = score - baseline if use_baseline else score
        for i in range(3):
            chosen = 1.0 if i == move else 0.0
            prefs[i] += LR * signal * (chosen - probs[i])
        baseline += (score - baseline) / ep
        if show_every and ep % show_every == 0:
            print(f"episode {ep:4d}:", [round(p, 2) for p in softmax(prefs)])
    return softmax(prefs)

print("no baseline")
reinforce(1, 1000, False, show_every=200)
print("with baseline")
reinforce(1, 1000, True, show_every=200)

Output

no baseline
episode  200: [0.03, 0.12, 0.85]
episode  400: [0.01, 0.01, 0.99]
episode  600: [0.01, 0.01, 0.98]
episode  800: [0.0, 0.01, 0.99]
episode 1000: [0.0, 0.0, 1.0]
with baseline
episode  200: [0.02, 0.04, 0.94]
episode  400: [0.01, 0.03, 0.95]
episode  600: [0.01, 0.02, 0.97]
episode  800: [0.02, 0.02, 0.96]
episode 1000: [0.01, 0.01, 0.98]

Each line shows the chances for bus, bike and tram. In both versions the tram's chance climbs fast: 0.94 by episode 200 with a baseline, 0.85 without. By episode 1000 it is 0.98 with one and 1.0 without. In this one run, both worked. Do not trust a single run, since one seed can hide the pattern. So we repeat it with 30 different seeds.

thirty_runs.py
import math
import random

MEANS = [5.0, 6.0, 7.0]
LR = 0.05

def softmax(prefs):
    top = max(prefs)
    exps = [math.exp(x - top) for x in prefs]
    total = sum(exps)
    return [x / total for x in exps]

def reinforce(seed, episodes, use_baseline):
    rng = random.Random(seed)
    prefs = [0.0, 0.0, 0.0]
    baseline = 0.0
    for ep in range(1, episodes + 1):
        probs = softmax(prefs)
        move = rng.choices(range(3), weights=probs)[0]
        score = rng.gauss(MEANS[move], 2.0)
        signal = score - baseline if use_baseline else score
        for i in range(3):
            chosen = 1.0 if i == move else 0.0
            prefs[i] += LR * signal * (chosen - probs[i])
        baseline += (score - baseline) / ep
    return softmax(prefs)

for use_baseline in (False, True):
    best = [reinforce(seed, 1000, use_baseline)[2] for seed in range(1, 31)]
    label = "with baseline   " if use_baseline else "without baseline"
    print(label, "average", round(sum(best) / 30, 2),
          "| worst", round(min(best), 2),
          "| runs under 0.5:", sum(p < 0.5 for p in best), "of 30")

Output

without baseline average 0.86 | worst 0.0 | runs under 0.5: 4 of 30
with baseline    average 0.99 | worst 0.97 | runs under 0.5: 0 of 30

Now the difference shows. With a baseline, the tram's average chance after 1,000 episodes was 0.99, and even the worst of the 30 runs reached 0.97. Without one, the average was 0.86 and 4 of the 30 runs ended under 0.5. The worst left the tram at a chance that rounds to 0.00: it locked onto a worse route and never came back.

Why? Without a baseline every score here is positive, so whichever route is picked early gets pushed up each time it is picked, and a middling route can snowball. With a baseline, a route that scores below the running average gets pushed down, which gives the better route its chance.

Quick check

What does the baseline change in the update?

What it does well

Why bother, when Deep-Q networks work so well? The Hugging Face course gives these answers.

  • Smooth changes

    In value-based methods a tiny change in the scores can flip the favourite and change behaviour at once. In policy gradient methods, 'stochastic policy action preferences (probability of taking action) change smoothly over time.'

  • Random strategies

    Policy gradient methods 'can learn a stochastic policy while value functions can’t.'

Why a random strategy can be the best one

Random sounds like a flaw, but Hugging Face lists two gains. First, 'We don’t need to implement an exploration/exploitation trade-off by hand.' Since the policy outputs odds, the agent keeps trying different moves on its own, with no epsilon needed.

Second, it handles perceptual aliasing: 'when two states seem (or are) the same but need different actions.' The course's example is a vacuum cleaner that senses only walls, so two different spots look the same. A fixed rule 'will either always move right when in a red state or always move left', and gets stuck. A random one 'will randomly move left or right in red (colored) states', so it reaches the goal 'with a high probability'.

Moves that are not a list

Our three routes are a short list. Many tasks are not. For a self-driving car, Hugging Face says there is 'a (near) infinite choice of actions (turning the wheel at 15°, 17.2°, 19,4°, honking, etc.).' A value method would need a score for every option, and 'taking the max action of a continuous output is an optimization problem itself!'

A policy gradient method sidesteps this: 'with policy-gradient methods, we output a probability distribution over actions.' Instead of scoring every angle, the policy can say 'turn about this much, give or take' and pick from that spread. Weng agrees: 'It is natural to expect policy-based methods are more useful in the continuous space.'

Where it struggles

It is not magic. Hugging Face lists three weaknesses: 'Frequently, policy-gradient methods converges to a local maximum instead of a global optimum.' 'Policy-gradient goes slower, step by step: it can take longer to train (inefficient).' And 'Policy-gradient can have high variance.'

  • A local peak

    The climb can stop on a small hill when a taller one stands nearby. Our worst run is a small version of this.

  • Slow

    Steps are small, and because it is on-policy each update needs fresh tries. Hugging Face says only data from 'our most recent version' of the policy is used.

  • Noisy

    One return is a rough guess at how good a move was. Our 30 runs showed how much that matters.

Where it leads: actor-critic and PPO

Our baseline was a plain average. Wikipedia says a better baseline is one close to the true value of the situation, and 'One common approach is to train a separate function that estimates the value function, and use that as the baseline. This is one of the actor-critic methods, where the policy function is the actor and the value function is the critic.'

So the two ideas meet. The policy is the player. A value score, like those from Deep-Q Networks, is the coach saying how well things are going. Hugging Face calls this 'the actor-critic method, which is a combination of value-based and policy-based methods.'

One more name: proximal policy optimization, or PPO. In Wikipedia's words, it 'avoids pushing the parameter update too hard, and avoids changing the policy too much.' Wikipedia adds that a version with a penalty 'has been used in training reasoning language models with reinforcement learning from human feedback', the RLHF idea from the Reinforcement Learning lesson.

Recap

  • What it is

    Learn the chance of each move directly, using softmax to turn preferences into odds.

  • How it learns

    REINFORCE: after each try, make the moves you made more likely if the return was high, less likely if it was low.

  • What makes it steadier

    Compare each return with a baseline. In our 30 runs, the average chance of the best route was 0.99 with one and 0.86 without.

Tip

When you meet a new policy gradient method, ask two questions. How does it cut the noise? And how does it keep each step from being too big?

Next: actor-critic methods, a player and a coach learning together.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.