Machine Learning Foundations / Core / 02.11
Learning from rewards, not answers
Reinforcement learning learns by trying things and getting a score. Meet the agent, reward and policy, then run a tiny experiment in plain Python.
14 min readFree
The idea in one minute
Picture a video game you have never played. There is no manual and nobody beside you. You press a button and your character jumps. You press another and you fall in a hole and lose a life. After a few tries you know what to avoid. After a few hundred you can finish a level.
Nobody handed you an answer sheet. You tried things, saw what happened, and got a score. That is reinforcement learning.
IBM defines it like this: 'Reinforcement learning (RL) is a type of machine learning process in which autonomous agents learn to make decisions by interacting with their environment.' Wikipedia says it is 'concerned with how an intelligent agent should take actions in a dynamic environment in order to maximize a reward signal.'
This lesson is about the idea only. The famous methods, such as Q-learning, deep Q-networks, policy gradients and actor-critic, come in later lessons. First we need the words, and a small experiment you can run yourself.
Key words
- Agent
- The learner and decision-maker. In the game, that is you. In a program, it is the piece of code that picks what to do.
- Environment
- Everything the agent acts on and gets answers from. In the game, it is the game world.
- State
- What the situation looks like right now: where your character stands, how many lives are left.
- Action
- One thing the agent can do: jump, run, wait.
- Reward
- A number the environment hands back after an action. Good outcomes give a high number. Bad ones give a low or negative number.
- Episode
- One full try from start to finish, such as one game from the first screen until you win or lose.
- Policy
- The agent's habit: a rule that says which action to take in which state. IBM describes it as 'mapping perceived environmental states to specific actions'.
- Value
- How good it is to be in a situation, counting the rewards that are likely to follow, not only the next one.
The loop: look, act, get a score
Every reinforcement learning problem runs the same small loop again and again. IBM puts it this way: 'The environment provides information on its current state. The agent then uses that information to determine which actions(s) to take. If that action obtains a reward signal from the surrounding environment, the agent is encouraged to take that action again when in a similar future state.'
- 1
1. The agent looks at the state
It sees where things stand: the game screen, the sensor readings, the board.
- 2
2. It picks an action
It uses its policy, or tries something new. At the start the policy is little more than a guess.
- 3
3. The environment answers
The world moves to a new state and sends back a reward, which may be zero.
- 4
4. The agent learns
It adjusts its policy so that actions that led to good scores become more likely.
- 5
5. Repeat until the episode ends
Then start a new episode and keep getting better.
Notice what is missing: a teacher who says 'the right move here was X'. The only feedback is the reward, and it often says very little about why.
Tip
A quick way to remember the cast: the agent acts, the environment reacts, the reward is the score. The policy is the agent's plan, and value is its hunch about how things will turn out.
The goal: the most total reward
The agent is not trying to win each single step. It is trying to collect the most reward over the whole run. IBM says: 'The agent’s only objective is to maximize its cumulative rewards from the environment.' Cumulative just means added up.
That small difference matters. Think about studying. Tonight, studying feels bad: you give up a film and get a small negative reward. Next month, passing the exam feels great. A learner who only chases the next step would never open the book. A learner who adds everything up would.
Wikipedia makes the same point. To act well, the agent must reason about long-term results, 'although the immediate reward associated with this might be negative.' It adds that reinforcement learning 'is particularly well-suited to problems that include a long-term versus short-term reward trade-off.'
Delayed reward: the score arrives late
Often the reward comes long after the action that earned it. In a board game, you may make fifty moves and only learn at the end whether you won. Which of the fifty moves was the good one? The agent has to work that out for itself.
Discounting: sooner is worth a little more
Most of us prefer ten pounds today to ten pounds in a year. Reinforcement learning builds that habit in with a discount. Each step further into the future, a reward counts a bit less.
Wikipedia writes the total as a sum: 'R 1 + γ R 2 + γ 2 R 3 + ⋯'. The Greek letter gamma (γ) is the discount rate, a number below 1. Wikipedia says: 'γ is less than 1, so rewards in the distant future are weighted less than rewards in the immediate future.'
In words: the first reward counts in full. The second is multiplied by gamma. The third is multiplied by gamma twice, and so on, so each step into the future shrinks it a little more. Let us see it with five rewards of 1 each.
rewards = [1, 1, 1, 1, 1]
for gamma in (1.0, 0.9, 0.5):
total = sum(r * gamma ** t for t, r in enumerate(rewards))
print(f"discount {gamma}: total = {total:.3f}")Output
discount 1.0: total = 5.000 discount 0.9: total = 4.095 discount 0.5: total = 1.938
With no discount (1.0), five rewards add to exactly 5. With 0.9, the later ones shrink and the total is about 4.1. With 0.5, the future barely counts and the total drops to about 1.9. A gamma near 0 makes a short-sighted agent. A gamma near 1 makes a patient one.
Reward versus value
These two words are easy to mix up. IBM keeps them apart: 'Reward signal differs from value function in that the former denotes immediate benefit while the latter specifies long-term benefit.'
Think of a map app. The reward is how pleasant this next street feels. The value is how good it is to be on this street, given where it leads. A shortcut through a car park may look quick now but end at a locked gate. Its value is low. IBM gives a road example too: a self-driving car could save time by driving on the sidewalk, but that would 'reduce its overall value function', so it accepts a slightly longer trip.
Explore or exploit?
You have just moved to a new town. The first cafe you try has good coffee. Do you go back every day? Or do you try the other cafes in case one is better?
That tension has two names. Exploitation means using what you know to take the best-known option. Exploration means trying something new to learn more. Wikipedia describes the agent choosing 'between trying new actions to learn more about the environment (exploration), or using current knowledge of the environment to take the best action (exploitation)', and calls the search for the right balance 'the exploration–exploitation dilemma'.
Neither extreme works. IBM says: 'The agent cannot exclusively pursue exploration or exploitation. It must continuously try new actions while also preferring single (or chains of) actions that produce the largest cumulative reward.'
All exploit
You stay loyal to the first decent cafe forever. You may miss a much better one two streets away.
All explore
You try a new cafe every day and never settle. You learn a lot and enjoy little.
A mix
Mostly go to your favourite, and now and then try somewhere new. This is what good learners do.
Epsilon-greedy: a simple mix
The simplest recipe is named epsilon-greedy. 'Greedy' means always take the best-known option. Epsilon (ε) is a small chance, such as 10 percent, of doing something random instead.
Wikipedia's bandit page puts it formally: 'The best lever is selected for a proportion 1 − ϵ of the trials, and a lever is selected at random (with uniform probability) for a proportion ϵ. A typical parameter value might be ϵ = 0.1, but this can vary widely depending on circumstances and predilections.'
So with epsilon at 0.1, the agent picks its favourite about nine times in ten, and picks any option at random about once in ten. The random ones are the exploring.
Tip
A bigger epsilon explores more and earns less right now. A smaller one earns more now and risks missing a better option. There is no perfect number. It depends on the problem.
How it differs from supervised learning
In supervised learning, every example arrives with its correct answer, and the model learns to match it. Reinforcement learning has no answer key. IBM says: 'In contrast to supervised learning, reinforcement learning does not use labeled examples of correct or incorrect behavior.' It also says RL 'differs from unsupervised learning in that reinforcement learning learns by trial-and-error and reward function rather than by extracting information of hidden patterns.'
Wikipedia counts reinforcement learning as one of 'the three basic machine learning paradigms, alongside supervised learning and unsupervised learning.' IBM sums up the difference in four words: 'reinforcement learning learns to act.'
Supervised learning
Gets the right answer for every example. It learns to predict. Feedback is instant and exact.
Unsupervised learning
Gets no answers at all. It looks for hidden patterns in the data.
Reinforcement learning
Gets a score for what it did, often late and rarely a full explanation. It learns to act, and its own actions change what it sees next.
That last point is the key. In supervised learning the next photo does not depend on your last guess. In reinforcement learning, what you do changes the world you see next. IBM describes the data as 'interdependent tuples', an ordered sequence 'organized as state-action-reward'.
Quick check
Which of these is a reinforcement learning setup?
Where it is used
Here is what our sources say, and no more.
Games and control
Wikipedia says reinforcement learning 'has been applied successfully to various problems, including energy storage, robot control, photovoltaic generators, backgammon, checkers, Go (AlphaGo), and autonomous driving systems.'
Robots
IBM says that because RL is 'centrally concerned with decision-making in unpredictable environments, it has been a core area of interest in robotics.' It also notes that deep reinforcement learning with robots 'relies heavily on collected datasets'.
Chat models
Hugging Face explains RLHF, short for reinforcement learning from human feedback: 'use methods from reinforcement learning to directly optimize a language model with human feedback.' It adds that 'RLHF's most recent success was its use in ChatGPT'.
The chat-model case is a good example of the idea. Hugging Face says writing a loss function for what makes good text 'seems intractable', which is a fancy way of saying it is too hard. So human annotators rank the model's answers, and the article says a reward model is trained so that it 'takes in a sequence of text, and returns a scalar reward which should numerically represent the human preference.' That number is then the score the language model learns to raise. A scalar is just a single number.
Worked example: three lunch stalls
Let us build the smallest honest reinforcement learner. It is called a multi-armed bandit. Wikipedia names it after a gambler at a row of slot machines, and says it 'is a classic reinforcement learning problem that exemplifies the exploration–exploitation tradeoff dilemma.' It also says that 'the selected actions in bandit problems do not affect the reward distribution of the arms', so it is a simpler cousin of full reinforcement learning: no changing states, only a choice.
We will use a gentler picture: three lunch stalls. Each has a hidden chance that your meal is good. Stall 0 is good 30 percent of the time, stall 1 half the time, and stall 2 70 percent of the time. We made these numbers up. The agent does not know them. It only tastes and scores: 1 for a good meal, 0 for a bad one.
- 1
1. Set the hidden rates and epsilon
The true rates are 0.3, 0.5 and 0.7. Epsilon is 0.1.
- 2
2. Keep a score card
For each stall, count the visits and keep an estimated value: the average reward so far. These start at zero.
- 3
3. Choose by epsilon-greedy
With a 10 percent chance, pick a random stall (explore). Otherwise pick the stall with the best estimate (exploit).
- 4
4. Taste and update
Get a 1 or a 0. Nudge that stall's estimate toward the new result.
- 5
5. Repeat 200 times
Then print what the agent believes and how often it went to each stall.
import random
random.seed(7)
true_rates = [0.3, 0.5, 0.7] # hidden from the agent
epsilon = 0.1
pulls = [0, 0, 0]
values = [0.0, 0.0, 0.0] # estimated win rate per arm
total_reward = 0
for step in range(200):
if random.random() < epsilon:
arm = random.randrange(3) # explore
else:
arm = values.index(max(values)) # exploit
reward = 1 if random.random() < true_rates[arm] else 0
pulls[arm] += 1
values[arm] += (reward - values[arm]) / pulls[arm]
total_reward += reward
for i in range(3):
print(f"arm {i}: true {true_rates[i]:.2f}, estimated {values[i]:.2f}, pulled {pulls[i]} times")
print("total reward:", total_reward, "out of 200")Output
arm 0: true 0.30, estimated 0.58, pulled 12 times arm 1: true 0.50, estimated 0.60, pulled 5 times arm 2: true 0.70, estimated 0.72, pulled 183 times total reward: 141 out of 200
Here the stall numbers are the arms. Stall 2 is best, and the agent went there 183 times out of 200. It scored 141 good meals. Always going to the best stall would average 140 (200 times 0.7), so this run was lucky, not magic.
Look at the estimates too. They are close for stall 2, which was visited a lot (0.72 against a true 0.70). But stall 0 shows 0.58 when the truth is 0.30. It was only tried 12 times, so a few good meals fooled the agent. Few visits give a noisy picture. That is why exploring matters.
Now let us see what exploring buys. The next script runs three agents, with the same seed. One never explores (epsilon 0.0), one uses 0.1, and one always picks at random (epsilon 1.0).
import random
random.seed(7)
true_rates = [0.3, 0.5, 0.7]
for epsilon in (0.0, 0.1, 1.0):
pulls = [0, 0, 0]
values = [0.0, 0.0, 0.0]
total_reward = 0
for step in range(200):
if random.random() < epsilon:
arm = random.randrange(3)
else:
arm = values.index(max(values))
reward = 1 if random.random() < true_rates[arm] else 0
pulls[arm] += 1
values[arm] += (reward - values[arm]) / pulls[arm]
total_reward += reward
print(f"epsilon {epsilon}: pulls {pulls}, total reward {total_reward}")Output
epsilon 0.0: pulls [200, 0, 0], total reward 62 epsilon 0.1: pulls [34, 8, 158], total reward 135 epsilon 1.0: pulls [66, 71, 63], total reward 93
The agent that never explores went to stall 0 all 200 times. All estimates start at zero and ties go to stall 0, so it began there, and nothing ever pushed another stall above it. It never learned the others existed. Its score was 62, around the 30 percent we expect from stall 0. The always-random agent explored forever and scored 93, close to the 100 we expect from the stalls' average. The mix scored 135. Each run is only one sample, so the exact numbers will move with the seed, but the pattern is the lesson.
Quick check
In the comparison, why did the epsilon 0.0 agent score so low?
Recap
What it is
An agent acts in an environment, gets a reward, and learns a policy that collects the most reward over time.
What makes it different
No answer key, and rewards often come late. Its own actions change what it sees next.
The key trade-off
Explore to learn, exploit to earn. Epsilon-greedy mixes the two with a small chance of a random move.
Tip
When you read about a new reinforcement learning system, ask three questions. What is the state? What are the actions? And who decided the reward?
Next: loading data into your program.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.