Machine Learning Foundations / Core / 02.07
Supervised learning: learning from answers
In supervised learning a model studies examples that already come with the right answer, then predicts the answer for new ones. You will build a tiny one yourself.
13 min readFree
What supervised learning is
Picture a child learning fruit names. A parent holds up a fruit and says 'apple'. Then another and says 'orange'. After enough of these, the child can name a fruit they have never held. Someone gave the right answer each time. That is the whole idea of supervised learning.
IBM defines it as a machine learning technique 'that uses labeled data sets to train artificial intelligence (AI) models to identify the underlying patterns and relationships'. Wikipedia explains the name: 'supervised' refers to 'the role of a teacher or supervisor who provides this training data'. The teacher is the person, or the measuring device, that supplied the answers.
The goal, IBM says, is 'to create a model that can predict correct outputs on new real-world data'. In plain words: learn from examples with answers, then answer questions where nobody has given you the answer yet.
Key words
- Example
- One item to learn from. Google: 'You could think of an example as analogous to a single row in a spreadsheet.'
- Feature
- A value the model looks at, like a fruit's weight. Google: features are 'the values that a supervised model uses to predict the label'.
- Label
- The answer for that example. Google: 'The label is the "answer," or the value we want the model to predict.'
- Labeled example
- An example that has both features and a label. An unlabeled example has features but no label.
- Model
- What training produces. Google calls it a collection of numbers that define the link from features to labels.
- Loss
- How far a prediction is from the real label. Zero would mean a perfect guess.
Two kinds of question
Wikipedia says supervised learning is commonly used for 'classification (predicting a category, e.g., spam or not spam)' and 'regression (predicting a numerical value, e.g., a house price)'. The scikit-learn guide puts it the same way: classification is 'for data with discrete labels', and regression is for 'data with continuous labels'. Discrete means separate boxes, like apple or orange. Continuous means any number on a scale, like 12.5 or 12.6.
Tip
A quick test: look at the label. If you could write the possible answers in a short list, it is classification. If the answer is a number that could land anywhere on a scale, it is regression.
Classification: pick a category
Google defines classification as 'the task of predicting which of a set of classes (categories) an example belongs to'. A class is just one of the possible answers.
IBM gives a picture example: a model that looks at photos of vehicles and says which type it sees. To build it you prepare 'a labeled training dataset containing numerous vehicle examples along with the corresponding vehicle type: car, motorcycle, truck, bicycle and more'. Wikipedia gives another: show a model many images explicitly labeled 'cat', so it learns to spot cats.
Two classes
Yes or no, true or false. Google calls this binary classification. Our apple or orange example is this kind.
Many classes
More than two answers, like the vehicle types above. Google calls this multi-class classification.
Your own idea
Is this screenshot a wireframe or a finished screen? Any question with a short list of answers works.
Regression: predict a number
IBM says that in regression problems 'the output is a continuous value'. Its examples are 'projections for sales revenue or financial planning'.
Google's linear regression lesson uses cars. The feature is the car's weight, in thousands of pounds. The label is miles per gallon. A 4.42 car does 14, and a lighter 2.37 car does 24. Plot the points and they slope downward: heavier cars go fewer miles. A regression model learns that slope and uses it for a car it has not seen.
Quick check
A model reads a flat's size, floor and street, and outputs its monthly rent. Which kind of task is it?
Where labels come from
Every supervised model is only as good as its answers, so the labels matter a lot. IBM calls a labeled data set 'ground truth': data 'verified against real-world outcomes, often through human annotation or measurement'. Annotation means a person marking each item with its answer.
That costs time. IBM notes that 'data scientists manually create ground truth training datasets containing input data along with the corresponding labels'. Someone has to look at each photo and type 'bicycle'. Google adds that 'large and diverse datasets produce a better model', so you want lots of labels, and that is exactly the expensive part.
Some labels need no person at all. A sensor can measure the answer. Measurement and annotation are the two sources IBM names.
Wrong labels
People get tired and sensors break. Wikipedia describes the problem: 'If the desired output values are often incorrect (because of human error or sensor errors), then the learning algorithm should not attempt to find a function that exactly matches the training examples.' Wrong labels are called label noise. The data cleaning lesson covered it: a wrong label is worse than a wrong feature, because the label is the answer key.
Wikipedia also lists a remedy: 'detecting and removing the noisy training examples prior to training'. Here is why that matters. We use a method we will build below, which copies the answer of the closest known fruit. A fruit of 155 g and 7.1 cm is closest to the known 150 g, 7.0 cm fruit. We run it twice: once with the right label for that fruit, and once after a typo changes it.
import math
train = [
(150, 7.0, "apple"),
(170, 7.5, "apple"),
(140, 6.8, "apple"),
(130, 6.5, "orange"),
(160, 8.0, "orange"),
(180, 8.5, "orange"),
]
def distance(a, b):
return math.sqrt((a[0] - b[0]) ** 2 + (a[1] - b[1]) ** 2)
def predict_label(rows, new):
return min(rows, key=lambda fruit: distance(fruit, new))[2]
new = (155, 7.1)
print("clean labels:", predict_label(train, new))
# one person typed the wrong answer for the 150 g fruit
noisy = [(150, 7.0, "orange")] + train[1:]
print("one wrong label:", predict_label(noisy, new))Output
clean labels: apple one wrong label: orange
One typo, and the answer for a new fruit flips.
The training loop
How does a model get from a pile of answers to a rule? Google describes it: the model 'finds the best solution by comparing its predicted value to the label's actual value'. Then, 'based on the difference between the predicted and actual values, defined as the loss, the model gradually updates its solution'. Parameters are the numbers inside the model, like settings on a dial.
- 1
1. Predict
Show the model the features of one example, or a whole batch. It gives its guess.
- 2
2. Compare
Check the guess against the label. The gap between them is the loss. Big gap, big loss.
- 3
3. Adjust
Change the model's numbers a little, in the direction that should shrink the loss. IBM names gradient descent as the most common family of methods for this, which a later lesson explains.
- 4
4. Repeat
Go round again. Google says the model may look at each example 'in some cases, multiple times', until it settles on a solution that makes 'the best predictions, on average'.
Here is the loop in a few lines. Our fruit shop prices fruit by weight. The price of four fruits is known. The model is just one number, w, and it guesses 'price = w times the weight'. It starts at 0, a bad guess. Each turn it measures the average gap (the loss) and nudges w up when its guesses are too low.
# price in rupees for a fruit, from its weight in grams
weights = [100, 150, 200, 250]
prices = [10, 15, 20, 25]
w = 0.0 # start with a bad guess: price = w * weight
step = 0.01
for turn in range(1, 6):
predictions = [w * x for x in weights]
loss = sum(abs(p - y) for p, y in zip(predictions, prices)) / len(prices)
print(f"turn {turn}: w = {w:.2f}, average error = {loss:.2f}")
# nudge w in the direction that shrinks the error
direction = sum((y - p) * x for x, p, y in zip(weights, predictions, prices))
w += step if direction > 0 else -stepOutput
turn 1: w = 0.00, average error = 17.50 turn 2: w = 0.01, average error = 15.75 turn 3: w = 0.02, average error = 14.00 turn 4: w = 0.03, average error = 12.25 turn 5: w = 0.04, average error = 10.50
The loss fell every turn, from 17.50 to 10.50. Bigger models have more numbers to adjust, but the rhythm stays the same.
Train, validate, test
A model that is right on the examples it studied has proved little, like a student who memorised the answer sheet. What matters is new data. Google's evaluation step does it this way: 'we use a labeled dataset, but we only give the model the dataset's features. We then compare the model's predictions to the label's true values.'
So the labeled data is split into parts. Wikipedia names two of the jobs: parameters 'may be adjusted by optimizing performance on a subset (called a validation set) of the training set', and the result 'should be measured on a test set that is separate from the training set'.
Training set
The examples the model learns from. Its labels drive the loop above.
Validation set
A part held back while you build, used to compare settings and choices.
Test set
Kept apart until the end. It tells you honestly how the finished model does on data it never saw.
Once the model passes the evaluation, Google calls the last step inference: using the model 'to make predictions, called inferences, on unlabeled examples'. A later lesson goes deep on splitting.
Quick check
Why do we test a model on examples it did not train on?
A tour of model families
The loop is the same everywhere. What changes is the shape of the model. Here is a map by name, one line each. Later lessons go deep, so do not memorise details yet. The scikit-learn guide for supervised learning lists most of these as sections.
- 1
Linear regression
IBM: it 'expresses the relationship between variables as a straight line'. Used for predicting numbers.
- 2
Logistic regression
IBM: it 'handles categorical dependent variables' with two outcomes and 'mainly solves binary classification problems'. Despite the word regression in the name, it is used to pick a category.
- 3
k-nearest neighbours
Look at the closest known examples and copy their answer. We build one below.
- 4
Decision trees and random forests
IBM: a random forest is 'a collection of uncorrelated decision trees which are merged to reduce variance and increase accuracy'.
- 5
Support vector machines
IBM: an SVM 'separates the classes of data points with a decision boundary or hyperplane' and it 'usually handles classification problems'.
- 6
Gradient boosting
Listed by scikit-learn among its ensembles. IBM describes ensemble learning as training 'multiple models' on the same task and aggregating their results.
- 7
Neural networks
IBM says they 'excel at handling complex classification problems'. They are built from layers of nodes.
Tip
No model wins everywhere. Wikipedia advises that 'it is often better to spend more time collecting additional training data and more informative features than it is to spend extra time tuning the learning algorithms'. Better data usually beats a fancier model.
Worked example: fruit by nearest neighbour
Now we build one ourselves, with plain Python and no libraries beyond math. The idea behind nearest neighbours, in scikit-learn's words: 'find a predefined number of training samples closest in distance to the new point, and predict the label from these'. The usual measure is 'standard Euclidean distance', which is the straight-line distance between two points. With one neighbour, we simply copy the label of the closest known fruit.
scikit-learn calls these 'non-generalizing' methods, since they 'simply "remember" all of its training data'. There is no learning loop here. The memory is the model.
Classifying a fruit
Our table has six fruits with weight in grams, width in centimetres and a label. We keep three more fruits aside as a test set, with their true labels hidden from the model. The function finds the nearest training fruit and returns its label.
import math
# weight in grams, width in cm, label
train = [
(150, 7.0, "apple"),
(170, 7.5, "apple"),
(140, 6.8, "apple"),
(130, 6.5, "orange"),
(160, 8.0, "orange"),
(180, 8.5, "orange"),
]
def distance(a, b):
return math.sqrt((a[0] - b[0]) ** 2 + (a[1] - b[1]) ** 2)
def predict_label(new):
nearest = min(train, key=lambda fruit: distance(fruit, new))
return nearest[2]
# fruits the model has never seen, with the true answer kept aside
test = [(155, 7.1, "apple"), (175, 8.4, "orange"), (135, 6.6, "orange")]
right = 0
for weight, width, truth in test:
guess = predict_label((weight, width))
right += guess == truth
print(f"{weight} g, {width} cm -> {guess} (truth: {truth})")
print(f"{right} of {len(test)} right")Output
155 g, 7.1 cm -> apple (truth: apple) 175 g, 8.4 cm -> orange (truth: orange) 135 g, 6.6 cm -> orange (truth: orange) 3 of 3 right
All three right. The weight counts for far more than the width, because grams are bigger numbers than centimetres. That is a trap with distance methods: a feature on a big scale drowns the others. A later lesson on feature scaling fixes it. IBM adds another limit for this family: 'as the test dataset grows, the processing time lengthens'.
Predicting a number the same way
The same idea works for numbers. The scikit-learn guide says the label for a new point 'is computed based on the mean of the labels of its nearest neighbors'. The mean is the plain average. Here we predict a fruit's price from its weight, by averaging the prices of the two nearest known fruits. The last step differs: a vote for classification, an average for regression.
# weight in grams -> price in rupees
prices = [(100, 10), (120, 13), (150, 15), (200, 22), (250, 26)]
def predict_price(weight, k=2):
nearest = sorted(prices, key=lambda row: abs(row[0] - weight))[:k]
return sum(price for _, price in nearest) / k
for weight in [110, 175, 240]:
print(f"{weight} g -> about {predict_price(weight)} rupees")Output
110 g -> about 11.5 rupees 175 g -> about 18.5 rupees 240 g -> about 24.0 rupees
Check one by hand. The nearest weights to 110 g are 100 g (10 rupees) and 120 g (13 rupees), and their average is 11.5.
Quick check
In the price example, the model predicts 24.0 rupees for a 240 g fruit. What is that number made from?
What to remember
Supervised learning learns from examples that come with answers. A category label means classification, a number label means regression. Labels come from people or measurement, and wrong ones teach wrong lessons. Training is a loop: predict, compare through the loss, adjust, repeat. Keep a test set apart, because the goal is new data.
Next: unsupervised learning. What can a model do when there are no answers at all?
Sources
- IBM: What is supervised learning?
- Google Machine Learning: Supervised learning
- Google Machine Learning Crash Course: Linear regression
- Google Machine Learning Crash Course: Classification
- scikit-learn user guide: Supervised learning
- scikit-learn user guide: Nearest neighbors
- Wikipedia: Supervised learning
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.