esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.18

Making predictions with a model

Training is done. Now the model has to answer for real. Here is what comes out, how to turn it into a decision, and how to judge it.

14 min readFree

Training is over. Now it counts.

A baker spends weeks testing a recipe. Then the shop opens, and customers line up. From that moment, the baker is no longer learning the recipe. They are using it, one customer at a time.

A model has the same two lives. First it trains on examples. Then it is used on new cases it has never seen. That second life is called prediction, and it is the whole reason the model exists. This lesson follows the one on tuning, and it shows what a prediction really is, how to act on it, and how to tell whether it is any good.

Key words

Prediction
The model's answer for one new case.
Inference
Another word for using a trained model to make a prediction. Wikipedia notes that in machine learning, the term inference is sometimes used to mean "make a prediction, by evaluating an already trained model".
Score
A number the model gives before any final decision, such as a chance between 0 and 1.
Threshold
A cut-off. Scores above it become one answer, scores below it become the other.
Baseline
A very simple guess you compare your model against, such as always saying the most common answer.
Error
The gap between a prediction and the true answer.

If "model", "label" and "features" feel hazy, the lesson on supervised learning covers them. If train and test data feel hazy, see Train and test data.

Two moments: training and prediction

Training is the slow, careful part. The model sees many examples with answers, and it adjusts its numbers until its guesses improve. Prediction is the quick part. The numbers are now fixed. A new case goes in, and an answer comes out.

Scikit-learn, a popular Python library, builds this split into its tools. Its glossary says the predict method "Makes a prediction for each sample". You call it only on a model that has been trained already. If the model was not fitted first, calling it should raise an error.

  • Training

    Happens once, or now and then. It uses labelled examples, takes the most time, and changes the model.

  • Prediction

    Happens again and again, for every new case. The model is not changed. It only reads its fixed numbers and gives an answer.

What a prediction looks like

The answer depends on the task. There are three common shapes. You met the two kinds of task in the supervised learning lesson. Here is what each hands back.

  1. 1

    1. A label

    A word from a fixed list, such as spam or not spam. Scikit-learn says that for a classifier, the prediction is in the same target space used in fitting, for example one of red, amber or green if those were the labels it trained on.

  2. 2

    2. A score or probability

    A number saying how strongly the model leans one way. Scikit-learn's guide says predict_proba gives conditional probability estimates for each class, while decision_function gives a decision score for each class.

  3. 3

    3. A number

    For regression tasks, the answer is a number on a scale, such as a price or an exam mark. The glossary says a regressor returns a numeric array.

Notice that the label comes last, even though it is the part people see. Scikit-learn's guide splits classification in two. One part is the statistical problem of learning a model to predict, ideally, class probabilities. The other is the decision problem to take concrete action based on those probability predictions.

Its own example is the weather. The first part answers "what is the chance that it will rain tomorrow?" The second answers "should I take an umbrella tomorrow?" The chance is the score. The umbrella is the decision. A model gives you the first. You, or your product, make the second.

From a score to a decision

Say the model scores one email 0.99 for spam and another 0.51. You must file each one: spam folder or inbox. You need a cut-off.

Google's Machine Learning Crash Course calls this a classification threshold. Examples with a probability above the threshold value are assigned to the positive class, the class you are testing for. Examples with a lower probability go to the negative class. In its spam example, a threshold of 0.5 classes both emails as spam, while a threshold of 0.95 classes only the 0.99 email as spam.

Is 0.5 always right? No. Scikit-learn says the default cut-offs (a probability above 0.5, or a decision score above 0) are most certainly not ideal for most use cases. It gives the example of a tool that helps doctors spot tumours. The doctors will most likely want to find everyone with cancer and miss no one, so they prioritise a high recall rate. That comes with the trade-off of potentially more false-positive predictions. In the guide's words, it is a risk worth taking because the cost of a missed cancer is much higher than the cost of further diagnostic tests.

Key words

Positive class
The answer you are looking for, such as spam or pass.
False alarm
The model says yes, but the true answer is no. Also called a false positive.
Miss
The model says no, but the true answer is yes. Also called a false negative.
Recall
Out of all the real yes cases, how many the model found.

So the threshold is a decision about mistakes. Raise it and the model says yes less often. Google's course puts it this way: with a threshold of 0.9999, a spam filter is highly unlikely to mislabel a legitimate email, but also likely to miss actual spam. Lower it and the model catches more, and raises more false alarms.

Tip

Ask a real person which mistake hurts more. A wrongly blocked email and a missed scam are not equal. The answer sets your threshold, not the model.

Worked example: pass or fail

Let us build a tiny model and watch it predict. The data is made up: how many hours each student studied (0 to 10), and whether they passed. A seed fixes the random numbers, so the run repeats exactly. The model is a small logistic model: it learns two numbers, w and b, and turns hours into a score between 0 and 1. (You do not need the maths yet. A later lesson covers it.)

Training uses 80 students. Then we meet 40 new students the model never saw. For each, it gives a score. We turn the scores into pass or fail at two thresholds, 0.5 and 0.8, and count the misses and false alarms.

Python
import math
import random

random.seed(3)

# Made-up data: hours a student studied (0 to 10) and whether they passed (1) or not (0).
def make(n):
    rows = []
    for _ in range(n):
        hours = random.uniform(0, 10)
        chance = 1 / (1 + math.exp(-(hours - 6.5)))
        rows.append((hours, 1 if random.random() < chance else 0))
    return rows

train, new = make(80), make(40)

# Training: find two numbers (w and b) by small repeated nudges.
w, b = 0.0, 0.0
for _ in range(2000):
    for hours, passed in train:
        score = 1 / (1 + math.exp(-(w * hours + b)))
        w -= 0.01 * (score - passed) * hours
        b -= 0.01 * (score - passed)

# Prediction: the trained numbers are fixed now. New data goes in, a score comes out.
def score(hours):
    return 1 / (1 + math.exp(-(w * hours + b)))

print("learned w =", round(w, 2), " b =", round(b, 2))
for hours, passed in new[:4]:
    print("hours", round(hours, 1), "-> score", round(score(hours), 2), " real answer:", passed)

def accuracy(threshold):
    hits = sum((score(h) >= threshold) == bool(p) for h, p in new)
    return hits / len(new)

passed_total = sum(p for _, p in new)

def count(threshold):
    caught = sum(score(h) >= threshold and p == 1 for h, p in new)
    flagged = sum(score(h) >= threshold for h, _ in new)
    return caught, flagged, passed_total - caught, flagged - caught

print("students who passed:", passed_total, "of", len(new))
for t in (0.5, 0.8):
    caught, flagged, missed, false_alarms = count(t)
    print("threshold", t, "-> says pass for", flagged, "| missed", missed, "| false alarms", false_alarms, "| accuracy", round(accuracy(t), 2))

majority = 1 if sum(p for _, p in train) * 2 >= len(train) else 0
base = sum(p == majority for _, p in new) / len(new)
print("baseline (always say", majority, ") accuracy", round(base, 2))

Output

learned w = 0.99  b = -6.19
hours 8.8 -> score 0.93  real answer: 1
hours 2.3 -> score 0.02  real answer: 0
hours 5.3 -> score 0.28  real answer: 1
hours 8.1 -> score 0.86  real answer: 0
students who passed: 17 of 40
threshold 0.5 -> says pass for 18 | missed 3 | false alarms 4 | accuracy 0.82
threshold 0.8 -> says pass for 14 | missed 5 | false alarms 2 | accuracy 0.82
baseline (always say 0 ) accuracy 0.57

Read the four sample rows first. The model gives 0.93 to a student who studied 8.8 hours, and that student passed. It gives 0.02 to a student with 2.3 hours, who failed. Good. But 5.3 hours scores only 0.28, yet that student passed. And 8.1 hours scores 0.86, yet that student failed. The model is a good guide, never a certain one.

  • At 0.5

    It says pass for 18 students. It misses 3 real passes and raises 4 false alarms. Accuracy, the share of right answers, is 0.82.

  • At 0.8

    It says pass for 14. It misses 5 real passes, with only 2 false alarms. Accuracy is again 0.82.

  • What changed

    The score of every student stayed the same. Only the cut-off moved, and the kind of mistakes moved with it. Same accuracy, different mistakes. If a missed pass hurts more, pick 0.5. If a false alarm hurts more, pick 0.8.

Always compare with a baseline

An accuracy of 0.82 sounds fine. But is it? You cannot tell until you know what a no-effort guess scores.

A baseline is that no-effort guess. For a yes or no task, the simplest one is to always say the most common answer. Scikit-learn's guide has the idea: when doing supervised learning, a simple sanity check consists of comparing one's estimator against simple rules of thumb. Its DummyClassifier has a most_frequent strategy that always predicts the most frequent label in the training set. The guide adds that with these strategies, the predict method completely ignores the input data.

In our example, most students in the training data failed, so the baseline says fail every time. That is the last line of the output above: 0.57. Our model's 0.82 beats it by a clear margin, so the model has learned something real.

  1. 1

    1. Pick the simple guess

    Most common answer for yes or no tasks. The average for number tasks.

  2. 2

    2. Score it the same way

    Use the same new data and the same measure as your model.

  3. 3

    3. Compare

    If the model does not clearly beat the baseline, it is not worth using yet.

Predicting a number, and its error

When the answer is a number, there is no threshold. The model's output is the prediction. The question is how far off it is.

That gap is the prediction error. For one case, it is the model's guess minus the true answer, with the sign ignored. Averaged over many new cases, it tells you the typical miss. In words: add up all the gaps, then divide by how many cases there are.

typical miss=1n∑i=1n∣guessi−truthi∣\text{typical miss} = \frac{1}{n}\sum_{i=1}^{n} \left| \text{guess}_i - \text{truth}_i \right|

The baseline for numbers is the mean, meaning the average of the training answers. Guess it for everyone. Here is the same idea on made-up exam marks. The model is a straight line fitted to 60 students, and then it is tested on 30 new ones.

Python
import random

random.seed(5)

# Made-up data: hours studied and the exam mark (out of 100).
def make(n):
    rows = []
    for _ in range(n):
        hours = random.uniform(0, 10)
        rows.append((hours, 30 + 6 * hours + random.gauss(0, 8)))
    return rows

train, new = make(60), make(30)

# Training: fit a straight line, mark = a * hours + c, by least squares.
mx = sum(h for h, _ in train) / len(train)
my = sum(m for _, m in train) / len(train)
a = sum((h - mx) * (m - my) for h, m in train) / sum((h - mx) ** 2 for h, _ in train)
c = my - a * mx

def typical_miss(guess):
    return sum(abs(guess(h) - m) for h, m in new) / len(new)

print("line: mark =", round(a, 1), "* hours +", round(c, 1))
print("model predicts for 8 hours:", round(a * 8 + c, 1))
print("typical miss, model:   ", round(typical_miss(lambda h: a * h + c), 1))
print("typical miss, baseline:", round(typical_miss(lambda h: my), 1), "(always the mean mark", round(my, 1), ")")

Output

line: mark = 6.0 * hours + 31.6
model predicts for 8 hours: 79.4
typical miss, model:    7.7
typical miss, baseline: 15.0 (always the mean mark 62.7 )

For a student who studied 8 hours, the line predicts 79.4. The typical miss of the model is 7.7 marks. The baseline, which says 62.7 for everyone, misses by 15.0 on average. So the model's error is about half the baseline's. That is a fair win.

Quick check

A model for exam marks has a typical miss of 7.7. The mean-mark baseline misses by 15.0. What is the best reading?

Why a model is never certain

Look back at the pass or fail output. A student with 8.1 hours got 0.86 and failed anyway. No bug caused that. It is how prediction works.

A model finds patterns in past examples. Real life has more going on than the model can see: sleep, stress, a bad exam day. The data also holds some luck. So the best a model can do is lean one way, and a score is a lean, not a promise.

  • Data has noise

    Two students with the same hours can end up with different results. The model sees only hours.

  • Models are simple

    A model is a small rule standing in for a messy world. It is an approximation.

Quick check

The model gives a student a score of 0.86. The student fails. What does this show?

New data it has never seen

One rule runs through this whole lesson. A prediction only counts when it is made on new data. In the pass or fail example, the 80 training students taught the model, and the 40 new students tested it. Every number above came from the new 40.

This is the idea of generalising, from the train and test lesson: doing well on cases beyond the ones you learned from. Wikipedia describes the thinking behind it. Inferential analysis assumes the observed data is sampled from a larger population, and tries to say something about that wider group. The students we have are a sample. The students who come later are the group we really care about.

A model that scores well on its training data but badly on new data has memorised, not learned. So always judge a model on cases it did not train on.

Tip

Before you trust any quoted number, ask one question: was this measured on data the model had never seen?

What to remember

  • Answers

    Prediction is the model working on new cases. It gives a label, a score or a number. Training happens first, and prediction does not change the model.

  • Decisions

    A score becomes a decision at a threshold. Move the threshold and the mix of misses and false alarms moves. Choose it by which mistake costs more.

  • Honesty

    Compare with a baseline, report the error, and judge on new data. A model is never certain, so say "likely", not "sure".

Next: classification, sorting things into groups.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.