Machine Learning Foundations / Core / 02.05
What is Machine Learning?
A second, deeper look: the formal definition, loss, generalisation, the steps of a real project, and when not to use machine learning at all.
14 min readFree
A second look
In the first lesson you met machine learning in plain words: a computer learns a pattern from examples instead of following rules a person typed in. You have since learned a lot about data: where it comes from, its formats, how to clean it, and how to scale it and trim it. Now we can say what machine learning is more carefully.
This lesson adds the exact words practitioners use, the idea of a loss, the steps of a real project, and an honest look at when not to use machine learning. You will also fit a line to data in plain Python and watch the loss fall.
Key words
- Task
- The job we want done, such as 'guess the price' or 'say spam or not spam'.
- Experience
- The data the program learns from.
- Performance measure
- A number that tells us how well the job is going.
- Model
- What training produces: a rule with its numbers filled in.
- Algorithm
- The recipe that finds those numbers.
- Parameter
- A number inside the model that training finds by itself.
- Hyperparameter
- A setting a person chooses before training starts.
- Loss
- A number for how wrong the model's guesses are. Lower is better.
- Generalisation
- How well the model does on data it has never seen.
The formal definition
Tom Mitchell wrote a well-known textbook, Machine Learning (McGraw Hill, 1997). His book page opens with a short line: 'Machine Learning is the study of computer algorithms that improve automatically through experience.'
Wikipedia says Mitchell also gave a 'more formal definition of the algorithms studied in the machine learning field', and quotes it in full:
"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E."It looks heavy, but it has three simple parts. T is the task. E is the experience. P is the performance measure. A program has learned when P goes up as E grows. Wikipedia adds that this definition is 'fundamentally operational rather than defining the field in cognitive terms'. In plain words: it does not ask whether the machine 'understands' anything. It only asks whether the machine gets better at a job, and it gives you a way to check.
Spam filter (our example)
T: mark each email as spam or not. E: emails that people already marked. P: the share marked correctly.
Price guesser (our example)
T: guess a price from a size. E: past sales. P: how far the guesses are from the real prices.
Photo sorter (our example)
T: say what a photo shows. E: photos with names. P: how many names it gets right.
Tip
Before any project, write one sentence: 'The program should do T, learn from E, and be judged by P.' If a part is missing, find it first.
Model, algorithm and parameters
Beginners often mix up two words: algorithm and model. The algorithm is the recipe for learning. The model is the result of following that recipe on your data. Like a recipe and a cake: the same recipe with other ingredients gives another cake.
Inside a model are numbers that decide its guesses. In a straight-line rule like 'score = slope × hours + start', the slope and the start are such numbers. Training finds them. These are the parameters.
Then there are settings that you choose before training begins, for example how many steps to take or how simple the model is allowed to be. These are hyperparameters. Training does not find them. You pick them, test them and change them.
A warning about words. The scikit-learn glossary says: 'We do not use parameters in the statistical sense, where parameters are values that specify a model and can be estimated from data.' In that library, 'parameters' are the options you pass in when you create an estimator. The glossary says these 'might be what statisticians call hyperparameters to the model: aspects for configuring model structure that are often not directly learnt from data.' So the same word can mean different things in different places. When you read a tutorial, check which meaning it uses.
Quick check
Which of these is a hyperparameter?
Loss: how wrong is the model?
To learn, a program needs a way to tell whether it is getting better. That is the job of the loss. Google's Machine Learning Crash Course says loss is 'a numerical metric that describes how wrong a model's predictions are'. It adds that loss 'measures the distance between the model's predictions and the actual labels' and that 'the goal of training a model is to minimize the loss, reducing it to its lowest possible value.'
Picture darts: the closer all the darts land to the bullseye, the lower the round's score. A perfect round scores zero.
Google points out that loss looks at distance, not direction. If a model predicts 2 and the real value is 5, we do not care that the difference is negative. We care that the distance is 3. So every loss method removes the sign, and the two most common ways are to take the absolute value of the difference or to square it.
Mean absolute error (MAE)
The average of the absolute differences between the actual values and the predictions.
Mean squared error (MSE)
The average of the squared differences. Google says that when the difference is large, 'squaring makes the loss even larger.'
Which to pick
Google says to choose MSE if you want to 'heavily penalize large errors', and MAE if your data has significant outliers that you do not want to overly influence the model.
In words: for each example, take the actual value minus the predicted value and square it. Add them all up. Divide by the number of examples N. A small result means the guesses are close.
Training is lowering the loss
Now we can say what training really is. Training means finding the parameters that give the lowest loss. Google calls gradient descent 'a mathematical technique that iteratively finds the weights and bias that produce the model with the lowest loss.' In our words, weights and bias are the parameters.
You met the ideas behind this in the calculus lessons. A derivative tells you how fast something changes, and the gradient points in the direction where a value rises fastest. If you go the opposite way, the loss goes down. Gradient descent uses exactly that. Google describes its loop like this:
- 1
1. Calculate the loss
Use the current parameters and measure how wrong the guesses are.
- 2
2. Find the direction
Work out which way to move the parameters so that the loss gets smaller.
- 3
3. Move a small amount
Change the parameters a little in that direction.
- 4
4. Repeat
Go back to step one, until the model cannot reduce the loss any further.
Think of walking down a foggy hill. You cannot see the valley, but you can feel which way the ground slopes under your feet. You take a small step downhill, feel the slope again, and repeat. The height is the loss and your position is the set of parameters.
Google adds that the model starts with 'randomized weights and biases near zero'. Linear models have a convex loss, which means gradient descent will always find the best model for the given data.
Try it: fit a line by hand
Here is a tiny, honest experiment in plain Python with no libraries. We have five students. For each, we know the hours they studied and their exam score. We guess that the score follows a straight line: score = slope × hours + start. To keep it simple we fix the start at 46 and try five slopes. For each one we print the mean squared error. The slope with the lowest loss wins. Then we ask the model about a student it has never seen.
hours = [1, 2, 3, 4, 5]
scores = [52, 58, 65, 69, 76]
def predict(slope, start, x):
return slope * x + start
def mse(slope, start):
total = 0
for x, y in zip(hours, scores):
error = predict(slope, start, x) - y
total += error * error
return total / len(hours)
best_slope = None
best_loss = None
for slope in [2, 4, 6, 8, 10]:
loss = mse(slope, 46)
print("slope", slope, "loss", round(loss, 2))
if best_loss is None or loss < best_loss:
best_slope, best_loss = slope, loss
print("best slope:", best_slope)
new_hours, new_score = 6, 83
guess = predict(best_slope, 46, new_hours)
print("prediction for 6 hours:", guess)
print("real score:", new_score, "off by:", abs(guess - new_score))Output
slope 2 loss 174.8 slope 4 loss 43.6 slope 6 loss 0.4 slope 8 loss 45.2 slope 10 loss 178.0 best slope: 6 prediction for 6 hours: 82 real score: 83 off by: 1
Read the output from top to bottom. A slope of 2 is far too flat, and the loss is 174.8. A slope of 4 is better, with 43.6. A slope of 6 is almost perfect, with 0.4. A slope of 8 overshoots, and the loss climbs to 45.2 again. The loss falls and then rises, so the best slope sits in the middle. That is training in miniature: score each try, keep the lowest.
Now map it to Mitchell's words. T: guess an exam score from study hours. E: the five students. P: the mean squared error, where lower is better. The program improved at T, as measured by P, when we let it try slopes on E. And the slope, 6, is the parameter that training found.
The last two lines are the important ones. The 6-hour student was not among the five we trained on. The model guessed 82, the real score was 83, and it was off by 1. That is a first taste of generalisation. This is only one extra student, and the data is tiny and made up for this lesson, so it proves nothing about the real world. It simply shows the habit: always try the model on a case it did not learn from.
Generalisation: the real goal
A low loss on the training data is easy to get. The real question is the loss on data the model has not seen. This is called generalisation. A model generalises well when it predicts accurately on new data. When it does not, it has overfit. Google describes overfitting as matching the training set so closely that 'the model fails to make correct predictions on new data'.
That is why data is split. Some rows are used to train and some are kept back to test, as the earlier lessons showed. Google names two causes of overfitting: training data that is not representative, and models that are too complex. In plain words, your test data should look like the real data your model will meet.
Quick check
A model has almost zero loss on its training data but a high loss on new data. What is the likely problem?
The life of a machine learning project
Real projects are more than training. Most of the work happens before and after. Here is a common outline, in our own words. Notice how many steps are about data, which is why this course spent so long on it.
- 1
1. Define the problem
Say what you want and how you will measure success. This is Mitchell's T and P. Decide if machine learning is even needed.
- 2
2. Get the data
Find or collect examples: from files, databases, APIs or sensors. This is the experience E.
- 3
3. Clean and prepare it
Fix missing values, wrong types and duplicates. Split the rows into training and test sets.
- 4
4. Build features
Turn raw columns into inputs the model can use: encode, scale, select, reduce.
- 5
5. Train
Run the algorithm to find the parameters that give a low loss. Try different hyperparameters.
- 6
6. Evaluate
Measure the model on data it has not seen, using your performance measure P.
- 7
7. Deploy
Put the model where people or other programs can use it, such as inside an app.
- 8
8. Monitor
Watch how it does in real life, and retrain when it gets worse.
The last step is easy to forget. Google's Rules of Machine Learning says to 'know the freshness requirements of your system', which means asking how much quality you lose if the model is a day, a week or a quarter old. It also warns about training-serving skew, 'a difference between performance during training and performance during serving'. One cause it lists is 'a change in the data between when you train and when you serve'. The world moves, so a model that was good last year can decay.
AI, machine learning and deep learning
These three terms are often used as if they meant the same thing. They do not. IBM describes them as a series from largest to smallest, each containing the next: 'AI is the overarching system. Machine learning is a subset of AI. Deep learning is a subfield of machine learning, and neural networks make up the backbone of deep learning algorithms.'
Artificial intelligence (AI)
The widest term. IBM says it is used for machines that mimic human intelligence and cognitive functions like problem-solving and learning.
Machine learning (ML)
The part of AI where programs learn from data, which is what this course teaches.
Deep learning
A part of machine learning built on neural networks. IBM says it is the number of node layers that separates a single neural network from a deep learning algorithm, which 'must have more than three'.
Think of nested boxes. Every deep learning model is machine learning, and all machine learning is AI, but not the other way round. The line-fitting program above is machine learning, not deep learning.
When not to use machine learning
Machine learning is a tool, not a goal. Google's Rules of Machine Learning opens with this advice: 'Don't be afraid to launch a product without machine learning.' It explains that 'machine learning is cool, but it requires data'. It also gives a rough sense of the gain: 'If you think that machine learning will give you a 100% boost, then a heuristic will get you 50% of the way there.' A heuristic is a simple rule of thumb. Google's examples are to rank apps by install count or to filter out publishers who have sent spam before.
The same guide says to choose machine learning over a complex heuristic, because a complex pile of rules is unmaintainable. Start simple, then move to machine learning once you have data and a clear goal.
A clear rule already exists
If a person can write the rule in a few lines, such as 'free shipping over 50', write it. A rule is cheaper, faster and easy to explain.
There is little or no data
Learning needs examples. With a handful of rows, a model will mostly memorise.
You cannot say what 'good' means
With no performance measure P, you cannot tell whether the model has improved. Define success first.
These three are our own summary of that guidance.
What to take away
Mitchell's definition has three parts: a task T, experience E and a performance measure P. A program learns when P improves as E grows. The algorithm is the recipe, the model is what it produces, parameters are found by training and hyperparameters are chosen by you. Loss says how wrong the model is, and training is the work of lowering it. The real goal is generalisation, meaning good results on data the model has not seen. A project runs from problem to data, features, training, evaluation, deployment and monitoring. Deep learning sits inside machine learning, which sits inside AI. And sometimes the best answer is a simple rule.
Next: the types of machine learning, side by side.
Sources
- Wikipedia: Machine learning (Tom Mitchell's definition)
- Tom Mitchell: Machine Learning (CMU book page)
- Google: Linear regression, Loss
- Google: Linear regression, Gradient descent
- Google: Overfitting
- Google: Rules of Machine Learning
- IBM: AI vs. machine learning vs. deep learning vs. neural networks
- scikit-learn: Glossary of common terms
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.