esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.17

Tuning a model's settings

Every model has dials you set before it learns. Here is how to find good settings without fooling yourself.

14 min readFree

The dials you set before learning

Think of a camera. You point it, and it records what it sees. But first you pick settings: how long the shutter stays open, how wide the lens opens. The camera cannot choose these for you. A bad choice gives a bad photo, however good the scene.

Machine learning models have the same kind of dials. This lesson is about finding good settings for them, and doing it honestly. It follows the lesson on model selection, where you learned to compare models fairly.

Key words

Parameter
A number the model works out for itself while it trains, such as a weight.
Hyperparameter
A setting you choose before training starts. The model does not learn it.
Tuning
Trying different hyperparameter values to find the ones that work best. Also called hyperparameter optimization.
Validation set
Data kept back from training, used to compare settings.
Test set
Data kept back until the very end, used once to check the final model.
Trial
One try: one set of hyperparameter values, trained and scored.
Budget
How many trials, or how much time, you allow yourself.

If these split-up datasets feel hazy, the lesson on train and test data covers them. If "model" and "learning" feel hazy, see What is Machine Learning?.

Parameters and hyperparameters

Google's Machine Learning Crash Course draws the line in one sentence: hyperparameters are values that you control, and parameters are values that the model calculates during training. Scikit-learn's guide says it another way: hyper-parameters are parameters that are not directly learnt within estimators. (An estimator is scikit-learn's word for a model.)

A recipe helps. The oven temperature and baking time are hyperparameters: you set them. How the cake turns out is the result of the baking. In a model, the learned weights are the result. The temperature is yours.

  • Parameters (learned)

    The weights and bias in a line-fitting model. The training process finds them. You never type them in.

  • Hyperparameters (chosen)

    The learning rate, the number of neighbours k, the depth of a decision tree, how strongly the model is held back from memorising. You pick them first.

  • Why it matters

    The same model with different hyperparameters can be great or useless. Nothing inside the model will fix a bad choice for you.

Examples you will meet

Scikit-learn lists typical examples, such as C, kernel and gamma for a Support Vector Classifier, and alpha for Lasso. You will meet those models later. Here are four easier ones.

  1. 1

    1. k in k-nearest neighbours

    To guess a new value, look at the k closest known examples and average them. Small k follows every wiggle. Large k smooths everything flat.

  2. 2

    2. Learning rate

    How big a step the model takes each time it improves. We come back to this one below.

  3. 3

    3. Tree depth

    A decision tree asks questions one after another. The depth is how many questions in a row it may ask. Deeper trees can memorise the training data.

  4. 4

    4. Regularisation strength

    A dial that holds a model back from memorising. Turn it up and the model stays simpler. Turn it down and it is freer to chase noise.

Tip

Scikit-learn notes that a small subset of the settings can have a large impact on a model, while others can be left at their defaults. Start by tuning the one or two that matter most.

The search loop

Scikit-learn describes a search as having five parts: a model, a parameter space (the values you allow), a method for choosing candidates, a way to split the data, and a score function. Wikipedia puts the goal plainly: find the hyperparameters that give a model with the lowest loss on a given dataset.

In plain words, the loop looks like this.

  1. 1

    1. Split the data

    Training set to learn from, validation set to compare settings, test set locked away for the end.

  2. 2

    2. Choose a setting

    Pick one set of hyperparameter values to try.

  3. 3

    3. Train

    Train the model on the training set with those values.

  4. 4

    4. Score

    Measure the error on the validation set. Not on the training set: a model can score perfectly on data it has memorised.

  5. 5

    5. Repeat

    Do steps 2 to 4 again until the budget runs out.

  6. 6

    6. Keep the best

    Choose the setting with the lowest validation error.

The big question is step 2: how do you choose which settings to try? There are two classic answers.

Grid search and random search

Grid search is the tidy answer. Wikipedia calls it the traditional method: an exhaustive search through a manually specified subset of the hyperparameter space. You list the values for each dial, then try every combination. With two dials and ten values each, that is 100 trials.

Random search is the scruffy answer. You say which values are allowed and let the computer draw them at random. Scikit-learn names two benefits: a budget can be chosen independent of the number of parameters and possible values, and adding parameters that do not influence the performance does not decrease efficiency.

Why might scruffy win? Picture two dials, where only one really matters. A 3 by 3 grid tries nine combinations, but only three different values of the dial that matters. Nine random draws give you nine different values of it. Same budget, more useful information.

This is no mere hunch. The Bergstra and Bengio paper (JMLR, 2012) says in its abstract that it "shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid." It adds that, for most data sets, only a few of the hyper-parameters really matter, but different hyper-parameters are important on different data sets. That, it says, makes grid search a poor choice for configuring algorithms for new data sets.

Wikipedia agrees, with a limit: random search can outperform grid search, especially when only a small number of hyperparameters affects the final performance. It can also explore many more values than a grid could for continuous settings. Random search is still an important baseline against which new methods are compared.

Tip

Neither always wins. Random search is a fair bet, not a guarantee. In the example below, you will see the grid come out on top.

Worked example: tuning k

Let us run both searches on a small problem. The data is made up: a wavy curve, with some noise added. The model is k-nearest neighbours for regression, meaning it predicts a number by averaging the y values of the k closest training points. The error is the root mean squared error (RMSE): a typical size of the miss. A seed fixes the random numbers, so the run repeats exactly.

Each search gets the same budget: five trials. The grid is five hand-picked values of k. The random search draws five values from 1 to 59. We also keep a test set that no trial may look at.

Python
import math
import random

random.seed(7)

# Made-up data: a wavy curve plus noise.
def make(n):
    rows = []
    for _ in range(n):
        x = random.uniform(0, 10)
        rows.append((x, math.sin(x) * 3 + random.gauss(0, 1)))
    return rows

train, valid, test = make(60), make(30), make(30)

def predict(k, x):
    nearest = sorted(train, key=lambda r: abs(r[0] - x))[:k]
    return sum(y for _, y in nearest) / k

def error(k, rows):
    return math.sqrt(sum((predict(k, x) - y) ** 2 for x, y in rows) / len(rows))

def search(name, ks):
    results = []
    for k in ks:
        e = error(k, valid)
        results.append((k, e))
        print(name, "k =", k, "validation error", round(e, 3))
    best = min(results, key=lambda r: r[1])
    print(name, "best: k =", best[0], "error", round(best[1], 3))
    return best

grid_best = search("grid  ", [1, 5, 15, 30, 59])
rand_best = search("random", sorted(random.sample(range(1, 60), 5)))

k = min([grid_best, rand_best], key=lambda r: r[1])[0]
print("final test error with k =", k, ":", round(error(k, test), 3))

Output

grid   k = 1 validation error 1.23
grid   k = 5 validation error 0.985
grid   k = 15 validation error 1.129
grid   k = 30 validation error 1.77
grid   k = 59 validation error 2.002
grid   best: k = 5 error 0.985
random k = 9 validation error 1.013
random k = 17 validation error 1.11
random k = 27 validation error 1.554
random k = 35 validation error 2.099
random k = 54 validation error 2.039
random best: k = 9 error 1.013
final test error with k = 5 : 1.572
  • The shape

    Error is high at k = 1 (it chases every wiggle of noise), drops to a low around k = 5 to 9, then climbs as k grows and the model smooths the curve flat. This U shape is typical.

  • The winner

    Here the grid found k = 5 (0.985) and the random search found k = 9 (1.013). Both are close, and the grid is slightly ahead. With only one dial, there is little for random search to gain. Its advantage grows as dials multiply.

  • The honest number

    The test error, 1.572, is higher than the validation error, 0.985. That is normal. We picked k because it did well on the validation data, so that score is a little flattering. The test set never took part in the choice, so its number is the one to report.

Quick check

Quick check

A model has ten hyperparameters, but only two really affect the result. You can afford 50 trials. What does the evidence favour?

Smarter ways to search

Grid and random search are "blind": each trial ignores what earlier trials showed. Two ideas try to be cleverer. You do not need to learn them now, only to know they exist.

  • Bayesian optimization

    Wikipedia says it builds a probabilistic model of how hyperparameter values map to the validation score, then repeatedly tries the setting that looks most promising and updates the model. It reports that this has been shown to give better results in fewer evaluations than grid and random search.

  • Successive halving

    Scikit-learn describes it as like a tournament among candidates: all are evaluated with a small amount of resources first, and only some go on to the next round, with more resources. Scikit-learn says its halving searches can be much faster at finding a good combination.

  • Where to start

    Random search is a sensible first move. Wikipedia calls it one of the important base-lines against which to compare new methods. Beat it before you reach for something heavier.

The learning-rate story

One hyperparameter deserves its own story, because it shows up in nearly every model that learns by gradient descent. If that phrase is new, the feature scaling lesson shows the bowl-shaped loss and the steps downhill.

The learning rate is the size of each step. Google says it is a number you set that influences how quickly the model converges, meaning how fast it settles on good values. Its rule: if the learning rate is too low, the model can take a long time to converge. If it is too high, the model never converges, but instead bounces around the weights and bias that minimize the loss.

Picture walking down a hill in fog to find the lowest spot. Tiny steps are safe, but you will be there all day. Giant leaps carry you across the valley and up the other side, again and again. A medium step gets you down quickly. Google's goal is the same: not too high, not too low.

Here is the idea in a few lines. The loss is the simple bowl 0.5 times w squared, so its lowest point is w = 0. Every run starts at w = 1 and takes 20 steps.

Python
# Loss = 0.5 * w * w, lowest at w = 0. The gradient (the slope) is w.
# Each step: w = w - learning_rate * gradient
for lr in [0.01, 0.5, 2.5]:
    w = 1.0
    for step in range(20):
        w = w - lr * w
    print("learning rate", lr, "-> w after 20 steps:", round(w, 4))

Output

learning rate 0.01 -> w after 20 steps: 0.8179
learning rate 0.5 -> w after 20 steps: 0.0
learning rate 2.5 -> w after 20 steps: 3325.2567

Too small crawls: after 20 steps w has only moved from 1 to 0.8179. A middle rate lands on the bottom. Too big overshoots, and each step lands farther away than the last: w ends at 3325.2567, far from 0.

Budgets and stopping

Tuning has no natural end. There is always another setting to try, so you must decide when to stop.

Random search makes this easy. Scikit-learn's RandomizedSearchCV takes a computation budget, the number of sampled candidates, called n_iter. Grid search has no such dial: its cost is the number of combinations, which multiplies with every dial you add.

  1. 1

    1. Set the limit first

    Decide the number of trials, or the minutes, before you start. For example: 30 trials.

  2. 2

    2. Start wide

    Early trials should cover a broad range of values, to find the right region.

  3. 3

    3. Zoom in

    Once a region looks good, try a narrower range inside it.

  4. 4

    4. Stop when gains are tiny

    If the last ten trials improved the score by a hair, more tuning will not change the story.

Tip

Beware of over-tuning. Each trial gives the validation set one more chance to flatter a setting by luck. The more you try, the less you can trust that the best score is real. That is why the test set matters.

Keep the test set untouched

Tuning uses data to pick settings, so it is a kind of training. Scikit-learn says it plainly: model selection by evaluating parameter settings can be seen as a way to use the labeled data to "train" the parameters of the grid.

So the final check needs data the search never saw. Scikit-learn recommends evaluating the resulting model on held-out samples that were not seen during the grid search process, and splitting the data into a development set (used for the search) and an evaluation set (used to measure performance).

Think of an exam. The validation set is your practice test. You may take it many times and change your study plan each time. The test set is the real exam. You sit it once. If you peek at the real exam and then study to it, your mark means nothing.

Quick check

Your tuned model scores 0.98 on the validation set, then 0.91 on the test set. What is the right reading?

Record every run

A tuning session produces many trials. Without notes, you will forget which setting gave which score, and you will not be able to repeat your best result.

  • What to write down

    The date, the data version, the random seed, every hyperparameter value, and the validation score.

  • Where

    A plain table or a spreadsheet is enough. One row per trial, including the bad ones. Failed trials teach you where not to look.

  • Why

    You can repeat the best run, show a teammate how you got there, and spot patterns, such as the error climbing whenever k passes 20.

Tip

Seeds matter. The example above used random.seed(7), so anyone running it gets the same numbers. Without a seed, the random search would pick different values each time, and you could not explain the result later.

What to remember

  • Dials

    Hyperparameters are settings you choose before training. Parameters are what the model learns. Tuning is trying values and keeping the best.

  • Search

    Grid search tries every combination. Random search draws values, and often finds as good a setting within the same budget. Bayesian optimization and successive halving are smarter options for later.

  • Honesty

    Compare settings on validation data, set a budget, keep a log of every trial, and touch the test set once, at the very end.

Next: what supervised learning looks like in practice, task by task.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.