esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.09

Semi-supervised learning

Labels are costly and raw data is cheap. Semi-supervised learning uses a few labelled examples to make sense of many unlabelled ones.

12 min readFree

Why mix labelled and unlabelled data

Picture a teacher with a huge pile of unmarked photos and only two minutes to sort them. She labels two photos herself: one cat, one dog. Then she notices that the other photos fall into groups. Photos that look like the cat go with the cat. Photos that look like the dog go with the dog. Two labels have sorted a whole pile. That is the idea of semi-supervised learning.

You already know two kinds of learning. In supervised learning, every example has a label, which is the right answer. In unsupervised learning, no example has one. IBM says semi-supervised learning 'combines supervised and unsupervised learning by using both labeled and unlabeled data' to train models for classification and regression tasks.

Why would anyone want this? Because labels cost effort and raw data does not. IBM says semi-supervised methods matter most when 'obtaining a sufficient amount of labeled data is prohibitively difficult or expensive, but large amounts of unlabeled data are relatively easy to acquire.' It gives a sense of the work: to tell cars from motorcycles, 'hundreds (if not thousands)' of training images must be labelled. For drug discovery, genetic sequencing or protein classification, IBM says labelling is 'extremely time-consuming' and 'requires very specific domain expertise.'

Key words

Labelled example
An example that comes with its answer, such as a photo marked 'cat'.
Unlabelled example
An example with no answer attached, such as a photo nobody has sorted yet.
Pseudo-label
A label the model guesses for an unlabelled example. The word 'pseudo' means 'not the real one'. IBM calls these 'pseudo-label predictions'.
Confidence
How sure the model is about a guess. A model that says '85 percent dog, 15 percent cat' is fairly confident it is a dog.
Similarity graph
A map where every example is a dot, and dots that look alike are joined by a line.

Wikipedia gives a school picture. Think of the labelled data as sample problems that the teacher solves for the class, to help with another set of problems.

What has to be true for it to work

Unlabelled data does not help by magic. Wikipedia says that to make any use of it, 'some relationship to the underlying distribution of data must exist.' scikit-learn says the same thing in its own words: 'Semi-supervised algorithms need to make assumptions about the distribution of the dataset in order to achieve performance gains.' Three assumptions come up again and again.

  • Smoothness

    Wikipedia calls it the continuity or smoothness assumption: 'Points that are close to each other are more likely to share a label.' If a photo looks almost the same as a labelled cat, it is probably a cat too. IBM adds that the idea works step by step: a label can travel from a labelled point to a close unlabelled point, and on to the next one.

  • Cluster

    'The data tend to form discrete clusters, and points in the same cluster are more likely to share a label.' A cluster is a bunch of points that sit close together. Wikipedia calls this a special case of the smoothness assumption.

  • Manifold

    'The data lie approximately on a manifold of much lower dimension than the input space.' In plain words, data with thousands of numbers may really be controlled by only a few things. Wikipedia's examples: the human voice is controlled by a few vocal folds, and facial expressions by a few muscles.

Tip

These are bets, not facts. If the bet is wrong for your data, the unlabelled examples will not help. Always ask: do things that look alike in my data really share a label?

Self-training

Self-training is the simplest method. Wikipedia describes it as a wrapper method, meaning it wraps around an ordinary supervised model. IBM says compatibility with 'nearly any type of supervised base learner' is a main benefit. Here is how it goes.

  1. 1

    1. Train on the labels you have

    Wikipedia: 'First a supervised learning algorithm is trained based on the labeled data only.' This may be only a handful of examples.

  2. 2

    2. Guess labels for the rest

    The model predicts a label for each unlabelled example. These guesses are the pseudo-labels.

  3. 3

    3. Keep only the confident guesses

    Wikipedia: 'Generally only the labels the classifier is most confident in are added at each step.' IBM describes accepting only predictions that 'exceed a certain confidence threshold'. A threshold is a cut-off line: guesses below it are thrown away for now.

  4. 4

    4. Retrain and repeat

    IBM: the model is re-trained using both the real labels and the pseudo-labels, 'not differentiating between the two.' Each round the pile of labelled data grows.

scikit-learn ships this as SelfTrainingClassifier, built on Yarowsky's algorithm. It works with any classifier that can report probabilities. In each round it labels some of the unlabelled samples, picked either 'using a threshold on the prediction probabilities, or by choosing the k_best samples'. You can also set max_iter, the most rounds it will run. scikit-learn adds a warning: the 'calibration of the classifier is important', which means the confidence numbers must be honest ones.

scikit-learn also asks you to mark unlabelled entries in your answer list y with the integer -1.

Label propagation

Label propagation takes a different route. Imagine a rumour spreading through a village: it travels along friendships. Here, a label travels along similarity. scikit-learn says its two models, LabelPropagation and LabelSpreading, 'both work by constructing a similarity graph over all items in the input dataset.'

IBM explains the graph. Every example, labelled or not, is a node. The closer two nodes are, by some chosen measure of distance, the heavier the line between them. Starting from the labelled points, labels then spread step by step to their unlabelled neighbours. This uses the smoothness and cluster assumptions from above. scikit-learn's picture shows the point well: 'the structure of unlabeled observations is consistent with the class structure, and thus the class label can be propagated to the unlabeled observations of the training set.'

Self-training asks a model to guess. Label propagation asks a map to carry the label. Both lean on the same belief: close things share a label.

Co-training in one line

Wikipedia: co-training is 'an extension of self-training in which multiple classifiers are trained on different (ideally disjoint) sets of features and generate labeled examples for one another.' IBM says this variety is meant to reduce 'the tendency to reinforce poor initial predictions', and gives an example: in video, train one learner on the pictures and another on the sound.

Where it goes wrong

Two risks matter most. Both come from the same fact: the model teaches itself, so a mistake can feed on itself.

  • Wrong guesses reinforce themselves

    If an early pseudo-label is wrong, the next round learns from it, and the error spreads. IBM says most wrapper methods add 'some regularization techniques to reduce the risk of reinforcing potentially inaccurate pseudo-label predictions.' A confidence threshold is one simple guard. Co-training is another.

  • Unlabelled data from the wrong place

    IBM says the unlabelled examples 'must be relevant to the task'. For a cat-versus-dog model, 'images of horses and motorcycles will not be helpful.' A 2018 study quoted by IBM found that 'adding unlabeled data from a mismatched set of classes can actually hurt performance compared to not using any unlabeled data at all.'

  • A bad assumption

    Wikipedia warns about one family of methods: if their assumptions are incorrect, 'the unlabeled data may actually decrease the accuracy of the solution relative to what would have been obtained from labeled data alone.'

Quick check

A self-training model guesses wrongly on its very first pseudo-label and adds it to the labelled pile. What is the risk?

Worked example: self-training by hand

Let's build a tiny self-training loop with nothing but plain Python. The data is invented for this lesson. Each example is a single number on a line, say the width of a shape. We have two labelled points: 1.0 is class A and 9.0 is class B. We also have eight unlabelled points.

Our 'model' is the simplest one possible: a nearest neighbour. To guess a label, find the closest labelled point and copy its label. The distance to that point is our confidence: the smaller it is, the more sure we are. Each round, we find the single most confident guess among all unlabelled points, add it to the labelled pile, and print what happened. The limit max_distance is our confidence threshold. When the best guess is farther than the limit, the loop stops.

self_training.py
labelled = {1.0: "A", 9.0: "B"}
unlabelled = [2.0, 2.5, 3.0, 4.5, 6.5, 7.5, 8.0, 8.5]
max_distance = 2.0  # the confidence threshold: farther than this, we stay silent

round_no = 0
while unlabelled:
    best = None  # (distance, point, label)
    for p in unlabelled:
        for q, label in labelled.items():
            d = abs(p - q)
            if best is None or d < best[0]:
                best = (d, p, label)
    d, p, label = best
    if d > max_distance:
        print(f"stop: the closest unlabelled point, {p}, is {d} away (limit {max_distance})")
        break
    round_no += 1
    labelled[p] = label
    unlabelled.remove(p)
    print(f"round {round_no}: point {p} gets {label} (nearest labelled point is {d} away)")

print("still unlabelled:", unlabelled)

Output

round 1: point 8.5 gets B (nearest labelled point is 0.5 away)
round 2: point 8.0 gets B (nearest labelled point is 0.5 away)
round 3: point 7.5 gets B (nearest labelled point is 0.5 away)
round 4: point 2.0 gets A (nearest labelled point is 1.0 away)
round 5: point 2.5 gets A (nearest labelled point is 0.5 away)
round 6: point 3.0 gets A (nearest labelled point is 0.5 away)
round 7: point 6.5 gets B (nearest labelled point is 1.0 away)
round 8: point 4.5 gets A (nearest labelled point is 1.5 away)
still unlabelled: []

Read the output like a story. The three points next to 9.0 (8.5, 8.0, 7.5) are taken first, each only 0.5 away from a labelled neighbour. In round 4, 2.0 and 6.5 are both 1.0 from a labelled point. This is a tie, and the code keeps the first one in the list, 2.0. Notice that labels travel: 3.0 gets A because 2.5 was labelled in round 5, not because of the original 1.0. That is smoothness at work, one step at a time. The hardest point, 4.5, comes last, since it sits in the middle of the gap.

Now change one line: set max_distance = 1.2. The loop refuses to guess about 4.5, because its nearest labelled neighbour is 1.5 away. Here is the result.

Text
round 1: point 8.5 gets B (nearest labelled point is 0.5 away)
round 2: point 8.0 gets B (nearest labelled point is 0.5 away)
round 3: point 7.5 gets B (nearest labelled point is 0.5 away)
round 4: point 2.0 gets A (nearest labelled point is 1.0 away)
round 5: point 2.5 gets A (nearest labelled point is 0.5 away)
round 6: point 3.0 gets A (nearest labelled point is 0.5 away)
round 7: point 6.5 gets B (nearest labelled point is 1.0 away)
stop: the closest unlabelled point, 4.5, is 1.5 away (limit 1.2)
still unlabelled: [4.5]

This is a confidence threshold in action. We trade coverage for care: one point stays unlabelled, but we never made a doubtful guess. In a real project, those left-over points can wait for a human to label them.

Quick check

In the example, why does 3.0 get the label A?

Where people use it

Semi-supervised learning suits any job where labelling is slow and raw data is plentiful. The sources name a few.

Text. scikit-learn's own examples include 'Semi-supervised Classification on a Text Dataset'. Wikipedia adds that in natural language processing a common self-training algorithm is the Yarowsky algorithm, used 'for problems like word sense disambiguation, accent restoration, and spelling correction.'

Expert-heavy fields. IBM names drug discovery, genetic sequencing and protein classification, where labelling needs very specific domain expertise.

Images. IBM reports that semi-supervised learning together with active learning (see below) 'more than halved the amount of labeled data required to effectively train a model for semantic segmentation', which is labelling an image pixel by pixel.

Tip

Before choosing a method, count your labels. If you have plenty, ordinary supervised learning is simpler. If you have almost none and cannot get more, semi-supervised learning is worth a try.

Cousins: self-supervised and active learning

Two similar names are easy to mix up.

Self-supervised learning also tries to avoid hand labelling. IBM says that 'whereas semi-supervised learning involves some labeled data, self-supervised learning methods like autoencoders are truly unsupervised.' It adds that self-supervised tasks 'derive the ground truth from the underlying structure of unlabeled samples.' That is the next lesson.

Active learning keeps humans in the loop. Wikipedia says the algorithm 'can interactively query a human user' to label new data points. IBM says active learning algorithms 'do not automate the labeling of data points', and instead decide 'which unlabeled samples would provide the most helpful information if manually labeled.' In short: semi-supervised learning guesses labels, and active learning asks for them.

Next: self-supervised learning.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.