Machine Learning Foundations / Core / 02.10
When the data labels itself
Self-supervised learning makes its own labels from raw data: hide a word, guess it, check. Here is how it works and why language models start this way.
14 min readFree
The idea in one minute
Imagine you are learning a language from a book. Nobody gives you a teacher or an answer sheet. So you invent your own quiz. You cover one word in a sentence, guess it, then lift your finger and check. You did not need anyone to mark your work. The book itself held the answer.
That is self-supervised learning. The model hides a piece of its own data, tries to guess it, and then checks the guess against the piece it hid. The data makes its own labels.
Wikipedia defines it as 'a paradigm in machine learning where a model is trained on a task using the data itself to generate supervisory signals, rather than relying on externally-provided labels.' IBM puts it this way: 'Rather than relying on labeled datasets for supervisory signals, self-supervised models generate implicit labels from unstructured data.'
Key words
- Label
- The correct answer attached to an example, such as the word 'cat' on a photo of a cat. Usually a person writes it.
- Ground truth
- The answer we compare the model's guess against. IBM calls it the 'correct' prediction for an input.
- Pretext task
- A practice quiz the model makes up from its own data, such as filling in a hidden word. IBM says it is useful 'only because it teaches models data representations that are useful for the purposes of subsequent downstream tasks.'
- Downstream task
- The real job you care about afterwards, such as sorting reviews into happy and unhappy.
- Pseudo-label
- A label made by the process itself, not by a person. IBM says pretext tasks produce these from unlabeled data.
- Representation
- The model's own compact summary of an example: the useful pattern it keeps after learning.
Why labels are the bottleneck
In supervised learning, every example needs a label. IBM says that annotating a lot of data is 'often a bottleneck in the research process.' Its example is a task like instance segmentation, where the model must say which pixels belong to which object. There, annotation 'must be done at the pixel level. This is costly and time-consuming.'
Think of marking exam papers. One teacher can mark 30 papers in an evening. Marking 30 million would take forever. Meanwhile, raw text, photos and audio pile up everywhere with no marks on them. IBM puts it simply: 'labeled data is relatively scarce and expensive, unlabeled data is abundant and relatively cheap.'
Self-supervised learning goes after that pile. If the data can grade itself, nobody has to mark anything.
Tip
Self-supervised learning is not 'no teacher at all.' There is still a right answer to compare against. It just comes from the data, not from a person. IBM says the model is still measured 'against a ground truth, albeit one implicitly derived from unlabeled training data.'
Pretext tasks: quizzes the data writes
A pretext task is the practice quiz. Each kind of data can make its own. Here are three that matter most.
Predict what comes next
Show the start of a sentence and ask for the next word. The real next word is the answer. Nobody typed a label.
Fill in the blank
Hide a word, or a part of an image, and ask the model to restore it. The original is the answer.
Same picture or not?
Show two edited copies of one photo, and a copy of some other photo. Ask which pair belongs together. This is contrastive learning.
Let us look at each one in turn.
Predict the next word
Your phone does this when it suggests the next word as you type. IBM explains the training version. In pre-training, language models 'are provided the beginning of sample sentences drawn from unlabeled training data and tasked with predicting the next word, with the "actual" next word of the sample sentence serving as ground truth.'
IBM notes that this approach is used 'prominently in causal language models like the GPT, LLaMa and Claude families of LLMs.' The sentence is its own answer key.
Fill in the blank
This is called masking. IBM describes it as 'masking certain parts of an unlabeled data sample and tasking models with predicting or reconstructing the missing information.' The original input, before it was hidden, is the ground truth.
For text, 'random words are omitted from sample sentences and models are trained to fill them in.' IBM adds that masked language models like BERT have a useful strength: they are 'bidirectional', meaning they 'can predict not only the next word, but also previous words or words found later on in a sequence.' It is like solving a crossword clue with letters on both sides of the gap.
Masking is not only for words. IBM describes it for any 'unlabeled data sample', not only text. Applied to a picture, it means hiding a patch and asking the model to restore it. IBM lists another picture example too: rotate an image, then ask the model to predict how far it was turned.
Same picture or not? (contrastive learning)
'Contrastive' just means comparing. IBM says contrastive methods 'train models to distinguish between similar and dissimilar things.' The pairs are often made by data augmentation: 'applying different kinds of transformations or perturbations to unlabeled data to create new instances or augmented views.' For images, common edits include 'rotation, random cropping, flipping, noising, filtering and colorizations.'
Picture a photo of a dog. Crop it one way and tint it blue. Crop it another way and flip it. Those two edits are a matching pair, because they came from the same photo. A photo of a bus is a non-matching one. IBM says the model learns to 'minimize the difference in vector representations between positive matches', which are 'pairs of augmented images derived from the same original image', and to 'maximize the difference between negative matches.'
Nobody told the model what a dog is. It only learned that two different-looking edits of one photo belong together. IBM says this makes the learned representations 'robust to trivial variations (like the color, perspective or visible parts in a specific image).'
Quick check
In contrastive learning, which two images count as a matching pair?
Why it matters: large language models
This is how many language models are first trained. Wikipedia says GPT-style models are 'pre-trained to predict the next word' and are 'then often fine-tuned to follow instructions and to behave as assistants.' In another place it says: 'GPT models are first pretrained to predict the next word on a large amount of data, before being fine-tuned.'
BERT, introduced by Google researchers in October 2018, took the fill-in-the-blank route. The paper's abstract says BERT 'is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.' Wikipedia says BERT 'learns to represent text as a sequence of vectors using self-supervised learning' and 'is trained by masked token prediction and next sentence prediction.'
The scale was large. Wikipedia says BERT's models 'were trained on the Toronto BookCorpus (800M words) and English Wikipedia (2,500M words).' That is about 3.3 billion words, and no labels were needed: the text itself supplied the quiz.
The result: the abstract says BERT 'obtains new state-of-the-art results on eleven natural language processing tasks.'
Images use the same idea. IBM names computer vision models like SimCLR and Momentum Contrast (MoCo) as trained with self-supervised learning. IBM also describes CLIP, which 'jointly trains an image encoder and text encoder to predict which caption goes with which image, using millions of readily available unlabeled (image, text) pairings collected from the internet.'
Pretraining, then fine-tuning
Self-supervised learning is usually the first half of a two-part story. First the model practises on the pretext task using piles of raw data. That is called pretraining. Then it is adapted to the real job. IBM says: 'Models pre-trained with SSL are often fine-tuned for their specific downstream tasks: this fine-tuning often involves true supervised learning (albeit with a fraction of the labeled data needed to train a model with supervised learning alone).'
Think of a new cook. First they spend years just tasting and chopping, with no recipes. Then they join your restaurant and learn your ten dishes in a week. The long practice made the quick learning possible.
- 1
1. Collect raw data
Gather lots of unlabeled text, images or audio. No annotation needed.
- 2
2. Make the pretext task
Hide a word, crop a photo or pick the next word. The data supplies the answer.
- 3
3. Pretrain
The model guesses, checks and improves, again and again. It builds up its representations.
- 4
4. Fine-tune
Add a small amount of labeled data for the real job, such as spotting spam or answering questions.
- 5
5. Use it on the downstream task
The model now does the job you care about. IBM calls reusing a pre-trained model on a new task 'transfer learning.'
The BERT abstract shows how little may need adding: the pre-trained model 'can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.'
Tip
A good test for any pretext task: would solving it force the model to learn something your real job needs? Filling in a word needs grammar and meaning. Guessing the file size of a photo does not.
Not the same as unsupervised or semi-supervised
Three names sound alike, so here is how to tell them apart.
Unsupervised
No labels, and no right answer to check against. IBM's example is a shop model that finds which products are bought together. It is not copying a human answer. It is 'discovering correlations not apparent to human observers.'
Self-supervised
No human labels, but there is a right answer: the hidden part of the data. IBM calls it a subset of unsupervised learning: 'all self-supervised learning techniques are unsupervised learning, but most unsupervised learning does not entail self-supervision.'
Semi-supervised
A mix. IBM says semi-supervised learning 'uses both labeled and unlabeled data to train models', for example a small set of labeled points used to infer labels for the rest.
IBM adds that the two approaches have similar motives, since both avoid the need for large labeled datasets, but 'their respective methodologies are different.' In semi-supervised learning, people still label some examples. In self-supervised learning, nobody does.
Quick check
A model hides random words in millions of sentences and learns to guess them. No person labels anything. What is this?
Worked example: a sentence makes its own quiz
Let us prove that no human labels are needed. We will take one plain sentence and turn it into training pairs two ways, with only Python's built-in features.
- 1
1. Split the sentence into words
sentence = 'the cat sat on the mat'.split() gives a list of six words.
- 2
2. Make next-word pairs
For each position, the words before it are the context and the word at that position is the answer.
- 3
3. Make masked-word pairs
For each position, swap that word for [MASK]. The hidden word is the answer.
sentence = "the cat sat on the mat".split()
print("Next-word pairs (context -> next word):")
for i in range(1, len(sentence)):
print(" ".join(sentence[:i]), "->", sentence[i])
print()
print("Masked-word pairs (sentence with a hole -> hidden word):")
for i, word in enumerate(sentence):
hidden = sentence[:i] + ["[MASK]"] + sentence[i + 1:]
print(" ".join(hidden), "->", word)Output
Next-word pairs (context -> next word): the -> cat the cat -> sat the cat sat -> on the cat sat on -> the the cat sat on the -> mat Masked-word pairs (sentence with a hole -> hidden word): [MASK] cat sat on the mat -> the the [MASK] sat on the mat -> cat the cat [MASK] on the mat -> sat the cat sat [MASK] the mat -> on the cat sat on [MASK] mat -> the the cat sat on the [MASK] -> mat
One sentence gave us 11 labeled examples: 5 next-word pairs and 6 masked-word pairs. Nobody wrote a single label. The left side of each arrow is the question and the right side is the answer. A real model trains on millions of such pairs.
Now a tiny next-word model. It counts which word follows which, across three sentences. Counting pairs of neighbouring words is called a bigram counter. Then it predicts using the most common follower.
from collections import Counter, defaultdict
sentences = [
"the cat sat on the mat",
"the cat ate the fish",
"the dog sat on the rug",
]
counts = defaultdict(Counter)
for s in sentences:
words = s.split()
for current, following in zip(words, words[1:]):
counts[current][following] += 1
print("after 'the':", dict(counts["the"]))
print("after 'cat':", dict(counts["cat"]))
print("after 'sat':", dict(counts["sat"]))
word = "the"
guess, n = counts[word].most_common(1)[0]
print(f"Prediction after '{word}': {guess} (seen {n} times)")Output
after 'the': {'cat': 2, 'mat': 1, 'fish': 1, 'dog': 1, 'rug': 1}
after 'cat': {'sat': 1, 'ate': 1}
after 'sat': {'on': 2}
Prediction after 'the': cat (seen 2 times)After 'the', the word 'cat' appeared twice and every other word once, so the model predicts 'cat'. It also learned that 'sat' is always followed by 'on'. All of this came from the three sentences alone.
This toy only looks at one word of context and just counts. Real large language models are, as Wikipedia puts it, 'typically a neural network', and they use much more context. But the shape is the same: raw text in, its own next word as the answer, and a guess that gets checked.
Quick check
In the bigram example, why did the model never need a human to write labels?
Limits to keep in mind
Self-supervised learning is powerful, but it is not magic.
The quiz must teach something useful
IBM says a pretext task is 'not (necessarily) useful unto itself.' It is useful only if it teaches representations that help later. A badly chosen quiz can teach the wrong thing.
It can be costly to run
Wikipedia says 'Substantial infrastructure is necessary for training the largest models.' It reports that training GPT-2 (1.5 billion parameters) in 2019 cost $50,000, and PaLM (540 billion parameters) in 2022 cost $8 million. IBM adds that some contrastive methods need larger batches and a memory bank to store negative samples, and that non-contrastive methods save 'memory and computational cost during pre-training.'
It still needs good data
Wikipedia notes that 'biased or inaccurate training data can make an LLM's output less reliable.' Raw data can carry mistakes and unfair patterns, and a model that learns from it learns them too.
Remember also that pretraining is not the finish line. As IBM says, fine-tuning often still uses a small set of real labels.
Quick check
Which statement is true about the pretext task?
Recap
What it is
The model builds its own labels by hiding or comparing parts of unlabeled data.
How it works
Pretext tasks: predict the next word, fill in a blank, or tell whether two views come from the same picture.
Why it matters
Unlabeled data is abundant and cheap. Language models such as GPT and BERT start with pretraining like this, then get fine-tuned.
Next: reinforcement learning.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.