esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.47

Lemmatization

Turning a word into its dictionary form, so was becomes be and mice becomes mouse, using a word list, grammar rules and the word's job in the sentence.

14 min readFree

Finding the dictionary form

The last lesson, Stemming, ended with a promise: lemmatization, finding the dictionary form of a word. Stemming chopped the ends off words. It was quick, but it left odd scraps behind, such as wolv or lazi. Lemmatization does the same job with more care. It hands back a real word.

Think of how you look a word up in a dictionary. You never search for ran or mice. Your brain turns them into run and mouse first, then you flip to the page. A lemmatizer does that small step for the computer. The dictionary entry is called the lemma.

By the end of this lesson you will know what a lemma is, why a lemmatizer must also know the job a word does in its sentence, how it differs from a stemmer, and how a tiny one is built from a word list and a few rules. You will run one yourself.

Key words

Lemma
The dictionary form of a word, the one you would look up. The lemma of walked, walks and walking is walk.
Inflected form
A word after its ending or shape has changed to show things like past tense or more than one. Walked and mice are inflected forms.
Lemmatization
Turning each inflected form into its lemma. Some people spell it lemmatisation. Both mean the same thing.
Stemming
Chopping the ends off words by simple rules, without looking at meaning. The result is a stem, which may not be a real word.
Part of speech (POS)
The job a word does in a sentence: noun (a thing), verb (an action), adjective (a describing word) and so on. A tag is a short label for it, such as NOUN or VERB.
Irregular form
A word that does not follow the usual pattern, like mice (not mouses) or was (not beed).
Lookup table
A list that pairs one thing with its answer, like a phone book pairs a name with a number.

What a lemma is

English words change shape. The verb to walk can show up as walk, walked, walks or walking. Wikipedia puts it this way: the base form, walk, is the one you would look up in a dictionary, and it is called the lemma. All four shapes belong to one family. Lemmatization groups them so a program can treat them as a single item.

Why would a program want that? Imagine a shop review site. One person writes that the shoes are comfortable and another writes that the shoe is comfortable. To a computer, shoes and shoe are two different strings. If you count words to find what people talk about, the count splits in two. Folding both into shoe fixes that.

  • Same family, one entry

    Walked, walks and walking all become walk. A count of walk now covers all of them.

  • Odd words handled too

    Better becomes good. Was becomes be. Mice becomes mouse. No chopping rule can do this, because the spelling changes a lot.

  • Real words out

    A lemma is a word you could find in a dictionary. You can read it, check it and show it to a person.

Tip

Wikipedia gives better and good as an example. The link between them is missed by stemming, because it needs a dictionary look-up.

Why the part of speech matters

Here is the catch. A word on its own can be ambiguous. Take meeting. In our last meeting it is a noun, a thing: a gathering of people. In We are meeting again tomorrow it is a verb, an action, and it comes from to meet.

Same spelling, two different lemmas. For the noun, the lemma is meeting. For the verb, the lemma is meet. Wikipedia says that unlike stemming, lemmatization depends on correctly identifying the intended part of speech and the meaning of a word in a sentence. A lemmatizer that does not know the job of the word can only guess.

Think of the word bat. Is it the animal or the thing used in cricket? You cannot tell from the letters. You need the sentence around it.

That is why real lemmatizers usually work with a tagger. A tagger is a small program that reads a sentence first and labels each word with its part of speech. The lemmatizer then uses that label. The spaCy documentation says the same: its lemmatizer assigns base forms using rules based on part-of-speech tags, or lookup tables, and in rule mode a component that assigns the part of speech must run before it.

Quick check

In the sentence "We are meeting again tomorrow", what is the lemma of meeting?

How it differs from stemming

A stemmer works on a single word. It looks at the letters and applies rules about endings. It does not know the sentence, and it cannot tell a noun from a verb. Wikipedia puts the difference like this: a stemmer operates on a single word without knowledge of the context, and so cannot tell apart words that mean different things depending on part of speech.

That is also why a stem can be odd. Wikipedia notes that many stemmers turn lazy into lazi. The aim of stemming is not to find the real word. It is to map different forms of a word to one shared form, so lazy and laziness end up matching.

So which is better? Neither always. Here is the trade, as described by Wikipedia.

  • Stemming

    Typically easier to build and faster to run. The answer may not be a real word. It works from spelling alone.

  • Lemmatization

    More accurate, because it uses a dictionary and the word's job in the sentence. It needs more data and more work, and getting it right is still an open area of research.

  • In search

    Wikipedia says that in information retrieval systems stemming improves recall, the share of matching documents you find, but lowers precision, the share of what you find that is really a match.

Picture a search box. You type walking shoes. A stemmer might also find pages about walked and walks, so you see more results. Some of them will be off topic. A lemmatizer is stricter about meaning, so fewer wrong pages sneak in, but it may also miss a few.

How a lemmatizer works

There are two main tricks, and real tools mix them.

  1. 1

    Look it up in a table

    Wikipedia calls a simple dictionary lookup a trivial way to do lemmatization. It works well for straightforward cases. The table pairs a word and its part of speech with the answer: better as an adjective maps to good.

  2. 2

    Apply rules when the table has no answer

    A rule says: if the word is a verb and ends in ing, drop the ending. Wikipedia notes that a rule-based system is needed for other cases, and that such rules can be hand-written or learned automatically from examples.

  3. 3

    Fall back to the word itself

    If no table entry and no rule fits, keep the word as it is. A word like cat is already a lemma.

Order matters. The table goes first, because irregular words break the rules. Rules come second. The word itself is the last resort.

The spaCy documentation names the same two styles. Its lemmatizer has a mode, set to lookup or rule. Lookup uses tables. Rule uses rules based on part-of-speech tags, and for English the default is rule.

Where does word knowledge like this live? WordNet is one example. Wikipedia describes it as a lexical database that links words into relations, grouping synonyms into sets called synsets, with short definitions and examples.

Build a tiny lemmatizer

Let us build one with plain Python and nothing else installed. It has a small table of irregular words, a few ending rules for each part of speech, and a rule for what to do when nothing fits. It is a toy, not a real tool, but it shows every idea above.

First, the table on its own. A Python dictionary pairs a key with an answer. Our key is the word together with its tag, so the table can tell meeting the noun from meeting the verb. If the key is missing, we say so.

Python
word, pos = "meeting", "NOUN"
IRREGULAR = {("better", "ADJ"): "good"}
print(IRREGULAR.get((word, pos), "not in the table, fall back to the rules"))
print(IRREGULAR.get(("better", "ADJ"), "not in the table, fall back to the rules"))

Output

not in the table, fall back to the rules
good

Meeting is not in the table, so we move on to the rules. Better is, and the table answers good at once. Next, the rules. They depend on the tag. Watch what the same word does when it is treated as a noun and as a verb.

Python
RULES = {"NOUN": [("s", "")], "VERB": [("ing", ""), ("s", "")]}
word = "meeting"
for pos in ["NOUN", "VERB"]:
    rules = RULES[pos]
    print(pos, [(e, n) for e, n in rules if word.endswith(e)])

Output

NOUN []
VERB [('ing', '')]

The noun rules have no ending that fits meeting, so a noun is left alone. The verb rules find ing and say to drop it. Same letters, two answers, because the tag is different. This is the whole reason the part of speech matters.

Tip

Each rule is a pair: the ending to look for, and what to put in its place. For example ("ies", "y") turns studies into study.

Run it on twelve words

Now the full program. It has the irregular table, rules for nouns, verbs and adjectives, and a lemmatize function that tries the table first, then the rules, then gives the word back. It also has a chop function, which does what a quick stemmer-style chop would do: cut a common ending and ask no questions. We run both on twelve tagged words and print them side by side.

Python
IRREGULAR = {
    ("better", "ADJ"): "good",
    ("was", "VERB"): "be",
    ("mice", "NOUN"): "mouse",
    ("wolves", "NOUN"): "wolf",
}

RULES = {
    "NOUN": [("ies", "y"), ("s", "")],
    "VERB": [("ied", "y"), ("ies", "y"), ("ing", ""), ("ed", ""), ("s", "")],
    "ADJ": [("ier", "y")],
}

def lemmatize(word, pos):
    if (word, pos) in IRREGULAR:
        return IRREGULAR[(word, pos)]
    for ending, new in RULES.get(pos, []):
        if word.endswith(ending) and not word.endswith("ss"):
            return word[: len(word) - len(ending)] + new
    return word

def chop(word):
    for ending in ["ing", "ied", "ies", "ed", "ier", "es", "s"]:
        if word.endswith(ending):
            return word[: len(word) - len(ending)]
    return word

tagged = [
    ("better", "ADJ"), ("was", "VERB"), ("mice", "NOUN"), ("wolves", "NOUN"),
    ("meeting", "NOUN"), ("meeting", "VERB"), ("walking", "VERB"),
    ("studies", "NOUN"), ("studies", "VERB"), ("cried", "VERB"),
    ("happier", "ADJ"), ("cats", "NOUN"),
]

print(f"{'word':<9}{'tag':<6}{'lemma':<8}chop")
for word, pos in tagged:
    print(f"{word:<9}{pos:<6}{lemmatize(word, pos):<8}{chop(word)}")

Output

word     tag   lemma   chop
better   ADJ   good    better
was      VERB  be      wa
mice     NOUN  mouse   mice
wolves   NOUN  wolf    wolv
meeting  NOUN  meeting meet
meeting  VERB  meet    meet
walking  VERB  walk    walk
studies  NOUN  study   stud
studies  VERB  study   stud
cried    VERB  cry     cr
happier  ADJ   happy   happ
cats     NOUN  cat     cat

Read it row by row. The lemma column is a real word every time: good, be, mouse, wolf, meet, walk, study, cry, happy, cat. The chop column is mixed. It leaves better and mice untouched, and it makes scraps such as wa, wolv, stud, cr and happ.

  1. 1

    Irregular words

    Better, was, mice and wolves are all in the table, so they come out right. The chop has no table and cannot do this.

  2. 2

    The two meetings

    Meeting as a noun stays meeting. Meeting as a verb becomes meet. The chop cannot tell them apart and gives meet to both, which is wrong for the noun.

  3. 3

    Regular words

    Walking, studies, cried and happier follow the rules, and the lemma comes out as a real word. Cats works the same way. Here the chop also gets cat and walk right, so on easy words the two agree.

Notice that studies gets the same answer as a noun and as a verb. That is correct: the plural of study and the he/she form of to study both lead back to study. The tag only changes the answer when the rules for the two jobs differ.

Quick check

In the printed table, why is the chop of mice just mice?

Where our toy falls short

Our lemmatizer is small on purpose. It gets these twelve words right, but it would stumble on many others. Look at the rules. Drop a final s from a noun, and bus becomes bu. We guarded against ss, so class survives, but bus does not. Drop ing from a verb, and sing becomes s. A real tool needs many more entries and care.

Also, the tag has to be right. If the tagger says meeting is a verb when it is a noun, the lemmatizer gives the wrong answer with total confidence. The quality of the lemma rests on the quality of the tag.

  • Not enough table entries

    Every irregular word needs a row. English has many of them, and some languages have long compound words that need rules of their own.

  • Rules that over-reach

    A rule fits many words, and some of them are exceptions. Each exception needs a check or a table entry.

  • A wrong tag

    Garbage in, garbage out. The lemmatizer trusts the label it is given.

This is also why Wikipedia calls the building of efficient lemmatization algorithms an open area of research. Context can stretch beyond one sentence, even to neighbouring sentences or a whole document.

When to use which

A simple rule of thumb helps. Ask what the output is for.

  • Search and matching

    You want to find many pages quickly, and a few off-topic ones are fine. A stemmer is fast and finds more. The stems are only used inside, and people never read them.

  • Analysis and reading

    You count words, group topics or show the results to a person. Real words matter, and meaning matters. Use a lemmatizer, so that meeting the noun and meeting the verb stay apart.

  • Speed above all

    You process a huge pile of text and every second counts. Stemming is typically easier and faster, as Wikipedia notes. Check whether the loss in accuracy matters for your job.

Tip

Wikipedia says the reduced accuracy of stemming may not matter for some applications. Try both on a small sample of your own data and look at the results with your own eyes.

You do not have to build any of this yourself. Libraries such as spaCy ship a lemmatizer as a ready-made piece. The spaCy documentation describes it as a component that assigns base forms to tokens, where a token is one word or piece of text, using rules based on part-of-speech tags or lookup tables. You switch it on and read the lemma from each token.

What to remember

A lemma is the dictionary form of a word. Lemmatization maps every shape of a word, walked, walks, walking, back to that one form, and it handles odd words like better and mice that no chopping rule can reach.

It needs the part of speech, because the same letters can have different lemmas, as meeting does. It works with a table of known cases first, then rules, then the word itself. Compared with stemming, it gives real words and better accuracy, at the cost of more work and more data.

Quick check

You are counting which topics shoppers write about in reviews, and a person will read your word list. Which tool fits better?

Next: embeddings, turning words into numbers that carry meaning.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.