Machine Learning Foundations / Advanced / 03.46
Stemming
Cutting word endings back to a rough root, so that connect, connected and connecting count as one word.
13 min readFree
Cutting words back to a rough root
The last lesson, Tokenization, ended with a promise: stemming, cutting words back to a rough root. This is that lesson. Tokenization gave us a text cut into small pieces. Now we look at those pieces and ask a plain question: are two of them really the same word wearing different clothes?
Think of a gardener pulling up a plant. Plants and planted and planting all grow from the same root. If you want to count how many kinds of plant you have, it helps to look at the root and not at every leaf. Stemming does that for words. It takes connect, connected, connecting and connection and cuts each one back to connect.
Key words
- Stem
- What is left of a word after its ending is cut off. It is a rough root, and it does not have to be a real word.
- Stemming
- Cutting words back to their stems, so that related word forms end up as the same string.
- Suffix
- The ending of a word, such as -ed, -ing or -s.
- Inflected form
- A word changed a little to show number or time, such as cats from cat or walked from walk.
- Stemmer
- A program that does stemming. You may also hear stemming algorithm.
- Vocabulary
- The list of different words in a text, each counted once.
What stemming does
Wikipedia describes stemming as the process of reducing inflected (or sometimes derived) words to their word stem, base or root form. Its example: a stemmer might reduce fishing, fished and fisher to the stem fish.
Snowball, a project that collects stemmers, gives a very clear picture. Its English stemmer maps connection, connections, connective, connected and connecting to connect. So a search for connected would also find documents which only have the other forms.
Here is the key idea, and it is easy to miss. The stem does not need to be a real word. Wikipedia says it is usually sufficient that related words map to the same stem, even if this stem is not in itself a valid root. What counts is that the related words end up the same. The stem is a label, like a folder name, and nobody has to read it.
Wikipedia gives a neat example. The Porter algorithm reduces argue, argued, argues, arguing and argus to the stem argu. Argu is not a word in any dictionary. It does not matter. All the forms land in the same folder.
Why bother?
Search finds more
Many search engines treat words with the same stem as synonyms. Wikipedia reports that Google Search adopted word stemming in 2003, and that before this a search for fish would not have returned fishing.
Counting gets fairer
If you count words in a text, run, runs and running count as three different words. After stemming they can count as one. That makes your word counts bigger for each word and your list of different words shorter.
Less to learn
Wikipedia notes that stemming is used to prepare text before text mining.
Stemming is not a magic fix. Wikipedia also notes that the benefit for English search turned out to be rather limited, and that early researchers even called stemming irrelevant in general. It may help more in other languages. So treat it as a tool that is sometimes useful, not a rule.
How a stemmer chops
Wikipedia lists a few ways to build a stemmer. The simplest is a lookup table: a big list that says cats goes to cat, ran goes to run, and so on. It is fast and handles odd words well. Its weakness is that every form must be written down in advance. A new word is not handled, even if it is perfectly regular, and the table can get huge.
The other way is suffix stripping, and it is the one we will use. Instead of a giant table, you keep a small list of rules. Wikipedia gives three examples of such rules: if the word ends in ed, remove the ed. If it ends in ing, remove the ing. If it ends in ly, remove the ly.
Rules are easier to look after than a table. But Wikipedia also says suffix stripping can look crude when it meets exceptions, like ran and run. A rule that cuts ing cannot know that ran and run are the same word, because ran has no ending to cut. Keep that in mind, because we will see it in the code.
The Porter stemmer
The best-known suffix-stripping stemmer was written by Martin Porter and published in the July 1980 issue of the journal Program, in a paper called An algorithm for suffix stripping. Porter's own home page describes it as a process for removing the commoner morphological and inflexional endings from words in English, mostly used to tidy up words when setting up search systems. Wikipedia says it became the de facto standard for English stemming.
How does Porter's stemmer work? In plain words, it is a short series of steps. Each step looks at the end of the word and, if a rule fits, cuts or swaps the ending. Only one rule fires per group, and it is the one with the longest matching ending. The paper, as Snowball reprints it, runs in five numbered steps.
- 1
Step 1: plurals and -ed, -ing
caresses becomes caress, ponies becomes poni and cats becomes cat. Then ed and ing come off, but only if a vowel is left in front: plastered becomes plaster, motoring becomes motor, but sing and bled stay as they are. A last small rule turns y into i if there is a vowel before it: happy becomes happi.
- 2
Step 2: long endings to short ones
Big endings are swapped for shorter ones: relational becomes relate, and hopefulness becomes hopeful.
- 3
Step 3: more swaps
electrical becomes electric, and hopeful becomes hope.
- 4
Step 4: remove common endings
Endings such as -ment, -ant and -ion are removed if enough of the word stays behind: replacement becomes replac, adoption becomes adopt.
- 5
Step 5: tidy up
A final e is dropped from probate to give probat, and a double l at the end becomes one l: controll becomes control.
Many rules carry a size condition. Porter measures a word part by counting how many times a group of vowels is followed by a group of consonants. He calls the count m. The word trouble has m of 1, and troubles has m of 2. A rule such as the one for -ement only fires when the word part left in front has m above 1. This stops the rule from chopping short words down to nothing.
Tip
Porter's page says his stemmer should be regarded as frozen, that is, strictly defined, and not open to further change. He recommends the newer Snowball English stemmer, also called Porter2, for practical work. The original is still good where experiments must be exactly repeatable.
Porter2, Snowball and Lancaster
After Porter wrote his stemmer, he built Snowball, a small language for writing stemming algorithms, with an improved English stemmer (Porter2) and stemmers for several other languages. Snowball is now maintained by a community. Its home page says it aims to join words with the same meaning, not all words with a common linguistic root, so awe and awful do not share a stem.
Snowball also states a choice: over-stemming is more problematic than under-stemming, so the stemmers tend not to stem cases that are hard to resolve. We will see both mistakes in the next section.
A rival is the Paice-Husk stemmer, often called Lancaster, written by Chris Paice at Lancaster University in the late 1980s. Wikipedia calls it an iterative stemmer. We will not use it here; knowing the name is enough.
Two kinds of mistake
Wikipedia names two errors every stemmer can make. They are easy to remember with a laundry picture.
Over-stemming
Two different words get the same stem when they should not. Like tipping a red sock and a white shirt into one pile. Wikipedia's example: the Porter stemmer turns universal, university and universe into univers, though their meanings are in widely different areas.
Under-stemming
Two forms of the same word get different stems when they should match. Like leaving one sock out of its pair. Wikipedia's example: Porter turns alumnus into alumnu, alumni into alumni and alumna into alumna, so these near-synonyms are not joined.
A trade-off
Wikipedia says stemmers try to reduce each type of error, but reducing one can increase the other. A stemmer that cuts hard joins more words but also joins the wrong ones.
Build a tiny stemmer
Now let us build one. This is our own simplified stemmer, written for this lesson. It borrows the flavour of Porter's steps, but it is not Porter's algorithm. It uses far fewer rules and rough size checks instead of the real measure m. It is only here to show the idea. All the code runs with plain Python and no extra packages.
VOWELS = "aeiou"
def has_vowel(s):
return any(c in VOWELS for c in s)
def stem(word):
w = word.lower()
# Step 1: plurals (longest match wins)
if w.endswith("sses"):
w = w[:-2]
elif w.endswith("ies"):
w = w[:-2]
elif w.endswith("ss"):
pass
elif w.endswith("s") and len(w) > 3:
w = w[:-1]
# Step 2: -ed and -ing, only if a vowel is left in front
if w.endswith("eed"):
w = w[:-1]
for end in ("ed", "ing"):
if w.endswith(end) and not w.endswith("eed") and has_vowel(w[: -len(end)]):
w = w[: -len(end)]
if w.endswith(("at", "bl", "iz")):
w += "e"
elif len(w) > 2 and w[-1] == w[-2] and w[-1] not in "aeioulsz":
w = w[:-1]
break
# Step 3: y to i, if there is a vowel before it
if w.endswith("y") and has_vowel(w[:-1]):
w = w[:-1] + "i"
# Step 4: long endings turned into shorter ones
for end, new in (("ational", "ate"), ("ization", "ize"), ("ation", "ate"),
("fulness", "ful"), ("ness", "")):
if w.endswith(end) and len(w) - len(end) >= 3:
w = w[: -len(end)] + new
break
# Step 5: drop common endings if enough of the word stays
for end in ("iti", "ive", "ion", "al", "er", "ate", "ize", "ful", "e"):
if w.endswith(end) and len(w) - len(end) >= 4 and w[-2:] != "ee":
w = w[: -len(end)]
break
return wThat block prints nothing. It only defines stem(), a function that takes one word and hands back its stem. Next we try it on fifteen words, with the file name stemmer.py for all three blocks: add each block under the one before and run the whole file.
Try it on 15 words
Add this under the first block. It loops over fifteen words and prints each word, an arrow, and its stem.
words = ["connect", "connected", "connecting", "connection", "caresses",
"ponies", "running", "ran", "hopping", "agreed", "relational",
"universe", "university", "universal", "happily"]
for w in words:
print(w, "->", stem(w))Output
connect -> connect connected -> connect connecting -> connect connection -> connect caresses -> caress ponies -> poni running -> run ran -> ran hopping -> hop agreed -> agree relational -> relat universe -> univers university -> univers universal -> univers happily -> happili
Look at the good news first. Connect, connected, connecting and connection all land on connect. Running and hopping become run and hop, because step 2 cut the ing and then dropped the doubled letter. Caresses becomes caress and ponies becomes poni. And poni is not a word, which proves the point from earlier: a stem is a label.
Now the mistakes. We have one of each.
An over-stem
Universe, university and universal all become univers. That is exactly the mistake Wikipedia describes for the Porter stemmer. Three words from different areas now look the same.
An under-stem
Running becomes run, but ran stays ran. They are the same word, yet they get different stems. A rule cannot cut an ending that is not there, just as Wikipedia warned about ran and run.
Stems that are not words
Poni and univers are not words. They are labels, and that is fine, as long as related words get the same label.
Tip
Try your own words: add rain, rained and raining, then a hard one such as mice. No suffix rule can fix mice.
Quick check
In our output, ran stays ran while running becomes run. What kind of mistake is this?
Count words before and after
Here is the payoff. We take a short text, count its different words, stem every word and count again. The text is split with plain Python: lower case, strip full stops and commas, cut at spaces.
Add this block under the other two and run the whole file.
text = ("She runs. He is running. They ran home. The runs were long, "
"and the running was fun. We connect, we connected, "
"and the connection stays. Connected people connect.")
tokens = text.lower().replace(".", "").replace(",", "").split()
print("words in text:", len(tokens))
print("distinct before:", len(set(tokens)))
stems = [stem(t) for t in tokens]
print("distinct after:", len(set(stems)))
print(sorted(set(stems)))Output
words in text: 28 distinct before: 20 distinct after: 17 ['and', 'connect', 'fun', 'he', 'home', 'is', 'long', 'peopl', 'ran', 'run', 'she', 'stai', 'the', 'thei', 'was', 'we', 'were']
The text has 28 words. Before stemming it has 20 different ones. After stemming it has 17. So stemming joined 3 words. The connect family (connect, connected, connection) became one stem, and runs and running became run. The count dropped by three, and a search for run would now also find runs.
Quick check
Why does the list of different words get shorter after stemming?
Other languages are harder
English is kind to stemmers because its words change in simple ways. Wikipedia says English stemmers are fairly trivial, with only occasional problems, such as dries being a form of dry while axes can be the plural of both axe and axis.
Other languages change their words much more. Wikipedia says stemmers become harder to design as the language gets more complex. An Italian stemmer is more complex than an English one because of a greater number of verb forms. A Russian one is more complex still, with more noun forms. Hebrew and Arabic are still considered difficult. Wikipedia also says stemmers may give greater benefits in other languages than in English.
Stemming and lemmatization
Stemming is quick and rough. It looks only at the letters and has no idea what the word means. Wikipedia says suffix stripping can perform poorly on exceptions, and that lemmatization attempts to improve on this.
Lemmatization looks words up and returns a proper dictionary word, such as run for ran. That is slower, but the result is a real word. The next lesson covers it. A good rule of thumb for now: if you need speed and a rough match, stem. If you need real words, lemmatize.
Quick check
Which of these is a real example of over-stemming from this lesson?
Next: lemmatization, finding the dictionary form of a word.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.