Machine Learning Foundations / Advanced / 03.44
Natural Language Processing
How a computer can work with words: the steps from raw text to numbers, and two simple ways to count them by hand.
14 min readFree
Teaching computers to work with words
The last lesson, Generative Adversarial Networks, ended with a promise: next, natural language processing, teaching computers to work with words. So far this course has worked mostly with numbers and pictures. Now we turn to text.
Words are tricky for a computer. A computer only does arithmetic. It does not know that "cat" is a pet or that "not good" is bad news. Natural language processing is the field that closes that gap. By the end of this lesson you will know what it is, why it is hard, the steps most text goes through, and how to turn four tiny documents into numbers with nothing but plain Python.
Key words
- Natural language
- The language people speak and write every day, like English or Tamil, as opposed to a programming language.
- NLP
- Natural language processing: getting a computer to work with human language.
- Document
- One piece of text: a sentence, an email, a review, a whole article.
- Corpus
- A collection of documents, the pile of text you work on.
- Token
- One small piece of text, usually a word. Cutting text into tokens is tokenisation.
- Vocabulary
- The list of every different word that shows up in the corpus.
- Bag of words
- A way of turning a document into numbers by counting its words and ignoring their order.
- TF-IDF
- A score that is high for words that matter in one document and low for words found everywhere.
What is NLP?
Wikipedia gives a short, plain definition: natural language processing is "the processing of natural language information by a computer". It describes NLP as a subfield of computer science that is closely associated with artificial intelligence.
IBM's guide adds more detail. It calls NLP a subfield of computer science and AI that uses machine learning to enable computers to understand and communicate with human language. It lets computers recognise, understand and generate text and speech.
Wikipedia lists four major jobs inside an NLP system: speech recognition, text classification, natural language understanding and natural language generation. In plain words: hearing speech, sorting text into groups, working out what a text means, and writing text back.
Where you meet it every day
You use NLP daily without noticing. IBM says it is already part of everyday life for many people, powering search engines, chatbots for customer service, voice-operated GPS systems and question-answering digital assistants on smartphones such as Amazon's Alexa, Apple's Siri and Microsoft's Cortana.
Search and assistants
IBM names search engines and voice assistants such as Alexa and Siri as everyday NLP.
Translation
IBM says NLP facilitates language translation, converting text from one language to another while preserving meaning, context and nuances.
Chatbots and paperwork
IBM says NLP chatbots can handle routine customer questions, and NLP tools can sort, pull key facts from and summarise documents.
Why language is hard
People read language without effort. For a computer, almost every sentence has traps. Three of them show up everywhere.
The first trap is ambiguity: one piece of text with more than one meaning. IBM gives examples. The word "make" is a verb in "I can make a paper plane" and a noun in "What make of car do you own?". The verb also changes meaning between "make the grade", which means to achieve, and "make a bet", which means to place one. Wikipedia adds that the grammar of natural languages is ambiguous, and a typical sentence may have thousands of possible analyses, most of which look completely nonsensical to a human.
The second trap is context: the meaning depends on the words around it. IBM mentions the pronoun problem, working out who "she" refers to, and idioms, where "bear" may not mean an animal at all but a large, hairy person.
The third trap is sound, when the text comes from speech. IBM says that sorting out "I will be merry when I marry Mary" takes a sophisticated NLP system. Wikipedia explains part of the reason for speech: in natural speech there are hardly any pauses between successive words, and the sounds blend into each other.
Slang and new words add one more trap. A bag-of-words model only knows the words in its vocabulary, so a brand-new slang word has no column at all.
The classic pipeline
Most NLP work follows a series of steps. IBM describes a typical pipeline that starts with text preprocessing, then feature extraction, then analysis. Here is a simple version of the same idea, in five steps. Each step with its own lesson is named.
- 1
Clean
Remove clutter. IBM says text cleaning removes unwanted elements such as punctuation, special characters and numbers. It also lists lowercasing, so that "Apple" and "apple" are treated the same.
- 2
Tokenise
Split the text into pieces. IBM: tokenization involves splitting the text into smaller units like words, sentences or phrases. The next lesson, Tokenization, covers it.
- 3
Normalise
Shrink different forms of a word into one. IBM says stemming or lemmatization reduces words to their root form, such as "running" becoming "run". Two lessons follow: Stemming, which chops word endings off, and Lemmatization, which uses a dictionary to find the base form.
- 4
Turn into numbers
A model needs numbers, not letters. IBM calls this feature extraction: converting raw text into numerical representations that machines can analyse. Bag of words and TF-IDF are the first methods, below. A richer way is word embeddings, which have their own lesson, Embeddings.
- 5
Model
Feed the numbers to a machine learning model that does the job: sort, label, translate, answer.
Tip
You do not need every step every time. Modern models such as transformers often do their own splitting and learn their own numbers. The five steps are still the clearest way to see what happens to text.
Bag of words: counting what is there
The simplest way to turn a document into numbers is to count its words. Wikipedia describes the bag-of-words model as a model of text that uses an unordered collection, a "bag", of words. It disregards word order, and so most of syntax or grammar, but captures multiplicity, meaning how many times each word appears.
The cost of that is real. Wikipedia points out that "man bites dog" and "dog bites man" give the same bag, so a program that uses only the bag must treat them the same. Even so, Wikipedia says the representation is fast and may be enough for simple tasks that do not need word order. The scikit-learn guide calls it quite simplistic but surprisingly useful in practice.
Here is the whole idea in plain Python, with four tiny made-up documents. We split each one into words, build the vocabulary, then count each word per document.
docs = [
"the cat sat on the mat",
"the dog sat on the log",
"the cat chased the dog",
"a bird sang a song",
]
# 1. tokenise: lower-case, split on spaces
tokens = [d.split() for d in docs]
# 2. vocabulary: every different word, in a fixed order
vocab = sorted({w for t in tokens for w in t})
# 3. bag of words: count each word in each document
bow = [[t.count(w) for w in vocab] for t in tokens]
print("words:", vocab)
for i, row in enumerate(bow):
print(f"doc {i}:", row)Output
words: ['a', 'bird', 'cat', 'chased', 'dog', 'log', 'mat', 'on', 'sang', 'sat', 'song', 'the'] doc 0: [0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 2] doc 1: [0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 2] doc 2: [0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 2] doc 3: [2, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0]
Each row is one document, each column one word. The words are in alphabetical order: a, bird, cat, chased, dog, log, mat, on, sang, sat, song, the. Look at doc 0, "the cat sat on the mat". It has a 1 under cat, mat, on and sat, and a 2 under the. A document became a list of 12 numbers. That is something a model can work with.
TF-IDF: counting what matters
Raw counts have a weak spot. Words like "the", "a" and "is" show up everywhere. The scikit-learn guide says such very frequent words carry very little meaningful information about the actual contents of a document, and that fed in directly they would shadow the rarer, more interesting words. Its fix is the tf-idf transform.
Wikipedia defines tf-idf as a measure of the importance of a word to a document in a collection, adjusted for the fact that some words appear more frequently in general. It is a refinement over bag of words. It has two parts.
Key words
- Term frequency (tf)
- How often a word appears in one document, as a share of that document's words.
- Document frequency
- How many documents in the corpus contain the word at least once.
- Inverse document frequency (idf)
- A number that is big for rare words and small for common ones.
Wikipedia writes idf as the logarithm of the total number of documents divided by the number of documents that contain the word, and tf-idf as tf multiplied by idf. In symbols:
In words: take the share of the document made up of the word, and multiply it by the log of (how many documents there are, divided by how many documents hold the word). Here N is the number of documents and n_t is the number that contain the word t. A word in every document gets a log of 1, which is 0, so its score drops to nothing. A word in just one document gets the biggest boost. Wikipedia summarises it: a high weight comes from a high term frequency in the document and a low document frequency across the collection, so the weights tend to filter out common terms. That is the stop word idea again, with no list needed.
import math
docs = [
"the cat sat on the mat",
"the dog sat on the log",
"the cat chased the dog",
"a bird sang a song",
]
tokens = [d.split() for d in docs]
n = len(docs)
def tf(word, doc):
# share of the document's words that are this word
return doc.count(word) / len(doc)
def idf(word):
# how rare: log of (all documents / documents holding the word)
holding = sum(1 for t in tokens if word in t)
return math.log(n / holding)
for word in ["the", "sat", "cat", "mat", "bird"]:
scores = [round(tf(word, t) * idf(word), 3) for t in tokens]
print(f"{word:5} idf={idf(word):.3f} tf-idf per doc: {scores}")Output
the idf=0.288 tf-idf per doc: [0.096, 0.096, 0.115, 0.0] sat idf=0.693 tf-idf per doc: [0.116, 0.116, 0.0, 0.0] cat idf=0.693 tf-idf per doc: [0.116, 0.0, 0.139, 0.0] mat idf=1.386 tf-idf per doc: [0.231, 0.0, 0.0, 0.0] bird idf=1.386 tf-idf per doc: [0.0, 0.0, 0.0, 0.277]
Read the first line. The word "the" is in 3 of the 4 documents, so its idf is only 0.288. In doc 0 it makes up 2 of 6 words, yet its score is 0.096. Now read the "mat" line. It is in a single document, so its idf is 1.386, and in doc 0 its score is 0.231, more than twice the score of "the", even though it appears only once. The same goes for "bird" in doc 3. TF-IDF found the words that make each document different.
Quick check
A word appears in every document of the corpus. What is its TF-IDF score?
How transformers changed the field
Bag of words and TF-IDF treat each word on its own. They cannot tell "not good" from "good". For a long time, people fixed that with more rules and more clever counting.
Wikipedia tells the story in three stages. Up until the 1980s, most NLP systems were based on complex sets of hand-written rules. From the late 1980s, machine learning algorithms brought a revolution. In the 2010s, representation learning and deep neural network methods became widespread, with state-of-the-art results in many language tasks.
The Transformers lesson covered the latest step. IBM describes transformer models as using tokenisation (the position of each token, whether words or subwords) and self-attention to capture how different parts of the language relate to one another, and says they can be trained efficiently with self-supervised learning on massive text. Self-attention was the idea in the Self-Attention and Multi-head Attention lessons. Before that, the Recurrent Neural Networks, LSTM and GRU lessons showed how a network could read text one word at a time.
Common tasks
Once text is numbers, you can point a model at a job. Four are very common.
Classification
Put a document into a group. Wikipedia lists text classification as a major NLP task, and says the bag-of-words model is commonly used for document classification, with word counts as the features.
Named-entity recognition
Find names in text. Wikipedia: given a stream of text, decide which items map to proper names, such as people or places, and what type each is: person, location, organisation. IBM's example: London is a location, Maria a person's name.
Sentiment analysis
Work out the mood. Wikipedia: identifying and classifying the emotional tone in text, as positive, negative or neutral. It says models for this often use inputs such as word n-grams and TF-IDF features.
A fourth task is summarisation. Wikipedia calls it automatic summarisation: produce a readable summary of a chunk of text, often for a known type of text such as research papers or articles in the financial section of a newspaper.
Quick check
Which task would you use to find that "Maria" is a person and "London" is a place?
A tiny sentiment score by hand
Now the third task, sentiment, in its simplest form. Add 1 for every word on a good list and take away 1 for every word on a bad list. This is a toy: a real system learns its word lists from data.
good = {"great", "love", "happy", "fast"}
bad = {"awful", "hate", "slow", "broken"}
sentences = [
"I love this app and it is fast",
"The screen is broken and the app is slow",
"This is not great",
]
for s in sentences:
words = s.lower().split()
score = sum(w in good for w in words) - sum(w in bad for w in words)
print(score, "|", s)Output
2 | I love this app and it is fast -2 | The screen is broken and the app is slow 1 | This is not great
Tip
Try it yourself: add a word to each list and rerun. Small changes show you how fragile a hand-made word list is.
The first two sentences work. "I love this app and it is fast" scores 2: love and fast are both on the good list. "The screen is broken and the app is slow" scores -2: broken and slow are both on the bad list.
The third is wrong. "This is not great" scores 1, which looks happy, because the program sees "great" and cannot see the "not" in front of it. This is the bag-of-words weakness from earlier, in action: counting words without their neighbours loses meaning. Wikipedia says that sentiment models typically use inputs such as word n-grams (short runs of neighbouring words) and TF-IDF features, or deep learning models designed to recognise both long-term and short-term dependencies. Those are the ideas behind the models in the earlier lessons.
Quick check
Why does "This is not great" score 1 in our toy sentiment program?
What to remember
The goal
NLP gets a computer to work with human language, which is full of ambiguity and context.
The steps
Clean, tokenise, normalise, turn into numbers, then model. Counting words with bag of words or TF-IDF is the first way to get numbers.
The limit
Counting loses word order, which is why embeddings and transformers took over for harder jobs.
Next: tokenization, cutting text into pieces a model can count.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.