esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Advanced / 03.44

Natural Language Processing

How a computer can work with words: the steps from raw text to numbers, and two simple ways to count them by hand.

14 min readFree

Teaching computers to work with words

The last lesson, Generative Adversarial Networks, ended with a promise: next, natural language processing, teaching computers to work with words. So far this course has worked mostly with numbers and pictures. Now we turn to text.

Words are tricky for a computer. A computer only does arithmetic. It does not know that "cat" is a pet or that "not good" is bad news. Natural language processing is the field that closes that gap. By the end of this lesson you will know what it is, why it is hard, the steps most text goes through, and how to turn four tiny documents into numbers with nothing but plain Python.

Key words

Natural language
The language people speak and write every day, like English or Tamil, as opposed to a programming language.
NLP
Natural language processing: getting a computer to work with human language.
Document
One piece of text: a sentence, an email, a review, a whole article.
Corpus
A collection of documents, the pile of text you work on.
Token
One small piece of text, usually a word. Cutting text into tokens is tokenisation.
Vocabulary
The list of every different word that shows up in the corpus.
Bag of words
A way of turning a document into numbers by counting its words and ignoring their order.
TF-IDF
A score that is high for words that matter in one document and low for words found everywhere.

What is NLP?

Wikipedia gives a short, plain definition: natural language processing is "the processing of natural language information by a computer". It describes NLP as a subfield of computer science that is closely associated with artificial intelligence.

IBM's guide adds more detail. It calls NLP a subfield of computer science and AI that uses machine learning to enable computers to understand and communicate with human language. It lets computers recognise, understand and generate text and speech.

Wikipedia lists four major jobs inside an NLP system: speech recognition, text classification, natural language understanding and natural language generation. In plain words: hearing speech, sorting text into groups, working out what a text means, and writing text back.

Where you meet it every day

You use NLP daily without noticing. IBM says it is already part of everyday life for many people, powering search engines, chatbots for customer service, voice-operated GPS systems and question-answering digital assistants on smartphones such as Amazon's Alexa, Apple's Siri and Microsoft's Cortana.

  • Search and assistants

    IBM names search engines and voice assistants such as Alexa and Siri as everyday NLP.

  • Translation

    IBM says NLP facilitates language translation, converting text from one language to another while preserving meaning, context and nuances.

  • Chatbots and paperwork

    IBM says NLP chatbots can handle routine customer questions, and NLP tools can sort, pull key facts from and summarise documents.

Why language is hard

People read language without effort. For a computer, almost every sentence has traps. Three of them show up everywhere.

The first trap is ambiguity: one piece of text with more than one meaning. IBM gives examples. The word "make" is a verb in "I can make a paper plane" and a noun in "What make of car do you own?". The verb also changes meaning between "make the grade", which means to achieve, and "make a bet", which means to place one. Wikipedia adds that the grammar of natural languages is ambiguous, and a typical sentence may have thousands of possible analyses, most of which look completely nonsensical to a human.

The second trap is context: the meaning depends on the words around it. IBM mentions the pronoun problem, working out who "she" refers to, and idioms, where "bear" may not mean an animal at all but a large, hairy person.

The third trap is sound, when the text comes from speech. IBM says that sorting out "I will be merry when I marry Mary" takes a sophisticated NLP system. Wikipedia explains part of the reason for speech: in natural speech there are hardly any pauses between successive words, and the sounds blend into each other.

Slang and new words add one more trap. A bag-of-words model only knows the words in its vocabulary, so a brand-new slang word has no column at all.

The classic pipeline

Most NLP work follows a series of steps. IBM describes a typical pipeline that starts with text preprocessing, then feature extraction, then analysis. Here is a simple version of the same idea, in five steps. Each step with its own lesson is named.

  1. 1

    Clean

    Remove clutter. IBM says text cleaning removes unwanted elements such as punctuation, special characters and numbers. It also lists lowercasing, so that "Apple" and "apple" are treated the same.

  2. 2

    Tokenise

    Split the text into pieces. IBM: tokenization involves splitting the text into smaller units like words, sentences or phrases. The next lesson, Tokenization, covers it.

  3. 3

    Normalise

    Shrink different forms of a word into one. IBM says stemming or lemmatization reduces words to their root form, such as "running" becoming "run". Two lessons follow: Stemming, which chops word endings off, and Lemmatization, which uses a dictionary to find the base form.

  4. 4

    Turn into numbers

    A model needs numbers, not letters. IBM calls this feature extraction: converting raw text into numerical representations that machines can analyse. Bag of words and TF-IDF are the first methods, below. A richer way is word embeddings, which have their own lesson, Embeddings.

  5. 5

    Model

    Feed the numbers to a machine learning model that does the job: sort, label, translate, answer.

Tip

You do not need every step every time. Modern models such as transformers often do their own splitting and learn their own numbers. The five steps are still the clearest way to see what happens to text.

Bag of words: counting what is there

The simplest way to turn a document into numbers is to count its words. Wikipedia describes the bag-of-words model as a model of text that uses an unordered collection, a "bag", of words. It disregards word order, and so most of syntax or grammar, but captures multiplicity, meaning how many times each word appears.

The cost of that is real. Wikipedia points out that "man bites dog" and "dog bites man" give the same bag, so a program that uses only the bag must treat them the same. Even so, Wikipedia says the representation is fast and may be enough for simple tasks that do not need word order. The scikit-learn guide calls it quite simplistic but surprisingly useful in practice.

Here is the whole idea in plain Python, with four tiny made-up documents. We split each one into words, build the vocabulary, then count each word per document.

Python
docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "the cat chased the dog",
    "a bird sang a song",
]

# 1. tokenise: lower-case, split on spaces
tokens = [d.split() for d in docs]

# 2. vocabulary: every different word, in a fixed order
vocab = sorted({w for t in tokens for w in t})

# 3. bag of words: count each word in each document
bow = [[t.count(w) for w in vocab] for t in tokens]

print("words:", vocab)
for i, row in enumerate(bow):
    print(f"doc {i}:", row)

Output

words: ['a', 'bird', 'cat', 'chased', 'dog', 'log', 'mat', 'on', 'sang', 'sat', 'song', 'the']
doc 0: [0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 2]
doc 1: [0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 2]
doc 2: [0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 2]
doc 3: [2, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0]

Each row is one document, each column one word. The words are in alphabetical order: a, bird, cat, chased, dog, log, mat, on, sang, sat, song, the. Look at doc 0, "the cat sat on the mat". It has a 1 under cat, mat, on and sat, and a 2 under the. A document became a list of 12 numbers. That is something a model can work with.

TF-IDF: counting what matters

Raw counts have a weak spot. Words like "the", "a" and "is" show up everywhere. The scikit-learn guide says such very frequent words carry very little meaningful information about the actual contents of a document, and that fed in directly they would shadow the rarer, more interesting words. Its fix is the tf-idf transform.

Wikipedia defines tf-idf as a measure of the importance of a word to a document in a collection, adjusted for the fact that some words appear more frequently in general. It is a refinement over bag of words. It has two parts.

Key words

Term frequency (tf)
How often a word appears in one document, as a share of that document's words.
Document frequency
How many documents in the corpus contain the word at least once.
Inverse document frequency (idf)
A number that is big for rare words and small for common ones.

Wikipedia writes idf as the logarithm of the total number of documents divided by the number of documents that contain the word, and tf-idf as tf multiplied by idf. In symbols:

tf-idf(t,d)=tf(t,d)×idf(t),idf(t)=ln⁡Nnt\text{tf-idf}(t,d)=\text{tf}(t,d)\times\text{idf}(t),\qquad \text{idf}(t)=\ln\frac{N}{n_t}

In words: take the share of the document made up of the word, and multiply it by the log of (how many documents there are, divided by how many documents hold the word). Here N is the number of documents and n_t is the number that contain the word t. A word in every document gets a log of 1, which is 0, so its score drops to nothing. A word in just one document gets the biggest boost. Wikipedia summarises it: a high weight comes from a high term frequency in the document and a low document frequency across the collection, so the weights tend to filter out common terms. That is the stop word idea again, with no list needed.

Python
import math

docs = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "the cat chased the dog",
    "a bird sang a song",
]
tokens = [d.split() for d in docs]
n = len(docs)

def tf(word, doc):
    # share of the document's words that are this word
    return doc.count(word) / len(doc)

def idf(word):
    # how rare: log of (all documents / documents holding the word)
    holding = sum(1 for t in tokens if word in t)
    return math.log(n / holding)

for word in ["the", "sat", "cat", "mat", "bird"]:
    scores = [round(tf(word, t) * idf(word), 3) for t in tokens]
    print(f"{word:5} idf={idf(word):.3f}  tf-idf per doc: {scores}")

Output

the   idf=0.288  tf-idf per doc: [0.096, 0.096, 0.115, 0.0]
sat   idf=0.693  tf-idf per doc: [0.116, 0.116, 0.0, 0.0]
cat   idf=0.693  tf-idf per doc: [0.116, 0.0, 0.139, 0.0]
mat   idf=1.386  tf-idf per doc: [0.231, 0.0, 0.0, 0.0]
bird  idf=1.386  tf-idf per doc: [0.0, 0.0, 0.0, 0.277]

Read the first line. The word "the" is in 3 of the 4 documents, so its idf is only 0.288. In doc 0 it makes up 2 of 6 words, yet its score is 0.096. Now read the "mat" line. It is in a single document, so its idf is 1.386, and in doc 0 its score is 0.231, more than twice the score of "the", even though it appears only once. The same goes for "bird" in doc 3. TF-IDF found the words that make each document different.

Quick check

A word appears in every document of the corpus. What is its TF-IDF score?

How transformers changed the field

Bag of words and TF-IDF treat each word on its own. They cannot tell "not good" from "good". For a long time, people fixed that with more rules and more clever counting.

Wikipedia tells the story in three stages. Up until the 1980s, most NLP systems were based on complex sets of hand-written rules. From the late 1980s, machine learning algorithms brought a revolution. In the 2010s, representation learning and deep neural network methods became widespread, with state-of-the-art results in many language tasks.

The Transformers lesson covered the latest step. IBM describes transformer models as using tokenisation (the position of each token, whether words or subwords) and self-attention to capture how different parts of the language relate to one another, and says they can be trained efficiently with self-supervised learning on massive text. Self-attention was the idea in the Self-Attention and Multi-head Attention lessons. Before that, the Recurrent Neural Networks, LSTM and GRU lessons showed how a network could read text one word at a time.

Common tasks

Once text is numbers, you can point a model at a job. Four are very common.

  • Classification

    Put a document into a group. Wikipedia lists text classification as a major NLP task, and says the bag-of-words model is commonly used for document classification, with word counts as the features.

  • Named-entity recognition

    Find names in text. Wikipedia: given a stream of text, decide which items map to proper names, such as people or places, and what type each is: person, location, organisation. IBM's example: London is a location, Maria a person's name.

  • Sentiment analysis

    Work out the mood. Wikipedia: identifying and classifying the emotional tone in text, as positive, negative or neutral. It says models for this often use inputs such as word n-grams and TF-IDF features.

A fourth task is summarisation. Wikipedia calls it automatic summarisation: produce a readable summary of a chunk of text, often for a known type of text such as research papers or articles in the financial section of a newspaper.

Quick check

Which task would you use to find that "Maria" is a person and "London" is a place?

A tiny sentiment score by hand

Now the third task, sentiment, in its simplest form. Add 1 for every word on a good list and take away 1 for every word on a bad list. This is a toy: a real system learns its word lists from data.

Python
good = {"great", "love", "happy", "fast"}
bad = {"awful", "hate", "slow", "broken"}

sentences = [
    "I love this app and it is fast",
    "The screen is broken and the app is slow",
    "This is not great",
]

for s in sentences:
    words = s.lower().split()
    score = sum(w in good for w in words) - sum(w in bad for w in words)
    print(score, "|", s)

Output

2 | I love this app and it is fast
-2 | The screen is broken and the app is slow
1 | This is not great

Tip

Try it yourself: add a word to each list and rerun. Small changes show you how fragile a hand-made word list is.

The first two sentences work. "I love this app and it is fast" scores 2: love and fast are both on the good list. "The screen is broken and the app is slow" scores -2: broken and slow are both on the bad list.

The third is wrong. "This is not great" scores 1, which looks happy, because the program sees "great" and cannot see the "not" in front of it. This is the bag-of-words weakness from earlier, in action: counting words without their neighbours loses meaning. Wikipedia says that sentiment models typically use inputs such as word n-grams (short runs of neighbouring words) and TF-IDF features, or deep learning models designed to recognise both long-term and short-term dependencies. Those are the ideas behind the models in the earlier lessons.

Quick check

Why does "This is not great" score 1 in our toy sentiment program?

What to remember

  • The goal

    NLP gets a computer to work with human language, which is full of ambiguity and context.

  • The steps

    Clean, tokenise, normalise, turn into numbers, then model. Counting words with bag of words or TF-IDF is the first way to get numbers.

  • The limit

    Counting loses word order, which is why embeddings and transformers took over for harder jobs.

Next: tokenization, cutting text into pieces a model can count.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.