AI engineering / Beginner / 01.36
Type of Models
The kinds of AI models you can pick from, what each takes in and gives back, and how to match one to a feature you want to build.
14 min readFree
Not every model does the same job
The last lesson, MCP, ended with a promise: next, types of models, from text-only to models that see and hear. Here it is.
Think of a toolbox. A hammer, a saw and a tape measure are all tools, but you would not measure a wall with a saw. Models are the same. Some take text and write text. Some take a photo and describe it. Some turn a voice recording into words. Picking the wrong kind is the most common beginner mistake, and it costs time and money.
By the end you will know the main kinds, what goes into each and what comes out, when to choose a small model over a large one, where to read a model's limits, and you will run a small program that picks a kind for a feature request.
Key words
- Model
- A program that learned patterns from lots of examples. You give it an input and it gives back an output.
- Modality
- A type of data: text, images, audio or video.
- Multimodal
- Able to handle more than one modality, for example text and images together.
- Embedding
- A list of numbers that stands for the meaning of a piece of text or an image.
- Reasoning model
- A model that writes out extra working steps before it gives its final answer.
- Parameters
- The numbers inside a model that it learned. More parameters usually means a bigger model.
- Model card
- A short document that comes with a model and says what it is for, how it was tested and where it falls short.
A map of the kinds
Before the details, one picture. Every kind of model is described by two things: what you give it (the input) and what you get back (the output). That pair is the fastest way to choose.
The Hugging Face Tasks page is a good place to see this. It calls itself the home for machine learning tasks and groups them by data type: Multimodal, Natural Language Processing, Computer Vision, Audio, Tabular and Reinforcement Learning. Inside those groups you will find jobs named Text Generation, Sentence Similarity, Text-to-Image, Automatic Speech Recognition and Text-to-Speech. Each name is just an input and an output in words.
Here is the short version we will work through.
Words in, words out
Chat models, reasoning models and code models. They read text and write text or code.
Other things in
Embedding models (text in, numbers out), vision models (image in, text out) and speech-to-text models (audio in, text out).
Other things out
Image generators (text in, picture out) and text-to-speech models (text in, audio out).
Text generation (chat) models
This is the kind most people mean when they say "AI". You give it text, such as a question or an instruction, and it writes text back. The Large language model page on Wikipedia describes the basic job as predicting the next piece of text, one token at a time. After more training, these models learn to follow instructions, which is what makes them feel like a chat.
Typical product uses: a help chat on a website, a button that rewrites a sentence in a friendlier tone, a summary of a long email, a draft of a product description.
The Tokens and How LLMs Work lessons in this course explain the inside. The Tokenization lesson in Machine Learning Foundations goes deeper on how text is cut up.
Tip
Start with a chat model. Most text features need nothing fancier, and a plain model is faster and cheaper than a thinking one.
Reasoning models
Some questions need several steps. "A train leaves at 9, another at 10, when do they meet?" A person would scribble working on paper first. A reasoning model does something like that: it writes out working steps before the final answer.
Wikipedia's page on reasoning language models says they are large language models trained to solve tasks that need several steps of reasoning. It says they tend to do better than standard models on logic, maths and programming tasks, can revisit and revise earlier steps, and make use of extra computation. The same page lists one model that has an adjustable amount of "thinking" tokens, so the working is counted in tokens like everything else.
That last point is the catch for a product. Extra thinking means extra tokens, and extra tokens mean a slower reply and a bigger bill. Typical uses: a maths tutor that shows its steps, a tool that checks a long contract for clashing rules, a planner that has to juggle many limits at once.
Use it when
The answer needs many steps and a wrong answer is costly.
Skip it when
The job is quick and simple, like rewording a line or sorting a message.
Remember
Thinking steps are not a guarantee. A reasoning model can still be wrong, so check what matters.
Embedding models
An embedding model does not write anything a person would read. It takes text and returns a list of numbers. Wikipedia describes a word embedding as a representation where words that are closer together in number-space are expected to be similar in meaning. The Hugging Face Tasks page has a Sentence Similarity group, so the same idea is used for whole sentences.
Why would you want that? Because numbers can be compared fast. Turn every help article into numbers once. When a user asks a question, turn it into numbers too, and find the closest articles. That is search by meaning, not by exact words. The Hugging Face Tasks page lists Feature Extraction and Sentence Similarity among its language tasks, which sit close to this.
Typical uses: smart search, "related articles", finding duplicate support tickets, and giving a chat model the right pages to read. The Embeddings lesson in Machine Learning Foundations goes deeper on how the numbers are made.
Vision and multimodal models
A vision-capable model accepts an image along with your text and answers in text. Wikipedia describes multimodal learning as a type of deep learning that combines several types of data, called modalities, such as text, audio, images or video. It lists tasks this helps with: visual question answering, image captioning, and text-to-image generation.
Typical uses: describing a photo for someone using a screen reader, reading text off a receipt, checking whether an uploaded picture matches the product you sell, answering a question about a chart.
The page also names two hard parts. Alignment means working out which part of one modality matches which part of another, such as which words go with which part of a photo. Fusion means deciding how to combine what each modality says. You do not need to solve these, but they are a reminder that mixing data types is harder than text alone.
In model names, the tag VL often means vision-language. The Large language model page lists it with other common tags.
Image generation
This is the mirror of vision. Instead of reading a picture, the model makes one. You write a short description, called a prompt, and it returns an image. The Multimodal learning page lists text-to-image generation as one of the tasks multimodal methods are used for, and the Tasks page has a Text-to-Image group, plus Text-to-Video.
Typical uses: a banner image from a short brief, mock-up pictures for a design idea, a set of icons in one style. For a designer, this is a sketching tool, not a final artist. Always look at what comes out.
Speech-to-text and text-to-speech
Two models cover talking and listening, and they are opposites.
Speech-to-text
Audio in, text out. Wikipedia says speech recognition is about methods that translate spoken language into text. Uses: captions, meeting notes, voice commands.
Text-to-speech
Text in, audio out. Wikipedia says a text-to-speech system converts normal language text into speech. Uses: reading an article aloud, a voice for an app.
Watch out
Speech recognition is hard because voices vary in accent, pitch, volume and speed, and recordings pick up background noise. Test with real voices, not only clean ones.
Wikipedia also says a speech synthesizer is judged by how close it sounds to a human voice and how clearly it can be understood. Both are things you listen to yourself before you ship.
Put them together and you can build a voice assistant: speech-to-text, then a chat model, then text-to-speech. Three models, each doing one job.
Small versus large
Any kind above comes in different sizes. Size is counted in parameters, the numbers a model learned. The Large language model page admits that the word "large" is vague: there is no fixed number of parameters that makes a model large.
So treat size as a dial, not a label. Big models usually know and do more, but they are slower, cost more to run and need more computer power. Small models are quicker and cheaper and can be enough for a narrow job. Wikipedia's reasoning page tells of an experiment where, with extra thinking at answer time, a small model beat a much larger one on hard reasoning tasks. That shows size is not the whole story.
A simple habit: try the smallest model that could work, test it on real examples, and move up only if it fails.
General versus specialised
A general model is trained to do many things. A specialised model is tuned for one area, such as programming, maths or a particular field. The Large language model page says a model may be specialised for specific applications or input types, and that names often carry a tag: Code or Coder for programming, Math, VL for vision-language, and Thinking for reasoning ones.
A code model takes a description and returns code, so it suits a tool that writes spreadsheet formulas or suggests fixes. A domain model, say one tuned on medical or legal text, may handle that vocabulary better than a general one. But a narrow model can be worse outside its area. Read what it was built for.
Where to read a model's limits
How do you know what a model is good at, and where it fails? Read its model card.
The paper "Model Cards for Model Reporting" (arXiv 1810.03993) opens by saying trained models are used more and more for high-impact tasks in areas such as law enforcement, medicine, education and employment. To make intended uses clear and cut misuse, it recommends that released models come with documentation about how they perform. It calls that a model card: a short document that gives benchmarked evaluation in different conditions, such as across different groups of people, that matter for the intended use.
In plain words, a model card is the label on the packet. Before you pick a model, look for the card and ask three things.
- 1
What is it for?
Find the intended uses. If your feature is outside them, be careful.
- 2
How was it tested?
Look for results on data like yours: your languages, your kinds of images, your accents.
- 3
Where does it fall short?
Look for known limits and warnings, and plan a check for each one.
Tip
No card, or a card that says nothing about limits, is itself a piece of information. Test the model harder before you trust it.
Worked example: a model-kind picker
Let us turn the map into code. Below is a small catalogue of model kinds as plain Python dictionaries: what each takes, what it gives, whether it thinks before answering, and an example use. A function called pick takes a feature's input and output types plus a needs-reasoning flag and returns the kinds that match.
This code does not call any model, so there is nothing to pay for and no key to set up. It only matches labels. It is a way to practise thinking in inputs and outputs.
KINDS = [
{"name": "chat model", "takes": "text", "gives": "text", "thinks": False,
"use": "a help chat that answers customer questions"},
{"name": "reasoning model", "takes": "text", "gives": "text", "thinks": True,
"use": "a tutor that works through a maths problem step by step"},
{"name": "code model", "takes": "text", "gives": "code", "thinks": False,
"use": "a button that writes a spreadsheet formula"},
{"name": "embedding model", "takes": "text", "gives": "numbers", "thinks": False,
"use": "finding help articles that match a question"},
{"name": "vision model", "takes": "image", "gives": "text", "thinks": False,
"use": "describing a photo for a screen reader"},
{"name": "image generator", "takes": "text", "gives": "image", "thinks": False,
"use": "making a banner picture from a short brief"},
{"name": "speech-to-text model", "takes": "audio", "gives": "text", "thinks": False,
"use": "turning a voice memo into notes"},
{"name": "text-to-speech model", "takes": "text", "gives": "audio", "thinks": False,
"use": "reading a blog post aloud"},
]
def pick(takes, gives, needs_reasoning):
found = []
for kind in KINDS:
if kind["takes"] != takes or kind["gives"] != gives:
continue
if needs_reasoning and not kind["thinks"]:
continue
if not needs_reasoning and kind["thinks"]:
continue
found.append(kind["name"])
return found
features = [
("Answer customer questions in a chat", "text", "text", False),
("Find help articles close in meaning to a question", "text", "numbers", False),
("Describe a photo the user uploads", "image", "text", False),
("Read every new blog post aloud", "text", "audio", False),
("Solve a tricky multi-step maths puzzle", "text", "text", True),
]
for label, takes, gives, thinks in features:
print(label, "->", pick(takes, gives, thinks) or "no match")Output
Answer customer questions in a chat -> ['chat model'] Find help articles close in meaning to a question -> ['embedding model'] Describe a photo the user uploads -> ['vision model'] Read every new blog post aloud -> ['text-to-speech model'] Solve a tricky multi-step maths puzzle -> ['reasoning model']
Read the output. Each request landed on exactly one kind. "Describe a photo" has an image going in, so only the vision model fits. "Solve a tricky multi-step maths puzzle" asks for text in and text out like a chat feature, but the reasoning flag is on, so it picked the reasoning model instead of the chat one.
What if nothing matches? Here is a smaller catalogue to show it.
KINDS = [
{"name": "chat model", "takes": "text", "gives": "text", "thinks": False},
{"name": "speech-to-text model", "takes": "audio", "gives": "text", "thinks": False},
]
def pick(takes, gives, needs_reasoning):
return [k["name"] for k in KINDS
if k["takes"] == takes and k["gives"] == gives
and k["thinks"] == needs_reasoning]
print(pick("audio", "image", False) or "no match: split the job into two steps")
print(pick("audio", "text", False))Output
no match: split the job into two steps ['speech-to-text model']
Nothing turns audio straight into an image, so the picker says so. The answer in real products is the same: split the job. Turn the audio into text with speech-to-text, then send that text to an image generator.
A real catalogue would hold many more kinds and many real models inside each one. And choosing a real model brings more questions than this picker asks: speed, cost, size, limits, and what its model card says. The next lessons come back to some of those.
Quick checks
Quick check
A user uploads a photo of a receipt and you want the text on it. Which kind fits best?
Quick check
What is the main cost of using a reasoning model?
Quick check
Where should you look to learn what a model is for and where it falls short?
Putting it together
Choosing a model starts with two questions: what goes in, and what should come out? Then ask whether the job needs several steps of thinking, how big a model you really need, and whether a specialised one fits. Finally, read the model card before you commit.
Remember that the models are tools, and a feature often uses several of them in a row. A voice assistant is speech-to-text, a chat model and text-to-speech. A smart help centre is an embedding model and a chat model.
Next: closed vs open models, renting a model or running your own.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.