AI engineering / Core / 02.01
Choosing the Right Model
Write down the job, shortlist a few models, then test them on your own examples and compare cost, speed and quality.
13 min readFree
Matching the job to the model
The last lesson, Data catalogs, ended with a promise: next, choosing the right model, matching the job to the model. Here it is. Your data is tidy and labelled. Now you have to pick which model reads it.
Picking a model is like picking a vehicle. A bicycle, a van and a lorry can all carry things. But you would not send a lorry to fetch milk, and you would not move a sofa on a bicycle. The right choice depends on the job.
By the end you will know how to write down a job, shortlist models, test them on your own examples, and route easy work to a small model. All the numbers in the code are made up.
Key words
- Model
- The program that reads your input and writes an answer. Each one has a name and a size.
- Candidate
- A model you are thinking of using. You compare several candidates before you pick one.
- Eval set
- A small set of questions with the right answers written next to them. You use it to score models. "Eval" is short for evaluation.
- Accuracy
- The share of eval questions a model got right. 5 right out of 6 is 83.3 percent.
- Latency
- How long you wait for one answer, in seconds.
- Routing
- Sending each request to a different model depending on how hard it is.
Start from the job, not the model
The most common mistake is to start with a famous model name and look for a use for it. Turn it around. Write down the job first, in plain words, and let the job point at the model.
Seven questions cover almost every feature.
Inputs
What goes in: short text, long documents, pictures, audio? A model that cannot take your input type is out before you start.
Outputs
What must come out: a label, a paragraph, a JSON object, a picture? Some jobs need a fixed shape, as the Structured Output lesson showed.
Quality bar
How often may it be wrong? A spelling hint can miss now and then. A medical or money answer cannot.
Speed
Will a person wait for the answer, or does it run overnight? Chat feels slow after a couple of seconds. A nightly report can take minutes.
Budget
How many calls per day, and what can each one cost? A small cost per call grows fast at a million calls.
Privacy
Is the text private, such as customer emails? Then you need to know where it is sent and who may keep it.
Languages
Which languages do your users write in? Test every one of them. A model can be strong in one language and weak in another.
Write the answers on one page. This page is your brief. It is the thing you will test every model against.
Tip
Be honest about the quality bar. If a wrong answer is cheap to fix, a small model is probably fine. If a wrong answer hurts someone, spend more and test harder.
Build a shortlist
Now you can look at what exists. You do not need to try every model in the world. Pick three to five candidates that match your brief on paper: they take your kind of input, they fit your budget, and they meet your privacy needs.
- 1
Cross out the impossible
Drop any model that cannot take your input, or that your privacy rules forbid.
- 2
Pick a spread of sizes
Keep at least one small, cheap model and one large, strong one. You want to learn how much quality the extra money buys.
- 3
Read each provider's docs
Docs say what the family is for, which inputs it takes and what it is called now. Names change, so read the current page.
- 4
Write the shortlist down
Three to five names, with the reason each one is on the list.
What hosted providers offer
The first three lines below come from model overview pages. They show that one provider usually sells several kinds of models: for chat, for reasoning, for turning text into numbers, and more. Providers are listed in the order of the course plan, with no ranking.
Anthropic (Claude)
The models overview page says: "All current models support text and image input, text output, multilingual capabilities, vision, and tool use." It describes the smallest family member as being "For high-volume, latency-sensitive tasks such as classification, extraction, and routing".
Google (Gemini)
The Gemini models page lists one embedding model as "High-dimensional vector representations for advanced semantic search, text classification, and RAG systems." It describes another as "Our fastest, most cost-effective 3.5 model for high-throughput execution."
OpenAI (GPT, reasoning)
The text generation guide separates kinds of models. "Reasoning models" are "generally slower and more expensive to use than GPT models". And "Large and small (mini or nano) models offer trade-offs for speed, cost, and intelligence."
Cohere
The overview page says its Command family are "the text-generation LLMs powering tool-using agents, retrieval augmented generation (RAG), translation, copywriting, and similar use cases". It also lists separate Embed and Rerank models.
Mistral
The overview says: "Mistral develops, or makes available, open-weight and commercial large language models." Its list includes a "Hybrid model unifying instruct, reasoning, and coding in a single efficient model" and embedding models that "turn text or code into vectors for retrieval and similarity search."
Tip
These provider facts were checked on 16 November 2026. Model names, features and limits change often, so open each provider's docs page before you choose. Treat this lesson as a way to think, not a price list or a leaderboard.
Test on your own examples
Public leaderboards tell you how models did on someone else's questions. Your users ask different things. A small set of your own examples beats a leaderboard because it measures your job.
Anthropic's guide on choosing a model gives the same advice in its own words. It suggests starting with a fast, cheap model, then to "Test your use case thoroughly" and "Evaluate if performance meets your requirements", upgrading "only if necessary for specific capability gaps".
- 1
Collect real examples
Take 20 to 50 real inputs from your product, emails, chat logs or support tickets. Remove private details first.
- 2
Write the right answer
For each input, write the answer you want. For a label job this is one word. For writing jobs, write a short checklist of what a good answer must include.
- 3
Run every candidate
Send the same inputs to each shortlisted model. Save every answer.
- 4
Score and compare
Count right answers. Also note cost per call and seconds per call.
- 5
Look at the misses
Read the wrong answers by eye. A model that fails on rare, harmless cases may be fine. One that fails on the important cases is not.
A stand-in model for the test
Real models cost money and need keys, so the code in this lesson uses a stand-in. A stand-in is a tiny fake that returns fixed answers. It lets you run the whole test for free on your own computer.
This is a stand-in so the code runs free. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. A token is a small piece of text a model counts; the Tokenization lesson in Machine Learning Foundations goes deeper.
Our eval set has six support messages. Each one has a right label: refund, shipping, billing or other. Four made-up models, Model A to Model D, have saved answers. The function fake_model just looks the saved answer up.
# Stand-in for a real model call: it just looks up a saved answer.
# A real model is slower, costs money for every token, can answer
# differently each run, and can fail or refuse.
expected = {
"q1": "refund",
"q2": "shipping",
"q3": "billing",
"q4": "refund",
"q5": "other",
"q6": "shipping",
}
# Made-up saved answers for four made-up models.
answers = {
"Model A": {"q1": "refund", "q2": "shipping", "q3": "billing", "q4": "refund", "q5": "other", "q6": "shipping"},
"Model B": {"q1": "refund", "q2": "shipping", "q3": "billing", "q4": "refund", "q5": "other", "q6": "billing"},
"Model C": {"q1": "refund", "q2": "shipping", "q3": "billing", "q4": "other", "q5": "refund", "q6": "shipping"},
"Model D": {"q1": "refund", "q2": "other", "q3": "billing", "q4": "refund", "q5": "other", "q6": "shipping"},
}
def fake_model(name, question_id):
return answers[name][question_id]
for name in answers:
right = sum(fake_model(name, q) == expected[q] for q in expected)
print(name, "got", right, "of", len(expected))Output
Model A got 6 of 6 Model B got 5 of 6 Model C got 4 of 6 Model D got 5 of 6
Read the output. Model A got all 6 right. Models B and D got 5 of 6. Model C got 4 of 6. These answers are invented, so none of this says anything about any real model.
Score cost, speed and quality together
Accuracy alone is not enough. A model that is slightly more accurate but costs four times more may be the wrong buy. So we add cost and speed to the table.
Everything here is made up, including the costs. Model A costs 0.0120 dollars per call, B costs 0.0030, C costs 0.0004 and D costs 0.0060. Latency is 2.4, 1.1, 0.4 and 0.9 seconds. Do not read these as real prices.
The program turns them into three numbers. Cost per 1,000 calls is the cost per call times 1,000. The weighted score gives each factor a share: 60 percent for accuracy, 25 percent for being cheap and 15 percent for being fast. Cheap and fast are each scaled from 0 to 1 against the worst candidate.
In words: start with 60 percent of the accuracy. Add 25 percent of how much cheaper the model is than the most expensive one. Add 15 percent of how much faster it is than the slowest one.
# All numbers below are made up. Cost is dollars per call, latency is seconds per call.
expected = {"q1": "refund", "q2": "shipping", "q3": "billing", "q4": "refund", "q5": "other", "q6": "shipping"}
answers = {
"Model A": {"q1": "refund", "q2": "shipping", "q3": "billing", "q4": "refund", "q5": "other", "q6": "shipping"},
"Model B": {"q1": "refund", "q2": "shipping", "q3": "billing", "q4": "refund", "q5": "other", "q6": "billing"},
"Model C": {"q1": "refund", "q2": "shipping", "q3": "billing", "q4": "other", "q5": "refund", "q6": "shipping"},
"Model D": {"q1": "refund", "q2": "other", "q3": "billing", "q4": "refund", "q5": "other", "q6": "shipping"},
}
cost_per_call = {"Model A": 0.0120, "Model B": 0.0030, "Model C": 0.0004, "Model D": 0.0060}
latency = {"Model A": 2.4, "Model B": 1.1, "Model C": 0.4, "Model D": 0.9}
rows = []
for name in answers:
right = sum(answers[name][q] == expected[q] for q in expected)
rows.append({
"name": name,
"accuracy": right / len(expected),
"cost_1000": cost_per_call[name] * 1000,
"latency": latency[name],
})
# Weighted score: 60% accuracy, 25% cheapness, 15% speed (each 0 to 1).
max_cost = max(r["cost_1000"] for r in rows)
max_lat = max(r["latency"] for r in rows)
for r in rows:
r["score"] = (0.60 * r["accuracy"]
+ 0.25 * (1 - r["cost_1000"] / max_cost)
+ 0.15 * (1 - r["latency"] / max_lat))
print("name accuracy $/1000 calls seconds score")
for r in rows:
print(f"{r['name']} {r['accuracy']*100:6.1f}% {r['cost_1000']:12.2f} {r['latency']:7.1f} {r['score']:.3f}")
passing = [r for r in rows if r["accuracy"] >= 0.80]
print("Cheapest that passes 80%:", min(passing, key=lambda r: r["cost_1000"])["name"])
print("Most accurate:", max(rows, key=lambda r: r["accuracy"])["name"])
print("Best weighted score:", max(rows, key=lambda r: r["score"])["name"])Output
name accuracy $/1000 calls seconds score Model A 100.0% 12.00 2.4 0.600 Model B 83.3% 3.00 1.1 0.769 Model C 66.7% 0.40 0.4 0.767 Model D 83.3% 6.00 0.9 0.719 Cheapest that passes 80%: Model B Most accurate: Model A Best weighted score: Model B
Now compare the two priorities. If you need the cheapest model that passes 80 percent, Models B and D both pass with 83.3 percent, and B costs 3.00 dollars per 1,000 calls against 6.00. The pick is Model B. If you want the most accurate model, the pick is Model A at 100 percent, but it costs 12.00 dollars per 1,000 calls, four times more than B.
Quick check
With a rule of "cheapest that scores at least 80 percent", which model do the made-up numbers pick?
Small for easy, large for hard
You do not have to pick one model for everything. Most products have easy steps and hard steps. Sorting a ticket into a label is easy. Explaining a tangled billing problem is hard.
Routing sends each request to a model that fits it. Anthropic's guide on choosing a model describes the same idea: "Multimodel strategies pair a lower-cost model with a frontier model so that most tokens are billed at the lower rate."
Here is a tiny router. A made-up rule decides what is hard: a request is hard if it has more than 12 words, or if it asks "why". Both models are stand-ins that only return text, so the code runs free. Real routers often use a small model, or a past score from your eval set, to decide. A real model is slower, costs money for every token, can answer differently each run, and can fail or refuse.
# Stand-ins so the code runs free. Real models are slower, cost money
# for every token, can answer differently each run, and can fail or refuse.
def small_model(text):
return "small model handled: " + text
def large_model(text):
return "large model handled: " + text
def is_hard(text):
# A made-up rule: long requests or ones that ask "why" go to the large model.
return len(text.split()) > 12 or "why" in text.lower()
def route(text):
return large_model(text) if is_hard(text) else small_model(text)
requests = [
"Where is my parcel?",
"Why was I charged twice when my plan says one payment per month and my bank agrees?",
"Reset my password",
]
big = 0
for r in requests:
print(route(r))
big += is_hard(r)
print(big, "of", len(requests), "went to the large model")Output
small model handled: Where is my parcel? large model handled: Why was I charged twice when my plan says one payment per month and my bank agrees? small model handled: Reset my password 1 of 3 went to the large model
One of the three requests went to the large model, and the two short ones stayed with the small one.
Tip
Routing adds a decision that can be wrong. Include some hard requests in your eval set and check that the router sends them to the right place.
A quick check
Quick check
Why use your own eval set instead of only a public leaderboard?
Habits to keep
Before you choose a model for any feature, walk through these five steps.
- 1
Write the brief
Inputs, outputs, quality bar, speed, budget, privacy and languages.
- 2
Make a shortlist
Three to five candidates, with at least one small and one large.
- 3
Build an eval set
20 to 50 real examples with the right answers written down.
- 4
Score the table
Accuracy, cost per 1,000 calls and seconds per call for each candidate.
- 5
Route if it helps
Send easy steps to a small model and hard steps to a large one.
Next: open-weight model families you can download and run.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.