AI engineering / Advanced / 03.14
Know your users and use cases
Before you write code, write down who will use your AI feature, what for, and what could go wrong, then check it in tests.
12 min readFree
The work before the code
The last lesson, Robust prompt engineering, ended with a promise: next, know your users and use cases, the safety work that starts before code. Here it is.
Think of a baker who opens a shop. Before baking, the baker asks: who walks in? Children, office workers, people with allergies? What will they ask for? What must never leave the shop? An AI feature needs the same questions, and the answers shape everything you build after.
By the end you will know how to describe the people who use your feature, list what it is for and what it is not for, judge how bad a wrong answer could be, and put it all on one short page. You will also run a small program that turns that page into rules.
Key words
- Use case
- One job a person wants done with your feature, such as "explain a school topic".
- Intended use
- A use case you built the feature for and have tested.
- Out-of-scope use
- A use case the feature is not built for. You decide ahead of time to say no to it.
- Risk level
- A label, such as low, medium or high, for how much harm a wrong answer could do in one use case.
- Rate limit
- A cap on how many requests one person can send in a set time.
- Abuse monitoring
- Watching how the feature is used so you can spot people misusing it.
- Usage policy
- The rules a model provider sets for what you may do with its model.
- Use-case card
- A short page that records users, intended uses, out-of-scope uses and risks for your feature.
Intended and out-of-scope uses
Start with the jobs. Write down what your feature is for, in plain words. A homework helper might be for explaining a school topic, checking a maths step and planning a study week. That is the intended list.
Then write the other list: what it is not for. This is the out-of-scope list. A homework helper is not a doctor, a lawyer or a money adviser. If someone asks it about medicine, the right move is to say so and point them elsewhere.
Why bother, when nobody asked for those uses? Because people will try them anyway. The NIST AI Risk Management Framework Playbook, in its MAP section, says organizations "should expect broadly deployed AI tools to be reused, repurposed, and potentially misused regardless of intentions." A shop sells bread, and still someone asks for a bus ticket. You decide the answer before they ask.
Intended uses
Jobs you built, tested and are happy to answer.
Out-of-scope uses
Jobs you will not do, written down so the feature and your team agree.
Grey areas
Jobs that sit between the two. Decide each one early, or send it to a person.
Tip
Write the out-of-scope list from the user's side. Ask: what would a bored, curious or desperate person type here? That list is usually longer than you expect.
Who the users are
The same answer can be fine for one person and risky for another. So describe your users. The NIST Playbook lists "specific set or types of users along with their expectations" among the things to understand and document. Three questions cover most of it.
- 1
How old are they?
A feature for adults is not automatically safe for teenagers or children. If young people may use it, say so on the card and plan for it.
- 2
How much do they know?
An expert reads a hedged answer and checks it. A beginner may take it as fact. The less people know, the more they trust the answer, so be stricter.
- 3
How vulnerable are they?
Someone who is ill, frightened, grieving or short of money is more likely to act on a wrong answer, and more harmed if they do.
Notice that these are guesses until you check. Talk to a few real users, or read how people describe the problem in their own words. Then write what you learned. OpenAI's Safety best practices page puts it simply: "Consider your customer base and the range of inputs that they will be using."
What could a wrong answer do?
A model can be wrong, and it can sound sure when it is. So for each use case, ask one question: if the answer is wrong and the user believes it, what happens?
A wrong fact in a trivia game costs nothing. A wrong step in a maths check costs a lower mark. A wrong medicine dose can hurt someone. These are very different, so give each use case its own risk level.
Low
A mistake is cheap and easy to spot, such as a wrong trivia fact.
Medium
A mistake costs money, time or trust, such as bad budget advice.
High
A mistake can harm health, safety, rights or livelihood, such as medical or legal advice.
High risk changes how you build. OpenAI's page says: "Wherever possible, we recommend having a human review outputs before they are used in practice." It adds that this is especially critical in high-stakes domains. Anthropic's Usage Policy has a stricter version for its own high-risk cases: a qualified person must meaningfully review the output before it is delivered as advice to someone who might rely on it. Read that policy for the exact wording; it can change.
Tip
Do not average the risk. A feature with nine safe uses and one dangerous one is judged by the dangerous one.
Access controls
Once you know who should use the feature, decide how to keep everyone else, and bad behaviour, out. Three tools do most of the work.
Sign-in
Knowing who is asking. OpenAI's page says "Users should generally need to register and log-in to access your service."
Rate limits
Capping how often one person can ask. It slows down people who try the same trick a hundred times.
Abuse monitoring
Watching for patterns, and a way for users to report problems that a human reads.
Rate limits are like a bakery that hands each visitor three free samples per hour. A normal customer never notices. Someone trying to empty the tray hits the limit fast.
Monitoring needs a way to tell users apart without storing their private details. OpenAI's page describes safety identifiers: a string that uniquely identifies each user, made by hashing the username or email address, so the provider receives no identifying information. It says they help your team respond to abuse faster.
The same page also says to make reporting easy: users should have an easily available way to report improper behaviour, and that route "should be monitored by a human and responded to as appropriate." A report nobody reads is worth nothing.
Read the usage policy
Your model provider has its own rules, and you agree to them when you use the model. They cover what the model may be used for, and they apply to you.
Anthropic's Usage Policy says it applies to anyone who submits inputs, and names "the end users of products or services integrating Claude". So your customers are covered too, and the rules reach them through your product. The same page says Anthropic's Safeguards Team implements detection and monitoring, and that if it suspects a violation it may warn you or "throttle, limit, suspend, or terminate" your access.
The policy is split into universal standards, high-risk requirements and extra guidelines for areas such as consumer chatbots and products serving minors. Other providers publish their own, with different details.
Tip
Read your provider's usage policy once, early, and again before launch. Copy the parts that touch your feature into your card, in your own words, and note the date you read it.
Write it down: the use-case card
Everything above is only useful if it is written in one place that everyone can read. The idea has a model in research. In 2018, a paper proposed model cards: short documents that travel with a trained model. Its abstract says the aim is to "clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited". Our use-case card does the same job for a feature, not a model.
Better still, make the card data your code can read. Then the card is not a forgotten document; it drives the app. The program below holds a card for the homework helper as a plain Python dictionary: users, intended uses with risk levels, out-of-scope uses with keywords and risk levels, and a rate limit.
A router reads the card for each request. It first checks the rate limit, then looks for out-of-scope keywords, and only then calls the model. A clock we control stands in for real time so the run is the same every time.
Important: fake_model is a stand-in so the code runs free. It returns fixed text and never calls anything. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. Eight made-up requests go through the router.
card = {
"feature": "Homework helper",
"users": "students aged 13 to 18, signed in",
"intended": {
"explain a school topic": "low",
"check a maths step": "low",
"plan a study week": "low",
},
"out_of_scope": {
"medical advice": (["dose", "symptom", "medicine"], "high"),
"legal advice": (["lawsuit", "sue"], "high"),
"money advice": (["invest", "loan"], "medium"),
},
"limit": 3,
"window": 60,
}
now = 0
def fake_clock():
return now
history = {}
def fake_model(messages):
return "(stand-in answer)"
def route(user, text):
t = fake_clock()
recent = [s for s in history.get(user, []) if t - s < card["window"]]
if len(recent) >= card["limit"]:
return "BLOCK too many requests"
history[user] = recent + [t]
for use, (words, risk) in card["out_of_scope"].items():
if any(w in text.lower() for w in words):
return f"REFUSE {use} ({risk} risk)"
fake_model([{"role": "user", "content": text}])
return "ALLOW sent to the model"
requests = [
(0, "ana", "Explain how plants make food"),
(5, "ana", "What dose of medicine is safe?"),
(10, "ben", "Check my step: 2x + 3 = 11"),
(12, "ana", "Plan my study week"),
(15, "ana", "Should I invest my savings?"),
(20, "ben", "Can I sue my school?"),
(70, "ana", "Explain how plants make food"),
(75, "ben", "Explain the water cycle"),
]
for when, user, text in requests:
now = when
print(f"t={when:>2} {user:<3} {text:<34} {route(user, text)}")Output
t= 0 ana Explain how plants make food ALLOW sent to the model t= 5 ana What dose of medicine is safe? REFUSE medical advice (high risk) t=10 ben Check my step: 2x + 3 = 11 ALLOW sent to the model t=12 ana Plan my study week ALLOW sent to the model t=15 ana Should I invest my savings? BLOCK too many requests t=20 ben Can I sue my school? REFUSE legal advice (high risk) t=70 ana Explain how plants make food ALLOW sent to the model t=75 ben Explain the water cycle ALLOW sent to the model
Read the output from the top. Ana's first request passes. Her second asks about a medicine dose, so it matches the medical words and is refused as high risk. Ben's maths check passes, and so does Ana's study plan.
At t=15 Ana asks a fourth thing. She already used three requests in the last 60 seconds (at 0, 5 and 12, and the refused one counts), so the rate limit blocks her before any keyword check. The block does not count as a new request. Ben's question about suing his school is refused as legal advice. At t=70 enough time has passed, so Ana is allowed again, and Ben's last question is a normal one.
Keyword matching is crude, and you should know its limits. The word "sue" would also match "issue", so a harmless question about a school issue gets refused. Real systems often use a classifier, a small model trained to sort text into labels. The Tokenization lesson in Machine Learning Foundations explains how text becomes numbers for such models. Keywords are fine for a first version, as long as you know they are a first version.
Quick check
A user asks your homework helper for the right dose of a medicine. What does your use-case card say to do?
Feed the card into tests and monitoring
A card on a shelf does nothing. Use it twice: before launch as tests, and after launch as monitoring.
- 1
Turn each line into a test
For every out-of-scope use, write a few sample requests and check the feature refuses them. For every intended use, check it still answers.
- 2
Test the edges
Include odd spellings, other languages and questions that sit between two uses. OpenAI's page recommends testing a representative set of inputs and ones from someone trying to break your app.
- 3
Watch after launch
Count how often each decision happens: allowed, refused, blocked. A jump in refusals or blocks is a signal to look.
- 4
Read the reports
Users will find cases you never imagined. Add each one to the card and to the tests.
- 5
Review on a date
Put a review date on the card. New users, new model or new policy means the card is out of date.
This is a loop, not a single task. The card drives the tests, the monitoring finds surprises, and the surprises update the card.
Quick check
Your homework helper is suddenly used by many adults asking money questions. What do you do?
That is the lesson. Know who uses your feature, write down what it is and is not for, judge the harm of a wrong answer, control access, read the usage policy, and keep one short card that your tests and monitors follow.
Next: tracing and logging, seeing every step your app took.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.