AI engineering / Advanced / 03.09
AI safety and ethics
The harms to plan for when you ship an AI feature, the public rules of thumb for trustworthy AI, and a small risk checklist you can run before launch.
13 min readFree
Building things that don't cause harm
The last lesson, Security and Privacy Concerns, ended with a promise: next, AI safety and ethics, building things that don't cause harm. Here it is.
Security asks, "Can someone break in?" This lesson asks a wider question: "If everything works as planned, can the feature still hurt someone?" It can. A recipe helper that gets a cooking time wrong wastes a dinner. A loan bot that gets a number wrong can cost someone their home.
Think of a bridge. Engineers do not only ask whether it stands today. They ask who will cross it, how heavy the load can get, and what happens in a storm. Shipping an AI feature deserves the same habit. The good news: you do not need a law degree or a big team. You need a short list of questions, asked before launch.
By the end you will know five harms to plan for, what a public framework says trustworthy AI looks like, how to be open with users, when a person must stay in the loop, and how to run a small risk review. You will also run a checklist in Python.
Key words
- Harm
- Something bad that happens to a person because of your product: lost money, lost privacy, being treated unfairly, or being misled.
- Risk
- How likely a harm is, and how bad it would be if it happened.
- Ethics
- Thinking about what is right and fair, not only what is possible to build.
- High-stakes
- A decision that can change someone's health, money, job, home or freedom. Mistakes here are costly and hard to undo.
- Human oversight
- A real person who can check, change or stop what the AI does.
- Transparency
- Being open about what the feature is, what it used, and what it cannot do.
Five harms to plan for
Most trouble with AI features falls into five groups. Learn them and you will spot problems early.
Wrong answers people trust
A model can sound calm and sure while being wrong. The Hallucinations lesson explains why. The harm grows when the answer is about health, money or law.
Unfair outcomes
A feature can work well for some people and badly for others. If it decides who gets an offer, a job or a loan, that gap becomes a real cost.
Privacy loss
People type private things into chat boxes. Where those words go, who can read them and how long they stay are your problem now.
Misuse is the fourth. Someone may use your feature for something you never meant: writing scam messages, cheating, or harassing others. Ask early, "How would a bad actor use this?"
Over-reliance is the fifth. When a tool is right most days, people stop checking it. Then the one wrong day does the damage. Think of a driver who trusts the map app and turns into a closed road.
Tip
Do not try to remove every risk. Write each one down, decide how bad it is, and choose what to do about it. A risk you named is a risk you can manage.
What the public rule books say
You do not have to invent your own list of good practice. Public bodies have written some. Two are worth knowing, and both are free to read.
NIST: seven traits of trustworthy AI
NIST is the U.S. National Institute of Standards and Technology. It released its AI Risk Management Framework on January 26, 2023. NIST says the framework is "intended for voluntary use", so no one is forced to follow it. It exists "to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems."
The framework lists traits that a trustworthy AI system needs. Here is its list, word for word:
Characteristics of trustworthy AI systems include: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed.In daily words:
- 1
Valid and reliable
It does the job it was built for, and does it dependably. NIST shows this one as the base the other traits stand on.
- 2
Safe
It should not endanger human life, health, property or the environment.
- 3
Secure and resilient
NIST describes it as being able to withstand unexpected adverse events and to degrade safely and gracefully. The last lesson covered the security side.
- 4
Accountable and transparent
People who use it can get information about it and its outputs. NIST says accountability presupposes transparency.
- 5
Explainable and interpretable
Its inner workings (explainable) and the meaning of its output (interpretable) can be understood, which helps the people who run it and use it.
- 6
Privacy-enhanced
It protects personal information.
- 7
Fair, with harmful bias managed
It treats people fairly. NIST adds that a system with harmful bias reduced is "not necessarily fair", so keep checking.
NIST also says that creating trustworthy AI "requires balancing each of these characteristics based on the AI system's context of use." A recipe helper and a loan bot need different amounts of each trait.
OECD: five values-based principles
The OECD is a group of governments that works on shared policy. Its AI Principles were adopted in 2019 and updated in May 2024. Its page lists five values-based principles:
Inclusive growth, sustainable development and well-being
Human rights and democratic values, including fairness and privacy
Transparency and explainability
Robustness, security and safety
AccountabilityThe two lists overlap. Fairness, privacy, safety, transparency and accountability show up in both. A review that asks about those five things touches the heart of both.
Be open with your users
Transparency is the cheapest safety tool you have. It costs a few lines of text and prevents a lot of harm. Three habits cover most of it.
- 1
Say it is AI
Never let someone think they are chatting with a human. A label like "AI assistant" is enough.
- 2
Show where the answer came from
If the feature used a document, link it. The RAG lesson showed how a model can look things up. Showing the source lets the reader check.
- 3
Say what it cannot do
A line such as "AI can be wrong. Check before you rely on it." fights over-reliance. Put it where people will see it, not in the footer.
Here is a tiny version of all three. It uses a stand-in model, a plain function that always returns the same fixed sentence, so the code runs free with no keys and no network. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The stand-in only lets you see the wrapper around the answer.
def fake_model(messages):
# Stand-in: always returns the same fixed text.
return "Most rice needs about twice its volume in water."
SOURCES = ["our-recipe-book.txt, page 12"]
def reply_to_user(messages, high_stakes):
answer = fake_model(messages)
if high_stakes:
return "A person will review this before you get an answer."
return f"[AI answer] {answer}\nSource: {SOURCES[0]}\nAI can be wrong. Check before you rely on it."
print(reply_to_user([{"role": "user", "content": "How much water for rice?"}], False))
print("---")
print(reply_to_user([{"role": "user", "content": "Can I afford this loan?"}], True))Output
[AI answer] Most rice needs about twice its volume in water. Source: our-recipe-book.txt, page 12 AI can be wrong. Check before you rely on it. --- A person will review this before you get an answer.
Look at the second call. For the high-stakes question the code does not answer at all. It sends the question to a person. That is human oversight, and it is the next idea.
Tip
Disclosure is also becoming a rule. The European Commission's AI Act page says that when using AI systems such as chatbots, "humans should be made aware that they are interacting with a machine so they can take an informed decision."
Keep a person in the loop
A person in the loop means a human looks at the AI's output before it affects someone. You do not need this for a recipe. You do need it when the answer can change a life.
A good test is to ask, "If this is wrong, can the user easily notice and undo it?" A wrong recipe step is noticed at once. A wrong loan estimate may not be noticed for months.
NIST's framework treats this as part of the work. One line in its Map function reads: "Processes for human oversight are defined, assessed, and documented in accordance with organizational policies from the govern function." In plain words: decide ahead of time who checks what, and write it down.
AI drafts, a person decides
The model suggests. A trained person approves or changes it.
AI answers, a person samples
For lower stakes, a person reads a share of the answers each week to catch drift.
AI stays out
For some decisions, the model should not be involved at all. Naming those is a valid result of a risk review.
A person in the loop only helps if the person has time and the power to say no. If they must approve a hundred cases a minute, they will click yes. That is over-reliance again, just with a human face.
A risk review you can run before launch
Here is a lightweight review. It is a list of questions, each tied to an area, with a weight for how serious the risk is. You answer yes or no for your feature, add up the weights, and read the result. It will not give you a perfect answer. It will make sure you asked.
- 1
Write the questions as data
Each question gets an area (accuracy, fairness, privacy, misuse, transparency or oversight) and a weight from 1 to 3.
- 2
Answer for each feature
A yes means the risk applies. Be honest. Ask someone who did not build it.
- 3
Add up and read the top risks
A high score means more to fix. The top three show where to start.
- 4
Flag high-stakes features
If the feature touches money, health, work or housing, a person must review it, whatever the score.
Below, two made-up features go through the same list: a recipe helper and a loan advice bot. The weights and the cut-off of 15 are invented for teaching. Tune them for your own product. There is no model call here, so no stand-in is needed.
# Each question is a risk. Answer True means "yes, this risk applies".
# Weight: 1 = small worry, 3 = serious worry.
QUESTIONS = [
("accuracy", "A wrong answer could hurt someone's health, money or safety", 3),
("accuracy", "Users are likely to trust the answer without checking it", 2),
("fairness", "The answer affects what a person is offered or denied", 3),
("fairness", "Different groups of people may get different quality", 2),
("privacy", "Users type personal details into the chat", 2),
("privacy", "Chats are stored or reused to improve the feature", 1),
("misuse", "Someone could use it to cheat, scam or harass", 2),
("transparency", "Users might not realise they are talking to AI", 1),
("oversight", "No person looks at a decision before it reaches the user", 3),
]
# For each feature: which questions are True, and is a big life decision involved?
FEATURES = {
"recipe helper": {
"high_stakes": False,
"yes": {1, 4, 5, 7},
},
"loan advice bot": {
"high_stakes": True,
"yes": {0, 1, 2, 3, 4, 5, 7, 8},
},
}
def review(name, feature):
hits = [QUESTIONS[i] for i in sorted(feature["yes"])]
score = sum(w for _, _, w in hits)
top = sorted(hits, key=lambda h: -h[2])[:3]
by_area = {}
for area, _, w in hits:
by_area[area] = by_area.get(area, 0) + w
print(f"{name}: score {score} of {sum(w for _, _, w in QUESTIONS)}")
print(" by area:", by_area)
print(" top risks:")
for area, text, w in top:
print(f" [{area}, weight {w}] {text}")
print(" needs human review:", "YES" if feature["high_stakes"] or score >= 15 else "no")
for name, feature in FEATURES.items():
review(name, feature)Output
recipe helper: score 6 of 19
by area: {'accuracy': 2, 'privacy': 3, 'transparency': 1}
top risks:
[accuracy, weight 2] Users are likely to trust the answer without checking it
[privacy, weight 2] Users type personal details into the chat
[privacy, weight 1] Chats are stored or reused to improve the feature
needs human review: no
loan advice bot: score 17 of 19
by area: {'accuracy': 5, 'fairness': 5, 'privacy': 3, 'transparency': 1, 'oversight': 3}
top risks:
[accuracy, weight 3] A wrong answer could hurt someone's health, money or safety
[fairness, weight 3] The answer affects what a person is offered or denied
[oversight, weight 3] No person looks at a decision before it reaches the user
needs human review: YESRead the result. The recipe helper scores 6 out of 19. Its biggest worries are that people trust the answer, and that they type personal details. Those are fixable with a label and a privacy note, and no one needs to review each reply.
The loan bot scores 17 out of 19. Its top risks are a wrong answer about money, an answer that decides what a person is offered, and no human check. It is also marked high-stakes, so it needs human review. Both reasons point the same way here, but the flag would still be on with a lower score.
Notice what the checklist did not do. It did not say "do not build the loan bot". It said where the danger is and who must look. That is the job of a risk review: make the choices visible, so people can make them on purpose.
A word on the law
Rules are arriving. The European Commission describes its AI Act as setting out "risk-based rules for AI developers and deployers regarding specific uses of AI", with four levels of risk: unacceptable, high, transparency (limited) and minimal. Its list of high-risk uses includes credit scoring, with the example of "credit scoring denying citizens opportunity to obtain a loan". That is the loan bot's world.
This is a short, neutral note, not legal advice. Laws differ by country and change over time. If your feature touches people in a regulated area, read the source page and ask a qualified person.
Quick check
Quick check
A support bot sometimes gives a wrong refund policy. Which step reduces the harm the most?
Quick check
Which feature most needs a person to review answers before users see them?
What to remember
Name the harms
Wrong answers, unfairness, privacy loss, misuse and over-reliance. Write each down for your feature.
Be open
Say it is AI, show sources, say what it cannot do.
Match care to stakes
Put a person in the loop where a mistake is costly, and give them time to say no.
Next: bias and fairness, checking who your AI works for.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.