AI engineering / Advanced / 03.11
Content Moderation APIs
Check what users send in and what the model sends back, score it, pick thresholds, and decide what happens when something is flagged.
13 min readFree
A guard at both doors
The last lesson, Bias and Fairness, ended with a promise: next, content moderation, checking what goes in and comes out. Here it is.
Think of a shop with two doors. One door lets customers in. The other lets deliveries out. A careful owner watches both. In an AI app, the customer door is what your users type. The delivery door is what your model says back. Moderation is the guard at both.
By the end you will know what moderation checks, the two ways to get a checker, how scores and thresholds work, what to do when something is flagged, and why you keep logs. You will also run a small pipeline of your own, free, with no account.
Key words
- Moderation
- Checking text (or images) for harmful content, then deciding what to do about it.
- Classifier
- A program that puts an input into a group. A moderation classifier says which harm categories, if any, an input falls into.
- Category
- One kind of harm the checker looks for, such as harassment or violence.
- Score
- A number for how sure the checker is that an input belongs to a category.
- Threshold
- The score line you pick. At or above it, you act. Below it, you let the input through.
- False positive
- A harmless input that gets flagged by mistake.
- False negative
- A harmful input that slips through unflagged.
- Escalate
- Hand the case to a person instead of letting the program decide alone.
What moderation is
Moderation is older than AI. Wikipedia defines content moderation, for sites with user-generated content, as "the systematic process of identifying, reducing, or removing user contributions that are irrelevant, obscene, illegal, harmful, or insulting." It adds that the process can mean removal, or putting a warning label on flagged material.
It also says how big sites do it: "Major platforms use a combination of algorithmic tools, user reporting and human review." Keep that mix in mind. A program does the first fast pass. People handle the hard cases.
AI apps add a twist. You moderate two things, not one: the user's message and the model's reply. A user can type something harmful. A model can also write something harmful on its own, even when the user asked nicely.
Two doors, one pipeline
- 1
Check the input
Before the message reaches your model, score it. If it is clearly over the line, stop here. You also save the cost of a model call.
- 2
Call the model
Only when the input passed. Your model may still write something you do not want.
- 3
Check the output
Score the reply before the user sees it. The OpenAI guide says to review the moderation results before you show the output to a user or take downstream actions.
- 4
Act and log
Allow, warn, block or escalate. Then write down what happened, so you can learn from it and answer questions later.
Tip
Checking only the input is a common gap. A friendly question can still get a bad answer. Check both doors.
Where the checker comes from
You do not have to build the classifier yourself. There are two common routes.
A provider's moderation endpoint
You send text to a web address and get scores back. The OpenAI guide says the omni-moderation-latest model accepts text and image inputs. It does not classify audio.
An open safety model you run yourself
Llama Guard is one. Its paper abstract calls it an "LLM-based input-output safeguard model geared towards Human-AI conversation use cases". It checks prompts and responses, and the authors say they are making its weights available.
Your own rules on top
Neither route knows your product. A kids' homework helper and an adult fiction tool need different lines. The guide says to use the results to enforce your application's policy. The policy is yours to write.
The Llama Guard abstract adds that its categories can be adjusted: it allows "the adjustment of taxonomy categories to align with specific use cases". A taxonomy is just a list of categories with names. That means an open model can be bent toward your policy, while a hosted endpoint gives you its fixed list.
Running a safety model yourself brings the trade-offs from the Running Models Locally lesson: you need hardware and upkeep. A hosted endpoint is easier to start but sends your text to someone else.
What an API call looks like
Here is the request from the OpenAI Moderation guide, copied as it appears there. It needs an account and a key, so it is shown here but not run.
# From the OpenAI Moderation guide, shown without its output (not run here)
curl https://api.openai.com/v1/moderations \
-X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "omni-moderation-latest",
"input": [
{ "type": "text", "text": "...text to classify goes here..." },
{ "type": "image_url", "image_url": { "url": "https://example.com/image.png" } }
]
}'And the guide's example response, for a frame from a war movie. We left out one field (category_applied_input_types) to keep it short.
# From the OpenAI Moderation guide, its example response for a war-movie frame, without the category_applied_input_types field (not run here)
{
"id": "modr-970d409ef3bef3b70c73d8232df86e7d",
"model": "omni-moderation-latest",
"results": [
{
"flagged": true,
"categories": {
"sexual": false,
"sexual/minors": false,
"harassment": false,
"harassment/threatening": false,
"hate": false,
"hate/threatening": false,
"illicit": false,
"illicit/violent": false,
"self-harm": false,
"self-harm/intent": false,
"self-harm/instructions": false,
"violence": true,
"violence/graphic": false
},
"category_scores": {
"sexual": 2.34135824776394e-7,
"sexual/minors": 1.6346470245419304e-7,
"harassment": 0.0011643905680426018,
"harassment/threatening": 0.0022121340080906377,
"hate": 3.1999824407395835e-7,
"hate/threatening": 2.4923252458203563e-7,
"illicit": 0.0005227032493135171,
"illicit/violent": 3.682979260160596e-7,
"self-harm": 0.0011175734280627694,
"self-harm/intent": 0.0006264858507989037,
"self-harm/instructions": 7.368592981140821e-8,
"violence": 0.8599265510337075,
"violence/graphic": 0.37701736389561064
}
}
]
}Read it from the top. flagged says whether the checker thinks the input is potentially harmful. categories gives a true or false for each kind of harm. category_scores gives a number for each one. The guide says each score "is between 0 and 1, where higher values denote higher confidence." Here violence scores about 0.86, so violence is flagged. Every other score is far lower.
The guide names these categories: sexual, sexual/minors, harassment, harassment/threatening, hate, hate/threatening, illicit, illicit/violent, self-harm, self-harm/intent, self-harm/instructions, violence and violence/graphic. Some are text only. Others cover images too.
Tip
The guide warns that the model underneath is upgraded over time, so custom policies that rely on category_scores "may need recalibration over time". If you hard-code a threshold, test it again after provider updates.
A small pipeline you can run
Now build the guard yourself, with no account. First, the classifier. This one is a stand-in so the code runs free. It is crude: it just adds up points for scary words in three categories. A real classifier reads meaning. This one only matches words, and you will see how that goes wrong.
# Crude stand-in classifier: counts scary words per category. It reads words, not meaning.
KEYWORDS = {
"harassment": {"stupid": 0.5, "idiot": 0.5, "loser": 0.4},
"violence": {"kill": 0.4, "hurt": 0.3, "fight": 0.3, "violent": 0.3},
"self-harm": {"hopeless": 0.4, "myself": 0.2, "disappear": 0.5},
}
def score(text):
words = [w.strip(".,?!") for w in text.lower().split()]
return {c: round(min(sum((t.get(w, 0.0) for w in words), 0.0), 1.0), 2)
for c, t in KEYWORDS.items()}
for text in ["How do I bake bread?", "You stupid idiot", "How do I kill a stuck process?"]:
print(text, "->", score(text))Output
How do I bake bread? -> {'harassment': 0.0, 'violence': 0.0, 'self-harm': 0.0}
You stupid idiot -> {'harassment': 1.0, 'violence': 0.0, 'self-harm': 0.0}
How do I kill a stuck process? -> {'harassment': 0.0, 'violence': 0.4, 'self-harm': 0.0}The stand-in gives 0.0 for the bread question, 1.0 for the insult, and 0.4 for a harmless computing question that happens to use the word "kill". That last one is a mistake, and it is on purpose.
Now the full pipeline. It has per-category thresholds: a warn line and a block line. Self-harm is different. The aim there is to get a person involved, so a high score means escalate, not a plain block. The stand-in model, fake_model, returns fixed text. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. This stand-in does none of that.
# Crude stand-in classifier: counts scary words per category. It reads words, not meaning.
KEYWORDS = {
"harassment": {"stupid": 0.5, "idiot": 0.5, "loser": 0.4},
"violence": {"kill": 0.4, "hurt": 0.3, "fight": 0.3, "violent": 0.3},
"self-harm": {"hopeless": 0.4, "myself": 0.2, "disappear": 0.5},
}
def score(text):
words = [w.strip(".,?!") for w in text.lower().split()]
return {c: round(min(sum((t.get(w, 0.0) for w in words), 0.0), 1.0), 2)
for c, t in KEYWORDS.items()}
def fake_model(question):
# Stand-in for a real model: fixed replies, so the code runs free.
if "neighbor" in question:
return "Hi neighbor, your music is stupid and so are you, loser."
return "Here is a short, calm answer."
# category -> (warn at, block at). Self-harm goes to a human instead of a plain block.
def make_rules(violence_block):
return {"harassment": (0.3, 0.6), "violence": (0.3, violence_block), "self-harm": (0.2, 0.5)}
SEVERITY = ["allow", "warn", "block", "escalate"]
def decide(text, rules):
worst = "allow"
for category, value in score(text).items():
warn_at, block_at = rules[category]
if value >= block_at:
action = "escalate" if category == "self-harm" else "block"
elif value >= warn_at:
action = "warn"
else:
action = "allow"
if SEVERITY.index(action) > SEVERITY.index(worst):
worst = action
return worst
# (id, user text, should the user text be stopped?)
INPUTS = [
("A", "How do I bake bread at home?", False),
("B", "You stupid idiot, answer me", True),
("C", "How do I kill a stuck process on my laptop?", False),
("D", "I feel hopeless and want to disappear", True),
("E", "Write a note to my neighbor about the loud music", False),
("F", "Summarize the fight scene in this violent movie review", False),
]
def run(rules):
false_positives = 0
for key, text, harmful in INPUTS:
in_action = decide(text, rules)
if in_action in ("block", "escalate"):
out_action = "-"
else:
out_action = decide(fake_model(text), rules)
print(f"{key} input={in_action:<8} output={out_action}")
if in_action in ("block", "escalate") and not harmful:
false_positives += 1
print("good requests wrongly stopped:", false_positives)
print("violence block at 0.4")
run(make_rules(0.4))
print()
print("violence block at 0.7")
run(make_rules(0.7))Output
violence block at 0.4 A input=allow output=allow B input=block output=- C input=block output=- D input=escalate output=- E input=allow output=block F input=block output=- good requests wrongly stopped: 2 violence block at 0.7 A input=allow output=allow B input=block output=- C input=warn output=allow D input=escalate output=- E input=allow output=block F input=warn output=allow good requests wrongly stopped: 0
Read the first run. Request B is a real insult and is blocked. D is a person in distress and is escalated. E is interesting: the request is harmless, so the input passes. But the stand-in model writes a rude note, and the output check blocks it. That is why you check both doors.
Now look at C and F. C asks how to stop a stuck program. F asks for a summary of a movie review. Both are fine, and both were blocked, because they use words like "kill", "fight" and "violent". Those are two false positives.
Thresholds and false positives
In the second run, the only change was one number: the violence block line moved from 0.4 to 0.7. Good requests wrongly stopped went from 2 to 0. C and F now get a warning instead of a block, and they pass the output check.
It sounds like a free win, but it is a trade. Raise the line and you stop fewer good messages. You also let more real ones through, and each false negative is a harm that reaches your user. Lower the line and you catch more, but you annoy more honest people.
This is like a smoke alarm. Set it too touchy and it screams at toast. Set it too sleepy and it misses a fire. There is no setting with no mistakes.
Pick lines per category
A missed self-harm message is worse than a blocked cooking question. Give that category a lower line.
Test on your own data
Collect real messages from your app (with permission), label a few hundred, and count both kinds of mistake at each threshold.
Use warn as a middle step
A warn zone lets borderline cases through with a note or a second check, instead of an all-or-nothing block.
Context also matters. A classifier that only sees one message cannot tell a joke between friends from a threat. Many errors come from missing context, and a human reviewer can fix them.
What to do when something is flagged
A flag is a signal, not a verdict. The OpenAI guide lists what to do with it: "filtering content, routing a request for review, or intervening with accounts that submit flagged content." Here are the usual actions.
Block
Do not send the input to the model, or do not show the output. Show a short, calm message that says what happened, without lecturing.
Rewrite or retry
For a flagged model output, you can ask the model again with a stricter instruction, or return a safe fixed reply. Check the new answer too.
Escalate to a human
Send borderline or serious cases to a person. Self-harm is the clearest case, where a real person should look quickly.
When the case is about someone's wellbeing, do more than say no. Show help resources that fit where your user lives, such as a local helpline, and keep the tone kind. A blunt block can feel like a door slammed on someone who needed to talk. Each region has different services, so check which ones you can point to.
Tip
Do not let a flag crash the app. If the moderation call fails or times out, decide ahead of time: let the message through, or hold it. Pick one on purpose, per category.
Logging and appeals
Every decision should leave a record: when, which door (input or output), the category, the score, the threshold used, and the action. Store as little of the user's text as you can. Logs help you tune thresholds and show what happened when someone asks.
People will sometimes be wrongly blocked, so give them a way to say so. Wikipedia notes that under the EU's Digital Services Act, "users may appeal through the internal complaint-handling system of platforms" and that platforms "are required to promptly review their decisions". Your app may or may not fall under that law. The idea still holds: a fair appeal path is good practice.
A simple appeal flow: the user clicks "this was a mistake", the case lands in a queue with its log entry, a person reviews it, and the answer goes back. Those reviews are also your best data. Every false positive that a person confirms tells you where a threshold is too tight.
Quick check
Quick check
A user asks how to kill a stuck program, and your pipeline blocks the message. What is this called?
Quick check
Why check the model's reply, not just the user's message?
Next: adversarial testing, attacking your own app before others do.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.