esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

AI engineering / Advanced / 03.22

People as the judges

When a person has to read the answers, how to ask them fairly, hide which version wrote what, and check that reviewers agree.

13 min readFree

When a person has to judge

The last lesson, Model-Based Evals, ended with a promise: next, human evals, when people must judge the answers. Here they are.

A rule can check that an answer is valid JSON. A second model can grade it with a marking sheet. But some questions only a person can settle. Is the tone warm without being fake? Is the medical advice right? Would a nervous customer feel helped, or brushed off? Think of a restaurant. A machine can count how fast the food arrives. Only a diner can say if it tastes good.

Key words

Human eval
A test where real people read your model's answers and mark them. It is a test of quality, and it can be repeated.
Reviewer (rater)
A person who reads an answer and gives it a mark.
Rubric
A short written guide that says what each mark means, with examples. It is the marking sheet reviewers share.
Blind review
The reviewer is not told which version, model or prompt wrote an answer, so they cannot favour the one they expect to win.
Side-by-side test
Two answers to the same question are shown together, and the reviewer picks the better one.
Agreement
How often two reviewers give the same mark to the same answer.
Cohen's kappa
A number that shows how much two reviewers agree after taking out the agreement you would get by luck alone.

OpenAI's evaluation guide says: "Human judgment evals provide the highest quality but are slow and expensive." Both halves matter. People are the best judges you have, and they are the scarcest. So you save them for the cases where nothing else works.

  • Tone and helpfulness

    Does it sound kind, clear and on-brand? A rule cannot tell. A reader can in seconds.

  • Right in a special field

    Law, health, money, engineering. A reviewer who knows the field spots a wrong detail that sounds perfect.

  • Safety edge cases and your own judges

    Odd, risky requests need human eyes. Also, people check your automated judges. OpenAI's tips say: "Use human feedback to calibrate automated scoring."

The Deterministic Evals lesson and the Model-Based Evals lesson are still your daily tools, because they are cheap and fast. People are the check on those tools.

Write the rubric first

Ask three people "is this answer good?" and you may get three different meanings of good. One cares about length, one about politeness, one about facts. A rubric fixes that.

Wikipedia's page on inter-rater reliability says: "Clearly stated guidelines for rendering ratings are necessary for reliability in ambiguous or challenging measurement scenarios." It adds that "Without scoring guidelines, ratings are increasingly affected by experimenter's bias, that is, a tendency of rating values to drift towards what is expected by the rater." In plain words: without a guide, people see what they expect to see.

  1. 1

    Pick two or three things to judge

    For example: is it correct, is the tone right, is it safe. Too many questions tire reviewers and blur the marks.

  2. 2

    Use few choices

    Pass or fail is the simplest. A 1 to 5 scale is common, but more choices give reviewers more room to differ. OpenAI's guide suggests a pass/fail threshold in addition to the numerical score.

  3. 3

    Show, don't only tell

    OpenAI recommends a "show rather than tell" policy: give examples of different score levels, such as 1, 3 and 8 out of 10. One real example is worth a paragraph of rules.

  4. 4

    Try it on a few answers, then fix it

    OpenAI suggests several rounds of detailed human review to refine the scorecard. Where reviewers disagree, the rubric is usually unclear, not the people.

Tip

Write down what a reviewer should do when they are unsure. A line like "if you cannot tell, mark it unclear" is better than a guess.

Hide who wrote it

Imagine a cooking contest where the judge knows which dish is the chef's famous recipe. They may like it before the first bite. Reviewers do the same with "the new prompt" or "the bigger model". So hide the label.

OpenAI's guide describes one way to run it: "create a randomized, blinded test in which employees, contractors, or outsourced labeling agencies judge the quality of system outputs". Wikipedia describes A/B testing, a method from user-experience research, as a randomized experiment that usually involves two variants, A and B. The side-by-side test borrows that idea.

  1. 1

    Pair the answers

    For each question, collect the answer from version A and the answer from version B.

  2. 2

    Shuffle the positions

    Randomly decide which one is shown first. If readers lean toward the first slot, shuffling makes that lean hit both versions equally.

  3. 3

    Show only "first" and "second"

    No version names, no model names, no hints in the file name.

  4. 4

    Save every pick, then unblind

    Only after all the picks are stored do you look up which version each pick was.

A blind side-by-side tally

This code runs a blind test with ten questions. It hides which version is first, tallies the picks, and works out a win rate: the share of questions a version won. The reviewer's picks are typed in, because there is no real reviewer here. The shuffle uses a fixed seed, so every run gives the same order.

Python
import random

# Ten questions. For each one we have an answer from version A and one from version B.
rng = random.Random(7)  # fixed seed: the same shuffle every run

pairs = []
for q in range(1, 11):
    versions = ["A", "B"]
    rng.shuffle(versions)  # which version appears first is decided by chance
    pairs.append({"question": q, "first": versions[0], "second": versions[1]})

# What the reviewer sees is only "first" or "second". They never see A or B.
# These picks were made before anyone looked at the hidden labels.
picks = ["first", "first", "second", "first", "second",
         "first", "first", "second", "first", "first"]

wins = {"A": 0, "B": 0}
for pair, pick in zip(pairs, picks):
    wins[pair[pick]] += 1  # unblind only now, after every pick is saved

total = len(pairs)
print("A shown first in", sum(p["first"] == "A" for p in pairs), "of", total, "pairs")
print("reviewer picked 'first' in", picks.count("first"), "of", total, "pairs")
print("A wins:", wins["A"], " B wins:", wins["B"])
print("A win rate:", wins["A"] / total)

Output

A shown first in 3 of 10 pairs
reviewer picked 'first' in 7 of 10 pairs
A wins: 4  B wins: 6
A win rate: 0.4

Version B won 6 and A won 4, so A's win rate is 0.4. Look at the other line, too. The reviewer picked the first answer on screen in 7 of 10 pairs. The picks are made up for this lesson, but a lean like this is easy to picture. Because the order was shuffled, it did not help one version.

Do not celebrate a 6 to 4 result from ten questions. The sample size section below shows why.

Do your reviewers agree?

One reviewer's mark is an opinion. If two careful reviewers give the same mark, you can trust it more. If they disagree a lot, the marks are noise, or the rubric is vague. OpenAI's guide lists "Disagreement among human experts" as one of the challenges of human evals, and says one way to combine several reviewers is to take consensus votes.

The simplest check is percent agreement: out of all answers, on how many did both reviewers give the same mark? Wikipedia calls this joint probability of agreement "the simplest and the least robust measure". It does not account for chance. With only two choices, two people who flipped coins would still agree about half the time.

Cohen's kappa fixes that. Wikipedia says it is "generally thought to be a more robust measure than simple percent agreement calculation", because it takes chance agreement into account. It runs from -1 to 1. A value of 1 means complete agreement, and 0 means no more agreement than chance would give.

κ=po−pe1−pe\kappa = \frac{p_o - p_e}{1 - p_e}

In words: p_o is the share of answers where the reviewers agreed. p_e is the share you would expect if each reviewer kept their own habits, such as saying "good" most of the time, but picked at random. Kappa is how far above luck they got, divided by the most they could have got above luck.

  1. 1

    Count the matches

    Both said good, or both said bad. That gives p_o.

  2. 2

    Work out luck

    For each label, multiply how often reviewer A used it by how often reviewer B used it. Add up the labels. That gives p_e.

  3. 3

    Plug in the formula

    Take p_o minus p_e, then divide by 1 minus p_e.

Python
# Two reviewers each mark the same 12 answers as "good" or "bad".
reviewer_a = ["good", "good", "bad", "good", "bad", "good", "good", "bad", "good", "good", "bad", "good"]
reviewer_b = ["good", "bad", "bad", "good", "bad", "good", "good", "good", "good", "bad", "bad", "good"]

n = len(reviewer_a)
labels = ["good", "bad"]

# 1. Percent agreement: how often did they pick the same label?
same = sum(1 for a, b in zip(reviewer_a, reviewer_b) if a == b)
p_o = same / n
print("answers:", n)
print("same label:", same)
print("percent agreement:", round(p_o, 3))

# 2. Chance agreement: if each reviewer just guessed using their own habits.
p_e = 0
for label in labels:
    share_a = reviewer_a.count(label) / n
    share_b = reviewer_b.count(label) / n
    print(label, "-> A says", round(share_a, 3), "B says", round(share_b, 3))
    p_e += share_a * share_b
print("chance agreement:", round(p_e, 3))

# 3. Cohen's kappa: how far above chance did they get?
kappa = (p_o - p_e) / (1 - p_e)
print("kappa:", round(kappa, 3))

Output

answers: 12
same label: 9
percent agreement: 0.75
good -> A says 0.667 B says 0.583
bad -> A says 0.333 B says 0.417
chance agreement: 0.528
kappa: 0.471

They agreed on 9 of 12 answers, so percent agreement is 0.75. That sounds good. But reviewer A said good about two thirds of the time and B said good about 58 percent of the time, so luck alone would give 0.528. After taking that out, kappa is 0.471, which is a fair amount lower than 75 percent suggested.

Is 0.471 good enough? Wikipedia is careful here. It says there is controversy over interpreting kappa, and that "factors other than agreement can influence its magnitude", such as how common each label is. Use kappa as a signal. If it is low, read the answers where the two reviewers disagreed, and improve the rubric. Do not treat any one cutoff as a pass mark.

Quick check

Two reviewers agree on 90 percent of answers, but nearly every answer is "good". Why is that high number not very reassuring?

Keep the sample honest

Ten questions feel like a lot of work for a person. They are very few for a statistic. Small samples are noisy: a handful of votes can swing either way on luck.

Wikipedia's A/B testing page says these tests "are sensitive to variance; they require a large sample size in order to reduce standard error and produce a statistically significant result." Standard error is how much a measured number can wobble from one try to the next. Here the two versions are exactly equal, so each vote is a fair coin flip. We run five tries at each size.

Python
import random

# Two versions that are EXACTLY equal: each vote is a fair coin flip.
rng = random.Random(3)

for votes in (10, 100, 1000):
    rates = []
    for trial in range(5):
        a_wins = sum(rng.random() < 0.5 for _ in range(votes))
        rates.append(round(a_wins / votes, 2))
    print(votes, "votes, five tries:", rates)

Output

10 votes, five tries: [0.6, 0.3, 0.4, 0.5, 0.4]
100 votes, five tries: [0.46, 0.51, 0.5, 0.51, 0.46]
1000 votes, five tries: [0.53, 0.49, 0.51, 0.51, 0.52]

With 10 votes, one version's win rate bounced between 0.3 and 0.6, though the true rate was 0.5. With 100 votes it stayed within 0.46 and 0.51. With 1,000 it stayed between 0.49 and 0.53. The made-up 6 to 4 result earlier could easily be this kind of luck.

  • Use more answers

    Pick a sample that covers your real traffic: easy questions, hard ones and odd ones.

  • Say how many

    Report "4 of 10", not just "40 percent". The count shows how little it rests on.

  • Treat a close result as a tie

    If the win rate is near 50 percent on a small sample, you have not learned which version is better.

Tip

Rubric work and a small test are a good way to begin. Do not stop there. Grow the set as you find new kinds of mistakes.

Feedback from real users

There is another kind of human signal: your users. A thumbs up or down, a short comment, a copy button that was pressed, a "try again" click. The Production Monitoring lesson showed where these live on a dashboard.

OpenAI's example of an eval for summarising transcripts uses "production data (collected from user feedback on generated summaries)" together with datasets made by domain experts. So user feedback can feed your test set.

But it is a weaker signal than a trained review. Think of a restaurant comment card. Only some diners fill one in, so you hear from a slice of people. Nobody tells them to use your rubric, they never see the other version, and a thumbs down may mean a slow page, not a bad answer. Use feedback to find where to look, then bring those answers to your reviewers.

Cost and speed

Human review is slow and costs money for every answer read, so spend it where it counts.

  • Spend it on

    Fresh rubrics, risky topics, tone, expert fields, and checking your automated judges every so often.

  • Save it by

    Letting rules and a model judge screen the easy cases first, and sending the unclear or close ones to people.

  • Plan for time

    A review round is slower than a rule that runs in seconds. Decide the sample, the rubric and the date before the release, not after.

Also use the right people. OpenAI mentions employees, contractors and outsourced labeling agencies. For a medical answer, a general reviewer may miss the error. Match the reviewer to the question.

Quick check

You compare two prompt versions with a blind test of 10 questions. Version A wins 6, B wins 4. What is the honest conclusion?

Recap

People judge what machines cannot: tone, expert correctness, safety edge cases, and whether your automated judges are right. Write a rubric with examples, hide which version wrote each answer, shuffle the order, and check agreement with percent agreement and Cohen's kappa. Keep the sample big enough, use user feedback as a weaker clue, and save people for where they matter.

Next: regression testing, making sure changes don't break what worked.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.