AI engineering / Beginner / 01.41
Harness Engineering
The code around a model that makes it dependable: tools, limits, checks, logs and tests. Build a tiny one in plain Python.
11 min readFree
The code around the model
The last lesson, Inference Engineering, ended with a promise: next, harness engineering, the code around the model that makes it dependable. Here it is.
Think of a race car. The engine gives the power. But nobody drives an engine alone down a track. You need a seat belt, brakes, a dashboard, a pit crew and a lap timer. The model is the engine. The harness is everything bolted around it so you can trust the ride.
By the end you will know what a harness is, the nine jobs it does, and how every earlier lesson in this course is one of those jobs. You will also read and run a small harness in plain Python.
Key words
- Harness
- All the code around a model call or an agent loop that makes it dependable. Not the model itself.
- Tool registry
- A list of the tools your code allows the model to use, by name.
- Budget
- A hard cap on how much a run may spend, in steps, tokens or money.
- Validation
- Checking that an output has the shape you expect before you use it.
- Trace
- A written record of every step a run took, so you can replay what happened.
- Eval
- A test for AI work: run a fixed set of tasks and score the answers.
- Stand-in
- A tiny fake used so code runs free. Here, a fixed script plays the model.
What a harness is
Wikipedia describes an agent harness as "the software infrastructure surrounding a large language model (LLM) that enables it to operate as an AI agent". It adds that the relationship can be written as "agent = model + harness". The page says a harness manages tool use, memory, state persistence, execution environments and feedback loops.
A model on its own only turns text in into text out. It cannot open a file, remember yesterday, or stop itself after ten tries. Your code does all of that. That code is the harness.
The same page says a minimal harness is unnecessary for a single prompt and reply, but becomes important as tasks grow multi-step, tool-oriented or long-running. So a one-shot chat needs very little. An agent needs a lot.
Tip
The word is used loosely. Different sources use harness for different things: an agent harness, a test harness, an evaluation harness. This lesson uses it the broad way, as the code around the model, and says when it means one of the narrower kinds.
Where the word comes from
The word is older than AI. In software testing, Wikipedia says a test harness can "call functions with supplied parameters and print out and compare the results to the desired value". It is also used when the real production setup is unavailable, so it swaps in pre-defined test data and smaller parts. A stand-in model in this course does the same job.
In AI there is the evaluation harness. The lm-evaluation-harness project from EleutherAI calls itself "a unified framework to test generative language models on a large number of different evaluation tasks". It notes that using publicly available prompts ensures reproducibility and comparability between papers. That is the testing side of a harness: same tasks, same scoring, comparable results.
Wikipedia also says the phrase harness engineering only emerged in early 2026 and that who coined it is contested. So treat the name as young. The jobs it covers are not.
The nine jobs of a harness
Sources group harness parts a little differently. Wikipedia lists tool dispatch, memory and state saving, a sandbox, context management, and guardrails such as scoped permissions and monitoring. Here is one plain split into nine jobs, each tied to a lesson you have already met.
- 1
Build the context
Decide what the model sees: instructions, history, documents. This is the Context Engineering lesson, plus Context Window, Context Compaction and Prompt Caching.
- 2
Register the tools
Keep a list of allowed tools and describe each one clearly. This is Function Calling. Anthropic says tool definitions should get "just as much prompt engineering attention as your overall prompts".
- 3
Run the tools
When the model asks for a tool, your code runs it and sends back the result. The model never runs anything itself.
- 4
Check permissions
Before running, ask: is this tool allowed, and does this action need a person's yes? This ties to Context Security and the guards in AI Agents.
- 5
Retry and recover
If the output is broken or a tool fails, try again a limited number of times, then give up politely.
- 6
Limit steps and budget
Cap the loop. Anthropic notes it is common to include "stopping conditions (such as a maximum number of iterations) to maintain control".
- 7
Log and trace
Write down every step. Without a trace you cannot tell why a run went wrong.
- 8
Validate the output
Check the answer has the right shape before anything downstream trusts it. This is Structured Output.
- 9
Save state and test
Keep progress outside the model's memory (Memory Systems, State and Historical Context), and run evals (Context Evaluation) so a change can be judged.
Streaming Responses and Constraints fit here too. Streaming is how the harness hands partial text to the screen. Constraints are the rules the harness enforces so the model cannot ignore them.
One loop, many guards
Inside the harness sits the loop from the ReAct lesson: ask the model, run what it asks for, show it the result, repeat. Anthropic describes agents as "typically just LLMs using tools based on environmental feedback in a loop". Each of the nine jobs is a guard placed at one point in that loop.
Why not trust the model alone
A real model is slower than a normal function, costs money for every token, can give a different answer each run, and can fail or refuse. A harness is how you build something steady on top of something that wobbles.
It can wander
A loop with no cap can run for a long time. Step and budget limits end it. Anthropic says the autonomy of agents means higher costs and the potential for compounding errors.
It can be wrong
Anthropic recommends "extensive testing in sandboxed environments, along with the appropriate guardrails". The harness is where guardrails live.
It forgets
Anthropic says long-running agents "must work in discrete sessions" and each new session begins with no memory of what came before. A harness saves notes so the next session can pick up.
That last point has a real example. In their long-running agent write-up, Anthropic had an agent write a feature list as a JSON file, over 200 features in one case, each marked as failing until tested. They say they chose JSON because the model is less likely to inappropriately change JSON files than Markdown files. They also had it keep a progress notes file and commit to git at the end of each session. All of that is harness, not model.
They also saw the model mark features as done without proper testing. Giving it browser automation tools to test as a human user would improved results. The lesson: a harness can include the checking tools, not only the limits.
Build a tiny harness
Now the code. Below is a small harness class in plain Python. It registers two tools, caps steps and tokens, validates the model's JSON, logs every step, and returns a report.
The model here is a stand-in: fake_model follows a fixed script, so the code runs free with no keys and no network. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The prices in the tool are made up. The token count is only a word count, a rough stand-in, since the Tokenization lesson in Machine Learning Foundations shows real tokens are not words.
import json
PRICES = {"pen": 2, "book": 9} # made-up prices
def fake_model(messages):
"""Stand-in model: follows a fixed script, no real AI."""
task = messages[0]["content"]
tools_used = [m for m in messages if m["role"] == "tool"]
if "forever" in task:
return '{"action": "tool", "tool": "lookup_price", "args": {"item": "pen"}}'
if not tools_used:
if not any("valid JSON" in m["content"] for m in messages if m["role"] == "system"):
return 'Sure! Let me look that up.'
return '{"action": "tool", "tool": "lookup_price", "args": {"item": "pen"}}'
if len(tools_used) == 1:
return '{"action": "tool", "tool": "multiply", "args": {"a": ' + tools_used[0]["content"] + ', "b": 3}}'
return '{"action": "final", "answer": "' + tools_used[1]["content"] + '"}'
class Harness:
def __init__(self, model, max_steps, max_tokens):
self.model, self.max_steps, self.max_tokens = model, max_steps, max_tokens
self.tools = {}
def register(self, name, fn):
self.tools[name] = fn
def run(self, task):
messages = [{"role": "user", "content": task}]
log, tokens = [], 0
for step in range(1, self.max_steps + 1):
reply = self.model(messages)
tokens += sum(len(m["content"].split()) for m in messages) + len(reply.split())
if tokens > self.max_tokens:
log.append(f"step {step}: stop, token budget used ({tokens})")
return self.report("budget", None, step, tokens, log)
try:
data = json.loads(reply)
if data["action"] not in ("tool", "final"):
raise ValueError("bad action")
except (ValueError, KeyError):
log.append(f"step {step}: bad output, asking again")
messages.append({"role": "system", "content": "Reply with valid JSON only."})
continue
if data["action"] == "final":
log.append(f"step {step}: final answer {data['answer']}")
return self.report("done", data["answer"], step, tokens, log)
name = data["tool"]
if name not in self.tools:
log.append(f"step {step}: blocked unknown tool {name}")
return self.report("blocked", None, step, tokens, log)
result = self.tools[name](**data["args"])
log.append(f"step {step}: ran {name} -> {result}")
messages.append({"role": "assistant", "content": reply})
messages.append({"role": "tool", "content": str(result)})
log.append(f"step {self.max_steps}: stop, step limit reached")
return self.report("step_limit", None, self.max_steps, tokens, log)
def report(self, status, answer, steps, tokens, log):
return {"status": status, "answer": answer, "steps": steps, "tokens": tokens, "log": log}
h = Harness(fake_model, max_steps=6, max_tokens=120)
h.register("lookup_price", lambda item: PRICES[item])
h.register("multiply", lambda a, b: a * b)
for task in ["What do 3 pens cost?", "Keep checking prices forever"]:
print(task)
print(json.dumps(h.run(task), indent=2))Output
What do 3 pens cost?
{
"status": "done",
"answer": "6",
"steps": 4,
"tokens": 87,
"log": [
"step 1: bad output, asking again",
"step 2: ran lookup_price -> 2",
"step 3: ran multiply -> 6",
"step 4: final answer 6"
]
}
Keep checking prices forever
{
"status": "budget",
"answer": null,
"steps": 5,
"tokens": 135,
"log": [
"step 1: ran lookup_price -> 2",
"step 2: ran lookup_price -> 2",
"step 3: ran lookup_price -> 2",
"step 4: ran lookup_price -> 2",
"step 5: stop, token budget used (135)"
]
}Read the first report. The stand-in first answers with plain chatter, not JSON. The harness catches it at step 1, writes "bad output, asking again" and sends a reminder. Then the stand-in looks up a pen price (2), multiplies by 3 (6), and gives a final answer of 6. Four steps, 87 tokens, status done.
Read the second report. This stand-in never stops asking for the price. A plain loop would run forever. The harness lets it go five steps, then sees the count at 135, over the budget of 120, and stops with status budget. The log shows exactly where. That is the whole point: the bad run ended cleanly, and you can see why.
Tip
Look at what is not in this class: nothing about prompts or the model's brain. Every line is ordinary code you could test. That is why harness work is a good place for designers and developers to add value.
Testing with evals
A harness without tests only feels safe. The test idea is the same as the test harness above: run a fixed set of tasks, compare to the expected answer, count passes. Because real models vary run to run, you run the set many times and look at the rate, not one result.
Here the stand-in makes the two reports above a tiny eval. Task one should end with status done and answer 6. Task two should end with status budget. If you change a limit and a report changes, you see it at once. The Context Evaluation lesson goes deeper on scoring.
Quick check
A model reply is not valid JSON. What should the harness do first?
Quick check
What does a step and budget limit protect you from?
What to remember
Model plus harness
The model gives answers. The harness makes the system dependable around it.
Guards everywhere
Context, tools, permissions, retries, limits, logs, validation, saved state and tests are all the harness.
Keep it testable
A good harness is plain code you can run with a stand-in model, log, and score.
Next: data catalogs, knowing what data you have before you feed it to a model.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.