AI engineering / Advanced / 03.15
Tracing and logging
How to record every step one request took through your AI app, so a strange answer becomes something you can open up and read.
13 min readFree
Seeing every step
The last lesson ended with a card that says who uses your feature and what it is for. A card tells you what should happen, not what did happen. Now a user writes: "your helper gave me a wrong answer yesterday." You open your app and there is nothing to look at. You can ask the same question again, but a model may answer differently the second time.
Think of a parcel. A good delivery service does not only say "delivered" or "lost". It shows every stop: picked up, sorted, loaded, out for delivery. When a parcel goes missing, you read the stops. This lesson gives your AI app the same list of stops. It is called a trace.
Key words
- Log
- One line of text your app writes when something happens, such as "model call failed". Logs are like a diary.
- Trace
- The full record of one request, from the moment it arrives to the moment the answer leaves. Like the stops on a parcel's tracking page.
- Span
- One step inside a trace, such as one search or one model call. It has a name, a start time and an end time.
- Attribute
- A small label stuck on a span, like a name and a value. For example: tokens used = 60.
- Sampling
- Keeping only some of your traces instead of all of them.
- Redact
- Hide or blank out a piece of private text before it is saved.
What a trace holds
Wikipedia describes tracing in software as "the process of capturing and recording information about the execution of a software program". It adds that programmers typically use it for debugging.
The OpenTelemetry docs call a trace "the path of a request through your application". A request is one thing a user asks your app to do, such as one question typed into a chat box.
Spans, parents and children
A trace is made of spans. The OpenTelemetry docs say a span represents "a unit of work or operation" and that spans are the building blocks of traces. The same page lists what a span includes: a name, a parent span ID (empty for root spans), start and end timestamps, a span context, attributes, span events, span links and a span status.
The parent link is what turns a pile of spans into a tree. Picture a family tree for one request. The root span is the whole request. Under it sit the steps: a search, a model call, a tool call. The docs say spans can be nested, and child spans represent sub-operations.
Attributes and status
Attributes are the labels. The OpenTelemetry docs describe them as key-value pairs that annotate a span with information about the operation it tracks. A key is the name of the label and a value is what is written on it. Keys must be non-null strings, and values must be a string, a boolean, a floating point number, an integer, or an array of these.
Every span also has a status. The docs list three: Unset, Error and Ok. Unset is the default, and it means the operation finished without an error. Error means something went wrong. Ok is a mark a developer can set on purpose, but the docs say that in most cases it is not necessary.
Why AI apps need it more
Any app with several moving parts benefits from tracing. AI apps need it more, for three reasons.
Answers vary
A real model can give a different answer each time you ask, so you cannot always rebuild a bad answer later. The record has to be kept at the time.
Many steps
One question may trigger a search, a model call, a tool call and another model call. An agent picks its own next step, so a bad answer could come from any of them.
Cost follows size
Models read and write tokens, and a real provider charges for them. A trace that records tokens on every model call shows which step is heavy. The Tokenization lesson in Machine Learning Foundations goes deeper on tokens.
Try it: a tiny tracer
Let's build the smallest tracer that works. We will trace one made-up request to a search-and-answer app (the pattern called RAG: search your own documents, then give what was found to a model). It does four things: a search, a model call, a tool call that looks up an order, and a second model call.
The tracer uses a Python feature called a context manager. It is the thing behind the word with. The Python docs say the contextmanager decorator lets you define a with-statement helper "without needing to create a class or separate __enter__() and __exit__() methods". We write with span(...) around a step. The span starts when the block starts and ends when it ends, even if the step crashes.
Two more details. Time comes from a fake clock that we move by hand, so every run gives the same numbers. And every attribute passes through a function that replaces anything that looks like an email address with [email].
The model and the search here are stand-ins, small fake functions with fixed answers, so the code runs free with no account or key. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The numbers below are made up.
import re
from contextlib import contextmanager
clock = {"ms": 0} # a fake clock: we move it by hand, so runs are repeatable
spans, stack = [], []
def redact(value):
# Hide anything that looks like an email address before it is stored.
if isinstance(value, str):
return re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[email]", value)
return value
@contextmanager
def span(name, **attrs):
parent = stack[-1]["id"] if stack else None
rec = {"id": len(spans) + 1, "name": name, "parent": parent,
"start": clock["ms"], "end": None, "error": None,
"attrs": {k: redact(v) for k, v in attrs.items()}}
spans.append(rec)
stack.append(rec)
def add(**more):
rec["attrs"].update({k: redact(v) for k, v in more.items()})
try:
yield add
except Exception as err:
rec["error"] = type(err).__name__
raise
finally:
rec["end"] = clock["ms"]
stack.pop()
def show():
depth = {}
for s in spans:
depth[s["id"]] = 0 if s["parent"] is None else depth[s["parent"]] + 1
ms = s["end"] - s["start"]
extra = ", ".join(f"{k}={v}" for k, v in s["attrs"].items())
flag = f" ERROR={s['error']}" if s["error"] else ""
print(" " * depth[s["id"]] + f"{s['name']} ({ms} ms) {extra}{flag}")
# Stand-ins so the code runs free. A real model is slower, costs money for
# every token, can answer differently each run, and can fail or refuse.
def fake_model(messages):
if any("ORDER" in m for m in messages):
return "Order 42 ships Friday.", 60, 6
return "Please wait for a reply.", 40, 5
def fake_search(question):
clock["ms"] += 40
return ["ORDER 42: ships Friday"]
def fake_tool(order_id):
clock["ms"] += 120
return {"order": order_id, "status": "ships Friday"}
question = "Where is order 42? Mail me at ana@example.com"
with span("request", user_input=question):
with span("retrieve", query="order 42") as add:
docs = fake_search(question)
add(documents=len(docs))
with span("chat stand-in-model", **{"gen_ai.operation.name": "chat"}) as add:
clock["ms"] += 300
text, tin, tout = fake_model(docs)
add(**{"gen_ai.usage.input_tokens": tin, "gen_ai.usage.output_tokens": tout})
with span("execute_tool lookup_order", **{"gen_ai.tool.name": "lookup_order"}):
fake_tool(42)
with span("chat stand-in-model", **{"gen_ai.operation.name": "chat"}) as add:
clock["ms"] += 250
text, tin, tout = fake_model(docs)
add(**{"gen_ai.usage.input_tokens": tin + 20, "gen_ai.usage.output_tokens": tout})
show()Output
request (710 ms) user_input=Where is order 42? Mail me at [email] retrieve (40 ms) query=order 42, documents=1 chat stand-in-model (300 ms) gen_ai.operation.name=chat, gen_ai.usage.input_tokens=60, gen_ai.usage.output_tokens=6 execute_tool lookup_order (120 ms) gen_ai.tool.name=lookup_order chat stand-in-model (250 ms) gen_ai.operation.name=chat, gen_ai.usage.input_tokens=80, gen_ai.usage.output_tokens=6
Read the output from the top. The first line is the root span: the whole request took 710 ms (milliseconds, thousandths of a second). Its four children are indented under it. The search took 40 ms, the first model call 300 ms, the tool call 120 ms and the second model call 250 ms.
The user typed an email address into the question. In the trace it shows as [email], because the redaction ran before the text was stored.
Reading the tree
A printed tree like this is a small version of what tracing tools draw. Wikipedia notes that the main way tools show traces is a swimlane view, where each step sits on its own bar along a time line, often with attributes attached.
- 1
Start at the root
The root span says how long the whole request took and what the user asked. Here: 710 ms.
- 2
Find the heavy step
Add up where the time went. The two model calls took 550 ms of the 710, which is more than three quarters. If the app feels slow, that is where to look first.
- 3
Check the labels
The second model call read 80 tokens and the first read 60. The second reads more because it also carries what the tool found. Tokens in the trace explain cost in a way a bill cannot.
- 4
Look for errors
A span that failed would end with ERROR and the name of the problem, and every span above it would show the same trouble.
OpenTelemetry and its AI names
You do not have to invent your own record format. Wikipedia describes OpenTelemetry as a CNCF open source project that provides comprehensive support for distributed tracing. The OpenTelemetry docs also describe trace exporters, which send traces to a consumer: standard output for debugging, a collector, or a backend you choose.
OpenTelemetry also publishes agreed names for common labels, called semantic conventions. They matter because if everyone calls the same thing by the same name, tools can understand traces from different apps. There is a set for generative AI. The pages say these conventions are in the Development status, which means they can still change. Check the current pages before you depend on a name.
Here is what they say about the AI steps. They cover several kinds of operation: client inference (a call to a model), embeddings, tool execution, retrieval and agent invocations. The span for a model call should be named with the operation and the model, in the form {gen_ai.operation.name} {gen_ai.request.model}. The span for running a tool should be named execute_tool {gen_ai.tool.name}, with the operation name set to execute_tool. Token counts go into gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.
What not to log
A record of everything is also a record of everything private. A trace store can fill up with names, emails, and whatever users type into a chat box. If you log it all, anyone who can open your traces can read it all.
The OpenTelemetry GenAI pages are careful about this. They state that model instructions, user messages and model outputs "are considered sensitive and are often large in size". Instrumentations should not capture them by default, and should give you an option to opt in. The first usage pattern they list, marked as the default, is to not record instructions, inputs or outputs. The opt-in attributes are named gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages.
Never log secrets
API keys, passwords, session tokens and anything that works like a key. If one reaches a trace, treat it as leaked and replace it.
Redact personal data
Emails, phone numbers, home addresses and ID numbers. Hide them before they are saved, as the tracer above did with the email.
Log sizes, not words
Token counts, time taken, the model used and the status are safe and very useful. Full prompts and answers are heavy and often private, so keep them off unless you really need them.
Sampling: keeping some traces
Traces take space and cost money to store. The OpenTelemetry sampling page says that if the large majority of requests are successful and finish with acceptable latency and no errors, you do not need 100% of your traces to meaningfully observe your system. It calls sampling one of the most effective ways to reduce the cost of observability without losing visibility. (Latency is how long something took.)
The same page says you might skip sampling if you generate very little data, and gives a rough size for considering it: 1000 or more traces per second. Most small apps are nowhere near that, so start by keeping everything.
There are two ways to decide. Head sampling decides early, as the request starts, without looking at the whole trace. It is easy and cheap, but the docs note you cannot make sure all traces with an error are kept. Tail sampling decides after the trace is finished, looking at the spans. The docs list examples: always keep traces that contain an error, and keep traces based on overall latency. The price is that it is harder to run.
Here is a small tail-style rule on 20 pretend traces. Traces 4 and 13 had errors, trace 9 was slow, and the rest were ordinary. The rule keeps every error, every slow one, and about 1 in 10 of the rest. A fixed seed makes the random part repeat.
import random
random.seed(3)
# 20 pretend traces: (trace number, had an error?, took how many ms)
traces = [(n, n in (4, 13), 2400 if n == 9 else 700) for n in range(1, 21)]
kept = []
for n, had_error, ms in traces:
# Always keep errors and slow ones. Keep about 1 in 10 of the rest.
if had_error or ms >= 2000 or random.random() < 0.10:
kept.append(n)
print("kept", len(kept), "of", len(traces), ":", kept)Output
kept 5 of 20 : [4, 7, 8, 9, 13]
Five of 20 were kept. Three were certain: 4, 9 and 13. The other two, 7 and 8, were picked by chance. A different seed would pick different lucky ones, but the errors and the slow trace always stay.
Using a trace to debug a bad answer
Now the payoff. A user says the helper gave a wrong answer about their order. You do not guess. You open that request's trace.
- 1
Find the trace
Look it up by request ID or the time of the report.
- 2
Read the search step first
If it shows documents=0, the model never saw the facts. That is a search gap, not a model mistake.
- 3
Check what the model read
Look at the input token count and, if you opted in, the input messages. Far fewer tokens than expected can mean something was cut off. The GenAI conventions also record why a model stopped generating, in gen_ai.response.finish_reasons.
- 4
Check each tool call
Did the tool run? Did it return an error?
- 5
Fix, then keep the case
Change one thing, then save this request as a test case so the same slip is caught next time.
Quick check
A trace shows the search span found 0 documents, yet the model answered with confidence. Where is the fix?
Quick check
You want to be sure every failed request keeps its trace. Which approach fits?
Putting it together
A trace is the full record of one request, built from spans. AI apps need traces more because answers vary and requests take many steps. OpenTelemetry is the open standard, and its generative AI names label model and tool calls, though they are still in development.
Log sizes and outcomes, hide private text, never log secrets, and sample only when volume makes you. Then a wrong answer comes with stops to read, not a guess.
Next: cost and latency monitoring, keeping speed and spend in check.
Sources
- OpenTelemetry docs: Traces
- OpenTelemetry docs: Sampling
- OpenTelemetry GenAI semantic conventions: overview
- OpenTelemetry GenAI semantic conventions: spans
- OpenTelemetry GenAI semantic conventions: client inference
- OpenTelemetry GenAI semantic conventions: execute tool
- Wikipedia: Tracing (software)
- Python docs: contextlib
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.