AI engineering / Advanced / 03.17
Seeing inside your AI app
Put traces, numbers, scores and thumbs up or down in one place, so you can see what your AI feature does and why it did it.
11 min readFree
From logs to a dashboard
The last lessons, Tracing and logging and Cost and latency monitoring, ended with a promise: next, LLM observability tools, dashboards for all of this. Here it is.
Think about a car. You can drive without a dashboard. But you will not know your speed, your fuel or the warning light until something breaks. An AI feature is the same. It can answer wrongly, slowly or at too high a cost, and nobody notices for weeks.
Observability means being able to look inside a running system and see what it is doing and why. This lesson shows how the tools do that, what five well-known tools say about themselves, and how to choose one. You will also run a small program that builds a dashboard from twelve pretend traces.
Key words
- Observability
- Being able to understand what a system is doing inside by looking at what it records on the outside.
- Trace
- The full record of one request: the question, every step taken, the answer, and how long it took.
- Span
- One step inside a trace, such as one model call or one search. A trace is a small tree of spans.
- Metric
- A number you can chart over time, such as average speed or tokens used.
- Evaluation (eval)
- A score for how good an answer was. It can come from a program, another model or a person.
- Prompt version
- A saved copy of your instructions to the model. Change the words, and you have a new version.
- Dataset
- A saved set of example inputs you can run again to compare changes fairly.
- Feedback
- What a real user tells you, most often a thumbs up or a thumbs down.
Four things in one place
Each of these has been a lesson of its own. The idea of an observability tool is to keep them together, tied to the same request, so you do not hunt through four places.
Traces
What happened. The prompt that went in, the answer that came out, and every step between.
Metrics
How much and how fast. Latency, tokens and cost, added up over many requests.
Scores and feedback
How good. Eval scores from tests and thumbs from real people.
Tokens are the small pieces a model reads and bills by. The Tokenization lesson in Machine Learning Foundations goes deeper. Here you only need to know that more tokens usually means more time and more cost.
Why bring them together? Because a number alone is not an answer. Say the thumbs-down rate jumped on Tuesday. That tells you something is wrong. Open the traces from Tuesday and you can see which prompt, which question and which step went bad. The number points. The trace explains.
Tip
Models can answer differently each time you ask. That is why you cannot just re-run a bug later and expect to see it again. A saved trace is the only reliable record of what really happened.
What a trace looks like
A trace is just a small record saved for every request. Here is one made by hand. It uses a stand-in for the model, a tiny function that always returns the same text, so the code runs free on your computer. A real model is slower, costs money for every token, can give a different answer each run, and can fail or refuse. The stand-in does none of that. It only lets you see the shape of a trace.
import json
def fake_model(messages):
# Stand-in: always returns the same text.
return "Your order ships in 3 days."
def traced_call(prompt_version, question):
messages = [{"role": "user", "content": question}]
answer = fake_model(messages)
return {
"prompt_version": prompt_version,
"input": question,
"output": answer,
"tokens": len(question.split()) + len(answer.split()),
"feedback": None,
}
trace = traced_call("v2", "When will my order arrive?")
trace["feedback"] = "up"
print(json.dumps(trace, indent=2))Output
{
"prompt_version": "v2",
"input": "When will my order arrive?",
"output": "Your order ships in 3 days.",
"tokens": 11,
"feedback": "up"
}Read the output from the top. It remembers the prompt version, what went in, what came out and a size. The size here is a rough word count, not real tokens. At the end, a thumbs up was added later, once the user reacted. Real tools save the same kind of record, plus the time each step took, and they do it in the background so your app does not slow down.
Five tools, in their own words
Many tools do this job. Below are five from the planned list for this course. Each fact comes from the tool's own documentation page, read on 16 November 2026. The order is the order we looked them up, not a ranking. Docs change, so check the page before you decide anything.
LangSmith
Its Observability page says it "provides full visibility into your LLM application: from individual traces to production-wide performance metrics." It also says you can choose "cloud, hybrid, or self-hosted" and that all options include observability, evaluation, prompt engineering and deployment.
Langfuse
Its overview page lists "LLM-as-a-Judge evaluation", "prompt management", "experiments and datasets" and "custom dashboards" as features made for AI engineering. It adds: "It's also open source and can be self-hosted."
Helicone
Its Platform Overview page is titled "Platform Overview - Helicone OSS LLM Observability". It says that "Every request is logged with costs, latency, and errors tracked". Its menu lists prompt management, sessions, a dashboard and an Open Source page.
Arize AI
The Phoenix docs say it "is built by Arize AI and the open-source community" and "is built on top of OpenTelemetry". Arize also offers "Arize AX, a managed enterprise platform built on the same open standards". Phoenix lets you group traces into datasets and rerun them through different versions of your application.
PostHog
Its AI Observability page says it captures every call your product makes to a model, listing "the prompt, the response, the tokens, the cost, the latency". It can also forward traces you already collect in Helicone or Langfuse, and attach user feedback: "Thumbs up and down, ratings, and comments attached to the exact trace they're about."
Tip
Tool facts were checked on 16 November 2026. Features, plans and wording change often, so read the tool's current docs before you pick one.
What most of them share
Read the five facts again and a pattern shows. Different names, similar toolbox.
- 1
Trace viewer
Open one request and see every step, with timing, inputs and outputs.
- 2
Prompt versions
Save each wording of your instructions, and see which version produced which traces.
- 3
Datasets
Turn real traces into a saved set of examples you can run again.
- 4
Evals
Run scores over traces or datasets, to catch a change that made answers worse.
- 5
Cost dashboards
Charts of tokens, spend and speed, split by prompt version, model or user.
- 6
User feedback
A thumbs up or down stored on the exact trace it is about.
Notice the loop. Feedback and bad traces become a dataset. The dataset feeds your evals. The evals tell you whether a new prompt version is safe to ship. Then the new version produces new traces, and you watch again.
How to choose
There is no one right tool. Ask these questions in this order.
Self-host or hosted?
A hosted tool is quicker to start and someone else runs it. Self-hosting means you run it yourself on your own servers. Some tools offer both, and some say so in their docs.
Where does the data live?
Traces hold real questions from real people, and they can contain names or private details. Ask where they are stored, who can read them, and whether you can delete them.
Does it fit your stack?
Check that it works with the language, model provider and framework you already use. A tool you cannot connect easily will not get used.
One more question deserves its own line: does it support OpenTelemetry? OpenTelemetry is a common, open way for software to send out its traces. Phoenix says it accepts traces over it. If your tool speaks it, you can change tools later without rewriting how your app sends data. That keeps you from being locked in.
Tip
Start small. Log traces first, then add feedback buttons, then evals. You do not need every feature on day one, but you do want the records from day one, because you cannot go back and record last week.
Build a small dashboard
Now the worked example. The data below is made up. It is twelve pretend traces, not from a real app, and no model is called at all. Each trace has a prompt version, a latency in seconds, a token count, an eval score from 0 to 1 and a thumbs up or down. The program groups them by prompt version, averages the numbers and prints a table. Then it looks for a version whose thumbs-up rate fell by more than 20 points compared with the one before.
- 1
Group
Put each trace into the row for its prompt version, and add up its numbers.
- 2
Average
Divide by the number of traces to get averages, and count the thumbs up.
- 3
Compare
Look at each version against the one before it, and flag a big drop in thumbs up.
# MADE-UP data: 12 pretend traces, not from a real app.
traces = [
# (prompt version, latency in seconds, tokens, eval score 0-1, thumbs)
("v1", 1.8, 410, 0.82, "up"),
("v1", 2.1, 450, 0.78, "up"),
("v1", 1.9, 430, 0.85, "up"),
("v1", 2.4, 470, 0.74, "down"),
("v2", 1.6, 380, 0.88, "up"),
("v2", 1.7, 395, 0.84, "up"),
("v2", 1.5, 360, 0.90, "up"),
("v2", 2.0, 420, 0.80, "up"),
("v3", 1.1, 250, 0.79, "down"),
("v3", 1.2, 270, 0.81, "up"),
("v3", 1.0, 240, 0.76, "down"),
("v3", 1.3, 280, 0.72, "down"),
]
rows = {}
for version, latency, tokens, score, thumbs in traces:
row = rows.setdefault(version, {"n": 0, "lat": 0.0, "tok": 0, "score": 0.0, "up": 0})
row["n"] += 1
row["lat"] += latency
row["tok"] += tokens
row["score"] += score
row["up"] += thumbs == "up"
print("version traces avg_latency avg_tokens avg_score thumbs_up")
rates = []
for version in sorted(rows):
r = rows[version]
rate = r["up"] / r["n"]
rates.append((version, rate))
print(f"{version:<7} {r['n']:>6} {r['lat']/r['n']:>10.2f}s {r['tok']/r['n']:>10.0f} {r['score']/r['n']:>9.2f} {rate:>8.0%}")
for (old, old_rate), (new, new_rate) in zip(rates, rates[1:]):
if new_rate < old_rate - 0.2:
print(f"FLAG: {new} thumbs-up fell from {old_rate:.0%} to {new_rate:.0%}")Output
version traces avg_latency avg_tokens avg_score thumbs_up v1 4 2.05s 440 0.80 75% v2 4 1.70s 389 0.85 100% v3 4 1.15s 260 0.77 25% FLAG: v3 thumbs-up fell from 100% to 25%
Read the table. Version v1 had four traces, 2.05 seconds on average, and 75% thumbs up. Version v2 was faster at 1.70 seconds, used fewer tokens and got 100% thumbs up. Good.
Version v3 is the trap. It is the fastest at 1.15 seconds and uses the fewest tokens, 260 on average, so a cost chart would love it. But its average eval score fell to 0.77 and only 25% of people gave a thumbs up. The last line of the output flags it: thumbs up fell from 100% to 25%.
This is why you watch everything together. Speed and cost alone would have said "ship v3". The feedback said "people hate it". With real traces you would now open the v3 ones and read them. Maybe the shorter prompt made answers too curt.
Tip
With only four traces per version, a few thumbs can swing the numbers a lot. The data here is tiny and made up. In a real app you would wait for many more traces before you trust a drop.
Quick check
Quick check
Your dashboard shows the new prompt version is faster and cheaper, but thumbs up dropped. What do you do first?
Quick check
What is a span?
Quick check
Why might OpenTelemetry support matter when you choose a tool?
What to remember
Observability is a dashboard for your AI feature. It joins traces, metrics, eval scores and user feedback in one place, so a number can point you to the exact request that explains it.
Tools such as LangSmith, Langfuse, Helicone, Arize Phoenix and PostHog offer a similar toolbox: trace viewer, prompt versions, datasets, evals, cost dashboards and feedback. Choose by where it runs, where your data lives, how well it fits your stack and whether it supports OpenTelemetry.
Remember the made-up dashboard. The fastest, cheapest version was the one users liked least. Always read speed, cost, scores and feedback side by side.
Next: production monitoring, keeping a live AI feature healthy.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.