esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Instrumentation: Making Your App Report How It Behaves

Add timers, counters, and trace IDs to your app so the raw signals that monitoring relies on actually exist.

In short

Instrumentation is adding timers, counters, and trace IDs to an app so it reports how it behaves and the signals monitoring relies on exist.

13 min readFree

What You Will Learn

Your app runs out of sight. When a page feels slow or an order fails, you need a way to look inside it. Instrumentation is the work of adding code, or agents that add code for you, so the running app reports data about its own behavior. In this lesson you will learn the three kinds of data an app can report, what to record first, and the trade-offs you face when you collect it.

Key words

Instrumentation
Adding code, or agents that add code, to an app so it reports on what it does.
Telemetry
The general word for data a running system sends out about itself.
Metric
A number that is added up or averaged over a stretch of time, such as errors per minute.
Log
A record of one event, with a time stamp and details such as an error message.
Trace
The full path of one request through your system, shown as a list of timed steps.
Span
One timed step inside a trace, such as a database query or a payment call.
Trace ID
A code that names one request, so every span from that request can be tied together.
Context propagation
Sending the trace ID along with each outgoing call, so the next service joins the same trace.
Overhead
The extra time, memory, and network use that telemetry adds to your app.
Sampling
Keeping only some of the telemetry, to save cost and storage.
Cardinality
How many different values a label can take. A label with many values creates many separate counts.
OpenTelemetry
A free, open-source project with tools and rules for creating and sending telemetry.

What Are the Three Kinds of Telemetry?

Instrumented code sends out telemetry in three main shapes, called signals. OpenTelemetry, an open-source framework for generating and exporting this data, works with the same three: traces, metrics, and logs.

  • Metrics

    A metric is a number that a system sums up or averages over a window of time, such as requests per minute or the count of failed payments. Use metrics to see whether something changed and how big the change is. Dashboards and alerts are built from them.

  • Logs

    A log is one record of one event, with a time stamp and details such as a severity and a message. Logs answer 'what did this step print when it failed?' On their own they often lack the context of which request wrote them.

  • Traces

    A trace follows one request from start to finish, as a chain of spans. Each span is a timed step. Use traces when one request is slow and you need to know which step took the longest.

Each kind has a job. Metrics are cheap to keep for a long time and good for spotting trouble early. Traces are detailed and tied to one request, so they are best for finding the slow step. Logs hold the words and details, so they are best for the final why.

How Do the Signals Work Together?

The three kinds work best side by side. A simple way to picture it: a metric shows that a symptom exists, a trace shows which request and which step carry it, and a log gives the detail behind that step.

They are joined by shared context. When a log is written while a span is active, the log can carry that span's trace and span IDs, so you can jump from a log line to its trace.

Tip

Add the trace ID to your log lines before you buy any fancy tool. It costs little, and it makes an incident much easier to read.

What and Where Should You Instrument?

Put instrumentation where work crosses a boundary. Most slow or failing requests show their trouble at a boundary: the point where a request enters your service, leaves it for a database or another service, or waits in a queue.

Key Points in a Request

  • Incoming requests

    Time and count every request that enters a service, grouped by its result. Framework tools often cover this part already, so check what they give you before you write it by hand.

  • Outgoing calls

    Time every call to a database or another service, and count its failures. A payment call, a stock check, or a cache read each deserves its own span.

  • Queues and background work

    A message placed on a queue runs later, often in another program. Unless you pass the trace context along on purpose, the trace breaks at that point.

Useful Signals to Record

  • Latency

    How long a piece of work takes. Record it for each request and for each outgoing call. Averages hide the worst cases, so many teams also look at high percentiles.

  • Error rate

    The share of requests that fail. Count failures with a label that says what kind of failure it was, not just that one happened.

  • Throughput

    How much work finishes over time, often counted as requests per second. A sudden drop can be as telling as a spike in errors.

Request IDs and Trace Propagation

A request ID is a unique label for one request. A trace ID is the label shared by all the spans of one request. Propagation means sending that label onward with each outgoing call. The first service creates the trace ID and puts it in the headers of later calls, so the next service can add its own spans to the same trace.

Without propagation, each service writes its own separate story, and you must guess how they connect. With propagation, the web request, the payment span, and the database span all share one trace ID.

Tip

Pass the trace ID into your queue messages too. Otherwise the background work starts a brand-new story and you lose the link to the customer's request.

Worked Example: Finding a Slow Checkout

An online shop has a checkout flow. The web app calls a payment service, and the payment service writes a payment record to a database. Some customers say checkout stalls. Here is how you would instrument the flow so the slow step can be found.

  1. 1

    Give each request a trace ID

    When the checkout request arrives at the web app, create a trace ID. If the request already carries one from an upstream service, reuse it.

  2. 2

    Time the payment call

    Wrap the call to the payment service in a timer and a span. Record how long it took and which trace it belongs to.

  3. 3

    Count payment failures

    When the payment call fails, add one to a payment error counter and mark the span as an error. The counter feeds the dashboard. The span shows which request failed.

  4. 4

    Pass the trace ID on

    Send the trace ID in the headers of the call to the payment service. The payment service then passes it on to its database call, so that span joins the same trace.

  5. 5

    Read one slow trace

    When the timer shows a 9-second payment call, open that trace. You can see whether the time was spent in the payment service's own code or in the database span inside it.

Python
payment_errors = {"count": 0}
payment_ms = []

def call_payment(trace_id, fail, took_ms):
    status = "ok"
    if fail:
        status = "error"
        payment_errors["count"] += 1
    payment_ms.append(took_ms)
    print(f"trace={trace_id} span=payment.charge status={status} ms={took_ms}")

call_payment("trace-42", False, 180)
call_payment("trace-42", True, 9000)
print("payment errors:", payment_errors["count"])
print("slowest payment ms:", max(payment_ms))

Output

trace=trace-42 span=payment.charge status=ok ms=180
trace=trace-42 span=payment.charge status=error ms=9000
payment errors: 1
slowest payment ms: 9000

The sample uses fixed numbers in place of a real clock, so the output is the same every time you run it. A real service would read the clock and send the spans to a backend. The output still shows the idea: both payment spans share trace-42, the second one failed, and the counter and the timer each caught it. Now an engineer can search for slow traces, find trace-42, and see that the payment step took 9000 ms.

Should You Instrument Manually or Automatically?

There are two main ways to add instrumentation. The choice affects how much work you do and how much detail you get.

  • Automatic

    Automatic, or zero-code, instrumentation uses agents or runtime hooks that watch popular frameworks, so you do not edit the source. It is quick and covers many services at once.

  • Manual

    Manual instrumentation means you write code with an SDK, a kit of library calls, to create spans, record metrics, and attach details. Business details, such as a cart ID or a payment provider's response code, are a good reason to write it by hand.

  • Both together

    Most teams mix the two. Automatic gives broad coverage, and manual adds depth where the business cares most. Start broad, then go deep where it counts.

Using a Shared Standard

A shared standard such as OpenTelemetry gives you one set of tools and data formats for traces, metrics, and logs. The code you write does not lock you to one storage or dashboard product, so you can choose where the data goes later.

  1. 1

    Turn on automatic coverage

    Enable automatic instrumentation for your web framework and your database client. Check that each incoming request now produces a trace.

  2. 2

    Look for gaps

    Open a few traces and find the places where time passes without a span to explain it. Those gaps mark where manual work is needed.

  3. 3

    Add business spans

    Add manual spans around business steps such as payment or stock checks. Give them names that a shop manager would understand.

  4. 4

    Recheck when you add code

    Each new service or endpoint needs matching instrumentation. Coverage can quietly shrink as the system grows, so review it whenever you ship something new.

What Are Overhead and Sampling?

Overhead is the cost of collecting telemetry. The amount depends on the application, the runtime, which instrumentations are enabled, and how many spans are created. Too much instrumentation can slow the app itself, so measure the effect after you add it rather than assuming it is small.

Sampling keeps only some traces, which cuts cost. The two main styles differ in when they decide.

  • Head sampling

    The decision is made at the start of a request, before the rest of the work happens. It is cheap and simple. Its weakness is that it cannot see how the request ends, so it cannot promise to keep every trace that has an error.

  • Tail sampling

    The decision waits until the spans of the trace are complete, so it can look at the whole trace. It can keep every error or every slow request.

Tip

Keep every error trace and a small share of normal traces. Many teams start with head sampling and add tail rules for errors and slow requests later.

Keeping Data Safe and Small

Telemetry is easy to add and easy to over-collect. Two habits keep it useful and safe. The first is to leave out private data. The second is to keep label values few.

Sensitive Data

Telemetry is copied into logs, dashboards, and backups that many people can read. Never put passwords, card numbers, session tokens, or full personal details into a span, a log, or a label. Record a safe reference such as an order ID instead, and remove private values before the data leaves the app.

High-Cardinality Labels

Cardinality is the count of different values a label can take. A status code label has a few values. A user ID label can have millions. Each new value creates another separate series that the tool must store and search, which slows queries and raises cost. Use a route template such as /orders/{id} instead of the full URL with its real ID, and keep the real ID in traces or logs, where one value per record is normal.

Tip

Before you add a label, ask how many different values it could take in a year. If the answer is 'a huge number', it belongs in a trace or log, not in a metric.

Check Your Understanding

Try these short scenarios. Each one needs the ideas from this lesson.

Quick check

Users say the 'Place order' button sometimes takes 10 seconds. Your server logs show every request finishing quickly. What would you instrument first, and which kind of telemetry would help you find where the time goes?

Quick check

Your service handles 20,000 checkouts an hour and keeps 5 percent of its traces to control cost. Last week a rare payment error happened, and no trace of it was kept. Which change fits best?

Quick check

A developer adds the customer's email address as a label on the order counter. Within a day, the metrics tool slows down. What is the best fix?

Key Takeaways

  • Record at the boundaries

    Time and count incoming requests, outgoing calls, and queue hand-offs. Carry one trace ID through all of them.

  • Use the three signals together

    Metrics show that something is wrong, traces show where, and logs show why.

  • Balance detail and cost

    Choose automatic coverage first, add manual spans for business steps, sample wisely, and keep private data and high-cardinality labels out of your metrics.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.