Skip to module content
🧠AI Academy
0/91
Sign in
Module 04 · ~8 min

Observability & Tracing

First in your build order, even though it's fourth on the page — error analysis needs traces to read.

Reading progress
0/6 · 0%

The big idea

💡Key idea
Traces tell you why; dashboards only tell you that. A dashboard can show you that p95 latency doubled — but only the trace reveals the agent making 12 retrieval calls per run because a prompt edit quietly removed the 'search at most twice' instruction. When you have an hour: read five traces. Don't stare at charts.
Quick check
1 question · instant feedback
0/1
  1. p95 latency doubled overnight. What's the fastest path to root cause?

Deep dive

2/2 open

**Cost tracking is per-trace, or it's fiction.**

Aggregate spend hides the one pathological run pattern that costs 40× the median. Track cost per trace, set alerts on outliers, and report cost-per-task to clients. 'Each account audit costs $0.31' is a sentence that closes deals. 'We spent $200 last month' is not.

**Latency budgets per span type** — retrieval under 500 ms, model calls under 8 s, total run under 90 s — turn 'it feels slow' into a diffable fact you can put in a PR description.

**Drift detection** doesn't require a fancy platform. Take a fixed slice of your golden eval set and run it on a schedule — weekly is usually enough. Chart judge pass-rates over time. Providers update models under the same name without announcing it. Your corpus changes. User behaviour shifts. None of those events show up in a single deploy, but a weekly probe run will catch the slow rot before a client does.

Tracing platforms — Langfuse, LangSmith, AgentOps and their kin — give you span capture, a UI for reading traces, and cost roll-ups. That's the core value proposition.

Two things to watch for when evaluating them:

**Data residency.** If client data flows through prompts (and it usually does), your traces contain client data. A self-hostable option like Langfuse matters when a client's risk register or regulatory posture means data can't leave their infrastructure. That's not a preference — it's a line-item requirement for regulated clients.

**System-of-record creep.** Some platforms also want to own your prompts and your evaluation datasets. That competes with your git-based workflow and creates a dependency that's painful to unwind. Take the tracing. Keep everything else in git.

Quick check
1 question · instant feedback
0/1
  1. You log summaries of tool responses only. What's the risk?

How to run it

  1. Model calls
    Full prompt (post-template, exactly as sent), full completion, model ID, token counts in/out, latency, computed cost.
  2. Tool calls
    Tool name, arguments, raw result — stored in the trace even when the observe step summarises it for context (this is the escape hatch that makes debugging possible) — plus error/status and latency.
  3. Run-level
    Trace ID propagated everywhere, prompt version (§6), model/config versions, total cost, total latency, stop reason.
  4. Outcome (when you can get it)
    Thumbs up/down, human edit made, task completed. Sparse but gold — it's what lets you find bad traces to error-analyse rather than sampling randomly.
Quick check
1 question · instant feedback
0/1
  1. Best way to report cost to a client?

In the field

🔬Worked example
Add tracing to §1's audit agent: every MCP tool handler gets a ~10-line wrapper — start span, record args, call through, record result/error/latency, close span, propagate trace ID. The loop harness records each model call the same way. Total instrumentation: an afternoon. First payoff, typically within a week: a run that 'worked but cost $4' turns out to be re-fetching the same 90-day performance report on every step because the observe step summarised away the fact it already had it — a bug invisible in any output, obvious in one trace read.
🚫When not to reach for it
Not defensible: building agents with no trace capture at all. A defensible v0 is very simple — structured JSON logs, one line per span, into the logging you already have; upgrade when reading them hurts. Decline the 'system-of-record creep' of platforms that also want to own your prompts or datasets — take the tracing, keep git as source of truth.
Quick check
1 question · instant feedback
0/1
  1. A regulated client can't ship trace data to a hosted platform. Options?

Pitfalls & takeaways

Failure modes

  • Logging summaries only. The digest is in the trace but the raw tool result isn't — now you can't tell whether the tool or the summary was wrong.
  • No trace-ID propagation. All the spans, no way to stitch a run together.
  • Metrics without traces. The dashboard-only failure: you'll know something broke, and nothing else.
  • Sensitive data in traces. Prompts contain client data; traces get shipped to a third-party platform nobody put on the risk register. Traces are data — residency, retention, and access rules apply.

Durable takeaways

  • Traces beat dashboards for causal understanding.
  • Cost per trace, not aggregate — outliers are 40× the median.
  • Traces are data: residency, retention, and access rules apply.
  • Weekly drift probes catch slow rot no single deploy caused.

Do the work

🏋️Prove you learned it

Instrument your MCP with span logging — platform or structured JSON — and write one paragraph defending your choice in infrastructure terms. Then produce a one-page cost & behaviour profile from 20 traced runs. That doc IS a client deliverable format.

0 chars
📦Artifact to produce
Cost & behaviour profile from 20 traced runs: cost per task (median + p95), latency per span type, tool-call distribution, one anomalous trace with a written diagnosis.