Observability & Tracing

Lesson preview

Traces tell you why; dashboards only tell you that. A dashboard can show you that p95 latency doubled — but only the trace reveals the agent making 12 retrieval calls per run because a prompt edit quietly removed the 'search at most twice' instruction. When you have an hour: read five traces. Don't stare at charts. Cost tracking is per-trace, or it's fiction. Aggregate spend hides the one pathological run pattern that costs 40× the median. Track cost per trace, set alerts on outliers, and report cost-per-task to clients. 31' is a sentence that closes deals. 'We spent $200 last month' is not. Latency budgets per span type — retrieval under 500 ms, model calls under 8 s, total run under 90 s — turn 'it feels slow' into a diffable fact you can put in a PR description. Drift detection doesn't require a fancy platform. Take a fixed slice of your golden eval set and run it on a schedule — weekly is usually enough. Chart judge pass-rates over time. Providers update models under the same name without announcing it. Your corpus changes. User behaviour shifts. None of those events show up in a single deploy, but a weekly probe run will catch the slow rot before a client does.

Key terms in this module

AI observability

AI observability is the practice of recording what an AI system does on every run, including each model call, tool call, cost and time taken, so you can see why it behaved the way it did. It is how teams debug and monitor AI once it runs on real work.

Latency (AI response time)

Latency is the time an AI system takes to respond, from the moment a request is sent until the answer, or its first words, arrive. In agent workflows it adds up across every model call and tool call in a run.

Trace

A trace is the step-by-step record of a single run of an AI workflow or agent, showing every model call and tool call in order with its inputs, outputs, errors and timing. Reading one shows exactly where a run went wrong.

Frequently asked questions

Who is "Observability & Tracing" for?

This Expert module in AI Delivery Skills & Workflows is for operators, founders and growth teams who run AI in real work and want production-grade judgement. It takes about 8 minutes and ends with a 4-question quiz.

Do I need to code?

Not to follow the lesson. The concepts are explained in plain language; if you build AI systems yourself, the exercise shows where the ideas map onto code.

What comes before and after this module?

Before "Observability & Tracing", take "The Eval Workflow" (module 3 of AI Delivery Skills & Workflows). Afterwards, continue with "Orchestration: n8n at Depth".

Related modules

• Previous
  The Eval Workflow

  Error analysis → judges → regression in CI. The skill clients cannot get from a demo.

• Next
  Orchestration: n8n at Depth

  You're not building workflows. You're building skeletons you clone — the difference between selling hours and selling assets.

• Same topic
  The Debugging Discipline

  A five-trace drill: hypothesis first, answer overleaf.

• Same topic
  Debugging AI Workflows

  Find the failing step, build the minimum repro, fix the actual problem.