Latency (AI response time)
Definition
Latency is the time an AI system takes to respond, from the moment a request is sent until the answer, or its first words, arrive. In agent workflows it adds up across every model call and tool call in a run.
By Ronan Liedmeier, fractional head of growth and course author · Published 17 October 2026 · Last updated 17 October 2026
Also called: AI response time, time to first token
People feel it as waiting, and in automated workflows it decides whether a job finishes in seconds or in minutes. It moves with the same things that drive cost: how many tokens go in, how many come out, and how many calls a task makes.
How it works
A single model call has two parts: the wait before the first token appears, and the time to generate the rest. Larger models are also slower. An agent loop multiplies this, because each step waits on a model call and often a tool call, and many loops re-send the full context on every turn.
Teams track a typical value and a slow-tail value such as p95, the time within which 95 percent of runs finish, because the slow runs are the ones people complain about. A dashboard shows that the slow tail got worse. A trace of one run shows which step caused it.
Example
A support lead builds a workflow that drafts replies to customer tickets. Drafts start arriving so late that agents write their own. Reading five traces shows the agent searching the help center six times per ticket, after a prompt edit removed the line that capped it at two searches. Putting the cap back brings response time down with the same model.
Common mistake
Chasing speed before checking that the answers are right. A smaller model or a trimmed context can cut latency and quality at the same time. The order that works: confirm correctness with an eval, cut waste such as repeated context, route simple high-volume tasks to faster models, then add caching.
Learn it in AI Academy
- Cost & Latency AwarenessAdvanced · 13 min
- Observability & TracingExpert · 8 min
Related terms
- AI observability
- AI observability is the practice of recording what an AI system does on every run, including each model call, tool call, cost and time taken, so you can see why it behaved the way it did. It is how teams debug and monitor AI once it runs on real work.
- Trace
- A trace is the step-by-step record of a single run of an AI workflow or agent, showing every model call and tool call in order with its inputs, outputs, errors and timing. Reading one shows exactly where a run went wrong.
- Prompt caching
- Prompt caching is an AI API feature that stores the processed start of a prompt you send repeatedly, so later requests beginning with exactly the same text are cheaper and faster. Only the repeated beginning is reused; the answer is still written fresh each time.
- Agent loop
- An agent loop is the repeating cycle an AI agent runs to finish a task: it plans a step, calls a tool, reads the result and decides whether to continue or stop. The loop ends when the goal is met or a limit is reached.