Cost & Latency Awareness

Every AI call is metered on tokens: what you send in (input) and what comes back (output). This sounds simple for a single chat message, but in agentic loops — where the model plans, acts, observes, and repeats — a large chunk of context often gets re-sent on every single turn, so a task that takes ten loop iterations can cost far more than ten times a single call, because each turn carries the accumulated history along with it.

Understanding this meter is the first step to controlling it: cost isn't just "how good is the model," it's "how much context does this workflow re-send, how often."

The most common cost leaks aren't exotic — they're mundane. A giant system prompt or instruction block re-sent on every single call, whole documents pasted in when only a section is relevant, and chatty agents that take many small steps instead of a few efficient ones, all quietly multiply spend without improving output quality.

These leaks are worth auditing specifically because they're invisible in day-to-day use — nobody notices a 3,000-token instruction block until they see it multiplied across thousands of runs on a bill.

Not every task needs the flagship model. Classification, labeling, and other low-judgment, high-volume tasks are often handled just as well by a cheap, fast model, while tasks requiring nuanced judgment, synthesis, or creative writing benefit from the frontier model. The skill is routing: sending each task to the cheapest model that can still hit your quality bar.

A practical pattern: a light model labels a batch of items into simple categories, and only the subset that actually needs deeper reasoning gets escalated to the frontier model — this can cut spend dramatically while keeping quality identical where it matters.

Many providers offer prompt caching: if the same stable prefix (like a long system prompt or set of instructions) is reused across calls, the provider can serve it more cheaply and faster than reprocessing it from scratch each time. This means stable, unchanging portions of your prompts are nearly free to repeat, while constantly-changing content pays full price every time.

The practical implication is to structure prompts so the stable part (instructions, examples, context that doesn't change) comes first and stays byte-for-byte identical across calls, while the variable part (the actual input for this run) comes last.

Latency and cost are cousins but not identical — a workflow can be cheap but slow, or fast but expensive. The main levers for speed are shorter prompts (less to process), smaller/faster models where quality allows, and running independent steps in parallel rather than sequentially when a workflow has multiple AI calls that don't depend on each other's output.

Parallelizing steps is often the highest-leverage latency fix and the one most people skip, because sequential pipelines are simpler to build even when they don't need to be sequential.

You can't optimize what you haven't measured. Pick your single busiest AI workflow and log, for a handful of runs, the tokens consumed (or the cost the platform reports) and the total time per run in a simple spreadsheet. This gives you a baseline before you touch anything.

This measurement habit matters more than any single optimization technique, because it turns "this feels expensive" into "this workflow costs $X and takes Y seconds per run," which is the only basis for deciding whether an optimization is worth the engineering time.

Optimize in this order, and don't skip ahead: cut waste first (trim bloated prompts, stop re-pasting whole documents), route models second (send tasks to the cheapest model that still passes your evals), and cache third (once the prompt structure is stable, let caching do its work on unchanging prefixes).

Critically, all of this happens after correctness is confirmed with your golden set from the evals module — a cheap, fast workflow that's wrong 20% of the time is more expensive once you count the cleanup than a pricier one that's right 95% of the time.