AI Agent Latency Engineering:Making Multi-Step Agents Feel Fast in 2026

A model provider's own benchmark page will tell you how fast a single call to a single model runs. That number is close to useless for predicting how fast your agent feels, because an agent is rarely a single call — it is a chain of model calls, tool invocations, and orchestration decisions, each contributing its own latency, and the chain's total time is not the model's per-call speed multiplied by the number of steps. It is that, plus every tool round trip, plus whatever time the orchestrator itself spends deciding what happens next, plus — in a parallel design — however long the slowest branch takes to finish.

Teams that optimize the model in isolation — switching providers, upgrading to a faster tier, fine-tuning for shorter outputs — routinely see no change in what users actually experience, because the model was never the bottleneck. The bottleneck was a serial chain of tool calls that could have run concurrently, a system prompt bloated enough to dominate time-to-first-token regardless of which model reads it, or an orchestration layer waiting on a slow sub-agent nobody had actually measured. This guide is about the levers that address the agent's architecture directly, not just the model underneath it.

If you are earlier in the decision of whether a multi-agent design is the right call at all — as opposed to one well-built agent — our multi-agent systems architecture guide covers that trade-off directly. This guide assumes you already have a multi-step agent, in production or close to it, and starts from the question of where its time is actually going.

Before touching any specific optimization, break your agent's total response time into its actual component parts. Most teams have never done this rigorously, which is exactly why the wrong lever gets pulled first.

Two of these deserve a closer look because they are the most commonly conflated. Time to first token is dominated by input processing — how much the model has to read, in the system prompt, the conversation history, and the tool definitions offered to it, before it can begin producing output — which is why a bloated system prompt or an enormous list of available tools quietly taxes every single call your agent makes, independent of the model's raw generation speed. Output token generation, by contrast, is dominated by response length and by how many reasoning tokens the model spends before its visible answer — a topic we cover in depth, from the cost side, in our LLM cost optimization guide, but which affects wall-clock time exactly as much as it affects the bill.

The honest exercise is instrumenting all five rows separately, in production, for each distinct workflow your agent runs — a one-step lookup and a five-step research task have completely different budgets, and averaging them together hides where either one is actually slow.

Two more contributors belong in this same budget even though they rarely get billed as "latency" line items. A retrieval-augmented agent pays a full search-and-rerank round trip before the model ever sees a token of context — our RAG for enterprise knowledge bases guide covers the retrieval architecture itself, but for this guide's purposes it is simply another row in the tool-call-round-trip category above, and often not a small one. And an agent with a large working memory — session history, a long-running task's accumulated state — pays a version of the same input-processing cost that drives time-to-first-token, which is exactly why the tiered memory and compaction strategies in our AI agent memory architecture guide double as a latency lever, not only a context-quality one.

Streaming does not make a model faster. It makes a slow response feel faster, by giving the user something to read while the rest of the answer is still generating — the gap between total completion time and perceived wait time is one of the largest, cheapest levers available, and it is purely a product decision, not a model or infrastructure change.

• Token-level streaming: The most basic form: render output tokens as they arrive rather than waiting for the full response. This alone converts a multi-second wait into a readable, progressively-appearing answer, and is table stakes for any chat-style agent interface in 2026.
• Reasoning/thinking-token visibility: Models that produce extended reasoning before a visible answer can surface that reasoning progressively rather than as one more silent delay — turning otherwise-invisible latency into something that reads as the agent visibly working, which measurably changes perceived wait even though it does not change actual completion time.
• Intermediate tool-call progress: On a multi-step agent task with no token stream to show — a background research or booking workflow — surfacing which step is currently running ("searching flights," "checking loyalty balance," "confirming price") does for a multi-step task what token streaming does for a single response: it replaces an opaque spinner with visible progress.

The limit of streaming is worth stating plainly: it improves perceived latency, not the underlying latency budget from the previous section. A background task with no user watching in real time gets none of this benefit, and total completion time — not TTFT, not perceived responsiveness — is the number that actually matters there.

Most agent frameworks and hosted APIs now support a model emitting several tool calls from a single turn and executing them concurrently rather than one after another — turning a serial chain of round trips into a single round trip bounded by the slowest individual call. The OpenAI Agents SDK, for instance, ships an explicit agents-as-tools pattern for exactly this kind of delegation, alongside its core handoff mechanism for passing control between agents.

The catch is that this only helps when the calls are genuinely independent. A workflow that looks up a customer, then that customer's specific order, then that order's specific shipment status is sequential by construction — each step needs the previous step's result as its input — and no amount of concurrency support changes that. Parallelizing dependent calls is not possible; the actual fix for a sequential chain like that is reducing the number of hops, not running unrunnable steps at the same time.

Frenchy Digital's own Beyond Points AI build is a concrete, published example of this in practice: its Claude-based orchestrator delegates to specialized loyalty, transfer, browser, flight-booking, and hotel-booking sub-agents, and its own measured numbers — 4.7 orchestrator calls and roughly nine HTTP calls per search on average, across thirty real pre-beta searches — are the kind of concrete tool-call graph this exercise is meant to produce for your own agent, not a hypothetical one.

Speculative decoding is the closest thing on this list to a free lunch, because it is a scheduling optimization, not a quality trade-off: a small, fast draft model proposes several candidate next tokens, and the large target model verifies all of them in a single parallel forward pass rather than generating them one at a time. Any proposed tokens that match what the target model would have produced on its own are accepted immediately; a mismatch simply falls back to normal token-by-token generation from that point forward. Because the target model still governs every accepted token, the technique preserves its exact output distribution — it is mathematically not an approximation.

It has moved from research paper to production default in the current generation of serving engines. vLLM supports several speculative decoding variants directly — including n-gram, suffix, and EAGLE-based drafting — alongside its PagedAttention memory management and continuous batching. SGLang lists speculative decoding as a core runtime feature alongside its RadixAttention prefix caching, which it separately reports delivering up to 5x faster inference on cache-hit workloads. Google's own research team, credited with introducing the technique in a 2022 paper, has reported deploying it in production surfaces including AI Overviews specifically for faster responses at unchanged output quality.

The commonly cited speedup is in the 2 to 3x range, though the actual gain on any specific request depends on how often the draft model's guesses land — a draft model well-matched to the target model's style and domain accepts more tokens per pass than a poorly matched one, and most production deployments today still use a fixed speculative token count rather than tuning it per request, which leaves further gains on the table for teams willing to tune it. Either way, if you are self-hosting on vLLM or SGLang and have not confirmed speculative decoding is switched on for your specific model pairing, that is a lower-effort win than most of what follows in this guide.

A distinct lever from anything algorithmic: running the same open model on inference hardware purpose-built for speed rather than general-purpose GPUs. Three vendors dominate this conversation, and all three changed corporate status meaningfully in the past year — details worth getting right before any of them factors into a build decision.

Groq's situation is the one most likely to be misreported, so it is worth stating precisely: on December 24, 2025, Groq and Nvidia announced a non-exclusive licensing agreement for Groq's inference technology, under which Groq founder Jonathan Ross, president Sunny Madra, and other senior leaders joined Nvidia to help scale the licensed technology, in a deal reported at roughly $20 billion. Groq itself was not acquired: it continues operating as an independent company under new CEO Simon Edwards, and its GroqCloud inference service was explicitly excluded from the transaction and continues without interruption. In February 2026, Groq separately distributed $7.6 billion to shareholders as the first payout under that agreement. Cerebras took a different path entirely, going public on Nasdaq on May 14, 2026 under ticker CBRS, raising $5.55 billion. SambaNova remains privately held, having closed the first tranche of a $1 billion Series F led by General Atlantic at an $11 billion valuation on July 8, 2026.

The decision this table actually informs is narrower than it looks: custom inference silicon is worth the integration effort — a different API surface, a less portable deployment, sometimes a different model catalogue — only when a workload is genuinely latency-bound at high enough volume that the speed difference changes the product, not merely when a benchmark chart looks impressive. For most teams, the standard GPU serving row in this table, with speculative decoding and modern attention optimizations already switched on, closes most of the practical gap without adding a new vendor dependency.

A different agent shape sidesteps this entire vendor decision by removing the network round trip altogether: running a small model directly on the user's own device. That trade-off — battery, thermal budget, and model-quality limits in exchange for zero-network latency — is a genuinely separate engineering question from anything in this table, and we cover it on its own terms, including when it wins and when it does not, in our on-device AI and edge inference guide.

Routing sends each request to the cheapest model capable of handling it well, and because a smaller model is both cheaper and faster per token, the same routing decision that saves money also tends to save time — the two effects are correlated, not separate levers to weigh independently.

RouteLLM, built by the LMSYS team behind Chatbot Arena, is the most-cited academic example: its routers are trained on human preference data from the Arena dataset, using a strong/weak model pair as the routing target, and the project reports reducing cost by up to 85% while retaining 95% of the stronger model's benchmark performance on MT-Bench — a specific, named methodology rather than an unsourced industry figure. Commercial products including Martian, Not Diamond, and OpenRouter's own Auto Router expose a similar capability as a drop-in, provider-agnostic endpoint, typically with a cost/quality trade-off dial rather than a fixed threshold.

The production pattern most teams converge on layers three tiers: a cheap rule-based or keyword pass to catch the obviously simple cases without invoking a model at all, an embedding or lightweight-classifier pass for the ambiguous middle, and a cascade for the long tail — answer with the fast, cheap model first, escalate to the expensive model only if a confidence check or an explicit verification step fails.

Prompt caching is usually discussed as a cost lever, and it is one — but it has a distinct, more limited latency effect worth separating out. When a cached prefix is reused (a long system prompt, a large tool-definition list, a document the agent references repeatedly across a session), the model skips reprocessing those tokens from scratch, which directly shortens time-to-first-token on that specific call, because TTFT scales with how much input has to be processed before generation can begin.

What caching does not do matters just as much: it has no effect on output-token generation speed, so it does nothing for total completion time once generation is underway, and it provides no benefit at all on the first call in a session, before any prefix has been warmed. For a background, multi-step agent task where most of the wall-clock time is spent on output generation and tool round trips rather than on repeated TTFT, caching's contribution to the overall latency budget can be genuinely small even when its contribution to the cost bill is large.

We cover the cost mechanics of prompt caching in full — the write-versus-read pricing multiplier, the break-even math, and where teams get the economics wrong — in our LLM cost optimization guide. This section exists specifically so that caching's latency effect and its cost effect are not conflated into one another, since a team can genuinely nail the cost side while barely moving the number users actually feel.

Every layer of orchestration between a user's request and a finished answer adds its own latency: the time the orchestrator itself spends deciding what to delegate and to whom, the round trips required to spin up and collect results from sub-agents, and — in any design that fans work out in parallel — the time the whole system waits on whichever branch finishes last, which is always at least as slow as the slowest individual sub-task and often slower once coordination overhead is added on top.

Anthropic's own published account of building a multi-agent research system is direct about the shape of this trade: parallelizing work across specialized sub-agents genuinely cuts wall-clock time on complex, breadth-first tasks — the kind where several independent lines of investigation can run at once — at a token cost the company has reported running several times higher than an equivalent single-agent approach, since every sub-agent carries its own context window and its own overhead rather than sharing one. That is not a criticism of the pattern; it is the actual price of the wall-clock win, stated honestly.

We go into the specific architectural patterns, the token multiplier in more detail, and the cases where a single well-built agent still wins outright in our dedicated multi-agent systems architecture guide rather than repeating that analysis here. The latency-specific takeaway is narrower: parallel delegation only pays off, on a wall-clock basis, when the delegated sub-tasks are genuinely independent of each other and the orchestrator's own decision-making overhead is small relative to the work being delegated — the same genuinely-independent test that applies to parallel tool calls, one layer up the stack.

A related, often-overlooked contributor: agents that reach tools through the Model Context Protocol add a discovery and negotiation round trip of their own on top of the tool call itself, since an MCP client and server exchange capability information before the actual tool invocation happens. That overhead is generally small per call, but it compounds across a workflow with many distinct tool calls in the same way every other per-hop cost in this guide does — worth including explicitly in the latency budget breakdown from earlier rather than assuming it is negligible without measuring it.

Given everything above, a reasonable build order treats measurement as the prerequisite step, the genuinely free wins as the obvious next move, and everything with a real trade-off — routing, custom silicon — as a deliberate decision made with your own data, not a default.

The order matters more than any individual step: teams that skip step one and jump straight to routing, custom silicon, or a faster model provider are optimizing blind, and frequently discover — after spending real engineering time and vendor-integration effort — that their actual bottleneck was a serial tool-call chain or an oversized system prompt that step two would have fixed for free.

An average latency number hides exactly the data that determines whether an agent has a reputation for being slow. Track p50 (typical experience), p95, and p99 (the slowest 5% and 1% of requests) as separate figures, for both time-to-first-token and total completion time, and do it per workflow type rather than as one site-wide blend — a one-step lookup and a five-step research task have entirely different acceptable latency profiles, and averaging them together produces a number that describes neither one accurately.

Multi-step agents are especially prone to a long tail: a single slow tool call, a single retry, or a single cascade escalation on an otherwise fast run can blow out that one run's total time while barely nudging the average across thousands of runs — which is exactly why the p99, not the average, is the number that correlates with users describing your agent as unreliable or slow, even when most runs are fine.

This measurement discipline belongs inside the same evaluation infrastructure covered in our AI agent evaluation and observability guide — eval sets, LLM judges, and cost-aware scoring — rather than existing as a separate, one-off latency dashboard nobody revisits after launch.

Whether the claim comes from a model provider, an inference-silicon vendor, or your own team's internal dashboard, the same scrutiny applies.

Latency engineering scopes the same way the rest of our agent-architecture work does: a discovery phase that measures before anything changes, a single-workflow build for teams with one clear bottleneck, a platform build for teams running several agent workflows against shared infrastructure, and an enterprise band for organizations with a real SLA to hit.

Senior-led delivery runs $150 to $225 per hour, retainers run $2,500 to $9,500 per month, and every engagement carries a 30-day post-launch warranty. Book a discovery call at calendly.com/frenchydigital/discovery-call or call +1 (424) 272-5601, and you receive a written, fixed-price phased proposal within five business days.

This guide is honest about what it does and does not establish. We did not run our own independent benchmark of Groq, Cerebras, SambaNova, or any standard GPU serving stack on a controlled workload — the throughput and latency figures attributed to each in the silicon section are the vendors' own published claims or named third-party comparisons, not audited by us, and are presented as such rather than as verified facts. Inference speed also shifts quickly and by workload: a figure accurate for one model, prompt length, and concurrency level in September 2026 may not transfer to a different model or a different traffic pattern, which is exactly why the guide repeatedly recommends reproducing any vendor's claim on your own actual workload before it factors into a build decision.

We also did not find a single authoritative, cross-vendor, apples-to-apples latency benchmark that controls for model, prompt length, and concurrency simultaneously across Groq, Cerebras, SambaNova, and standard GPU serving — the comparisons that exist, including our own sources below, each use their own methodology, and we have not reconciled them into one ranking, because doing so honestly would require running the comparison ourselves rather than aggregating others' numbers. A reader making a real vendor decision should treat every figure in the silicon table as a starting point for their own reproduction, not a final answer.