From LLM Inference to Agentic Workloads

TL;DR

• Conventional LLM serving assumes one request = one short, stateless model call. Agentic apps break that assumption: one user request becomes a long-running session that loops through model calls, tool calls, sandboxes, browsers, and stored state.
• The authors built AgentSysBench: ten real agentic applications (RAG, DeepResearch, Claude Code, Codex, browser/GUI agents, and more) wired into one instrumented, swappable serving stack, plus 24-hour production traces from three deployed agents.
• They measured six properties that make agentic workloads different, the biggest being: non-LLM components dominate latency in half the apps, sessions sit idle-but-live for minutes to hours holding expensive state, and a “control-plane tax” (tool schemas, observations, safety checks, re-prefills) quietly burns tokens and money.
• The headline: the bottleneck is not a fixed property of an app. It shifts with the model you pick, the payload size, the CPU cores, and where you place components. They formalize this as Y = Φ(W, S) — behavior is an interaction of the workload and the serving system, not a property of either alone.
• Four simple, characterization-guided fixes prove the findings are actionable: task-aware serving (−29 to −40% latency), communication-aware placement (up to 4.5× faster), state offloading (4.6× less memory), and tool-result caching (removes 35.2% of redundant search calls).

Problem & Motivation

If you serve agents for clients, you have already felt this pain even if you never named it: your GPU dashboard looks calm, but end-to-end latency is terrible and the cloud bill is ugly. That is the exact gap this paper fills.

The concrete pain: serving systems for agents are still designed with assumptions inherited from single-shot chatbot inference. A chatbot request is one stateless model call the system can batch, finish, and forget. An agentic request is a long-lived execution graph — chains, branches, loops, parallel sub-tasks — that interleaves model calls with tool calls, sandbox commands, browser actions, and persistent state updates. A single coding or research request can fire dozens of model calls and run for minutes to hours.

Because nobody had systematically measured these workloads, builders could not answer basic questions:

• What fraction of my latency is actually in the model versus the tools?
• How big does session state grow, and how long does it stick around?
• Does the bottleneck stay put, or move as conditions change?
• How much of my token spend is productive versus framework overhead?

Prior work left three specific gaps the authors call out:

1. Agent-serving systems (Parrot, Autellix, ALTO, Teola/Ayo) each optimized one to three workflows under incompatible assumptions, so you cannot tell which findings are fundamental versus quirks of one setup.
2. Capability benchmarks (SWE-bench, AgentBench, WebArena, OSWorld) measure whether the agent solved the task but record almost no systems data — no latency breakdown, no memory footprint, no token accounting.
3. Recent measurement studies got closer but each ran on a single fixed serving stack, so a reported bottleneck might just be an artifact of that one hardware/placement choice.

The core insight behind the gap: you cannot characterize an agent from its code alone, because the behavior you observe depends on how it is served. The same app looks LLM-bound with a slow model, tool-bound with a slow sandbox, or network-bound with remote placement. To measure the workload honestly, you have to control both sides at once.

What’s New (Core Contribution)

Four genuine contributions. None is a new algorithm — the novelty is measurement done right at a scale and breadth nobody had done.

• A formal model of agentic serving behavior: Y = Φ(W, S). Before: studies fixed one side and reported bottlenecks as if they were app properties. Now: behavior Y (latency breakdown, bottleneck, cost) is explicitly an interaction of the workload W = ⟨R, T, M, O⟩ and the serving system S = ⟨H, C, A⟩. This turns “where is the bottleneck?” into “which factor am I holding fixed?” (details below).
• AgentSysBench — a benchmark suite that varies both sides. Before: 1–3 apps on a frozen stack. Now: ten representative agents chosen to span the workload space, plus a modular serving stack where you can swap the model engine (vLLM vs SGLang), move a component from in-process to a remote service, change CPU/GPU allocation, and co-locate or distribute — all while holding everything else constant (“factor isolation”).
• A unified, component-level measurement toolkit. Before: end-to-end numbers only. Now: every LLM call, tool call, and state operation emits a normalized trace record (type, component, start/end, I/O size, tokens, cost). It works on white-box apps (via code annotations) and black-box third-party agents (via LLM/tool proxies and SSH sandbox hooks) into the same schema — so a Claude Code run and a RAG run become directly comparable.
• Production-trace complementarity. Before: synthetic, run-to-completion benchmarks. Now: 24-hour traces from three deployed agents (178,799 sessions in one day) surface three behaviors lab benchmarks structurally cannot: long idle-but-live gaps, the control-plane tax, and cross-request redundancy.

The unifying finding — the one sentence worth remembering — is: model inference is no longer the sole cost center, and the dominant cost shifts across requests, models, and deployments. Efficient agent serving therefore needs coordinated management of models, tools, state, and communication, not model-centric optimization alone.

How It Works (Technically)

There are three things to understand: (1) the Y = Φ(W, S) model that frames everything, (2) how the benchmark actually captures a run, and (3) the six measured properties that are the real payload.

1. The framing model: behavior is an interaction, not a property

The paper models the serving system as S = ⟨H, C, A⟩:

• H — Hardware. GPUs, CPUs, DRAM, storage, network bandwidth.
• C — Component-serving mechanisms. How each piece is served: which LLM engine (vLLM/SGLang), which vector DB, how the sandbox manager runs, batch sizes, prefill/decode strategy.
• A — Deployment architecture. Where components live: co-located on one box, separately containerized, or remote cloud services.

And the workload as W = ⟨R, T, M, O⟩:

• R — Request distribution. Task type, difficulty, payload size.
• T — Tools and environments. Sandboxes, browsers, vector DBs — this sets your I/O, state footprint, and non-LLM bottlenecks.
• M — Model choices and inference policies. Which models, which decode settings — sets latency, token volume, cache behavior.
• O — Orchestration structure. Predefined pipeline vs ReAct loop vs planner–executor — sets how dynamic, parallel, and stateful the run is.

The measured behavior is Y = Φ(W, S). Read it in plain English: what you observe is a function of the workload run through the serving system. Change any single letter — a faster model (M), a bigger document (R), more CPU cores (H), remote placement (A) — and the bottleneck can move to a different component. This is the paper’s spine; every experiment is “hold all letters fixed, wiggle one, watch Y move.”

2. How a run is captured

At runtime a generated request flows: workload generator → orchestrator → (LLM calls, tool calls, environment interactions) → components. The toolkit instruments this path at four layers (orchestrator, proxy, sandbox, container) and links every operation to its resource footprint. Two collection paths feed one schema:

• White-box apps (RAG, HuggingGPT, DeepResearch, Mini-SWE, WebAgent): source is modifiable, so lightweight annotations emit trace records directly.
• Black-box apps (Codex, GUIAgent, Claude Code, Openclaw, Pi-AutoR): can’t touch the code, so an LLM/tool proxy intercepts API traffic and SSH sandbox hooks intercept shell actions.

Both paths produce the same normalized per-operation record. On top of that, standard observability (cAdvisor for CPU/mem/disk/net, NVIDIA DCGM for GPU, Prometheus for time-series) captures resource usage. After a run, the analysis library normalizes traces and computes the component-level latency and cost breakdown. That breakdown — attributing every millisecond to a component — is the core technical primitive.

Architecture & data flow

flowchart LR
  WG[Workload generator<br/>owns R: task + arrival] --> ORCH[Orchestrator<br/>owns O: pipeline/ReAct/plan-exec]
  ORCH -->|LLM calls M| LLM[LLM engine<br/>vLLM / SGLang · GPU]
  ORCH -->|tool calls T| SBX[Sandbox<br/>shell / filesystem · CPU]
  ORCH -->|retrieval T| VDB[(Vector DB + embed<br/>memory-bound)]
  ORCH -->|search/fetch T| EXT[External services<br/>search · web · network-bound]
  LLM -. trace .-> COL[Instrumentation<br/>per-op trace records]
  SBX -. trace .-> COL
  VDB -. trace .-> COL
  EXT -. trace .-> COL
  COL --> AN[Analysis library<br/>Y = component latency / cost breakdown]

Schematic of Figure 4 — end-to-end time split by component across the ten apps. Blue is the LLM; everything else is tools, sandboxes, search, and orchestration. In half the apps the non-LLM slices dominate. Bars reflect the paper's qualitative claims (e.g. GUIAgent sandbox >70%, Pi-AutoR runtime ~90%), not exact per-app figures.

3. The six measured properties (the real payload)

Finding 1 — Heavyweight, non-LLM-dominated execution. Agentic runs span seconds to hours (Mini-SWE coding tasks often exceed 10 minutes; Pi-AutoR research runs reach hours). Decomposing the time: in 5 of 10 apps, non-LLM components dominate the critical path. GUIAgent’s desktop sandbox is >70% of total time; Pi-AutoR’s experiment runtimes hit 90%. Implication: model-only optimization leaves most of the clock untouched, and long runs make simple retry-on-failure prohibitively expensive — you need cheap checkpoint/resume.

Finding 1b — Token usage grows super-linearly. In a ReAct loop, every step re-appends the whole history (system prompt + all prior tool outputs + observations) to the context. So input size scales roughly quadratically with turn count. This makes prefix caching mandatory, not optional: apps with static prompts and append-only history (Claude Code) hit 99% prefix-cache hit rates; apps that reshuffle context every turn (DeepResearch) drop to 1% or lower. The cost then shifts to managing the memory of those caches.

Finding 1c — Three kinds of state, managed differently. This distinction is genuinely useful for builders:

• Performance state — the KV cache. Evictable without breaking correctness (you can re-prefill), but re-prefill is slow. A Claude Code session on a big model can hold up to 11 GB of KV cache.
• Persistent correctness state — per-session vector DB collections (median 7.4 MB, up to 49 MB), filesystem changes, installed dependencies, and external side effects (a sent email). Lose it and you break execution semantics.
• Active working-set memory — transient DRAM during a command. Median sandbox peak ~0.8 GB, but peaks at 28 GB during compilation and unit tests. This is a provisioning spike, not something you checkpoint.

The clever move: because agents alternate active steps and idle gaps, snapshot correctness state during idle intervals (when the agent is waiting on the LLM), never during the 28 GB compile peak. Symmetrically, page out the GPU KV cache of a session that is idling on a long tool call.

Finding 2 — Cross-stack heterogeneity. One app mixes GPU-bound LLMs, CPU-bound sandboxes, memory-bound vector DBs, and network-bound search. Even tasks on the same component diverge wildly: in DeepResearch, Embed-Doc is 32× slower than Embed-Query (its payload is 325× larger); within one Mini-SWE trace a pip install sandbox call is 171× slower than a sed. The driver is input/output scale, not the component. Mixing these in one queue causes head-of-line blocking (a 128K-token embed job makes a small query wait, up to 35.7× slowdown), co-batching interference (one short LLM request slows 1.8× when co-batched with two long-context ones), and shared-resource contention (8 sandboxes sharing 128 CPUs run 1.22× slower than pinned).

Finding 3 — Shifting bottlenecks (the thesis). The bottleneck is emergent, not intrinsic. Wiggle one factor, watch it move:

• Request type (R): Claude Code on MCP-Atlas is LLM-bound (up to 90%) for Movie/BI/DB tasks but tool-bound (up to 84%) for ETL/Wiki/Map.
• Payload size (R): growing RAG documents shifts the bottleneck from vector-DB storage → embedding.
• Tool set (T): giving WebAgent all three observation formats vs one raises per-step input tokens 4.8× and pushes LLM share from 46.9% → 61.6%.
• Model (M): swapping DeepResearch’s writer from fast Flash to slow Pro moves the bottleneck from embedding → LLM, and total time from 10.0h → 14.0h — driven by model speed, not token count.
• Hardware (H): more CPU cores turns a sandbox-bound Codex task LLM-bound.
• Serving mechanism (C): raising SGLang batch size 1→4 makes GUIAgent’s LLM 4× slower and turns it into the bottleneck — a shift from one engine knob.
• Deployment (A): distributing RAG across machines at concurrency 10 congests the network, which grows from negligible to 67.5% of latency.

Interactive: this is Y = Φ(W, S). Toggle a factor and watch which component becomes the bottleneck. The point of the paper in one widget — static profiling can't track this, so serving must adapt per-request.

Finding 4 — Long idle-but-live intervals (production only). Across 35,037 coding-agent sessions, the median session executes for only 20% of its lifetime; 70% execute for less than half. Idle gaps run from seconds to hours (overnight pauses), most between 1–10 minutes — all while the sandbox, terminal, KV cache, and history stay allocated. Implication: a binary “running vs finished” lifecycle is wrong; you need a third “waiting” state so you can reclaim resources without losing resumability.

Finding 5 — The LLM control-plane tax. Beyond productive work, tokens burn on framework machinery in three forms:

• Context-capacity tax: at session step 1, system messages (role, tool schemas, memories, directives) are 99.7% of input; by late steps, accumulated history is 84.3%. One late step emitted 151 output tokens but had to prefill 166,721 context tokens.
• Auxiliary-compute tax: the harness makes extra LLM calls for context compaction (summarizing history), safety guardrails, and loop detection. One compaction averages 176K input + 5K output tokens and 156 s latency (p99 775 s). Across 35,037 sessions, auxiliary tasks alone added 2,684 calls, 6.5M input tokens.
• Cache-reprefill tax: KV cache has a fixed 5-minute TTL, but human-paced idle gaps of 1–10 minutes routinely exceed it. 59.4% of sessions hit at least one eviction, and evictions account for 31.5% of aggregate monetary cost.

Finding 6 — Exploitable cross-request redundancy. Across a production search agent’s 373,678 search queries, 27% of distinct queries recur and account for 67.3% of all search calls. In an Openclaw-like agent, 24% of distinct URLs recur and account for 64% of fetches. This is invisible in single-task benchmarks (it only appears across users/sessions) and is a huge caching opportunity at the shared tool boundary.

The algorithm, simplified

There is no single “algorithm” — the contribution is the measurement loop. Here is its heart: instrument an agent’s ReAct loop so every operation becomes a trace record, then attribute end-to-end latency to components. This is what turns “my agent is slow” into “embedding is 45% of my clock.”

# The core primitive: run an agent, emit a per-operation trace, then
# decompose end-to-end latency by COMPONENT. This is Y = Φ(W, S) made concrete.

def run_and_trace(task, tools, llm):
    trace = []                                  # normalized per-op records
    ctx = build_initial_context(task)           # system prompt + tools schemas (control-plane!)
    t_start = now()

    while not done(ctx):
        # --- LLM call (component = "llm") ---
        t0 = now()
        step = llm(ctx)                          # reason + choose next action
        trace.append(rec("llm", t0, now(),
                         in_tok=count(ctx), out_tok=count(step)))

        if step.action is None:                  # agent decided it's finished
            break

        # --- tool / sandbox / search call (component varies) ---
        comp = step.action.component             # "sandbox" | "search" | "vecdb" | "embed" ...
        t0 = now()
        obs = tools[comp].call(step.action.args) # execute in the real environment
        trace.append(rec(comp, t0, now(),
                         in_kb=size(step.action.args), out_kb=size(obs)))

        ctx = ctx + step.text + obs              # <-- append-only growth: input scales ~quadratically

    total = now() - t_start
    # Attribution: what fraction of the wall clock did each component own?
    by_comp = {}
    for r in trace:
        by_comp[r.component] = by_comp.get(r.component, 0) + (r.end - r.start)
    breakdown = {c: dur / total for c, dur in by_comp.items()}
    return breakdown                             # e.g. {"llm": 0.31, "embed": 0.45, "vecdb": 0.18, ...}

The whole paper is: run this across ten apps and many settings of W and S, and read breakdown. The six findings are what you see when you do.

Built on Prior Work

Results & Evidence

Scale of the study: 4,641 benchmark requests, 64,924 LLM calls, 118,274 tool calls in controlled experiments; 178,799 production sessions in one day across three deployed apps.

The four design explorations (deliberately simple, to prove the findings are actionable rather than to build a full system):

What the evidence establishes: across ten real apps and production traces, the non-LLM stack is a first-class cost center and the bottleneck genuinely shifts with each factor. This is robust because it’s shown from multiple independent angles.

What it does NOT establish — read this before you over-index on the numbers:

• The four fixes are proof-of-concept, not a system. They’re isolated interventions against workload-oblivious baselines. The gains don’t compose automatically, and the baselines are intentionally naive.
• State offloading’s 4.6× uses an inter-component proxy, not the human pauses. The paper is explicit: the Mini-SWE experiment offloads during LLM planning calls (seconds), not the minutes-to-hours production idle gaps that motivate it. The production-scale win is argued, not measured.
• Memory numbers aren’t apples-to-apples. The 4.6× offloading figure is aggregate resident memory across concurrent sandboxes, not the 28 GB per-session peak — the paper flags this itself.
• Models are mostly a specific stack. Heavy reliance on DeepSeek-V4 variants and Alibaba-Bailian/Qwen APIs; some numbers (Claude Opus 4.6 pricing for the cost model) are estimates under public list prices, not billed costs.
• Production traces are three apps, one day each. Redundancy rates (27%, 24%) are real but app- and day-specific; your cache hit rate will differ.
• No open artifact yet at time of writing — “will be released as open source.” You can’t reproduce it today.

Net: trust the qualitative claims (non-LLM dominates, bottlenecks shift, idle-but-live is real, control-plane tax is large, redundancy is exploitable). Treat the specific multipliers as existence proofs, not benchmarks for your stack.

How You’d Use It

This paper is unusually direct fuel for an AI-services business, because it names the exact places margin leaks when you operate agents for clients. Five concrete moves:

• Sell an “agent serving audit” as a productized offering. The measurement toolkit is your methodology. Instrument a client’s agent (proxy + sandbox hooks for black-box, annotations for white-box), produce the component-level latency/cost breakdown, and hand them the one chart they’ve never seen: where their money actually goes. Almost every client believes their spend is “tokens”; you show them it’s sandbox idle time, embedding payloads, and cache re-prefills. That chart sells the remediation engagement.
• Deploy tool-result caching first — it’s the highest ROI, zero-risk fix. No change to agent logic. A shared exact-match query cache plus URL-fetch dedup with a short TTL. If a client’s redundancy is anywhere near the paper’s 27%, you cut a fifth of their search latency and a big chunk of external-API bills in an afternoon.
• Fix the KV-cache TTL mismatch. The single cleanest finding: a fixed 5-minute cache TTL against human-paced 1–10 minute pauses causes 59% of sessions to eat re-prefill costs, ~31% of spend. Tuning TTL / adding reuse-aware retention to match your session idle distribution is a direct, measurable bill reduction.
• Add a “waiting” lifecycle state to your orchestrator. If you run your own agent harness (you built ARC MAS, so you do), stop treating sessions as binary running/finished. A third quiescent state lets you offload sandbox + terminal state to cheaper storage during idle gaps and reclaim GPU KV memory — directly raising how many concurrent client sessions one box holds.
• Right-size provisioning off the three-state model. Provision for the 28 GB working-set peak separately from the steady 0.8 GB, and snapshot correctness state during idle windows. This is the difference between an agent fleet that OOMs unpredictably and one you can capacity-plan.

The strategic read for positioning: “model-centric” optimization (cheaper tokens, faster GPUs) is a commoditizing race. The moat this paper points at is workload-aware orchestration of tools, state, and communication — harder to copy, and exactly the layer a services company owns.

Build Your Own (Minimal Recipe)

You don’t need the whole benchmark. The 80%-value version is an instrumentation + attribution layer around one agent, plus the two cheapest fixes. Build order:

1. A per-operation tracer (1–2 days). Wrap every LLM call and tool call to emit rec(component, t0, t1, in_size, out_size, tokens). For your own agents, annotate directly. For black-box agents, put an OpenAI-compatible proxy in front of the model endpoint and hook the sandbox (the paper uses SSH hooks). Dump records to JSONL.
2. An attribution report (half a day). Sum durations by component, divide by wall-clock. That’s the breakdown dict from the pseudocode. Render it as a stacked bar per session. This alone is a sellable deliverable.
3. Resource time-series (half a day). cAdvisor + DCGM + Prometheus in Docker, exactly as the paper does. Now you can overlay memory/GPU on the latency trace and see idle-but-live gaps.
4. Tool-result cache (1 day). A dict/Redis keyed by exact search query and by URL, with a TTL. Wrap your search/fetch tool. Log hit rate. This is the first fix and it pays for the whole project.
5. Idle-aware state offload (the hard part, ~1 week). Detect when a session is waiting (no active LLM/tool op) and snapshot its sandbox to disk, restoring on the next tool step. Getting checkpoint/restore correct across the LLM engine and the sandbox is the genuinely hard bit — filesystem, installed deps, and process state must all come back.

Libraries/models to reach for: Docker (component isolation), vLLM or SGLang (LLM engine with prefix caching — non-negotiable given super-linear token growth), Milvus or similar (vector DB), E2B or Firecracker microVMs (sandboxes with fast snapshot), Prometheus/Grafana (observability), LiteLLM (a clean proxy point to intercept model traffic). The two genuinely hard parts are (a) correct sandbox checkpoint/restore and (b) a normalized schema that unifies white-box annotations and black-box proxy traces into one comparable record.

How to Improve It

Limitations are leverage. Five testable directions, ordered by payoff:

1. Measure state offloading against real human pauses. The paper’s biggest gap: it proxies idle-but-live with second-scale inter-op gaps. Build a scheduler that offloads on the actual 1–10 minute production distribution, with resume prediction/prefetch to hide restore latency, and measure the true concurrency gain. This is a paper and a product.
2. Reuse-aware KV retention instead of fixed TTL. Replace the 5-minute TTL with a policy that predicts a session’s return time (from its idle-gap distribution) and keeps/offloads/evicts KV accordingly. Directly attacks the 31.5%-of-cost finding. Testable: cost per session vs baseline TTL.
3. Agent-native tool interfaces. The control-plane tax comes partly from re-injecting full tool schemas and raw observations every turn. Design tools that expose stable schema IDs, typed size-bounded observations, incremental deltas, and retrievable handles to big artifacts. Measure token/prefill reduction per session — the paper proposes this but doesn’t build it.
4. Semantic (not just exact) tool-result caching. The query cache is exact-match. Add an embedding-similarity tier so paraphrased queries hit too. Risk to measure: staleness and false-positive hits. Could push far past the 35.2% exact-match ceiling.
5. A cross-request scheduler that co-optimizes the fixes. The four interventions are isolated. Combine task-disaggregation, communication-aware placement, and state offload under one scheduler and measure whether gains compose or interfere — the obvious “full system” follow-up the authors deferred.

Glossary

• Agentic workload — the distribution of long-running, tool-using, stateful executions an agent app produces, versus one short model call.
• Y = Φ(W, S) — the paper’s model: observed behavior is a function of the workload W and the serving system S; the bottleneck belongs to neither alone.
• W = ⟨R, T, M, O⟩ — workload factors: Request distribution, Tools/environments, Models/inference policy, Orchestration structure.
• S = ⟨H, C, A⟩ — serving factors: Hardware, Component-serving mechanisms, deployment Architecture.
• ReAct loop — reason → act → observe, repeated; the standard agent pattern whose append-only context drives quadratic token growth.
• KV cache — the model’s stored keys/values for past tokens, so it doesn’t recompute them; large (up to 11 GB/session here) and evictable-but-costly-to-rebuild.
• Prefix caching — reusing the KV cache for an unchanged context prefix across turns; hit rate ranges 1%–99% depending on how the app manages context.
• Prefill vs decode — prefill processes the input prompt (scales with input length); decode generates output tokens one at a time (scales with output length). Agent input grows every turn, so prefill cost climbs.
• Head-of-line (HOL) blocking — a big slow request stuck at the front of a shared queue makes small fast ones wait behind it.
• Co-batching interference — the GPU advances a batch at the pace of its slowest/longest member, so a long-context request slows the short ones batched with it.
• Performance state — evictable cached artifacts (KV cache); losing it costs recompute, not correctness.
• Persistent correctness state — data you cannot lose without breaking the run: per-session vector DBs, filesystem changes, external side effects.
• Active working-set memory — transient DRAM a live command uses (peaks at 28 GB in compile/test); a provisioning spike, not a checkpoint.
• Idle-but-live — a session that is logically alive and resumable but doing no active compute, holding state while it waits (median session is idle 80% of its life).
• Control-plane tax — the extra serving cost of the agent harness itself: tool schemas and observations filling context, auxiliary LLM calls (compaction, safety, loop detection), and re-prefills after cache eviction.
• Context compaction — the harness asking the model to summarize a long history into a shorter replacement; cuts context >70% but averages 156 s per event.
• Task disaggregation — deploying each logical task (embed-query, embed-doc, llm-judge) as its own service with dedicated resources, so heterogeneous tasks stop blocking each other.
• Cross-request redundancy — different sessions issuing the same search query or URL fetch; ~27% of distinct queries cause 67% of calls — cacheable at the shared tool boundary.