JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution

harness — memory, planning, action, and tool-orchestration code — for an off-the-shelf LLM on the spot, then repairs and evolves that harness from execution feedback, turning harness design itself into a learned, transferable skill instead of hand-engineering.

TL;DR

An LLM agent’s real capability comes from the model and its harness — the memory, planning, tool-exposure, and control-loop code wrapped around it. Today that harness is hand-built per task, which doesn’t scale. JIT-Agent is a compact model trained to generate a task-specific harness at inference time (Just-in-Time, not Ahead-of-Time), validate and self-repair it when generation fails, and evolve an archive of harness designs as it sees more tasks. Bolting JIT-Agent onto off-the-shelf backbones is a big lever: DeepSeek-V4-Flash + JIT-Agent beats GPT-5.6 on DeepSearchQA (+9.1 pts), PinchBench (+8.7), and OdysseyBench (+4.3), and GLM-5.2 + JIT-Agent gains up to +20.2 points over its own vanilla scaffold. Under a controlled, same-backbone comparison, JIT-Agent-generated harnesses also beat or match production runtimes like Claude Code and OpenCode while using 36% less tokens/cost on average — the harness, not just the model, is doing real work.

Problem & Motivation

Two agents running the same model can perform very differently depending on what history they keep, how they plan, what tools they can see at each step, and how they recover from failure — that wrapper is the “harness.” Right now harnesses are built by hand, one design per task family: a lean ReAct loop for terminal use, parallel search fan-out for web research, hierarchical memory for long documents, and so on. A growing line of work (“harness optimization”) automates this, but almost all of it is Ahead-of-Time (AOT): search or edit a single durable harness against a stream of past experience, then hope it generalizes to whatever task shows up next. That’s a bad fit when the right harness is genuinely instance-dependent — a shopping task with five constraints wants very different state-tracking than a six-clue identification puzzle. AOT methods must accumulate enough trajectories before they can commit to a design, and then bet that design transfers. The paper asks: instead of pre-compiling one harness, what if a trained model just generates the right harness for this exact task, right now?

What’s New (Core Contribution)

1. Model-as-a-Harness / Just-in-Time synthesis. Before: a harness is a durable artifact tuned ahead of time over accumulated experience. Now: a trained generator emits a fresh, task-conditioned harness at inference time, per instance, for whatever off-the-shelf backbone LLM you hand it.
2. A fixed, machine-generatable harness protocol. Before: harnesses are unconstrained agent programs — hard to generate reliably. Now: every harness is factored into four typed modules — memory (M), planning (P), action (A), capability orchestration (F) — under one shared interface (HarnessFactory), so generation is assembly over a typed design space, not free-form coding.
3. Learned repair, not just learned generation. Before: a failed generation is discarded. Now: Stage II trains the model on bounded repair trajectories (compiler errors, interface mismatches, runtime exceptions → patch), so JIT-Agent can fix its own broken output.
4. Evolvability as a trained objective (Evo-GDPO). Before: “evolution” in prior harness work means an external search/editing loop bolted onto a fixed scaffold. Now: the generator itself is RL-trained to propose harnesses that beat the current archive frontier on reward, latency, and cost simultaneously — evolution is a capability of the model, not a wrapper around it.

How It Works (Technically)

The four-module contract. Every harness h = (M, P, A, F) is required to obey the same runtime interface, so wildly different agent designs (a lean ReAct loop, a recursive sub-agent-spawning orchestrator, a DAG planner) all become coordinates in the same space instead of incomparable programs. At each step t the protocol runs, in order, M → P → F → A:

• v_t = M(history, s_t) — memory: compress raw event history + controller state into a view — the part of the past the model actually gets to see this step. Could be “everything” (FullHistory), a summarized rolling window, a retrieved subset, or a hierarchical fold.
• d_t = P(task, s_t, v_t) — planning: turn that view into a local directive (the current subgoal). Could be nothing at all (P∅, the “no explicit planner” case that keeps the type system consistent), a todo item, or the next node of a DAG.
• C_t = F(registry, s_t, v_t, d_t) — capability orchestration: filter the full tool/skill registry down to what’s actually exposed right now, conditioned on the directive — e.g. “only search tools until the evidence checklist is complete.”
• (s_{t+1}, e_t) = A(s_t, task, v_t, d_t, C_t) — action: the actual backbone LLM call. Reads the view, directive, and exposed capabilities; updates the controller state; emits either a tool call or a terminal answer.

A kernel then executes tool calls and appends the result to history, or — if the emitted action is terminal — stops and returns the answer. This is exactly what Rollout() in the paper computes: run the loop until a terminal output, producing a trajectory of (state, action, observation) triples. HarnessFactory reimplements 13 well-known scaffolds (ReAct, Plan-and-Execute, ReSum, Flash-Searcher, GAM, HiAgent, ROMA, AOrchestra, and others) inside this one interface — e.g. canonical ReAct is just (M_full, P∅, A_react, F_all) — which both proves the protocol is expressive enough and gives JIT-Agent 13 seed examples to imitate and recombine.

Architecture & data flow

flowchart TD
  H[(Event history ξ_<t\n+ controller state s_t)] --> M["Memory M\nhistory → view v_t"]
  M --> P["Planning P\nview → directive d_t\n(or null P∅)"]
  P --> F["Capability orchestration F\nfilters registry C_τ → C_t"]
  F --> A["Action A\nbackbone LLM call:\nupdates state, emits e_t"]
  A -->|"e_t = tool call"| K[Kernel: Exec]
  K -->|"observation o_t"| H
  A -->|"e_t = terminal output"| Y[Answer y]

The four-module loop running one step at a time: watch a task move through memory → planning → capability orchestration → action, then either loop back (tool call) or exit (terminal answer). This is Equations 3–7 in motion.

The rollout loop, simplified

# One step of executing a generated harness h = (M, P, A, F) under the fixed protocol.
def harness_step(h, s_t, history, task, registry):
    M, P, A, F = h
    v_t = M(history, s_t)                 # memory: raw history -> compact "view"
    d_t = P(task, s_t, v_t)                # planning: view -> local directive (or null)
    C_t = F(registry, s_t, v_t, d_t)       # which tools/skills are exposed right now
    s_next, e_t = A(s_t, task, v_t, d_t, C_t)   # backbone LLM call -> new state + action

    if is_tool_call(e_t):
        o_t = execute(e_t, C_t)             # kernel runs it
        return s_next, history + [(s_t, e_t, o_t)], None    # continue rollout
    return s_next, history, e_t             # e_t is the terminal answer

Training: three stages, one lifecycle. JIT-Agent’s training mirrors exactly what it does at inference — synthesize, then repair, then evolve. Every stage optimizes the same underlying objective: pick harness parameters θ that maximize expected task utility U (a mix of reward, latency, and cost) when a harness sampled from p_θ(h | task, protocol, tools, few reference harnesses) is run through a frozen backbone. Because the generator can output invalid harnesses, every candidate first passes through a protocol validator that returns pass/fail plus a structured diagnostic — this validator is what makes Stage II possible.

• Stage I — Customize (SFT + preference learning). A stronger frozen teacher model is shown the task, protocol, tool registry, and 3 reference harnesses sampled from the task-type-matched seed bank, and asked to write a harness; only protocol-valid, executable outputs are kept. First, standard next-token cross-entropy trains JIT-Agent to imitate these accepted harnesses (this is what teaches basic protocol-compliant structure). Second — because “compiles and runs” isn’t the same as “good” — pairs of harnesses for the same task are compared and one is preferred over the other only if it strictly improves reward and is no worse on latency or cost (a Pareto-style dominance rule, not a raw reward comparison). The margin of that dominance (Δval, a weighted sum of the reward/latency/cost gaps) then scales a DPO-style loss: log-probability ratios of the preferred vs. dispreferred harness (policy vs. a frozen SFT reference) are pushed apart, harder when the dominance margin is larger. Net effect: JIT-Agent learns to write valid harnesses, then learns to prefer the ones that are simultaneously better and cheaper, not just better.

• Stage II — Repair. Some Stage-I generations fail validation. Instead of throwing them away, the teacher is given the failed harness plus its diagnostic report (compiler error, interface mismatch, runtime exception) and proposes a structured patch; the patch is deterministically applied and re-validated. Only trajectories that become executable within two repair rounds are kept — this deliberately caps supervision to realistic, locally-fixable mistakes rather than full redesigns. JIT-Agent is then trained (plain cross-entropy again) to imitate the sequence of patches given the running history of (failed harness, diagnostic) pairs. This is what lets JIT-Agent recover from its own bad generations at deployment without a human in the loop.

• Stage III — Evolve (Evo-GDPO). This is the RL stage, and the part worth being careful about. At each training round the model samples a group of G candidate harnesses for a task (same group-sampling idea as GRPO), each is executed against the frozen backbone, and reward, latency, and cost are measured. Critically, the comparison isn’t just “better than the rest of my sampled group” (as in vanilla GRPO) — it’s better than the archive’s current best harness for that task (the incumbent). The reward channel gets a bonus only for beating the incumbent’s reward; the latency and cost channels only count at all if the candidate matched or beat the incumbent’s reward first (so you can’t buy an efficiency reward by getting worse). All three channels are z-score normalized separately within the group (so none numerically dominates), combined with weights that keep reward dominant, renormalized once more at the batch level, and fed into a standard PPO-clipped update with a KL penalty back to the frozen Stage-II checkpoint (to keep the policy from drifting too far chasing reward). After each round, a candidate harness is added to the persistent archive/bank only if it matches-or-beats the current reward frontier and strictly improves at least one of reward, latency, or cost — i.e. it must be a genuine Pareto improvement, not just “good enough.”

# One Evo-GDPO training round: sample a group of candidate harnesses, score them
# against the archive's current best (the incumbent), and update both the policy
# and the archive.
def evo_gdpo_round(policy_old, task, bank, backbone, G=8):
    incumbent = best_harness(retrieve(task, bank))          # top reward; ties -> lower latency, cost
    b_r, b_l, b_k = incumbent.reward, incumbent.latency, incumbent.cost

    candidates = [policy_old.sample(task, bank) for _ in range(G)]
    stats = [validate_and_run(h, backbone, task) for h in candidates]   # (reward, latency, cost)

    R_rew, R_lat, R_cost = [], [], []
    for r, l, k in stats:
        R_rew.append(r + LAMBDA_EVO * max(r - b_r, 0))       # bonus only for beating the archive
        beat_reward = r >= b_r
        R_lat.append(max(b_l - l, 0) if beat_reward else 0)  # efficiency only counts if quality held
        R_cost.append(max(b_k - k, 0) if beat_reward else 0)

    A_rew, A_lat, A_cost = zscore(R_rew), zscore(R_lat), zscore(R_cost)   # per-channel, in-group
    A_total = zscore(W_REW * A_rew + W_LAT * A_lat + W_COST * A_cost)     # W_REW dominates; re-normalize

    loss = ppo_clip_loss(policy_old, candidates, A_total) + BETA_KL * kl_to(REF_POLICY)
    update(policy_old, loss)

    for h, (r, l, k) in zip(candidates, stats):              # conservative archive update
        if r >= b_r and (r > b_r or l < b_l or k < b_k):
            bank = bank.add(task, h, r, l, k)
    return bank

A subtlety worth flagging explicitly: after Stage III, θ is frozen. “Evolving” at deployment time does not mean the model keeps training. What evolves is the archive — the bank of retained harnesses. As more tasks are seen, better harnesses get added to the bank, and because future generations condition on a handful of harnesses retrieved from that bank (E_τ), the generated harnesses keep improving even though the generator’s weights never move again. It’s evolution via retrieval/in-context reference, not via gradient updates — closer to a growing few-shot library than to online fine-tuning.

Inference: static vs. streaming. JIT-Agent supports two deployment modes:

• Static — generate N candidate harnesses in parallel for a task, pick one (test-time scaling through diversity of candidates, not more environment rollouts), execute it once. Independent per task; nothing carries over.
• Streaming — for task n, retrieve reference harnesses from the current bank B_n, generate and execute one harness, evaluate (reward, latency, cost), and conditionally fold the result into B_{n+1} using the Stage-III retention rule. The next task then retrieves from the updated bank. This is what lets JIT-Agent get better over a session without touching model weights.

sequenceDiagram
  participant Bank as Harness bank B_n
  participant JIT as JIT-Agent (frozen θ)
  participant BB as Backbone LLM (frozen)
  Note over Bank,BB: Task τ_n arrives
  Bank->>JIT: retrieve reference harnesses E_τ,n
  JIT->>JIT: generate + select harness h†_n
  JIT->>BB: wrap backbone with h†_n
  BB-->>JIT: trajectory ξ_n (reward, latency, cost)
  JIT->>Bank: retain h†_n only if it improves the frontier
  Note over Bank,BB: Task τ_n+1 retrieves from the (possibly) updated bank

Built on Prior Work

Results & Evidence

Two comparisons matter here, and they answer different questions.

Same backbone, default scaffold replaced (Table 3). Across all 18 backbone–benchmark pairs tested, swapping the default scaffold for a JIT-generated harness always helped. GLM-5.2’s nine-benchmark average rose 74.1 → 81.8 (+7.7); DeepSeek-V4-Flash rose 66.7 → 75.5 (+8.8). The biggest jumps were on tasks needing sustained state and constraint tracking — DeepSeek-V4-Flash +24.8 on DeepPlanning-Shopping, GLM-5.2 +20.2 on DeepPlanning-Travel — which lines up with the paper’s own qualitative cases (§6.7): the generated harnesses that win big are the ones that invent explicit typed state (a coverage checklist, a candidate×clue matrix) rather than just re-prompting. JIT-equipped GLM-5.2/DeepSeek-V4-Flash also beat proprietary frontier endpoints (GPT-5.6, Gemini 3.1 Pro) on 8 of 9 benchmark columns — a striking claim, but treat it as directional: those comparisons hold neither the underlying model provider’s own harness/product layer nor prompting constant, so it’s really “open backbone + JIT harness vs. proprietary product,” not a clean backbone-vs-backbone test.

Same backbone, harness varied (Table 4) — the more rigorous test. Holding DeepSeek-V4-Flash and Qwen3.6-Flash fixed, JIT-Agent-generated harnesses were compared against five production harnesses (Claude Code, Codex, OpenCode, Hermes, NanoBot). JIT-Agent won 4 of 6 backbone-benchmark settings outright, and in the two it didn’t win, it trailed by only 3.1 and 3.9 points while using far fewer tokens. It had the lowest token consumption and cost in all six settings — 14.9–54.1% cheaper than the cheapest fixed harness in each case (avg. 36.0% reduction), without sacrificing accuracy. That’s the paper’s strongest evidence: the gains aren’t “spend more compute,” they’re “spend it more selectively.”

What the evidence does not establish:

• No ablation isolating Stage I vs. Stage II vs. Stage III’s individual contribution to the headline gains — you can’t tell from the paper how much of the +7–9 point average lift comes from customization alone vs. the RL evolution stage.
• The reported per-case “Cost ↓” and “#Tokens” figures appear to measure the backbone’s execution only. JIT-Agent is itself a 27B model that has to run at inference time to generate/repair the harness — it’s not clear that generator inference cost is folded into the reported numbers, which would understate JIT-Agent’s true cost if not included.
• Streaming test-time evolution (Fig. 6) is only evaluated on 3 of the paper’s 9 benchmarks, over ~100–250 task streams; there’s no discussion of how large the bank grows or whether/how it’s pruned, so long-run behavior (does streaming degrade after 10,000 tasks?) is untested.
• Several benchmarks (AgentIF’s “weighted rubric score,” PinchBench’s “average score”) are themselves LLM-judged or aggregate metrics, which carries its own judge-noise and doesn’t decompose easily into “why did the harness help.”

How You’d Use It

This is directly a harness-generation layer you could sit between whatever LLM you’re already calling and the workflow you’re running it in — model-agnostic by design, since JIT-Agent wraps an arbitrary frozen backbone rather than requiring a specific one. Two concrete angles for your own stack:

• Your harness — a cost lever independent of model choice. Table 4’s 36% average cost reduction at equal-or-better accuracy is the real finding here: same model, different scaffold around it. If you’re running one fixed loop (a single ReAct setup, a single memory/planning policy) across every task type, this says some of those tasks are paying for context and tool exposure they don’t need. Latency and cost drop while quality holds or rises on the tasks that actually need it (constraint-heavy planning, multi-step workspace tasks) — worth testing if you already log per-task tokens and can compare against your current fixed harness as a baseline.
• Your automations — a meta-layer over your existing multi-agent pipeline. Instead of hand- designing memory/planning/tool policies for every new automation or task type (which is what extending a hand-built multi-agent system usually looks like today), a generator emits the four-module scaffold, a validator catches bad output before it reaches production, and a bounded repair loop patches it — a genuinely different build pattern from “engineer one harness per workflow.”

The realistic near-term move is not to reproduce the paper’s full RL training pipeline (see below) — it’s to prompt a strong model for Stage-I-style customization plus a validator and an archive, which captures a meaningful fraction of the value with none of the RL infrastructure.

Build Your Own (Minimal Recipe)

You can get most of the value without training anything — Stages II and III are refinements on top of Stage I’s core idea (task + protocol + reference examples → generated harness).

1. Define your own compact protocol. Pick 4–6 concrete implementations per module — e.g. for memory: full-history vs. summarized; for planning: none vs. todo-list vs. DAG; for action: plain ReAct; for capability orchestration: full tool registry vs. task-conditioned subset. This is your mini HarnessFactory and doubles as your seed bank.
2. Build a protocol validator. A static schema check (does the generated harness fill in every module correctly?) plus a dry-run executor that catches interface mismatches before a real task runs, returning a structured error message — not just pass/fail.
3. Prompt-based generation (Stage I without training). Give a strong model (Claude, GPT, DeepSeek) the task, your protocol spec, the available tools, and 2–3 example harnesses retrieved from your seed bank; ask it to emit a harness as structured config/code in your schema.
4. Wrap and execute. Instantiate the generated harness around your chosen backbone, run the task, and log reward, latency, and cost.
5. Bounded repair (Stage II without training). On a validator failure, feed the diagnostic back to the same prompted model and ask for a patch; retry up to 2 times before falling back to a known-good default harness.
6. A simple archive (streaming inference without RL). Log every (task type, harness, reward, latency, cost). Retrieve the top-k harnesses for similar past tasks as few-shot context for future generations. This alone gives you most of “streaming evolution” — no gradient updates required, just retrieval over what worked before.
7. (Optional, the hard part) Train your own generator. Only reach for Stage I SFT+preference tuning, Stage II repair-trajectory training, and Evo-GDPO if steps 1–6 aren’t enough — this requires rollout infrastructure (a frozen-executor training loop), a working reward/latency/cost evaluator per task, and the three-channel normalization + archive-relative bonus described above.

The genuinely hard parts: (a) a validator and bounded-repair loop trustworthy enough to run in production without silently shipping a broken harness, and (b) if you do go the RL route, getting the archive-relative reward (compare to the incumbent, not just your sampled group) and the per-channel normalization right — that’s the actual novel training machinery in this paper, and it’s easy to reduce to plain GRPO by accident if you skip the incumbent-comparison step.

How to Improve It

1. Fold generator cost into the accounting. Report (or measure yourself, if reproducing) the JIT-Agent generator’s own inference cost alongside backbone cost — the paper’s efficiency numbers look better if generation is “free.”
2. Ablate the three training stages. Run Stage-I-only, Stage-I+II, and the full pipeline against the same benchmarks to attribute the +7–9 point average gain across customization, repair, and evolution separately.
3. Add bank management. Introduce pruning, clustering, or decay to the harness archive so it doesn’t grow unbounded or become stale as the task distribution shifts over a long deployment.
4. Extend the module set. The paper deliberately keeps the protocol to four modules “as a starting point”; a fifth module — an explicit verifier/critic, or a role-assignment layer for multi-agent delegation — is a natural, testable extension given how much of the gain already comes from generated verification/state-tracking behavior (e.g. the SelfVerifyingAction and is_complete() gating patterns in the appendix cases).
5. Learn the retrieval policy, not just the generator. E_τ (which reference harnesses get shown to the generator) is currently a fixed top-3/top-k sample from a task-type-matched subset; jointly training what to retrieve alongside what to generate could squeeze more value out of a given archive size.

Glossary

• Harness — the memory/planning/tool-exposure/control-loop code wrapped around a frozen LLM that turns it into a closed-loop agent.
• Backbone / executor (π_ψ) — the frozen, off-the-shelf agentic LLM that actually does the reasoning; the harness is generated around it, not instead of it.
• Protocol (Π) — the fixed schema/interface every generated harness must satisfy (module types, validation rules, execution semantics) — what makes harnesses machine-generatable at all.
• Module (M/P/A/F) — memory, planning, action, and capability-orchestration; the four typed components every harness is assembled from.
• Controller state (s_t) / view (v_t) / directive (d_t) — the mutable state carried between steps; the compressed slice of history the model actually sees this step; the local subgoal formed from that view.
• Teacher model (q_φ) — a stronger, frozen model used only to generate training supervision (Stage I and II); not used at deployment.
• SFT (supervised fine-tuning) — training on next-token prediction over accepted example outputs; Stage I’s first loss.
• Preference / DPO-style optimization — training to increase the model’s relative log- probability of a preferred output over a dispreferred one, without an explicit reward model.
• Protocol validator — the static/dynamic check that flags whether a generated harness is executable, and produces a diagnostic report when it isn’t.
• GRPO / Evo-GDPO — Group Relative Policy Optimization (score a sampled group of candidates relative to each other) and this paper’s variant, which additionally scores candidates against a persistent archive incumbent and normalizes reward/latency/cost as separate channels.
• PPO-clipped objective / KL penalty — the standard trust-region RL update (clip how far the policy can move per step) plus a penalty that discourages drifting too far from a reference policy, used to stabilize Stage III training.
• Advantage — how much better (or worse) a sampled output is than some baseline, used to weight the RL gradient; here computed per reward/latency/cost channel and combined.
• Archive / bank (B_n) — the persistent, growing store of retained harness designs, their tasks, and their measured reward/latency/cost; what “evolves” at deployment time.
• Pareto frontier / dominance — a design is on the frontier if no other design is at least as good on every axis (reward, latency, cost) and strictly better on one; the retention rule for both training preferences and the archive.
• Static vs. streaming inference — generate-and-run once per task with no memory across tasks, vs. retrieving from and updating a persistent bank across a sequence of tasks.
• AOT (Ahead-of-Time) vs. JIT (Just-in-Time) — optimizing one durable harness offline before seeing the next task, vs. generating a fresh harness for each task at the moment it arrives.