EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents

TL;DR

Long-horizon agents (multi-step household tasks, coding agents, browser agents) lean on external scaffolding to stay coherent: something to remember environment state, something to track what’s done vs. pending, something to reuse lessons from past attempts. Today that scaffolding — the “harness” — is built by hand: prompts, heuristics, one-off conventions, and the agent never actually decides when to use it. EvoHarness-RL turns harness use into something the agent learns. It defines three compact memory slots — Belief (world state), Progress (subgoal tracker), Experience (cross-episode skills), together “BPE” — and gives the agent four small actions (track, commit, recall, note) to touch them, each costing a turn exactly like a real environment action. A two-stage recipe (supervised fine-tuning, then a cost-aware version of GRPO reinforcement learning) teaches a Qwen3-8B model both the vocabulary of these actions and, more importantly, when they’re worth the cost. On ALFWorld household tasks this pushes success from 47.9% (plain ReAct) to 96.9% — beating both frozen memory baselines and other trainable skill-curation methods — and along the way the training process shows two interesting behaviors: the agent calls the harness less and less as it internalizes routine patterns (“harness annealing”), while the stored experience self-curates into a compact, non-redundant skill bank (“harness evolution”).

Problem & Motivation

Give an LLM agent a long, multi-step task and it needs three things beyond a single prompt: a persistent sense of what’s true in the world (it can’t just re-derive object locations from a 2,000-token context every step), a record of what it has already tried (so it doesn’t loop or forget a subgoal), and a way to reuse what worked last time it saw a similar task. Frameworks today bolt all of this on as external infrastructure — memory modules, state trackers, verifiers, skill libraries — and the decision of when the agent should touch any of it is left to a system prompt or a fixed if/else convention written by a human. The agent itself is never trained to decide “is checking my memory worth a turn right now, or should I just act.”

That’s two separate, coupled problems the paper names explicitly:

1. State formation — turning noisy interaction traces (raw text observations, tool outputs) into something structured and useful.
2. Runtime control — deciding, step by step, when to read that state, when to write to it, and when to leave it alone.

Prior “harness engineering” work (e.g. Harness-1, Meta-Harness, HarnessX) attacks problem 1 by optimizing the harness itself offline — better prompts, better search-state formats, trace-driven adaptation — but still treats using it as a fixed convention. Prior self-evolving/memory agents (Reflexion, ExpeL, Voyager, SkillOS) mostly attack cross-episode knowledge reuse, but keep it separate from within-episode state tracking (belief about the current scene, progress on the current plan) — and few of them are trained end-to-end with RL to decide when access is worth it. EvoHarness-RL’s contribution is making “when to touch the harness” a first-class, RL-trainable decision, unified across all three roles.

What’s New (Core Contribution)

• A unified three-slot harness abstraction (BPE). Before: harness implementations are a grab-bag of domain-specific pieces — state trackers, execution logs, verifiers, skill libraries — with no shared interface. Now: everything is compressed into exactly three policy-facing roles — Belief (environment estimate), Progress (subgoal/execution status), Experience (cross-episode knowledge) — small enough to be part of what a policy conditions on and learns over.
• A compact, learnable action protocol. Before: an agent “uses” its memory implicitly — a RAG call injected by the framework, a memory block silently prepended to the prompt — invisible to the policy’s own decision process. Now: four explicit actions (track, commit, recall, note) sit in the same action space as real environment actions and cost the same interaction budget, so the model can be trained (not just prompted) to use them well.
• A two-stage, cost-aware training recipe. Before: RL agents that touch memory either don’t train the memory-access decision at all (frozen scaffolds) or optimize task success alone (ignoring that memory calls aren’t free). Now: supervised fine-tuning teaches the vocabulary; GRPO with an explicit efficiency/diversity/anti-spam reward teaches restraint — when a harness call pays for itself.
• Two named training dynamics, empirically demonstrated. Harness annealing: the policy’s harness-call rate drops sharply during RL as routine behaviors get baked into the model itself. Harness evolution: the experience store isn’t append-only — it expands, then self-prunes (merge/evict) into a compact, diverse skill bank. Both are shown with training curves, not just claimed.

How It Works (Technically)

The workspace. At every step t, the harness is a triple:

H_t = (B_t, P_t, E_t)

• Belief B_t — task-relevant facts inferred from interaction: object states, locations, spatial relations. Grounded in ALFWorld as a rule-based tracker that updates silently (no LLM call) after every environment step from the action/observation pair. It’s not auto-shown to the agent — the agent has to ask for it.
• Progress P_t — a bounded list of (subgoal, status) pairs. Externalizes “what have I attempted, what’s still open, where am I stuck” instead of leaving it implicit in the model’s running text.
• Experience E_t — a cross-episode skill store, split into four categories: general skills, task-specific skills, common mistakes, and object-location search priors. This is the only piece of BPE that persists across episodes, not just within one.

The actions. The agent’s full action space is A = A_env ∪ A_bpe, where A_bpe = {track, commit, recall, note}:

• track [object] — read from Belief (a specific object, or track [world] for a compact global summary).
• commit [subgoal] — write the current plan step into Progress.
• recall [query] — read from Experience (search hints, procedures, or common mistakes, depending on how the query is phrased).
• note [insight] — write a new observation into a temporary buffer that later gets folded into Experience.

At each step the policy sees the raw observation o_t, the currently-rendered harness views H_t (only what was last tracked/recalled — not the full internal store), and task context c_t, and samples one action:

a_t ~ π_θ(· | o_t, H_t, c_t)

If a_t is an environment action, it executes in ALFWorld and returns a new observation. If it’s a harness action, it queries/updates BPE and returns an updated harness view instead. The key mechanical point: both action types cost exactly one turn out of the same budget. That’s what makes “when to use the harness” a real trade-off the policy has to learn, rather than something for free.

Architecture & data flow

flowchart LR
  subgraph Harness["BPE Harness (external workspace)"]
    B["Belief B_t\n(world-state tracker)"]
    P["Progress P_t\n(committed subgoal list)"]
    E["Experience E_t\n(cross-episode skill bank)"]
  end
  CTX["Observation o_t + Task c_t"] --> POL["Policy π_θ (Qwen3-8B)"]
  Harness -->|"rendered views H_t"| POL
  POL -->|"env action"| ENV["Environment (ALFWorld)"]
  POL -->|"track[obj]"| B
  POL -->|"commit[subgoal]"| P
  POL -->|"recall[query]"| E
  POL -->|"note[insight]"| E
  ENV -->|"new observation"| POL
  B -->|"read result"| POL
  E -->|"read result"| POL

A schematic step-through of the agent choosing between an environment action and one of the four harness actions from a single shared turn budget. Watch how each choice routes into a different BPE slot — this is the mechanism the RL reward is shaping.

Training, stage 1 — supervised harness fine-tuning. A teacher model (Claude Opus) is run on 500 ALFWorld training tasks using the exact same BPE interface; only the 87 successful trajectories are kept (1,153 next-action examples, ~26.5 turns/episode). At every step the teacher outputs <think>...</think><action>...</action> — a short reasoning span plus one action, which may be an ALFWorld command or a BPE call. Qwen3-8B is fine-tuned to imitate this next-action prediction. This is plain supervised learning (cross-entropy on the target action tokens) — nothing RL-specific yet. Its job is just to teach the base model the syntax and rough semantics of the four harness actions and get it into the right ballpark of behavior, plus it seeds the initial skill bank from the teacher’s own accumulated experience.

Training, stage 2 — cost-aware GRPO. This is where “when is a harness call worth it” actually gets optimized. Some RL vocabulary first, translated to what it means here:

• Policy — the LLM itself, viewed as a function that outputs a probability distribution over the next action given the current context. Training it means nudging those probabilities toward actions that led to good outcomes.
• Trajectory (τ) — one full episode: the whole sequence of environment and harness actions from task start to success/failure/timeout.
• Reward — a single number scoring how good a trajectory was, computed after it finishes (not per-step).
• Advantage — how much better (or worse) a trajectory did than a baseline of “typical” performance on that same task. Reinforcing the good and suppressing the bad requires comparing to something — the choice of baseline is the main difference between RL algorithms.
• GRPO (Group Relative Policy Optimization) — instead of training a separate neural network to predict the baseline (as PPO does with a value function), GRPO samples a group of G trajectories for the same task, computes their rewards, and uses the group’s own mean and standard deviation as the baseline: advantage_i = (reward_i - mean(rewards in group)) / std(rewards in group). Trajectories that scored above the group average get their action-probabilities pushed up; those below get pushed down. This is “group-relative” — cheaper than PPO (no extra value network to train) and works well when you can afford to sample several attempts per task, which you can here since ALFWorld episodes are cheap to run. A KL-divergence penalty keeps the updated policy from drifting too far from the SFT checkpoint, which prevents it from degenerating into gibberish or reward-hacking in ways that break coherent behavior.

The reward itself is where the “cost-aware” part lives:

R(τ) = R_succ(τ) + λ_eff · R_eff(τ) + λ_div(u) · R_div(τ) − λ_spam · R_spam(τ) − λ_inv · R_inv(τ)

Term by term, in plain English:

• R_succ(τ) = 10 · 1[solved] — a big flat bonus only if the task was actually completed. This is the gatekeeper: nothing else matters if the task fails, so the model can’t game the other terms without also solving the task.
• R_eff(τ) = max(0, 1 − |τ|/T_max) — an efficiency bonus, but only paid out on success. Shorter successful trajectories score higher. Since harness calls consume turns just like environment actions, every unnecessary track/recall/commit/note directly eats into this bonus. This is the mechanism that punishes “checking memory just in case.”
• R_div(τ) = |{distinct action verbs used}| / |τ| weighted by λ_div(u) = (λ_max/2)·(1 + cos(πu/U)) where u is the current training epoch and U is a fixed annealing horizon — a diversity bonus whose importance is scheduled to fade over training. Early on (u small, cos(·)≈1) the weight is near its max, encouraging the model to actually try track, commit, recall, and note rather than collapsing to only environment actions. As training proceeds (u→U, cos(·)→−1), the weight decays toward zero, letting the model specialize and stop exploring once it knows which harness actions actually pay off for a given situation. This cosine schedule is exactly the “explore broadly first, then commit to what works” curriculum idea, applied to a discrete action-type distribution instead of a learning rate.
• R_spam and R_inv — fixed penalties for degenerate behavior: spamming the same action over and over (a classic RL exploit — “diversity” alone doesn’t stop looping if the model just varies which token pads out the loop), or emitting malformed <action> output that can’t be parsed. Without these, a policy chasing R_div and R_eff naively could find cheap tricks; these two terms close that loophole.

Experience consolidation (the harness’s own learning). Within a batch of GRPO rollouts, the skill bank is kept frozen — it would be unstable to have every parallel rollout editing shared memory mid-batch. Instead, every note output and completed-trajectory summary is buffered. At the end of each training epoch, a separate LLM (again Claude Opus, used only for this — it’s never inside the rollout loop) reads the buffer against the current skill bank and, for each candidate note, decides: add (genuinely new), update (refines an existing entry), remove (contradicts and invalidates an existing entry), or skip (redundant/trivial). Each skill category is capacity-bounded and uses LFU (least-frequently-used) eviction, so entries that keep getting recalled survive and stale ones get pushed out. This consolidation step is what makes Experience evolve rather than just grow — it’s the mechanism behind “harness evolution” in the results.

Tracing one example end-to-end (from the paper’s appendix case study — task: “clean the kettle and put it on the dining table”):

1. commit [find kettle] — registers the first subgoal before acting.
2. recall [how to do clean task] and recall [where to find kettle] — pulls the general clean-task procedure and a search prior; the prior says “countertop.”
3. Environment actions: go to countertop 1, go to countertop 2 — both empty. The prior is wrong.
4. Rather than looping on the stale hint, the agent falls back on general household priors and tries go to stoveburner 3 — finds the kettle there.
5. note [kettle found at stoveburner 3; recalled hint said countertop] — writes corrective evidence.
6. commit [clean kettle with sinkbasin], then executes clean + place actions to finish.
7. At the next epoch boundary, the consolidation model reads this note against the existing “kettle → countertop” search prior and updates it (or removes it and re-adds a corrected one) — so the next episode’s recall returns better information.

That loop — commit → recall → act → note, with the harness treated as revisable evidence rather than a fixed oracle — is the behavioral signature the whole training pipeline is trying to produce.

The algorithm, simplified

# One GRPO training step for EvoHarness-RL (cost-aware harness coordination).
# Real names match the paper: Rsucc, Reff, Rdiv, Rspam, Rinv.

def grpo_step(policy, task, group_size=8, T_max=70, epoch=0, U=150, lam_max_div=0.5):
    trajectories = [rollout(policy, task, T_max) for _ in range(group_size)]  # sample a GROUP
    rewards = [trajectory_reward(tau, epoch, U, lam_max_div) for tau in trajectories]

    mean_r, std_r = np.mean(rewards), np.std(rewards) + 1e-8
    advantages = [(r - mean_r) / std_r for r in rewards]     # group-relative baseline (no value net)

    loss = 0.0
    for tau, adv in zip(trajectories, advantages):
        for step in tau.steps:
            logprob = policy.logprob(step.action, step.context)      # how likely was this action?
            loss += -(adv * logprob)                                  # push up good, push down bad
    loss = loss / group_size + kl_coef * kl_to_reference(policy, sft_reference)  # stay near SFT init
    return loss

def trajectory_reward(tau, epoch, U, lam_max_div,
                       lam_eff=1.0, lam_spam=0.1, lam_inv=0.1):
    r_succ = 10.0 if tau.solved else 0.0
    r_eff  = max(0, 1 - len(tau) / tau.T_max) if tau.solved else 0.0   # efficiency only pays if you won

    verbs = {action_verb(a) for a in tau.actions}                     # unique harness/env action types used
    r_div = len(verbs) / len(tau)
    lam_div = (lam_max_div / 2) * (1 + math.cos(math.pi * epoch / U))  # explore early, specialize late

    r_spam = count_degenerate_repeats(tau.actions)      # fixed penalty for looping the same call
    r_inv  = count_malformed_actions(tau.actions)        # fixed penalty for unparsable <action> output

    return r_succ + lam_eff * r_eff + lam_div * r_div - lam_spam * r_spam - lam_inv * r_inv

Built on Prior Work

Results & Evidence

Headline (ALFWorld, seen split, Qwen3-8B): plain ReAct scores 47.9% average success across six task families. Just adding the BPE harness at inference time with no training at all (EvoHarness-Base) already reaches 56.4% (+8.5 points) — the structure alone helps. Supervised fine-tuning on BPE-using teacher traces (EvoHarness-SFT) reaches 68.6% (+20.7). The full cost-aware GRPO stage (EvoHarness-RL) reaches 96.9% (+49.0 over ReAct), matching or beating frontier models — including Claude Opus 4.5 with the same prompt-time harness (98.5%) — with an 8B open model. It also beats the closest trainable competitors: SkillOS (80.2%) and SkillRL (89.9%).

BPE helps frontier models too, not just the trained small model. Giving GPT-4.1 the prompt-time harness (no fine-tuning) lifts it from 47.2% to 69.3% (+22.1); GPT-5 goes from 60.7% to 86.4% (+25.7). Even Claude Opus 4.5, already near ceiling, gains from 96.4% to 98.5%. This is decent evidence that the BPE structure is doing real work independent of the RL training — it’s not just an artifact of fine-tuning a weaker model.

Ablation confirms all three BPE roles matter, and for different reasons. Removing Belief (no object/state tracking) hurts most on tasks needing localization and state verification (Clean, Cool). Removing Progress (no subgoal commitment) disproportionately hurts multi-subgoal tasks (Pick2). Removing Experience (no recall/notes/skill bank) gives the worst overall average (48.6%) and hits state-change tasks like Heat hardest. None of the three is redundant with the others.

Generalization result is the most interesting nuance in the paper. On unseen tasks: ReAct scores 50.0%, prompt-time BPE alone jumps to 77.6%, but SFT alone actually drops performance to 69.4% — the authors’ explanation is that imitating a teacher’s specific harness-use patterns from seen trajectories doesn’t teach when access is worthwhile in novel situations, it just memorizes surface patterns. The full RL-optimized policy recovers and improves to 86.6%. In other words: SFT teaches the vocabulary; only RL teaches the judgment, and that judgment is what transfers. This is a meaningfully honest result for a paper to report — SFT alone looking worse than the non-trained baseline on generalization is not the kind of number papers usually highlight, and its inclusion strengthens the paper’s case that the RL stage (not just “any training”) is doing the real work.

Training dynamics. Mean harness calls per episode start around 4–6 early in GRPO and stabilize near 1 per episode later (“harness annealing”) — with recall decaying slowest (cross-episode search priors keep paying off) and commit/note decaying fastest (once stable strategies are learned, externalizing every plan step stops being worth a turn). The skill bank expands rapidly early, then plateaus and self-prunes via the consolidation step as redundant/rarely-used entries are merged or evicted (“harness evolution”). The paper also shows EvoHarness-RL’s training reward climbing faster and to a higher plateau than standard GRPO (task reward only, no BPE), evidence the reduction in harness calls is deliberate specialization rather than the model just giving up on using the harness.

What this does NOT establish — caveats worth being clear-eyed about:

• Single environment family. Everything is ALFWorld: a text-based, fairly clean, deterministic embodied simulator. The authors themselves flag (Appendix A) that which BPE component matters most is “environment-dependent” — they explicitly guess Belief would matter more in visually grounded environments and Progress more in software-engineering/workflow settings, but don’t test either.
• Single trainable base model (Qwen3-8B). No evidence yet that the RL recipe transfers cleanly to other model families or sizes.
• The teacher/consolidation model (Claude Opus) is an uncosted dependency. SFT data collection and every epoch’s skill-bank consolidation both require Claude Opus calls. The “efficiency” reward term only counts environment/harness turns taken by the trained policy — it doesn’t count the real dollar/latency cost of the Opus calls that make the whole system work. A genuinely cost-aware framing would price that in.
• Small, filtered SFT set. Only 87 successful trajectories out of 500 attempted teacher episodes (1,153 examples total) — the SFT stage inherits whatever biases the teacher’s successful-trajectory distribution has.
• No wall-clock or compute-cost comparison against baselines like SkillOS/SkillRL — the win is reported purely in success-rate terms, not efficiency terms, even though “cost-awareness” is the paper’s own framing.
• R_succ, the gatekeeper term, requires a binary success signal. ALFWorld gives you this for free (task-completion checker built in); most real business tasks (a coding PR, a customer support resolution) don’t have a clean automatic verifier, which is the actual hard prerequisite for reproducing this outside a benchmark.

How You’d Use It

This paper is less “go deploy this exact system” and more “steal the design pattern for your own agent’s memory.” The direct, low-cost takeaway for anyone running production LLM agents (coding agents, browser agents, ops-automation agents) is the EvoHarness-Base result: splitting one undifferentiated “memory” or “context” blob into three typed slots — what’s true right now (Belief), what have I done / what’s left (Progress), what have I learned before (Experience) — and exposing each as an explicit, costed action the model chooses to call, rather than something silently injected every turn, gave a real +8.5pp lift with zero training. That’s a redesign of your prompt/tool scaffold, not a research project, and it’s the first thing worth trying on any long-horizon agent in your harness that’s underperforming.

Your harness — a concrete diagnostic. This is direct evidence for the claim “my agent’s reliability problem is a harness-design problem, not a model problem” — the same base model (Qwen3-8B) went from 47.9% to 56.4% just from restructuring its memory access, before touching a single weight. The ablation table is a useful diagnostic tool too: it lets you check which of the three roles your own setup is missing (no persistent world-state tracker? no explicit subgoal ledger? no curated cross-run knowledge base?) and predict roughly what kind of failures that gap produces.

The RL stage (SFT + cost-aware GRPO) is the expensive, harder-to-justify part. It’s worth building only when you have (a) a repeated, well-scoped long-horizon task at real volume — the same class of coding ticket, the same class of ops runbook, run hundreds/thousands of times — and (b) a reliable automatic success signal to gate R_succ. Without a verifier, you don’t have a reward, and without volume, the fine-tuning cost doesn’t amortize. That combination (verifiable + high-volume + long-horizon) is a narrower slice of real-world automation than “agents in general,” but it exists in your own automations and business processes — think an internal coding-agent fleet running the same category of ticket repeatedly, or a browser agent doing the same class of repetitive multi-step web task at scale.

Build Your Own (Minimal Recipe)

Phase 1 — no training, get most of the value fast.

• Define three stores in code: a belief dict (facts about the world/task), a progress list (ordered (subgoal, status) entries), an experience store (a list of {text, category, usage_count} records, category ∈ {general, task-specific, mistakes, search-priors}).
• Add four tool functions to your agent’s existing tool list — track(query), commit(subgoal), recall(query), note(insight) — each just reads/writes the stores and returns a short string. Make each cost exactly one turn/tool-call, same as any real action, so the model actually has to trade off using them.
• Borrow the paper’s exact system-prompt language (Appendix D.1) for when to use each action — especially the “note is MANDATORY when a recalled hint was empty or turned out wrong” rule, which is what drives self-correcting behavior even with zero training.
• Measure success rate before/after. This alone is the EvoHarness-Base experiment and it’s cheap to run this week.

Phase 2 — if the task is repeated enough to be worth fine-tuning.

• Collect trajectories from your strongest available model (or a frontier model as teacher) using the same four actions, keep only the successful ones, and format them as (context, harness_view) → (think, action) pairs.
• Supervised fine-tune a smaller open model (Qwen3-8B-class) on those pairs. This gets you the vocabulary and rough behavior, but — per this paper’s own unseen-split result — don’t expect it to generalize well past the training distribution on its own.

Phase 3 — if you want the actual coordination judgment, not just the vocabulary.

• Stand up GRPO (open implementations exist in TRL, verl, OpenRLHF) with rollouts sampling a group of G trajectories per task.
• Reward = task success (large, sparse, success-gated) + efficiency bonus (only on success, penalize trajectory length) + a time-annealed diversity bonus over which harness-action types get used (cosine-decay the weight over training) − fixed penalties for degenerate repetition and malformed actions.
• Run a separate, async consolidation step (one extra LLM call) at epoch/batch boundaries: feed it the batch’s buffered note outputs plus the current skill bank, ask it to decide add/update/remove/skip per note, and apply LFU eviction per category so the bank stays bounded.

The genuinely hard parts:

1. Getting a reliable success signal. R_succ needs a verifier. ALFWorld ships one; your domain probably doesn’t. Building this (a test suite, a human-in-the-loop rubric, an LLM judge you trust) is the actual prerequisite, and it’s harder than anything RL-specific in this recipe.
2. Tuning the reward weights (λ_eff, λ_max_div, λ_spam, λ_inv, the annealing horizon U) so the model neither ignores the harness nor spams it. The paper’s values are ALFWorld-specific; expect to re-tune per domain, and expect some ugly intermediate policies (loops, ignored memory) before it converges.

Libraries/models to reach for: vLLM for fast rollout serving, TRL or verl for the GRPO training loop, an open 7–8B model as the trainable policy, and a frontier model (Claude, GPT) as the one-time teacher and the ongoing (but off-critical-path) consolidation model.

How to Improve It

1. Test outside embodied/text-game environments. The authors explicitly flag that their per-action annealing pattern (recall persists, commit/note decay fast) is “environment-dependent” and guess it would differ for coding or web agents — but never test it. Running the same recipe on a coding-agent or browser-agent benchmark with a real verifier (e.g., passing tests) would tell you whether the BPE decomposition itself transfers, or whether it’s tuned to ALFWorld’s clean subgoal structure.
2. Price the teacher/consolidation cost into the reward. Right now R_eff only counts the trained policy’s own turns; the Claude Opus calls for SFT-data collection and every epoch’s consolidation are free in the objective but not free in dollars. A genuinely cost-aware version would fold consolidation frequency/cost into the reward, which could change how aggressively the harness evolves.
3. Replace the rule-based Belief tracker with a learned one. ALFWorld’s clean text observations make a hand-written parser for Belief cheap and reliable; most real environments (messy web pages, noisy tool output, multi-modal observations) won’t afford that. Swapping in a learned belief-updater (even a small model) and re-measuring how often track gets called is a natural next experiment.
4. Make the diversity-annealing schedule adaptive instead of fixed. λ_div(u) is a hand-set cosine curve over a fixed horizon U. A schedule that reacts to the actual rate of harness-call collapse (or plateau in reward) rather than a pre-set epoch count would likely transfer across tasks of different horizon length without retuning U per domain.
5. Turn Experience into a live, shared, multi-agent blackboard. Right now Experience only persists across episodes of the same agent. Given this reader’s own multi-agent-systems background: the natural extension is letting note from one agent in a cooperating team become a recall-able entry for a different agent in the same episode — turning cross-episode knowledge reuse into within-episode coordination, which the paper doesn’t explore at all.

Glossary

• Policy — the model, viewed as a function mapping “current situation” to a probability distribution over next actions.
• Trajectory (τ) — one complete episode: the full sequence of actions from task start to finish.
• Reward — a single score assigned to a finished trajectory, used to train the policy toward better ones.
• Advantage — how much better/worse a trajectory scored than a baseline; positive advantage gets reinforced, negative gets suppressed.
• GRPO (Group Relative Policy Optimization) — an RL method that estimates the advantage baseline from the mean/std of a group of sampled trajectories for the same task, instead of training a separate value-prediction network (as PPO does).
• SFT (Supervised Fine-Tuning) — training a model by direct imitation of example input→output pairs (here: context → next action), no reward or exploration involved.
• Rollout — running the current policy through an environment to generate one trajectory, used either for evaluation or as RL training data.
• KL penalty — a term that discourages the trained policy from drifting too far (in probability-distribution terms) from a reference policy (here, the SFT checkpoint), preventing collapse into degenerate behavior.
• Sparse reward — a reward signal that’s mostly zero and only nonzero at rare events (here: only paid at task success), as opposed to a reward given at every step.
• LFU eviction — “least frequently used” — a cache-style policy for removing entries: the least-often-accessed items get deleted first when a store is full.
• Cosine annealing — smoothly decreasing (or increasing) a hyperparameter over training following a cosine curve, rather than a sudden or linear change.
• Harness — the surrounding runtime infrastructure (prompts, tools, memory, state trackers, control flow) that supports an LLM agent beyond the raw model call.
• ALFWorld — a text-based simulated household environment used as an embodied-agent benchmark; the agent completes multi-step tasks (find, clean, heat, place objects) via text commands.
• ReAct — a common agent pattern that interleaves explicit reasoning (“think”) with actions, without persistent external state beyond the context window.
• Teacher model — a stronger model used to generate example trajectories/data for training a smaller model, without itself being trained.