WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

TL;DR

Agent skills (reusable SKILL.md procedure packs) are increasingly discovered automatically: run the agent, look at what failed, edit a skill, keep the edit if it helps. But existing pipelines (EvoSkill, Trace2Skill, SkillOpt) keep everything that’s “learned” tangled up inside the skill documents and their edit history — there’s no separate, durable record of why something worked or failed. WikiSkill adds exactly that: a Wiki Layer that consolidates execution traces into structured, root-caused knowledge, sits between raw traces and skills, and — critically — is never reset or rolled back even when a resulting skill edit is rejected. Across five benchmarks and five models, this consistently beats prior skill-evolution methods and no-skill baselines. Two findings matter beyond the leaderboard: skill evolution helps larger models more, not less (skills and scale compound rather than substitute), and skills evolved by one model transfer to — and sometimes beat — skills a different model evolved for itself, meaning skill discovery and skill execution are separable capabilities.

Problem & Motivation

Hand-writing agent skills doesn’t scale — you’d have to anticipate every workflow the agent will need. So recent work automates skill discovery: roll the agent out on training tasks, analyze what happened, propose an edit to a skill, validate it, keep it if it helps. The problem is what survives between iterations. EvoSkill keeps a flat feedback history of past proposals and their outcomes. Trace2Skill distills lessons straight into a skill patch and doesn’t keep the lesson anywhere else. SkillOpt keeps rejected-edit feedback and epoch-level “meta guidance,” but again folded into the optimization loop, not as an independent artifact. In all three, whatever the system “knows” about the domain lives only inside the skill text and its edit history — the very things that get rejected and rolled back when a proposal doesn’t pan out. If a rejected proposal contained a genuinely correct diagnosis, that diagnosis has no home; the next iteration has to rediscover it or hope it survives buried in a growing log the proposer must re-read every time. There’s no answer to “what have we permanently learned about this domain, independent of which skill edits we’ve kept.”

What’s New (Core Contribution)

• A three-layer workspace that decouples knowledge from the artifact under evaluation. Before: skill text is the knowledge, and it’s a mutable object competing on validation score every iteration. Now: a Wiki Layer holds structured, root-caused patterns and an audit trail that persist unconditionally, while the Skill Layer is a separate, gated, roll-backable object. Rejecting a skill edit no longer costs you the insight that motivated it.
• A dedicated Wiki Maintainer, separate from the Skill Proposer. Before: one agent (or one pipeline stage) both diagnoses failures and writes the skill edit, so diagnosis quality and edit quality are entangled. Now: the Maintainer’s only job is root-cause analysis and pattern consolidation; the Proposer’s only job is turning accumulated knowledge into one atomic skill change.
• A wiki-informed, tool-calling (ReAct) Skill Proposer instead of a fixed context dump. Before: proposers get handed a batch of pre-selected traces/feedback. Now: the Proposer starts with just a wiki index, an audit log of prior accept/reject decisions, and pass/fail summaries, then actively chooses which pattern pages and raw traces to read_file before writing a proposal — and explicitly avoids re-proposing things already marked rejected.
• An empirical result, not just a mechanism claim: skill evolution complements model scale and transfers across models. This is a genuine finding (not a repackaging) — it shows skill discovery and skill execution are distinct capabilities, which has real deployment implications (see How You’d Use It).

How It Works (Technically)

WikiSkill runs an outer evolution loop over iterations k = 1..K. At every iteration there are four components, and the state carried forward is the pair (S_k, W_k) — the active skill set and the wiki — where only S_k can be rolled back.

1. Inference Agent — produce trajectories. τ_i ~ π(x_i; S_{k-1}) just means: run the agent on task x_i with the last accepted skill set injected directly into its system prompt (no retrieval — full injection, to remove “did the agent find the right skill” as a confound), and record everything it did as a trajectory τ_i (observations, actions/tool calls, final answer). During training rollouts the Inference Agent is deliberately denied wiki access — more on why in Results.

2. Wiki Maintainer — consolidate traces into knowledge. W'_k ← M_WM(W_{k-1}, T_sample,k) means: one LLM call reads the entire current wiki plus a stratified sample of this iteration’s traces (up to 5 failing + 3 passing, capped at 15,000 characters each, so it fits in context) and returns structured edits — new pattern pages under wiki/patterns/, patches to existing ones, an updated index.md catalog, and an appended entry in the evolution log logs.md. It does root-cause analysis, not surface pattern-matching: “the agent picked the item back up and put it down again” is a symptom; “the agent doesn’t check whether the target location already satisfies the goal before repeating an action” is the root cause the wiki page should capture. Pattern pages are updated by patch operations (append / replace / insert_after against exact text spans), not full rewrites — this keeps them stable and diffable across iterations.

3. Skill Proposer — turn knowledge into one atomic skill change. P_k ← M_P(W'_k, S_{k-1}, T_train,k) means: a multi-turn ReAct agent (Yao et al., 2023 — reason, then act, then observe, repeat) that starts with the wiki index, the skill-impact.md audit trail (which includes the full content of rejected proposals, so it doesn’t retry them), and a pass/fail summary of every training task. It then actively calls read_file on specific pattern pages and raw traces to build its own evidence before calling finish() with one proposal — either create a new skill or apply an incremental patch to an existing one. “Atomic” matters: each iteration changes at most one skill, so gating (next step) can cleanly attribute a validation-score change to that one edit.

4. Gating and Rollback — validate, keep or discard. The proposal is applied to produce a candidate skill set S'_k = Apply(S_{k-1}, P_k). The system reruns the agent on the validation split to get R(T_val,k), and:

S_k = S'_k        if R(T_val,k) > R_best     (strictly better — accept)
S_k = S_{k-1}      otherwise                  (roll back the skill only)

R_best only ratchets upward when a proposal is accepted. Crucially, the wiki update is unconditional — W_k ← Update(W'_k, P_k, R(T_val,k), a_k) runs regardless of accept/reject, appending the proposal’s diff, validation score, and outcome to skill-impact.md. This is the load-bearing design decision: the skill set can be reverted, the wiki never is.

Architecture & data flow

flowchart TD
  subgraph Raw["Raw Layer (raw/) — immutable"]
    T[Execution traces]
  end
  subgraph Wiki["Wiki Layer (wiki/) — persists forever"]
    P[patterns/*.md]
    L[logs.md]
    SI[skill-impact.md]
  end
  subgraph Skill["Skill Layer (skills/) — gated, roll-backable"]
    SK[SKILL.md + PURPOSE.md]
  end

  SK -->|inject into prompt| IA[1. Inference Agent: rollout on train split]
  IA -->|write traces| T
  T -->|stratified sample: 5 fail + 3 pass| WM[2. Wiki Maintainer: root-cause analysis]
  Wiki -->|full wiki context| WM
  WM -->|patch patterns, index, log| Wiki
  Wiki -->|index + skill-impact + outcomes| SP[3. Skill Proposer: ReAct agent]
  T -->|on-demand read_file| SP
  SP -->|one atomic proposal| G[4. Gating: validate on Dval]
  G -->|accept: score improves| SK
  G -->|reject: revert skill only| SK
  G -->|append diff + outcome, always| SI
  G -->|next iteration k+1| IA

The key asymmetry in WikiSkill: click through an iteration and watch what happens on accept vs. reject. The Skill Layer (right) snaps back to its prior state on rejection; the Wiki Layer (left) always keeps growing, on both outcomes.

The algorithm, simplified

# Core WikiSkill evolution loop (Algorithm 1, simplified).
# wiki_maintainer, skill_proposer, inference_agent are LLM calls / agent loops.

def wikiskill_evolve(train, val, K, R):
    skills, wiki = {}, Wiki()             # S_0 = empty, W_0 = empty
    r_best = R(rollout(inference_agent, val, skills))   # baseline validation score

    for k in range(1, K + 1):
        if r_best >= 1.0:
            break                          # perfect on validation: stop early

        # 1. Inference Agent: roll out on training tasks with CURRENT skills only
        #    (wiki access deliberately withheld here -- see ablation)
        traces = [rollout(inference_agent, x, skills) for x in train]

        # 2. Wiki Maintainer: root-cause the sample, consolidate into the wiki.
        #    Wiki edits are unconditional -- they happen whether or not the
        #    resulting skill proposal below gets accepted.
        sample = stratified_sample(traces, max_fail=5, max_pass=3, char_cap=15_000)
        wiki = wiki_maintainer(wiki, sample)         # patch patterns/, index.md, logs.md

        # 3. Skill Proposer: ReAct agent, reads wiki + traces on demand,
        #    proposes exactly ONE atomic create-or-patch.
        proposal = skill_proposer(wiki, skills, traces)   # multi-turn tool use, then finish()
        candidate_skills = apply_proposal(skills, proposal)

        # 4. Gating: validate the candidate; strict improvement required to keep it.
        val_traces = [rollout(inference_agent, x, candidate_skills) for x in val]
        score = R(val_traces)
        if score > r_best:
            skills, r_best, outcome = candidate_skills, score, "accepted"
        else:
            outcome = "rejected"           # skills revert to last accepted state

        # Wiki always records what was tried and what happened -- this is what
        # lets the proposer avoid re-proposing rejected ideas next iteration.
        wiki.append_skill_impact(proposal, score, outcome)

    return skills, wiki

Built on Prior Work

Results & Evidence

Headline. Across 5 benchmarks (LiveMathematicianBench, SealQA, SpreadsheetBench, OfficeQA, ALFWorld) and 5 models (Qwen-3.5-4B/9B, Qwen-3.6-27B, Gemma-4-31B, Gemini-3.5-Flash), WikiSkill has the best average performance for every model, beating the strongest competing method by 3.3–12.0 points depending on model, and improving over no-skill baselines in most model×dataset cells. Scores are averaged over 3 independent full evolution runs, with paired bootstrap significance testing (p < 0.05) to call ties.

Scaling complements skills, doesn’t substitute for them. Within the Qwen family, WikiSkill’s improvement over no-skill grows with model size: +12.3 (4B) → +17.5 (9B) → +23.9 (27B) points average. Yet a 9B model with WikiSkill (47.4% avg) beats a 27B model with no skills at all (39.4%) — smaller+skilled beats bigger+bare.

Real Table 1 averages across all five benchmarks. Bars: no-skill baseline vs. WikiSkill for each Qwen model size. The dashed line marks Qwen-3.6-27B's no-skill score — notice Qwen-3.5-9B+WikiSkill clears it.

Cross-model transfer works, and can beat self-evolution. Skills evolved by Qwen-3.6-27B and deployed on Qwen-3.5-9B reach 50.5% on SpreadsheetBench vs. 33.6% from that model’s own self-evolved skills. But transfer isn’t universally safe: Qwen-3.5-4B’s SpreadsheetBench skills — full of single-line Python workarounds needed by a weak model — hurt Gemini-3.5-Flash (50.5% → 18.1%), apparently because they block it from writing the comprehensive scripts it’s actually capable of, and add redundant tool calls that exhaust its interaction budget. The authors’ takeaway: skill discovery (finding the right procedure) and skill execution (running it well) are separable capabilities, and a skill can encode either a general procedure or a model-specific crutch.

Real Table 2 data: SpreadsheetBench accuracy by (inference model, skill source) pair. Rows = who runs the skill, columns = who evolved it. Diagonal = self-evolved. Watch Qwen-3.5-4B's skills (leftmost column) tank Gemini-3.5-Flash while Qwen-3.6-27B's skills (middle column) lift everyone.

The wiki, not extra reasoning, is what drives the gain. The ablation (Table 3, Gemini-3.5-Flash) is the cleanest piece of evidence for the paper’s central claim. Giving the Skill Proposer wiki access (with the Inference Agent still denied it) lifts average score from 48.7% → 63.7% — a +15.0 point jump from persistent knowledge alone. Counterintuitively, also giving the Inference Agent wiki access during training rollouts hurts, dropping average score to 60.9% (LiveMath alone falls 72.6% → 64.8%). The authors’ explanation: if the agent can lean on the wiki directly during training, its trajectories stop being an honest signal of what the skills can and can’t do, so the traces feeding the next iteration’s diagnosis get less informative. This is why the default configuration deliberately withholds wiki access from the Inference Agent.

Caveats the paper is upfront about. Validation splits are small (16–80 examples per benchmark), which the authors mitigate with 3 reruns and bootstrap testing rather than eliminate — gating decisions on any single run could still be noisy. Gating requires strict improvement, so a proposal that’s validation-neutral today but would compound with future edits gets discarded outright — a design choice inherited from prior work for fair comparison, and flagged by the authors as worth loosening. All comparisons run skills fully injected into the prompt (no retrieval), so the paper says nothing about how this behaves once a skill library is large enough to need triggering/retrieval. And the wiki has no pruning mechanism — Table 4 shows 6–10 new patterns created per iteration on average, unbounded over a longer run.

How You’d Use It

If you run agents in production, three things here are directly usable:

• Your harness — the wiki is an audit trail you get for free. skill-impact.md is literally a unified-diff changelog of every proposal, its validation score, and why it was kept or discarded. That’s exactly the kind of “show your work” artifact you want when you’re nervous about a black-box self-improving system — you don’t need to build separate observability tooling for it.
• Your business — evolve expensive, deploy cheap. The cross-model transfer result is a direct cost lever: run the (expensive) skill-evolution loop once with a strong model against your own task suite, then ship the resulting skill pack to a cheaper model at inference time. It won’t always be free lunch (see the SpreadsheetBench negative-transfer case), so budget for a validation pass of the transferred skill on the target model before shipping it — but the default expectation, per this paper, is that it works.
• Your business — “small model + skills” is a legitimate cost/performance lever. Qwen-3.5-9B+WikiSkill beating Qwen-3.6-27B+nothing is the kind of number that argues against reflexively upgrading to the biggest model. Skill evolution against your actual workflows can be a cheaper lever than a model upgrade.
• Where it slots in your agent stack: this sits entirely in the offline tuning phase — you run the evolution loop against a labeled training/validation task set before deployment, then ship a static skill pack for inference. It’s orthogonal to (and composable with) skill retrieval systems and broader “harness optimization” work the paper explicitly calls out as complementary — WikiSkill only touches skill quality, not how skills get selected or how the rest of your agent harness is tuned.

Build Your Own (Minimal Recipe)

Components:

1. Raw store — a directory of execution logs (one file per task per iteration), append-only.
2. Wiki — index.md (one line per pattern), patterns/*.md (root-cause + evidence + fix per page), logs.md (chronological iteration summary), skill-impact.md (append-only: proposal diff, target skill, validation score, accept/reject).
3. Skills — one directory per skill, each with SKILL.md (trigger conditions + instructions) and PURPOSE.md (which wiki patterns motivated it).
4. Gating harness — plain code, no LLM: apply a proposal, rerun the validation split, compare to R_best, keep or revert.

Build order:

1. Get an Inference Agent + eval harness working on one benchmark with a hand-written static skill first — this validates your skill-injection prompt plumbing before any evolution logic exists.
2. Add the Raw store / trace log format.
3. Build the Wiki Maintainer first, as a single non-agentic LLM call (JSON-in, JSON-out: current wiki + sampled traces → patch operations). Per the ablation above, this is the highest-leverage half of the whole system — ship it before the fancier ReAct proposer.
4. Add the ReAct tool-calling Skill Proposer once the wiki actually has content worth reading.
5. Wire up gating/rollback — mostly glue code, but be careful to keep the wiki-append on the path that runs regardless of accept/reject.

The genuinely hard parts:

• Patch-based editing reliability. replace/insert_after operations need an exact substring match against existing markdown. The paper doesn’t spell out what happens when a patch target doesn’t match — in your own build, plan for a fallback (retry the LLM call, or degrade to append) rather than silently dropping the edit.
• Tuning the sampling budget. 5 failing + 3 passing traces at a 15,000-character cap is what this paper used; too little starves root-cause analysis, too much blows your context window. This will be domain- and task-length-specific — start here and adjust based on whether the Wiki Maintainer’s pattern pages look genuinely diagnostic or generic.

What to reach for: any tool-calling LLM API (paper used Qwen3.5/3.6, Gemma-4, Gemini-3.5-Flash via vLLM for the open-weight ones); a minimal hand-rolled ReAct loop is enough — no RL, no gradient updates, no vector DB. “Retrieval” here is just read_file/grep-style access to markdown files, not embeddings.

How to Improve It

1. Wiki pruning. The authors flag this as an open problem — the wiki only grows (6–10 new/edited patterns per iteration in their runs). Try periodic LLM-driven summarization/dedup passes, or evidence-recency-weighted eviction, and measure whether proposer quality holds up on a compacted wiki.
2. Softer gating. Strict “must beat R_best” discards neutral proposals that might compound later — the paper says as much in its limitations. Try patience-based acceptance (tolerate ties, or small regressions if the wiki log shows a positive multi-iteration trend) and see if it recovers value the strict gate is leaving on the table.
3. Combine with skill retrieval. Everything here uses full-prompt skill injection. As a wiki+skill library grows across many client domains, pair this with retrieval-based skill selection (the paper cites several: Cho et al., Su et al., Zheng et al.) — and specifically test whether wiki-informed routing (using pattern pages to pick the right skill) beats plain embedding retrieval.
4. Transferability tagging. The paper’s “general procedure vs. model-specific workaround” distinction (Sec 4.2.2) is currently a manual post-hoc read of the results. Have the Wiki Maintainer tag each pattern/skill at write time — general procedure vs. workaround-for-a-weak-model — and use that tag to gate whether a skill is a transfer candidate before shipping it to a different inference model.
5. Test-time wiki reads for long-horizon tasks. The wiki currently only updates between training iterations, never mid-episode. For genuinely long-horizon tasks (hundreds of steps, multi-hour sessions — explicitly out of scope here), a controlled test-time-only “read the wiki” tool for the Inference Agent could extend the idea within a single session. Note the ablation’s warning: giving the Inference Agent wiki access during training hurt skill development, so this would need to stay strictly test-time, or be a separate short-term-memory channel, not a rerun of the same setup.

Glossary

• Agent skill — a filesystem directory (SKILL.md + resources) packaging reusable procedural instructions an LLM agent can load into its prompt.
• Skill evolution — automatically discovering/refining skills by running an agent, analyzing its trajectories, and updating skills based on outcomes.
• Rollout / trajectory — one run of the agent on a task: the sequence of observations, actions/tool calls, and the final answer.
• Root-cause analysis (here) — diagnosing why a failure happened (a missing check, a wrong assumption) rather than just describing what went wrong.
• Patch-based editing — modifying a document via small, targeted operations (append / replace / insert-after an exact text span) instead of rewriting it wholesale.
• Validation gating — accepting a proposed change only if it improves a held-out validation score; otherwise discarding it.
• Rollback — reverting to the last accepted state when a proposed change fails gating.
• ReAct — an agent pattern that interleaves reasoning (“what should I check next”) with acting (calling a tool) and observing the result, repeated in a loop.
• Stratified sampling (here) — deliberately sampling a mix of failing and passing traces (not purely random) so the Wiki Maintainer sees both problems and working strategies.
• Cross-model skill transfer — using skills evolved on one model to run inference on a different model.
• Full-batch vs. minibatch — whether one evolution iteration processes the entire training split at once (full-batch, what WikiSkill uses) or a smaller chunk (minibatch, used by the baselines).
• Bootstrap significance testing — a resampling-based statistical test used here to decide whether one method’s score is reliably better than another’s, not just noise from a small test set.
• ALFWorld — a text-based simulated household environment used to test multi-step embodied task planning.