A Survey of Self-Evolving Agents: On Path to Artificial Super Intelligence

TL;DR

Today’s LLM agents are frozen: the weights and the scaffolding you ship are the weights and scaffolding the client gets forever, no matter how much the agent sees in production. This survey is the first to treat “agents that keep improving themselves” as a field in its own right and gives it a coordinate system. What to evolve: the model, the context (memory + prompts), the tools, or the architecture. When to evolve: during a single task (intra-test-time) or between tasks (inter-test-time). How to evolve: rewards (scalar or textual), imitation/demonstration learning, or population/evolutionary search — across the usual ML axes (online/offline, on-policy/off-policy, reward granularity). The payoff for a builder isn’t a new algorithm; it’s a checklist that turns “we should make our agent learn” from a vibe into a specific design decision with named, citable techniques behind each cell of the grid.

Problem & Motivation

The concrete pain: a shipped LLM agent does not get better at your client’s actual work. It can retrieve documents, call tools, and reason in a loop, but the moment a task ends, everything it learned evaporates except what you manually stuffed back into a prompt or a vector DB. Three failures follow from that:

1. No adaptation to drift. The client’s codebase, product catalog, or support policies change; the agent’s competence does not.
2. No compounding from experience. The agent solves the same class of ticket badly 1,000 times instead of getting good at it.
3. No on-demand skill acquisition. Hit a problem outside the model’s competence and the agent fails rather than learning the missing skill in the moment.

Prior surveys treated “evolution” as a footnote inside a giant agent taxonomy, or only covered self-improving language models (weight updates) and ignored the agent-level machinery — memory, tools, multi-agent topology. So practitioners had no shared vocabulary: “self-improving,” “continual learning,” “self-refining,” and “agentic RL” were used interchangeably for very different mechanisms. This paper’s bet is that you can’t engineer what you can’t name, and it supplies the names.

What’s New (Core Contribution)

This is a survey, so the contribution is organization and synthesis, not a new algorithm. The genuinely new parts:

• A three-axis taxonomy (what / when / how). Before: techniques described in isolation, each paper its own island. Now: every self-evolving method lands at a coordinate (component, timing, driver). This is the load-bearing idea and the thing worth stealing.
• A formal definition of a self-evolving agent as a POMDP-plus-mutable-policy. Before: “self-evolving” was a marketing word. Now: the agent is Π = (Γ, {ψᵢ}, {Cᵢ}, {Wᵢ}) — topology, models, contexts, tools — and evolution = any process that mutates one of those four during deployment. That single sentence is the cleanest definition in the field.
• Disentangling timing from mechanism. Before: “test-time” lumped everything post-training together. Now: a crisp split between intra-test-time (improve while solving the current task; goal is this instance) and inter-test-time (improve between tasks; goal is future instances). This distinction predicts cost, risk, and where the gains land.
• A cross-cutting RL map for agents. They thread online/offline, on-policy/off-policy, and reward granularity (outcome vs. process) through the agent setting, plus a comparison table of feedback type, source, learning method, updated components, and update timing.

Honest read: the taxonomy is the real product. The benchmark/application/challenge sections are competent literature roundups you’d skim, not study.

How It Works (The Taxonomy, Technically)

A survey has no single mechanism, so the “how it works” is the framework. First, the formalism — because it makes everything else precise.

The environment is a POMDP E = (G, S, A, T, R, Ω, O, γ):

• G goals (the user’s task), S hidden world states, A actions (here actions = reasoning text + tool calls + retrieval — that union is the agentic twist).
• T(s'|s,a) transition dynamics, R(s,a,g) the feedback — and crucially R can return a scalar OR a string. Textual feedback as a first-class reward is the whole reason LLM agents can “learn” from a critique.
• Ω, O observations (partial — the agent never sees full state s), γ discount.

The agent is Π = (Γ, {ψᵢ}, {Cᵢ}, {Wᵢ}). Read this as the four things you could mutate:

• Γ — the architecture/topology: how nodes (sub-agents) are wired (a graph or a code structure).
• ψᵢ — the model at node i (the LLM weights).
• Cᵢ — the context: prompt Pᵢ + memory Mᵢ.
• Wᵢ — the tools/APIs available at node i.

The policy at a node is π_θᵢ(·|o) with θᵢ = (ψᵢ, Cᵢ). Plain English: the agent’s behavior is parameterized by its weights and its context, so you can change behavior by fine-tuning weights OR by editing prompts/memory — both count as evolution. That is the insight that unifies “RL fine-tuning” and “append a reflection to the scratchpad” under one frame.

Now the three axes.

Axis 1 — WHAT to evolve

The practical lesson: most teams should evolve C (memory/prompt) and W (tools) first — no GPUs, no training loop — and only reach for ψ (weight updates) when context engineering plateaus.

Axis 2 — WHEN to evolve

• Intra-test-time — synchronous, during the current task; the objective is to nail this instance.
  • ICL flavor: Reflexion / AdaPlanner — the agent writes a verbal critique mid-task and conditions the next step on it. No weights move.
  • SFT flavor: “self-adaptive LM” generates self-edits and triggers an immediate fine-tune for the current task.
  • RL flavor: LADDER’s TTRL — hit a hard problem, auto-generate variants of it, and run a burst of RL right there to acquire the missing skill (“just-in-time skill acquisition”).
• Inter-test-time — retrospective, between tasks; the objective is future performance. This is where most real systems live: finish task → harvest feedback (rewards, gradients, metrics) → consolidate → improve.
  • ICL flavor: induce reusable workflows from past trajectories and prepend them next time.
  • SFT flavor: STaR / SELF / SiriuS — bootstrap a training set from your own correct attempts, fine-tune, repeat.
  • RL flavor: RAGEN, WebRL, DigiRL — on-policy RL over thousands of episodes with self-generated curricula.

The cost/risk read: intra-test-time spends compute now and is risky (you’re mutating during a live request); inter-test-time spends it offline and is safer to gate behind eval.

Axis 3 — HOW to evolve (the driver)

Three families:

1. Reward-based. The signal can be:
   • Textual feedback — natural-language critique (Reflexion’s “verbal reinforcement learning,” Self-Refine, TextGrad). Rich, sample-efficient, no reward model needed.
   • Internal reward — the model’s own confidence (CISC weights reasoning paths by certainty; Self-Rewarding LMs act as their own judge).
   • External reward — environment/tool signals, majority vote, or rule checks (SWE-Dev unit tests, math verifiers).
   • Implicit reward — the claim (“Reward Is Enough,” Endogenous Reward) that next-token prediction already encodes a reward function you can extract from logits.
2. Imitation / demonstration learning. Learn from exemplars — self-generated (STaR, V-STaR), cross-agent (SiriuS sharing trajectories), or hybrid (RISE recursive self-correction).
3. Population-based / evolutionary. Keep many agent variants; apply selection/mutation/crossover.
   • Darwin Gödel Machine — agents that rewrite their own Python codebase, keep an archive of all past versions, and branch from any ancestor (open-ended, not linear).
   • GENOME — genetic algorithms directly on model weights (gradient-free).
   • Self-play — SPIN (model vs. its old self), SPC (adversarial “sneaky generator” vs. “step critic”).

And the cross-cutting ML axes every one of these inherits: online vs. offline, on-policy vs. off-policy, and reward granularity (sparse outcome reward at the end vs. dense process reward per step).

A note on the RL vocabulary the reader should hold, since it recurs: a policy is the agent’s action-picking function π_θ; a reward scores a trajectory; the advantage is how much better an action did than the baseline expectation; on-policy means you train on data the current policy generated (stable, sample-hungry), off-policy reuses older data (efficient, can destabilize). GRPO (Group Relative Policy Optimization, what the survey cites for the rollout-based methods) skips a learned value network: it samples a group of answers, sets each answer’s advantage to (its reward − the group mean) / group std, and reinforces the above-average ones. That “compare within a batch instead of training a critic” trick is why GRPO is the default for cheap agentic RL.

Architecture & data flow

flowchart TD
  subgraph Agent["Agent system Π"]
    G[Topology Γ]
    M[Models ψᵢ]
    C[Context Cᵢ: prompt + memory]
    W[Tools Wᵢ]
  end
  Task[Task = Env E + goal g] --> Agent
  Agent -->|trajectory τ| Env[Environment / POMDP]
  Env -->|feedback r: scalar OR text| Driver{How to evolve}
  Driver -->|reward-based| Upd
  Driver -->|imitation| Upd
  Driver -->|population/evolutionary| Upd
  Upd[Update step] -->|WHAT: mutate ψ / C / W / Γ| Agent
  Timing[/WHEN: intra-task vs between-tasks/] -.gates.-> Upd

The what×when×how cube as an interactive grid: hover a cell to see which named techniques live there. This is the survey's mental model in one picture — most production-ready methods cluster in the low-cost corner (evolve Context, between tasks, via textual feedback).

How group-relative advantage (GRPO) turns a batch of sampled answers into a learning signal with no value network — drag the reward bars and watch which answers get reinforced vs. suppressed. This is the engine under most "self-play from multiple rollouts" methods in the survey.

The algorithm, simplified

The single most reusable loop in the whole survey is inter-test-time self-improvement via self-generated demonstrations (the STaR / SiriuS pattern). It captures ~80% of what “self-evolving” means in practice and needs no RL infra:

# Inter-test-time evolution by bootstrapping your own training data.
# Mutates ψ (weights) via SFT; driver = external reward (a verifier);
# timing = between tasks. Maps directly onto §4.2 + §5.2.1 of the survey.

def self_evolve(agent, tasks, verify, rounds=3):
    archive = []                                   # successful (task, trajectory) pairs = memory
    for _ in range(rounds):
        new_demos = []
        for task in tasks:
            traj = agent.run(task)                  # policy π_θ produces reasoning + tool calls
            if verify(task, traj):                  # external reward: tests pass / answer correct
                new_demos.append((task, traj))      # keep the win
            else:
                # STaR's "rationalization": show the answer, ask for the reasoning
                fixed = agent.run(task, hint=task.answer)
                if verify(task, fixed):
                    new_demos.append((task, fixed)) # salvage a failure into a demo
        archive += new_demos
        agent.finetune(archive)                     # SFT on accumulated self-made demos -> new ψ
    return agent

Swap agent.finetune for append_to_memory and you’ve converted the same loop into the no-training (C-evolving) version — the same skeleton, a different WHAT axis. That interchangeability is exactly what the taxonomy is trying to make visible.

Built on Prior Work

Results & Evidence

This is a survey, so “results” = the quality of the synthesis and the evaluation it surveys, not experiments the authors ran.

What it establishes well:

• A coherent, near-exhaustive taxonomy with hundreds of citations correctly placed. The comparison table (feedback type / source / learning method / updated components / update timing) is genuinely useful as a lookup.
• A real distinction in evaluation maturity: short-horizon adaptation (success-rate-by-iteration, learning curves, adaptation speed — well covered by ADAS, AWM, WebEvolver) vs. long-horizon lifelong learning (catastrophic forgetting, cross-task transfer — barely any benchmarks exist; LTMBenchmark, MemoryAgentBench’s “Test-Time Learning” are early stabs).

What it does not establish (be honest with clients):

• No head-to-head numbers. The survey doesn’t tell you whether memory-evolution beats RL fine-tuning on any task — it can’t, because the underlying papers don’t share benchmarks.
• Selection/recency bias. It’s a 2025 snapshot of a field moving monthly; “first comprehensive survey” also means “first draft of the map.”
• The ASI framing is aspirational. The title’s “path to Artificial Super Intelligence” is motivation, not a result. Treat it as hype scaffolding around a solid taxonomy.
• Safety is named, not solved. Self-modifying-code agents (DGM) editing their own Python is flagged as risky and left as an open problem.

How You’d Use It

For an AI services company, this survey is most valuable as a discovery and scoping tool, not a build spec:

1. Client capability menu. “Self-evolving” is a sellable upgrade. The three axes let you scope it: “We’ll add inter-test-time memory evolution (no retraining, low risk) to your support agent so it compounds on resolved tickets” is a concrete, bounded offer. The grid keeps you from over-promising weight-level RL when prompt/memory evolution would do.
2. Triage by cost/risk. Use the WHEN axis as a risk gate: intra-test-time mutation in a live customer request is dangerous (sell it cautiously); inter-test-time, eval-gated improvement is safe and easy to bill as a “continuous improvement” retainer.
3. Right-size the mechanism. Reach for the cheapest HOW that works — textual feedback (Reflexion-style) before RL, self-generated demos (STaR) before population search. The survey’s feedback-type table is your decision tree.
4. Multi-agent angle (your ARC MAS background). The Architecture-evolution row (AFlow, ADAS, GPTSwarm, EvoMAC’s “textual backpropagation,” Puppeteer’s RL orchestrator) is directly relevant — it’s about evolving the topology of a multi-agent system, which is exactly the pain you’ve felt getting agents to cooperate. EvoMAC treating test failures as a loss signal to rewire the team is a pattern worth prototyping.

Where it slots in: it’s the literature backbone behind a “Continuous Learning Agent” service line — you cite it to justify the architecture, then implement the cheap corner of the grid.

Build Your Own (Minimal Recipe)

Smallest thing that captures the value: an inter-test-time, memory-evolving agent with a verifier-gated demo archive — the no-GPU version of the pseudocode above.

Build order:

1. Instrument trajectories. Log every (task, reasoning, tool_calls, outcome). You can’t evolve what you don’t record.
2. Add a verifier. A cheap external reward: did the tests pass / did the user accept / did a judge-LLM approve? This is the single most important component — bad reward signal = the agent “evolves” toward garbage.
3. Build the archive (memory). Store verified-good trajectories; on a new task, retrieve the k most similar and prepend them (this is Agent Workflow Memory / Expel in practice). No training yet.
4. Add reflection (textual feedback). On failure, have the agent write a one-paragraph critique and store that too (Reflexion). Now you’re evolving on both wins and lessons.
5. (Optional, hard) Graduate to weights. Once memory plateaus, batch the archive into an SFT dataset (STaR) and fine-tune a small open model. This is where the real difficulty lives.

The two genuinely hard parts: (a) the verifier — designing a reward that’s cheap, automatic, and not gameable; and (b) catastrophic forgetting — once you fine-tune, the agent can lose old skills, and there’s no clean off-the-shelf fix.

Libraries/models to reach for: DSPy or TextGrad for prompt/context optimization; a vector store (or Mem0) for the archive; LangGraph for the trajectory loop; for the optional RL step, TRL/verl with GRPO on a small model (Qwen/Llama-class) so a single batch of rollouts gives you advantages without a value network.

How to Improve It

Limitations as leverage — testable directions:

1. Build the missing long-horizon benchmark. The survey admits lifelong-learning evaluation barely exists. A benchmark that injects controlled distribution shift over a long task stream and measures forgetting directly would be a citable contribution and a sales differentiator (“we can prove our agent doesn’t degrade”).
2. Cross-axis ablations. Nobody has cleanly tested memory-evolution vs. weight-evolution vs. tool-evolution on the same task. Pick one client domain and run all three; the result is a decision rule the field lacks.
3. Cheap forgetting mitigation for agents. Adapt continual-learning tricks (replay from the archive, parameter-efficient adapters per skill) specifically to the agentic setting and measure retention vs. plasticity.
4. Verifier robustness. Self-rewarding loops drift when the agent games its own judge. A study on adversarial self-evaluation (does the SPC “sneaky generator” idea generalize beyond math?) would harden every reward-based method.
5. Topology evolution under cost. EvoMAC/Puppeteer evolve multi-agent structure but ignore $/latency budgets. A version that evolves topology subject to a cost constraint is directly productizable.

Glossary

• Self-evolving agent — an agent that mutates one of its four parts (model, context, tools, topology) during deployment based on feedback, rather than staying frozen after training.
• POMDP — Partially Observable Markov Decision Process; the math frame for “agent acts in a world it can only partly see and gets feedback.”
• Intra-test-time — improving while solving the current task, aiming to win this instance.
• Inter-test-time — improving between tasks, aiming to win future instances (where most production systems operate).
• Policy (π_θ) — the function that maps an observation to a distribution over next actions; “the agent’s behavior.”
• Reward (scalar / textual) — the feedback signal; here it can be a number or a natural-language critique.
• Advantage — how much better an action did than the expected baseline; the quantity RL reinforces.
• On-policy / off-policy — train on data the current policy made (stable, costly) vs. reuse old data (efficient, riskier).
• GRPO — Group Relative Policy Optimization; computes advantage as a sample’s reward minus the group mean, dividing by std — no value network needed.
• STaR — Self-Taught Reasoner; bootstrap an SFT dataset from your own correct reasoning chains, then fine-tune and repeat.
• Reflexion — “verbal reinforcement learning”: the agent writes a language critique of its failure and conditions the next attempt on it.
• Reward granularity — sparse outcome reward (only at the end) vs. dense process reward (per reasoning step).
• Catastrophic forgetting — losing old skills when learning new ones; the central risk of weight-level evolution.
• Darwin Gödel Machine — an agent that rewrites its own Python codebase and keeps a branching archive of all past versions.
• Textual backpropagation — (EvoMAC) using compile/test errors as a “loss” to rewrite which agents and prompts a multi-agent team uses.