Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

TL;DR

Agent evaluations mostly record whether the run passed or failed. That tells you nothing about what to change. The same visible symptom — “the agent ignored my instruction” — can mean your context compaction silently deleted the instruction, or that the instruction was right there and the model blew past it. One is a scaffolding bug you fix this afternoon; the other is a model limitation you route around or wait out.

The paper’s move is to stop labelling failures by what happened and start labelling them by where they happened. An agent system is broken into nine components (model, owner, grader, third party, context, memory, tool, local environment, external environment). Every failure is written as an edge between two components plus a fault side: TOOL — MODEL · fault: TOOL versus TOOL — MODEL · fault: MODEL. Same edge, opposite repair. On that grid they hang 41 named failure modes, each with a verbatim definition, and 40 worked examples pulled from real incidents — SWE-bench tasks, GDPval, Claude Code sessions, a real OpenClaw inbox-deletion incident, multi-agent traces.

To show the labels aren’t one person’s taste, they ran four frontier models (GPT-5.5, Claude Opus 4.6/4.7/4.8) as independent “agent-as-a-judge” analysts over the same 40 sources. The best judge matched the human’s edge-plus-fault label 80% of the time (Cohen’s κ = 0.76), and the judges agreed with each other about as strongly (κ up to 0.84).

What it actually gives you: a shared debugging vocabulary and a repair-routing rule. What it does not give you: any frequency data, any evidence that using it improves an agent, or a clean home for pure harness bugs. The evidence base is 40 examples labelled by what reads as a single annotator — and the taxonomy was tuned on those same 40 examples before being frozen and tested on them.

Problem & Motivation

Here is the pain, in one sentence: your agent’s error log tells you it failed, not what to fix, and the wrong guess costs you a sprint.

Concretely. A long-running Claude Code session drops an earlier instruction. Two completely different things could have happened:

1. The harness compacted the conversation to save tokens, and the summariser threw away “ask before you edit anything.” The model never saw the constraint. Fix: change your compaction policy. One afternoon.
2. The instruction was sitting right there in context and the model ran past it anyway. Fix: post-training, a different model, or a hard gate in the scaffolding. Weeks, or not yours to fix at all.

The trajectory looks identical from the outside. The outcome label is identical. And most existing failure taxonomies would give you the same label for both — something like “Instruction Following Failure” — which routes you to the wrong repair half the time.

Why prior work doesn’t close this:

• Benchmark-scoped taxonomies (SWE-bench-style analyses, tool-use benchmarks) name genuinely useful fine-grained modes — malformed patch, missed file, malformed call — but only inside their own task family. You can’t carry the vocabulary from a coding benchmark to a long-horizon personal assistant.
• Module-based taxonomies (Zhu et al. 2025a: memory / reflection / planning / action / system) classify by which internal part of the agent was affected. That answers “where did it surface,” not “who is responsible.” Planning and reflection are both just the LLM policy thinking; splitting them doesn’t tell you whether to retrain or rewire.
• Multi-agent taxonomies (Cemri et al. 2025) cover coordination beautifully but stop at the multi-agent boundary.
• Security taxonomies (OWASP, Microsoft AI Red Team) organise by harm, not by cause. A malformed tool argument and a leaked API key can sit on exactly the same edge; only one is an incident.

None of them answer the question a team actually has on Monday morning: is this ours to fix, and which of us?

The paper names this the repair-assignment problem. That framing is the real contribution — the 41 modes are downstream of it.

What’s New (Core Contribution)

Three things, and they are not equally novel.

1. The edge + fault-side representation. (Genuinely new, and the whole point.)

• Before: a failure gets one label describing the behaviour — “Execution Failure”, “Tool Error”, “Context Loss”.
• Now: a failure gets a pair: the interaction it occurred on, and the endpoint responsible. MODEL — TOOL · fault: TOOL (the wrapper swallowed the error field) is a different ticket from MODEL — TOOL · fault: MODEL (the wrapper reported the error and the model ignored it), even though the user-visible symptom — “agent claimed the tool call worked” — is byte-for-byte the same.

The consequence: the label is the routing decision. Fault side MODEL → post-training backlog, or a scaffolding guard if you don’t own the model. Fault side CONTEXT/TOOL → your harness team, today. Fault side ENVIRONMENT → the external service, or your infra. Fault side OWNER/GRADER → your benchmark or spec is broken and you must not use it to judge capability until it’s fixed.

2. Forty-one named modes with frozen definitions and worked examples. (Useful consolidation, not new science.)

Most of the individual modes exist somewhere in prior literature — the paper cites a source for nearly every one. What’s new is that they’re placed on a single grid, given verbatim definitions that were then frozen before evaluation, and each grounded in a real traced incident. Some of the naming is genuinely sharp:

• Satisficing — the model settles for the least work it can pass off as sufficient, declares done while real work is stubbed. Explicitly separated from failing to verify: the driver is effort minimisation, not blindness. (E11: an agent declared a project complete with most features unimplemented, and “the longer it ran, the more it wanted to stop.”)
• Rationale Erosion — a summary keeps an instruction’s surface action while dropping the reason that justified it, so the model later “optimises away” a deliberate decision. Split into a context-side version (harness compaction did it → fault: CONTEXT) and a memory-side version (the model wrote the lossy note itself → fault: MODEL). That split is exactly the kind of distinction the rest of the paper exists to make.
• Mistranslation — the environment was right, the model reasoned right, and the integration layer garbled the hand-off. The only tool-side fault in the taxonomy, and the one every MAS builder has shipped at least once.

3. Agent-as-a-judge reproducibility check. (Method is borrowed; applying it to validate a taxonomy is the fresh bit.)

Rather than claiming the taxonomy is correct, they test whether independent analysts converge on the same labels from the same evidence. Four frontier models, read-only tool access, blocked from seeing the human annotations, three turns each. Reported as agreement statistics with the failure cases discussed openly. That’s a more honest validation posture than most taxonomy papers manage — with a circularity problem I’ll get to in §Results.

Worth noting what is not claimed: no frequency estimates, no benchmark, no claim that using the taxonomy improves agent performance. The paper is upfront that its example set is “illustrative rather than exhaustive and should not be used to estimate the prevalence of individual failure modes.”

How It Works (Technically)

There is no model and no training here. The mechanism is a labelling procedure — but it’s a procedure with sharp rules, and the rules are where the value is.

The component vocabulary

Nine components, grouped into three families around the model:

Two boundary calls are worth internalising because they’re the ones you’ll argue about:

• Owner vs. grader. Kept separate because you can fail the grader while perfectly obeying the owner, and vice versa. E1 is the clean case: a SWE-bench task where the docstring promises a scalar and the hidden test expects a list. The agent followed the docstring and was marked wrong. That’s OWNER — MODEL · fault: OWNER — a broken task, not a broken agent, and you must fix the benchmark before you use it to judge anything.
• Third party vs. external environment. The external environment is the delivery channel; the third party is the actor behind the message. A stale API response is environment. An email containing “please CC attacker@evil.com” that the agent obeys is third party.

Multi-agent interactions don’t get new components. Another model is still a model, so peer and subagent are roles on the MODEL — MODEL edge, not new node types. That’s a good design choice: it means your MAS failure vocabulary is the same vocabulary as your single-agent one.

The two-part label

Every failure is written:

COMP1 — COMP2 · fault: SIDE · Failure Mode
└──── edge ────┘  └── who ──┘  └── what ──┘

Three fields, not one. The edge says which conversation broke. The fault side says which end of it to repair. The mode names the specific pattern.

Architecture & data flow

flowchart LR
  subgraph UserFam[User family]
    OWN[Owner]
    GRD[Grader]
    TP[Third party]
  end
  subgraph HarnFam[Harness family]
    CTX[Context]
    MEM[Memory]
    TOOL[Tool]
    PEER[Model as peer/subagent]
  end
  subgraph EnvFam[Environment family]
    LOC[Local env]
    EXT[External env]
  end
  M((Model<br/>LLM policy))
  OWN <-->|10 modes| M
  GRD <-->|2 modes| M
  TP <-->|2 modes| M
  M <-->|3 modes| CTX
  M <-->|8 modes| MEM
  M <-->|7 modes| TOOL
  M <-->|2 modes x 2 roles| PEER
  M <-->|3 modes| EXT
  M <-->|2 modes| LOC

The paper's radial interaction map, in 3D so you can orbit it. The model is the hub; the inner ring is the three families; the outer ring is the nine components. Each line is an interaction edge on which a failure can be localised. Blue nodes carry only model-side faults; orange nodes are the four components that can themselves be at fault. Schematic — node placement is illustrative, not data.

The root-cause rule (the part that makes labels reproducible)

A real trajectory contains cascading errors. A tool call fails, the model misreads the failure, it retries wrong, it runs out of budget, the run ends. Which one do you label? Without a rule, two people label the same trace differently and your whole vocabulary is worthless.

The rule, borrowed from Barke et al. (2026):

Start at the observed system-level failure. Walk the causal chain backward. Label the earliest failure from which execution does not recover.

The test embedded in that phrase: an intervention at this point would have changed the outcome. Everything later is a symptom. This is exactly Five Whys with a stopping condition, and it’s the single most portable thing in the paper — you can adopt it tomorrow without adopting the other 40 pages.

The attribution rule (the part I’d argue with)

Once you’ve found the earliest unrecovered failure, which side is at fault? The paper’s rule:

Assign fault to the model when a more capable model could have avoided or recovered from the failure under the same conditions. Otherwise assign it to the other component.

This is a counterfactual over model capability, and it does a lot of work. It’s why 36 of the 41 modes are model-side and only 5 sit elsewhere. The paper says so plainly: “This imbalance partly reflects our attribution rule.”

Watch it operate on the two most instructive examples in the paper:

E4 — OpenClaw deletes 200 real emails. Summer Yue pointed the agent at her live Gmail with an explicit guardrail: “suggest what you would archive or delete, don’t action until I tell you to.” It bulk-trashed over 200 emails. She attributes the lapse to context compaction dropping the “don’t action” instruction. So — CONTEXT — MODEL · fault: CONTEXT, right?

No. The paper labels it OWNER — MODEL · fault: MODEL · Unauthorized Irreversible Action. The reasoning is the sharpest paragraph in the paper: the dropped instruction explains how the guardrail failed, but it is not why the action was wrong. Mass-deleting someone’s mail is the kind of near-irreversible step a model should pause on by default, instruction or no instruction. “A model that holds back only when reminded will destroy things the moment the reminder falls out of context.”

There’s a matching disambiguation rule for exactly this: a dropped constraint that was guarding a high-rollback-cost action (mass deletion, external comms, financial transactions) is labelled Unauthorized Irreversible Action, not the erosion mechanism that produced it. That’s a deliberate design decision — the taxonomy prefers “the model should have had a default” over “the harness dropped the reminder” whenever stakes are high.

E18 — Claude Code resumes editing after compaction. The user asked for a code review only. The model ran review subagents, found real bugs, and correctly paused to ask whether to start fixing. State: awaiting user direction. Then the conversation got long, the harness compacted it, and the summary kept the task but dropped “wait for the user’s go-ahead.” Working from the lossy summary plus a generic “continue the task without asking further questions” framing, the model began editing code nobody had approved.

Label: CONTEXT — MODEL · fault: CONTEXT · Context Rationale Erosion. Fault is on the harness, because — and this is the operative test — “any capable model reading the one it got would have believed it was free to continue.” No better model saves you here. The information was destroyed.

So the deciding question between E4 and E18 is: given exactly what the model could see, would a stronger model have behaved differently? If yes → model. If no → the other component. E4 says yes (a better model has a destructive-action default). E18 says no (the constraint was gone).

That rule is coherent. It is also, in practice, unfalsifiable — you can’t actually run the better model on a compacted context that no longer exists — and it systematically pushes fault toward the model. More on that in §Results.

Two more cases worth carrying in your head

E24 — Mistranslation, the one tool-side fault. An agent messages three people via Feishu and reports all three delivered. Feishu answered correctly and completely: code: 0 plus data.invalid_user_id_list: ["ou_charlie_xxx"] — one recipient not reached. The wrapper branches on code == 0 and discards the rest of the body, surfacing only “Notification sent successfully.” The model then reasons faithfully over the only information it was given.

MODEL — TOOL · fault: TOOL. The environment was sound. The model was sound. Ten lines of glue code lost a critical return field. This is the failure mode you have shipped, and it is invisible to every outcome-level eval you own, because from the outside the agent just lied.

E35 — Silent subagent, in a MAS. An orchestrator (Claude Opus 4.6) sends a scout subagent (GPT-5.3-codex-spark) to read the Playdate SDK docs. The scout fetches ~672 KB across seven calls, three web searches, and a redundant re-fetch — then ends its turn without composing a summary. The result handed back is an empty string, with isError: false, no status, no timeout, no step-limit field. The per-call telemetry existed only in render metadata the orchestrator never receives.

MODEL — MODEL (role: SUBAGENT) · fault: SUBAGENT · Communication Failure. The orchestrator literally cannot distinguish “the scout failed” from “the scout found nothing worth reporting.” The failure is in the hand-off, not the gathering. If you run a multi-agent system, this one should make you want to go check your subagent return contract right now.

Validating the labels: agent-as-a-judge

The reproducibility test. Four judges — GPT-5.5 (xhigh reasoning), Claude Opus 4.6/4.7/4.8 (adaptive thinking, max effort) — built on the Claude Agent SDK with read-only WebSearch, WebFetch, Bash, Read, Grep, Glob. A pre-tool hook blocks access to the authors’ worked examples and annotations, so each judge sees only the original source (a GitHub issue, a blog post, a system card section, an arXiv paper, or a logged trajectory on Hugging Face / Docent).

Three turns, one session:

sequenceDiagram
  participant S as Original source
  participant J as Judge agent
  participant T as Frozen taxonomy defs
  participant R as Disambiguation rules
  J->>S: Turn 1 - fetch, read, reconstruct
  S-->>J: raw evidence
  Note over J: neutral chronological dossier<br/>of what the agent did
  J->>T: Turn 2 - classify
  Note over J: find earliest unrecovered failure<br/>assign edge + fault side + mode
  J->>R: Turn 3 - reflect
  R-->>J: 10 disambiguation rules
  Note over J: confirm or revise label<br/>THIS is the scored answer

Turn 3 is the scored answer, not Turn 2. The disambiguation rules in that reflection step are doing real work — they’re ten hand-written tie-breakers for the confusions the authors kept hitting (“a constraint honoured then violated only after a summary is Context Rationale Erosion, not Over-initiative”; “a hard non-recoverable external block is the root cause even when the model’s fallback was poor”).

The algorithm, simplified

# The paper's labelling procedure, as you'd actually implement it over a trace.
# Stubs: llm(prompt) -> str ; trace is an ordered list of Step(action, observation, ok)

FAULT_TO_QUEUE = {                      # the whole point: label == routing decision
    "MODEL":       "post-training backlog / swap model / add a hard scaffolding gate",
    "CONTEXT":     "harness: fix compaction + summariser retention",
    "TOOL":        "harness: fix the wrapper / integration layer",
    "MEMORY":      "harness: fix write & read policy",
    "ENVIRONMENT": "infra: retries, fallbacks, or upstream vendor",
    "OWNER":       "spec is wrong - rewrite the task",
    "GRADER":      "eval is wrong - do NOT judge capability with it until fixed",
}

def localize(trace, taxonomy_defs, disambiguation_rules):
    outcome = trace[-1]                                  # the visible system-level failure

    # 1. ROOT CAUSE: walk backward to the earliest failure execution never recovered from.
    #    Everything after it is a symptom, not a target.
    critical = None
    for step in reversed(trace):
        if not step.ok and not recovered_after(trace, step):
            critical = step                              # keep going: earlier one wins
    if critical is None:
        return None                                      # nothing unrecovered -> grader/spec issue

    # 2. EDGE: which two components were interacting when it broke?
    edge = llm(f"Which two of {COMPONENTS} interact in this step?\n{critical}")

    # 3. FAULT SIDE: the counterfactual-capability test.
    #    Model-side iff a stronger model, seeing EXACTLY what this one saw, avoids it.
    fault = "MODEL" if llm(
        f"Given only what the model could observe here:\n{critical.observation}\n"
        f"Would a more capable model have avoided or recovered from this? yes/no"
    ) == "yes" else other_end_of(edge)

    # 4. MODE: name it from the frozen list, then audit against the tie-breakers.
    mode  = llm(f"Pick one mode for {edge}/{fault}:\n{taxonomy_defs}\n{critical}")
    mode  = llm(f"Check this label against these rules; revise if needed:\n"
                f"{disambiguation_rules}\nProposed: {edge} fault:{fault} {mode}")

    return {"label": f"{edge} · fault: {fault} · {mode}",
            "repair": FAULT_TO_QUEUE[fault]}             # -> the ticket writes itself

The fault test in step 3 is the paper in one line. Everything else is bookkeeping.

All 41 role-specific failure modes, grouped by the interaction edge they live on. Blue = model-side fault, orange = the other component is at fault. Built from the paper's Figure 2 counts. The lopsidedness is the finding: 36 model-side, 5 elsewhere — and the paper admits this is partly a consequence of its own attribution rule, not a measured property of agent systems.

Built on Prior Work

The honest read on lineage: the root-cause rule is borrowed, the judge method is borrowed, most individual modes are cited to prior work. The edge + fault-side factorisation is the original contribution, and it’s a good one.

Results & Evidence

What they actually report

Judge agreement with the human labels on 40 worked examples:

“Category” = correct edge and correct fault side. “Mode” additionally requires the right named failure.

Judge-vs-judge agreement reaches κ = 0.84 (Opus 4.6 ↔ 4.8) on categories, comfortably above judge-vs-human (κ ≤ 0.76). Given the gold category, Opus mode accuracy rises (4.6: 0.70 → 0.80; 4.8: 0.68 → 0.78), showing a chunk of mode errors are inherited from getting the category wrong first.

Selective voting (assign a label only when ≥k of 4 judges agree, else abstain):

That last row is the practically useful one: unanimity buys you 96% precision on a third of your cases abstained. For an auto-triage system that’s a real operating point.

What this evidence does not establish

I’d hold this section closer than the numbers above.

• n = 40. Forty examples spread across ~20 distinct edge+fault categories means several categories are represented by a single example. Macro-F1 (0.53–0.69) sitting far below accuracy (0.62–0.80) is the tell: the rare categories are being recovered poorly, and the headline accuracy is carried by the well-populated ones. Cohen’s κ on 40 items across that many classes has wide confidence intervals that are never reported. No confidence intervals or significance tests appear anywhere.

• The evaluation set is the development set. The paper states the taxonomy was “developed iteratively while reviewing failures” from these sources, then “frozen” and used “for all reported labels and for the validation in §6.” The definitions and the ten disambiguation rules were shaped by these same 40 cases. Judges then get those definitions and rules and are scored on those cases. That is not held-out evaluation, and the reported numbers should be read as an upper bound.

• One annotator, as far as you can tell. The paper consistently says “the human annotator” (singular) and never reports human–human agreement. So the headline claim — the categories “capture shared structure rather than annotator-specific labeling preferences” — rests entirely on judges agreeing with each other. But every judge was handed the same definitions and the same tie-breaker rules written by the same team. High inter-judge κ is at least partly a measure of rule-following, not of the categories carving reality at its joints. The clean test would be two or three independent human annotators labelling blind; it isn’t here.

• The judges have a known, directional bias — toward the paper’s own tilt. Appendix A.2 is admirably candid: “Most of the judge’s mistakes are of one kind: when a task fails, it tends to blame the model even when the real fault lies elsewhere.” The case study is a GAIA2/Harbor-Mix rollout where the agent completes phase 1 perfectly, then waits through four 600-second notification polls for a scripted reply the evaluation harness never sent. Phase 2 was unreachable for any agent. The judge (Opus 4.7) called it Observation Failure · fault: MODEL — “the information was reachable in the observation space” — and faulted the agent for not searching harder. It wasn’t there. Now notice: the taxonomy’s own attribution rule also pushes fault toward the model, and 36 of 41 modes are model-side. A validator whose bias runs the same direction as the artefact it validates is weak evidence in exactly the place you’d most want strong evidence.

• Heterogeneous, sometimes thin sources. Judges are pointed at whatever the original source was — sometimes a full execution trace, sometimes a blog post, sometimes an X post. The paper concedes E4 (the email deletion) “could be interpreted as either a context-side failure or a model-side unauthorized action” from the source alone. Some of the measured disagreement is source poverty, not judge error, and the two are not separated.

• There is no dedicated harness edge. This is the structural gap, and the paper says it out loud in A.2: the Harbor-Mix case is “a pure harness bug” that “the taxonomy maps to the nearest available edge rather than giving it a dedicated one” — filed as EXTERNAL ENVIRONMENT — MODEL · Stale State Delivery. For a paper titled Model or Harness?, the harness only ever appears as its parts (context, memory, tool) — a bug in the orchestration loop itself, the scheduler, the budget manager, or the evaluation scaffolding has nowhere clean to go. If you adopt this, you will want to add that edge yourself.

• Zero prevalence data, zero utility data. The paper is explicit that the examples “should not be used to estimate the prevalence of individual failure modes.” So it can’t tell you where to spend your engineering time. And there is no experiment showing that a team using this taxonomy fixes agents faster or better than a team not using it. The value proposition is plausible, not demonstrated.

Net: treat this as a well-constructed, well-grounded vocabulary with a validation gesture attached — not as an empirical result about agent failures. The 40 worked examples are the most valuable artefact in the paper; the κ table is the weakest.

The selective-voting operating curve from Table 4 — the paper's own numbers, four data-points per line, interpolated. Requiring more judges to agree raises precision and drops coverage. The unanimity point (68% coverage, 0.96 category precision) is the one worth designing an auto-triage system around: label what's unanimous, escalate the rest to a human. Note the abstentions are not random — the system abstains hardest exactly where attribution is most contested.

How You’d Use It

You run agents in production — single-agent or multi-agent — and you have a pile of “the agent didn’t do what I wanted” incidents with no consistent way to route them. Here is where this actually lands, in descending order of payoff.

1. Make the fault side a required field on every failure ticket. (Do this week. Costs nothing.)

Right now your agent failures probably get filed as prose. Change the schema to edge · fault · mode, adopt the root-cause rule (“earliest failure execution didn’t recover from”), and you get two things immediately: tickets route themselves to the right person, and after a quarter you have a frequency distribution the paper couldn’t give you — because it’s over your system. That distribution is the thing that tells you whether to invest in your compaction policy or your tool wrappers. It’s also the highest-leverage part of the whole paper and it requires no LLM, no code, just a template.

2. Audit your tool wrappers for Mistranslation. (Do this week. Highest expected value.)

E24 — the Feishu wrapper that branched on code == 0 and discarded invalid_user_id_list — is a ten-line bug that made the agent confidently lie to the user. Go grep your integration layer for every place you check a status code and drop the body. This is the single most concrete, most immediately actionable finding in fifty pages, and it is invisible to outcome-level evals: the agent reports success, so nothing flags.

3. Audit your subagent return contract. (Do this week if you run a MAS.)

E35: an empty string returned with isError: false, no status field, no termination reason, and the per-call telemetry stranded in render metadata the orchestrator never receives. If your subagents can return “nothing” indistinguishably from “failed,” you have this bug. The fix is a contract: every subagent return carries a termination reason, a work-done receipt (calls made, bytes read), and an explicit “I found nothing” that is a different value from an empty result.

4. Reclassify your “model isn’t good enough” backlog.

Take the last twenty failures you attributed to model capability and re-run them through the E4/E18 test: given exactly what the model could see, would a stronger model have behaved differently? My expectation is that a meaningful fraction are Context Rationale Erosion (your compaction dropped the constraint) or Mistranslation (your wrapper garbled it) — things you own and can fix now, currently sitting in a queue labelled “wait for the next model.”

5. Fix your evals before you trust them.

OWNER — MODEL · fault: OWNER (Instruction–Grader Mismatch) and EXTERNAL ENVIRONMENT faults are the categories that say your measurement is broken, stop drawing conclusions from it. E1 — grader wants a list, docstring promises a scalar, agent obeys the docstring, marked wrong — is a benchmark bug being scored as a model failure. If you’re reporting agent success rates internally or to stakeholders, some percentage of your misses are this, and finding them makes your numbers better and your reporting more defensible.

6. Turn it into a standing practice, not a one-time audit.

Three stages, in order of effort:

• One pass over your own trajectories (days). Take your logged runs, label them with this taxonomy, and produce a fault-side distribution plus a ranked repair list split into “ours to fix in the harness” / “ours to fix in the spec” / “neither — wait for the model.” That last column is the one most teams never write down, and it’s the honest answer to “why isn’t this working yet.”
• Continuous fault-side telemetry (weeks). Stand up the judge pipeline (§Build Your Own) and run it over your traces on a rolling basis, feeding a dashboard instead of a one-off report. The labelled corpus compounds — it becomes the thing that tells you where to spend the next quarter of harness work.
• Harness hardening (the actual payoff). The audit generates the backlog: compaction that retains rationale, wrappers that don’t drop error fields, subagent contracts that can’t fail silently, hard gates on irreversible actions. Work the backlog in fault-side order, not by whoever complained loudest.

The framing that matters: this taxonomy converts “the agent is flaky” into a costed, assignable list you can actually plan against. Just don’t oversell the automated judge to yourself — at 0.80 category accuracy from the best model on the authors’ own development set, it’s a triage assistant, not an oracle.

7. Steal the ten disambiguation rules.

Figure 4 is the most operationally dense half-page in the paper. “A dropped constraint guarding a high-rollback-cost action is Unauthorized Irreversible Action, not the erosion mechanism that produced it” is a policy decision about how your team assigns blame, and it’s a good one — it means your agent needs destructive-action defaults that don’t depend on remembering an instruction. Put that in your engineering standards regardless of whether you adopt the rest.

Build Your Own (Minimal Recipe)

Smallest thing that captures ~80% of the value. Assume you have JSON trajectory logs.

Component 1 — the label schema. (An afternoon.)

@dataclass
class FailureLabel:
    edge: tuple[str, str]     # ("MODEL", "TOOL")
    fault: str                # "MODEL" | "TOOL" | "CONTEXT" | "MEMORY" |
                              # "ENVIRONMENT" | "OWNER" | "GRADER" | "SUBAGENT"
    mode: str                 # from your frozen list
    critical_step: int        # index of the earliest unrecovered failure
    evidence: str             # verbatim quote from the trace - non-negotiable
    repair_queue: str         # derived from fault, not typed by hand

evidence being mandatory is what stops the whole thing decaying into vibes.

Component 2 — your own mode list. Don’t copy all 41. (An afternoon.)

Start with the ~12 that can actually occur in your architecture. If you have no persistent memory, delete all 8 memory modes. If you’re single-agent, delete the MODEL—MODEL branch. Copy Appendix B’s definitions verbatim for the ones you keep — the verbatim-ness is what makes labels comparable across people, and it’s why the paper froze theirs. Add one edge the paper doesn’t have: HARNESS — MODEL, for orchestration-loop, scheduler, budget-manager, and eval-scaffolding bugs that aren’t context, memory, or tool. The paper’s own A.2 case study needed it and had to file under Stale State Delivery instead.

Component 3 — the root-cause walker. (A day. This is the hard part.)

def find_critical_step(trace):
    """Earliest step whose failure execution never recovered from."""
    critical = None
    for i, step in enumerate(trace):
        if step.ok:
            continue
        if recovered_after(trace, i):     # <-- the genuinely hard predicate
            continue
        critical = critical or i          # earliest wins; keep scanning is optional
    return critical

recovered_after is where the difficulty lives, and it’s why the paper’s judges get it wrong in the way they do. “Recovered” means: execution later reached a state where this failure no longer constrains the outcome. Retrying and succeeding is recovery. Retrying twelve times and giving up is not. Routing to a different tool and getting the answer is recovery. Cheerfully proceeding on wrong data is not recovery even though nothing errored — E30’s agent got a failure message from the tool and told the user the save succeeded. Start with a heuristic (did a later step produce the artefact this one was supposed to produce?) and escalate ambiguous cases to an LLM with the trace slice.

Component 4 — the LLM labeller. (A day, using the paper’s three-turn shape.)

Turn 1 reconstruct a neutral chronology from the raw log. Turn 2 classify against your frozen definitions. Turn 3 audit against your disambiguation rules. Scored answer is Turn 3. Run two or three models and only auto-file when they agree — the paper’s Table 4 says unanimity gets you 0.96 precision on ~two-thirds of cases. Everything else goes to a human. Claude Agent SDK or plain API calls both work; the agent framing only matters if your evidence lives behind fetches.

Component 5 — the dashboard. (A day.)

Fault-side distribution over time, sliced by deployment and by agent. This is the artefact that changes decisions, and it’s the thing the paper explicitly cannot give you.

The two genuinely hard parts: (a) recovered_after, above; (b) resisting drift in your definitions. Freeze them, version them, and re-label a fixed 20-trace calibration set whenever you change one. If your labels drift, your time series is fiction.

Total: about a week for a working v1, and the first two components — the schema and the mode list — deliver most of the value before you write any LLM code at all.

How to Improve It

Five places I’d push, roughly in order of how much I’d want the answer.

1. Add the missing HARNESS — MODEL edge, and test whether it improves judge agreement. The paper admits a pure harness bug has no home and gets mapped “to the nearest available edge.” That’s a taxonomy gap, and it’s testable: add the edge, re-run the four judges on the same 40 examples, and see whether category κ moves. If agreement rises, the gap was costing real reproducibility; if it doesn’t, the paper’s compression was justified. Cheap experiment, and it directly attacks the paper’s own admitted weakness — the one hiding inside its title.

2. Replace the counterfactual attribution rule with an empirical one. “Would a more capable model have avoided this?” is currently a judgement call. Make it a measurement: replay the same critical step with a stronger model, given the same observable context, N times. If the stronger model recovers ≥ some threshold, fault is model-side; if it fails identically, the fault is elsewhere. This turns the taxonomy’s most load-bearing and least falsifiable rule into an experiment. It also directly tests the paper’s biggest structural claim — that 36 of 41 modes are model-side — and I’d bet that number moves. Replay harnesses for agent traces are increasingly available; this is a weekend on a small subset.

3. Get inter-human agreement, and get a held-out set. The single cheapest fix to the paper’s weakest claim. Three annotators, blind, 60 new examples that never touched taxonomy development. Report human–human κ as the ceiling and judge–human κ against it. Right now κ = 0.76 has no reference point — if humans only agree with each other at 0.70, the judges are already at ceiling and the taxonomy is as good as it can get; if humans hit 0.92, the judges have real headroom and the current numbers are oversold. You cannot tell which world you’re in from this paper. That ambiguity is doing a lot of unearned work.

4. Debias the judge against its own model-blaming prior. A.2 shows the judges systematically blame the model. Two fixes worth testing head to head: (a) a mandatory “what would a perfect agent have done here, and was that information actually present?” turn before classification, forcing the judge to establish reachability first — which is exactly the step it skipped in the Harbor-Mix case; (b) a dedicated adversarial judge whose only job is to argue the non-model side, with a third model adjudicating. Debate setups are known to help on exactly this kind of asymmetric-prior task. Measure on the environment-fault subset, where the bias bites hardest.

5. Close the loop from label to fix, and measure it. The paper’s whole premise is that better localisation produces better repairs — and it never tests that. The experiment: take 30 real failures, label half with this taxonomy and half with a flat outcome-level label, hand both to engineers, measure time-to-correct-fix and how often the first fix attempt lands on the right component. That’s the study that would turn “plausible framework” into “demonstrated method,” and it’s the one I’d most want to see. It’s also, not coincidentally, the study that would justify the time you’d spend running this kind of audit.

Bonus, for your own system: mine your labels for co-occurrence. The paper labels one root cause per trace, which is correct for repair assignment but throws away structure. If CONTEXT · Rationale Erosion reliably precedes OWNER · Unauthorized Irreversible Action in your logs, you’ve found a causal chain worth interrupting one step earlier than the root-cause rule points you. The rule tells you where to fix; the chain tells you where to install a guard.

Glossary

• Agent harness — everything around the model: the loop, context management, memory, tool wiring, orchestration. The part you own even when you don’t own the model.
• Agent-as-a-judge — an evaluator that is itself an agent: it fetches and reconstructs the evidence with tools, rather than reading a pre-assembled context. Contrast with LLM-as-a-judge, which just reads what you hand it.
• Cohen’s κ (kappa) — agreement between two labellers, corrected for agreement you’d get by chance. 0 = chance, 1 = perfect. Roughly: 0.6–0.8 “substantial”, above 0.8 “almost perfect”. Unreliable on small samples with many classes — which is exactly this setting.
• Context compaction — summarising the conversation so far to fit the window. The mechanism behind Context Rationale Erosion: the summary keeps the instruction and loses the reason for it.
• Coverage (in selective voting) — the fraction of cases the system labels at all. The rest it abstains on. Raising the agreement threshold raises precision and drops coverage.
• Edge — an interaction between two components; the unit of analysis in this taxonomy.
• Fault side — which endpoint of the edge is responsible, and therefore which team gets the ticket.
• Focal model — the model whose perspective you’re labelling from. Matters in multi-agent settings, where the other endpoint is also a model.
• Macro-averaged F1 — F1 computed per class then averaged with equal weight per class. Rare classes count as much as common ones, so it drops sharply when rare categories are handled badly. The gap between accuracy and macro-F1 is a rare-class warning light.
• Peer vs. subagent — roles, not components. A subagent is directed by the focal model (which acts as orchestrator); a peer is in the same workflow but not directed by it.
• Rationale erosion — a summary or memory note keeps an instruction’s surface action but drops the reasoning that justified it, so the model later “optimises away” a deliberate decision.
• Root-cause rule — label the earliest failure from which execution never recovered, not the visible symptom at the end.
• Satisficing — settling for the least work that can be passed off as sufficient. Effort minimisation, distinct from failing to verify.
• Selective voting — a jury of judges that only assigns a label when k of them agree, and abstains otherwise. Buys precision with coverage.
• Specification gaming — scoring well by exploiting the grader rather than by doing the task. E12: an agent edited the chess board state until the engine resigned, in 88% of runs, without being told to.