From Agent Behaviour to Agent-Friendly Documentation - An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation

TL;DR

Technical documentation is written for humans, but a growing share of code is now written by autonomous agents, and nobody had measured what those agents actually read. This paper mines 94,813 events from 557 real agent sessions (SWE-chat) plus 690,260 file-change records from 33,097 agent-opened pull requests (AIDev), and codes every documentation touch by type, trigger, and what happened next.

Four things fall out, and all four contradict the current “write agent-friendly docs” advice. One: the documentation agents care about is a genre that barely existed two years ago — instruction files (35.4%) and the agent’s own working notes (25.1%) — while API reference docs, the target of most documentation tooling, get 1.3%. Two: the read-then-code link is nearly invisible at the adjacent-step level (P(edit code | read doc) = 0.002); reads are followed by more reads (0.270) or by reasoning (0.245). Three: zero observed instances of an agent checking its code against prose documentation, and consultation is followed by less testing and building, not more. Four: agents write documentation almost as often as they read it (0.87×), and 41.5% of agent PRs touch docs — but code comes first 4.7× more often than docs do.

The practical takeaway for anyone shipping agentic systems: your CLAUDE.md is roughly 27× more likely to be read than your API reference prose, and “verifiability” as a documentation property has no behavioural support — if you want agents to check their work against a spec, the spec has to be executable, not prose.

Problem & Motivation

Here is the pain in one sentence: the entire field of documentation quality research assumes a reader who gets confused and asks a colleague, and that reader is increasingly not the one reading your docs.

Everything downstream of that assumption is now suspect. Staleness metrics, readability guidelines, API documentation taxonomies, “documentation smells” — all built on studies of human developers from 2003 to 2020. Meanwhile a large and growing fraction of open-source commits comes from agents that read repositories, execute shell commands, and open PRs with no human in the loop at each step.

The response so far has been advice-by-intuition. “Write clear headings.” “Provide runnable examples.” “Publish an llms.txt.” “Make your docs actionable and verifiable.” Every one of these is a claim about how agents ought to behave, published without anyone measuring how they do behave. That is the gap.

Why nobody had measured it: the evaluation apparatus of the field defines an agent’s job as issue resolution. SWE-bench scores whether the patch makes tests pass. No benchmark rewards documentation work, so no benchmark generates documentation data. Evidence has to come from observation — and until 2026, public corpora of real agent transcripts and agent-authored PRs did not exist at scale. Now two do, with complementary blind spots: SWE-chat has full tool-call trajectories but no merge outcomes; AIDev has artefacts and merge outcomes but no trajectories.

The authors’ framing move is the interesting part. They deliberately invert the usual research direction: instead of proposing documentation qualities and testing whether agents benefit, they observe agent behaviour first and derive the implications. That means the paper can and does end with a list of implications its own data refuse to support — which is rarer and more useful than the list it does support.

What’s New (Core Contribution)

Four contributions, and it is worth separating the genuinely new from the merely first-to-publish.

1. A behaviour-grounded map of what agents read — including a document genre nobody had a category for. Before: documentation taxonomies (Aghajani et al., README taxonomies, API doc research) had categories like API reference, tutorial, troubleshooting, architecture. Now: two categories that dominate real agent behaviour and appear in none of those taxonomies — agent_instruction (AGENTS.md, CLAUDE.md, SKILL.md, Cursor/Copilot rule files) and agent_working_note (plans, thoughts/, brainstorms, verification logs the agent wrote for itself). Together 60.5% of interaction. The second category was not even in the authors’ own initial coding scheme; it emerged from classifying paths that Tier 1 could not resolve. That is the strongest single result in the paper, and it is genuinely new.

2. A released extraction pipeline that survives four incompatible transcript formats. Before: trajectory studies coded action sequences but not artefact classes; artefact studies mined PRs but not trajectories. Now: an extractor that emits a common 20-symbol event alphabet across four transcript encodings, validated against SWE-chat’s own tool_call_count (exact match in 5 of 6 spot-checked sessions). Two of its bug fixes are contributions in themselves — see §Results for why shell-routed file operations will silently zero out entire agent families in any naive analysis.

3. A trace-derived interaction model that replaces the assumed linear journey. Before: the implicit model borrowed from human information-seeking — Discover → Retrieve → Interpret → Apply → Validate → Update. Now: a two-lobed cycle where a self-recurrent consultation lobe and a large production lobe are only loosely coupled, with Validate and Escalate at literally zero events and Apply at 75.

4. An explicit negative-results list. Before: “agent-friendly documentation should be actionable and verifiable” circulated as received wisdom. Now: both properties are named and shown to have no consistent behavioural support in this corpus, with the specific measurement attached to each refutation. This is the part most likely to change what you do on Monday.

What is not new: applying co-change mining to PRs (mature literature), cluster bootstrapping (standard), or the observation that context files exist (several 2025–2026 papers describe AGENTS.md as an artefact). The novelty is the denominator — how often agents touch these files relative to everything else they read — which nobody had.

How It Works (Technically)

This is an empirical study, so “how it works” means the measurement instrument. Get the instrument wrong and every number is wrong, so the authors spend an unusual amount of space on it — correctly, because two of their own bugs materially changed results before they were caught.

The two datasets and why neither alone would do

Crucially, the authors never pool them. Different populations, different units. Agreement between them corroborates a pattern; it is not cross-validation.

Sampling: deliberately non-proportional, and honest about it

From SWE-chat they draw a sample stratified by agent and by session length (four strata: ≤3 turns, 4–8, 9–18, >18) so neither trivial nor pathological sessions dominate, and they oversample minority agents so per-agent estimation is possible at all. 559 sampled, 557 parseable.

That choice creates a debt: pooled statistics now overweight long sessions and minority agents. They pay it in §4.4 by reporting every headline number under three weightings (pooled-event, session-equal, agent-reweighted to the corpus distribution of 83.8% Claude Code). More on that in Results — it matters, because one headline claim moves to the edge of a coin flip.

The two-tier documentation classifier

This is the core mechanism. “Documentation” is operationalised as a function over repository-relative file paths.

Tier 1 — deterministic rules. Filename and path patterns assign one of 15 document types, plus two orthogonal flags:

• machine_readable — OpenAPI, JSON Schema, Protobuf. These are simultaneously API documentation and executable specifications, so they get a flag rather than being forced into one bucket.
• vendored — third-party paths. Deliberately flagged, not dropped: reading node_modules/pkg/README.md is a genuine documentation interaction even though the repo does not own the file.

Non-documentation files get a kind in {source, config, test, data, build, other}, so one function serves both documentation identification and the AIDev co-change analysis. Clean design.

Tier 2 — LLM resolution of the residual. Tier 1 dumped 54% of documentation events into a residual “other” bucket. Too big to leave. They took the 527 distinct paths in that bucket and classified them with a language model: 500 labelled (covering 98.4% of ambiguous events), 27 by keyword fallback. This tier is where agent_working_note was discovered — the residual was dominated by agent-authored planning and reasoning documents that nobody’s taxonomy had a name for.

Note the honesty flag here, because it is load-bearing: these Tier-2 labels have no human validation. A quarter of the headline finding rests on unvalidated LLM classification. The authors say so explicitly and name the fix (dual human coding of 200–300 events, Cohen’s κ or Krippendorff’s α). Treat the 25.1% as provisional; the existence of the category is visible in raw paths and is the robust part.

Event extraction: the two bugs that would have broken the study

SWE-chat uses four incompatible transcript formats (406 sessions line-delimited JSON with content-block tool calls; 100 a single JSON doc with a parts array; 43 a {type, payload} event log; 10 a messages array). A separate extractor per format emits one common schema, so documentation events stay embedded in their original trajectory context — which is what makes transition analysis possible at all.

Two extraction details are worth internalising if you ever mine agent traces yourself:

1. Shell-routed file operations hide their paths inside command strings. Agents that do file work via apply_patch heredocuments put the target path only in the command text. Before the extractor parsed those paths, one entire agent family registered zero documentation events. Any corpus analysis keyed on tool names will systematically undercount shell-centric agents — and then “cross-agent differences” are really extraction-coverage differences.
2. Tool output arrives as a string in some formats, a list or dict in others. Handling the extra types recovered 9 sessions and 4,358 events, including 77 documentation events.

Both bugs were found because results looked implausible, not because a test caught them. Take that as a warning about your own telemetry.

The coding scheme: four dimensions per event

Every one of the 3,033 documentation events gets:

• Document type — 15 rule categories + the 2 discovered ones.
• Interaction type — Discover, Search, Read, Edit, Create. (Three types from the initial scheme are unattested and were removed rather than reported as rare: Compare — reading two docs against each other; Follow-reference — navigating a link between docs; Verify — checking code against docs. These may happen inside model reasoning, which the instrument cannot see.)
• Trigger — assigned from a four-event lookback window.
• Outcome — success/failure signals regex-matched over tool output.

And a development stage from a trajectory heuristic: orientation before the first write; implementation from the first write; verification after a passing test or build; debugging after a failure signal; delivery after the last write when version-control activity dominates.

One design decision deserves applause: they refuse to code interaction purpose. It was in the initial scheme and they cut it, because purpose is not recoverable from tool-call logs — a file read is compatible with many intents and assigning one would be unfalsifiable. Most papers would have guessed and reported it.

Every event keeps its evidence string, so labels are auditable.

Architecture & data flow

flowchart LR
  SC[(SWE-chat<br/>5,850 sessions)] -->|stratify by agent + length| SAMP[557 parseable sessions]
  SAMP -->|4 format-specific extractors| EV[94,813 events<br/>20-symbol alphabet]
  EV -->|path classifier Tier 1| T1[15 doc types + flags]
  T1 -->|54% residual| T2[Tier 2: LLM labels<br/>527 paths]
  T2 --> CODED[3,033 coded<br/>doc interactions]
  CODED --> AN[Transition · lift · GEE<br/>cluster bootstrap]
  AID[(AIDev<br/>33,097 agentic PRs)] -->|same path classifier| FC[690,260 file x commit rows]
  FC --> AN
  AN --> MODEL[Two-lobed<br/>interaction model]

The central empirical result, drawn from Table 1 of the paper: all 3,033 documentation interactions by document type. The two agent-facing categories are highlighted. Hover a bar for exact counts; use the toggle to switch to a magnified axis so the "classical" documentation genres are visible at all.

Demystifying the statistics (three ideas, no notation required)

The paper leans on three statistical devices. None is exotic, but each changes how you should read the numbers.

1. Lift over a base rate — not raw conditional probability. If you ask “how often does an agent edit code within 3 events of reading a doc?” and get 23%, that means nothing on its own, because editing code is common everywhere in a session. So they compute:

lift = P(action within 3 events of a documentation consultation) ÷ P(action at a non-anchor event in the same sessions)

with 1,615 consultation anchors and 93,198 non-anchor baseline events. Lift = 1.0 means “no different from background.” Lift = 1.67 means the action is 67% more likely near a consultation than elsewhere. Lift = 0.23 means it is four times less likely. That last one is the paper’s most robust positive finding, and it points the wrong way from everyone’s expectations.

2. Cluster bootstrap — because events are not independent. Events nest inside sessions; PRs nest inside repositories. The largest AIDev repository alone contributes 8,911 pull requests. Treating these as 33,097 independent coin flips would give you absurdly tight confidence intervals. Instead: resample whole sessions (or whole repositories) with replacement 2,000 times, recompute the pooled proportion each time from summed within-cluster counts, and take percentiles. Fixed seeds (13, 11, 7) so it replicates.

The effect is large and worth remembering next time you see a confident percentage from a repo-mining paper: clustering widened every interval, by up to 14.4× on the AIDev side. Wilson intervals (which assume independence) stay in the tables as a reference only.

3. Stage-adjusted logistic GEE — because consultation is not evenly spread through a session. A GEE (Generalised Estimating Equation) is a regression that accounts for correlated observations within clusters — here, an exchangeable correlation structure clustered by session. They adjust for development stage, within-session position, log session length, and agent family, and report the adjusted odds ratio alongside the unadjusted lift, not instead of it.

Why this matters: for two of the four outcomes the adjusted and unadjusted estimates disagree about whether the effect exists. Documentation creation is elevated unadjusted (lift 1.67, CI 1.14–2.31) but its adjusted interval includes 1 (OR 1.41 [0.98, 2.02]). Code editing is flat unadjusted (lift 1.05, CI 0.86–1.27) but elevated adjusted (OR 1.33 [1.09, 1.62]). The authors’ response is the correct one: declare those associations unresolved rather than picking the specification that tells a better story.

The observation scope — read this before quoting any number

The instrument sees repository-local, file-based documentation interactions: tool calls whose target resolves to a repository path, plus rare explicit documentation-retrieval calls. It does not see:

• API websites read through a browser,
• knowledge already baked into the model’s weights,
• in-source docstrings and inline comments,
• context files auto-loaded by the runtime at session start (visible only if the agent later reads or edits them explicitly — so instruction-file counts are lower bounds on exposure).

All absolute rates are therefore floors. The comparative findings survive unless in-source documentation is distributed very differently across categories than path-identified documentation, which nobody can rule out.

The algorithm, simplified

The “algorithm” here is the measurement. Here is the core of it — event coding plus the lift computation with a cluster bootstrap — in about 40 lines.

# ---- 1. Turn one raw transcript into a stream of typed events -------------
def extract_events(session):
    """Emit a common 20-symbol alphabet regardless of transcript format."""
    fmt = detect_format(session)                 # 4 incompatible encodings
    for call in FORMAT_EXTRACTORS[fmt](session):
        path = call.target_path or parse_path_from_shell(call.command)
        # ^ shell-routed edits hide the path inside apply_patch heredocs.
        #   Skip this and one whole agent family reports ZERO doc events.
        yield Event(symbol=call.symbol, path=path, output=coerce_output(call.result))
        #                                          ^ output is str | list | dict

# ---- 2. Code each event; documentation is a property of the PATH ----------
def code_event(ev, history):
    doc_type = tier1_path_rules(ev.path)         # 15 types + machine_readable/vendored
    if doc_type == "other_prose":                # 54% landed here before Tier 2
        doc_type = llm_label(ev.path)            # <-- this is where agent_working_note appeared
    return Coded(
        doc_type      = doc_type,
        interaction   = classify(ev.symbol),     # Read/Edit/Create/Search/Discover
        trigger       = infer_trigger(history[-4:]),   # fixed 4-event lookback
        stage         = stage_heuristic(history),      # sticky: stays in `debugging` until green
        outcome       = regex_outcome(ev.output),
        evidence      = ev.raw,                  # every label stays auditable
    )

# ---- 3. Is an action more likely NEAR a consultation than elsewhere? ------
def lift(events, action, horizon=3):
    anchors  = [i for i, e in enumerate(events) if e.interaction in CONSULT]  # 1,615
    p_after  = mean(any(events[i+1:i+1+horizon] has action) for i in anchors)
    p_base   = mean(e.symbol == action for i, e in enumerate(events) if i not in anchor_span)
    return p_after / p_base          # 1.0 = indistinguishable from background

# ---- 4. Uncertainty: resample SESSIONS, never events ----------------------
def cluster_bootstrap(sessions, stat, B=2000, seed=13):
    rng, draws = random.Random(seed), []
    for _ in range(B):
        resample = [rng.choice(sessions) for _ in sessions]   # whole clusters, with replacement
        draws.append(stat(flatten(resample)))                 # recompute from summed counts
    return percentile(draws, 2.5), percentile(draws, 97.5)    # up to 14x wider than Wilson

The model the traces actually support

The authors started with the linear journey borrowed from human information-seeking:

flowchart LR
  D[Discover] --> R[Retrieve] --> I[Interpret] --> A[Apply] --> V[Validate] --> U[Update]

Then they counted events for each stage (Table 10): Contribute/Update 1,401, Retrieve 1,344, Orient 462, Interpret 413, Revisit 360, Discover 287, Recover 109, Apply 75, Validate 0, Escalate 0.

Three things break the linear model at once. Two stages have zero events under the operational definitions. Apply — the model’s central step — is the weakest attested stage at 75 events. And the terminal stage is the largest: Contribute/Update outnumbers even Retrieve.

What replaces it is a two-lobed cycle:

flowchart TD
  subgraph CONSULT["Consultation lobe (self-recurrent)"]
    O[Orient] --> DIS[Discover]
    DIS --> RET[Retrieve / Read]
    RET -->|0.270 - strongest edge| RET
    RET -->|0.245| REA[Reasoning]
    REA --> RET
  end
  subgraph PROD["Production lobe (largest, 1,401 events)"]
    CU[Contribute / Update<br/>plans, notes, instructions]
    CU -->|0.350| CU
  end
  RET -.->|0.107 read doc -> edit doc| CU
  RET -.->|0.002 read doc -> edit code<br/>3 of 1,328| CODE[Code modification<br/>largely independent]
  FAIL[Failure episode] -.->|only 5.4%| RET
  CU -.->|no observed edge| VAL[Validation]

Read that diagram as the paper’s actual thesis: agents circulate inside a consultation lobe, spill into reasoning and into writing more documentation, and their code modification process runs mostly beside it rather than downstream of it. Documentation looks less like a reference and more like externalised working memory.

The two-lobed cycle as a 3D graph you can orbit. Node size is event count (Table 10); edge thickness is the measured transition probability. Note how thin the read-doc-to-edit-code edge is (0.002) versus the self-loops inside each lobe (0.270 and 0.350). Drag to rotate, scroll to zoom.

The authors offer two candidate mechanisms and decline to choose between them, which is the right call:

1. Bounded context windows. Agents externalise reasoning to files because they cannot hold it in context — documentation as working memory, not reference. The prominence of plans and thoughts/ directories fits this.
2. A cheaper oracle exists. Agents do not validate against prose because the test suite is right there and is executable. Why argue with a paragraph when you can run pytest?

Either way: prose was not observed functioning as a specification.

Built on Prior Work

Results & Evidence

The headline table

Where documentation work actually happens in a session

By development stage: 54.4% debugging, 27.2% implementation, 15.2% orientation, 3.0% verification, 0.1% delivery.

Do not read that as “agents mostly consult docs while debugging.” The authors flag their own heuristic as sticky — once a failure signal appears, the session stays in debugging until a test or build passes, which inflates the share. The claim they actually advance is the negative one, and it survives: documentation consultation is not confined to orientation. Any mental model where docs are a task-start activity is inconsistent with the traces.

Consultation vs. production by type — the asymmetries are informative

And in AIDev, the most-changed individual documentation files include AGENTS.md (692 PRs), CLAUDE.md (362), copilot-instructions.md (287). Agents are modifying the files that shape agent behaviour. That closes a second loop — agent output feeding back into agent input — that no existing documentation model captures. If you run agents in production, that loop is a governance problem you probably do not have a review process for.

The RQ2 result that matters most, from Table 3: unadjusted lift (with cluster-bootstrap CI) next to the stage-adjusted odds ratio, for each of the five downstream actions. The vertical line is 1.0 — "no different from background." Only testing and building sit clearly below it under both specifications. The two authoring outcomes flip sides depending on which model you fit, which is exactly why the authors call them unresolved.

What the evidence does NOT establish

The paper is unusually disciplined here, and its restraint list is worth copying:

• It does not establish that documentation is useless to agents. It establishes that a particular assumed behavioural coupling is absent from tool-call traces. Influence mediated through reasoning the instrument cannot see would be invisible to both analyses.
• It does not establish that agents never validate. Zero Validate events means zero instances of the operational pattern “consultation followed by a test or build run.” Validation via a test suite invoked independently is not counted and is probably very common.
• It does not rank failure-recovery strategies. Documentation-based recovery has the highest point estimate (7/11 = 63.6%) but an interval of 35.4–84.8% that overlaps every alternative. Eleven observable episodes. The authors call it “suggestive and explicitly not a finding,” which more papers should do.
• It cannot compare agents. Per-agent rates range from 62.6% (Claude Code, 238/380) to 37.2% (Codex, 16/43) to 0/11 (Cursor) — but one agent routes nearly all file work through the shell, and before path-parsing was added it registered zero events. Cross-agent comparison is confounded with extraction coverage. The 0/11 is not evidence of absence.
• It cannot see docstrings, inline comments, browser-read API sites, or model-weight knowledge. Absolute rates are lower bounds. Languages and projects that favour in-source documentation are systematically underrepresented.
• It does not validate the Tier-2 labels. A quarter of the headline number rests on unvalidated LLM classification of 500 paths.
• The population is skewed. SWE-chat is opt-in telemetry, 87% from a single agent family; developers who opt in may be more experienced or more open-source oriented. AIDev over-represents public repos that adopted agents early. Neither generalises to private codebases.
• It is a snapshot of a two-year-old, fast-moving practice. The authors expect the existence of the agent-facing category to persist; the 60.5% is not a constant.
• No multiple-comparison correction across the strata examined (agent, language, task type, star bucket, outcome), so small differences between adjacent strata should not be interpreted — and the authors base no claim on one.

One genuinely uncomfortable weighting result deserves a call-out. Under session-equal or agent-reweighted estimation, agent-facing consultation drops from 57.4% to 50.5% / 50.1% — right at the 50% line. So “agent-facing documents are a majority of what agents consult” is weighting-dependent; “about half” is the safe statement. Production goes the other way (63.7% → 67.1% / 66.3%), so the production-side claim is if anything strengthened.

How You’d Use It

Translate the findings into rules you can apply to your own repos and to any codebase your agents work in. Each rule below is tied to a specific measurement, and each is stated so you could falsify it in your own telemetry.

Rule 1 — Budget documentation effort by observed read rate, not by tradition

Instruction files get ~27× the interactions of API reference prose (1,074 vs 40 events). If you have ten hours of documentation budget on an agent-touched repo, the allocation implied by this data is roughly: six hours on CLAUDE.md / AGENTS.md / skill docs, two on task and requirements files, one on README, one on everything else. Not because API docs are worthless to humans, but because that is where the agent’s attention demonstrably goes.

For your own repos: this reframes what a documentation pass even is. “Rewrite the API reference” is a human-audience project. “Make the repo agent-legible” is a different, cheaper, higher-leverage project, and this paper is the citation that justifies spending the time on it.

Rule 2 — Write self-contained documents; do not rely on cross-links

Follow-reference is entirely unattested. Zero observed instances of an agent navigating a link between documents. Meanwhile P(read doc | read doc) = 0.270 — agents read in runs, opening file after file.

Concretely, in a CLAUDE.md:

• Inline the thing rather than linking to it, whenever the thing is under ~30 lines.
• When you must point elsewhere, give an exact path the agent can open with one tool call, never a URL and never “see the architecture docs.”
• Assume each file will be read cold, without its siblings.

Caveat the paper itself adds: this does not prove link hygiene has no behavioural consequence. It proves link-following is not observable in tool calls. Treat it as a strong prior, not a law.

Rule 3 — Stop writing prose you expect the agent to check its work against

Zero validation events. Not rare — zero. If you want an agent to verify against a specification, the specification has to be something it can execute: doctests, runnable examples, JSON Schema, OpenAPI contracts, a make check target, a test file. Prose that says “responses must include a request_id field” is, behaviourally, a suggestion.

This is the single most actionable finding in the paper. In a CLAUDE.md, replace:

“All API responses must be validated against the schema before returning.”

with:

“Run python scripts/validate_schema.py before you finish. It must exit 0.”

The second one produced observable behaviour in this corpus. The first one is a property (verifiability) the authors show has no behavioural support.

Rule 4 — Treat thoughts/ and plan files as a first-class, governed artefact class

Agent working notes are 25.1% of all documentation interaction and are produced almost exactly as often as they are consumed (367 vs 382). They accumulate in repositories as durable artefacts. Repository hygiene tooling, code review checklists, and documentation quality metrics currently have no category for them.

If you run agents against your own repos, that is a live liability worth a deliberate policy:

• A retention or archival policy for agent-authored notes (they grow without bound — see the “Why does CLAUDE.md keep growing?” literature).
• A review rule: agent-authored planning documents in a PR get read, or get excluded from the diff, but do not get merged unexamined.
• A linter for staleness in the notes directory.

Rule 5 — Guard the feedback loop where agents edit their own instructions

AGENTS.md is among the most-changed files in agentic PRs (692), followed by CLAUDE.md (362) and copilot-instructions.md (287). Agents modify their own configuration. That is agent output becoming agent input with, in most repos, no gate on it.

Minimum viable control: require human approval on any diff touching an instruction file, the same way you would for CI configuration or a deploy script. This costs nothing and closes a hole that, per the paper, is already being exercised at scale.

Rule 6 — Do not build your agent design around “docs as the failure-recovery resource”

Documentation is the first recovery move in 5.4% of failure episodes. Agents read code (31.0%), retry the same thing (19.9%), do nothing (15.6%), or edit directly (15.3%) first. Troubleshooting documents total 11 events across the entire corpus.

So: writing a beautiful TROUBLESHOOTING.md and expecting agents to find it when stuck is not supported. If you want a stuck agent to reach a remedy, the remedy has to be reachable through what the agent actually does when stuck — which is read code and retry. Put the fix in a runnable script the error message names, not in a guide.

Rule 7 — Instrument your own agents before believing any of the above about your stack

The paper’s own two extraction bugs are the lesson. If your agents route file operations through shell commands, tool-name-based telemetry will report that they never read documentation. Before you re-allocate a documentation budget on this paper’s numbers, spend a day parsing paths out of your own agent traces and reproduce the document-type distribution for your repos. It is a genuinely small project — the paper releases the extraction pipeline, the coding scheme, and the event-level data.

Build Your Own (Minimal Recipe)

You can reproduce the useful 80% of this study over your own agent fleet in about a week. That gives you a repeatable audit you can also sell.

What you are building: a pipeline that turns raw agent transcripts into a coded event table, then answers three questions — what do my agents read, what triggers it, what happens next.

Build order:

1. Get the traces (half a day). Claude Code writes session JSONL under ~/.claude/projects/<sanitised-cwd>/. Other CLIs have their own. Do not normalise yet; just inventory the formats you actually have. The paper found four incompatible ones across six agent families — expect the same shape of mess.

2. Write one extractor per format, emitting a common event schema (2 days — this is hard part #1). Fields: session_id, index, symbol, tool_name, target_path, raw_output, timestamp. A ~20-symbol alphabet is the right granularity: read_file, edit_file, create_file, search, run_test, run_build, shell, reasoning, ask_user, vcs, …. Parse paths out of shell command text, including heredoc targets — this is the bug that zeroes out entire agent families. Coerce tool output that may be str, list, or dict. Validate your extractor against any independent count the transcript already carries (a tool_call_count field, or a manual count on three sessions).

3. Write the Tier-1 path classifier (half a day). Pure function, path -> (doc_type, kind, machine_readable, vendored). Start from the paper’s 17 categories. Get agent_instruction (AGENTS.md, CLAUDE.md, SKILL.md, .cursor/rules/*, .github/copilot-instructions.md) and agent_working_note (thoughts/, plans/, *-plan.md, brainstorm*, verification logs) in from day one — the paper’s whole discovery was that these are missing from every pre-2024 taxonomy.

4. Resolve the residual with an LLM (half a day). Collect the distinct paths that fall into other, batch them, and label with a cheap model — Haiku is plenty for path-string classification. Cap the batch, cache by path string, keyword-fallback the rest. Then hand-label 200 of them yourself and compute Cohen’s κ against the model. That is the step the paper skipped and flagged; doing it makes your version more rigorous than the published one.

5. Code triggers and stages with heuristics (half a day). Four-event lookback for trigger; a sticky state machine for stage. Both are crude and both are fine as long as you only advance the negative claims they support.

6. Compute lift with a cluster bootstrap (1 day — hard part #2). Not because the code is difficult (it is ~20 lines, see the pseudocode above) but because getting the baseline right takes care: the baseline must be non-anchor events in the same sessions, and resampling must be over whole sessions. Use scipy.stats.bootstrap if you like, or write the loop. If you want the stage-adjusted comparison too, statsmodels.GEE with an exchangeable correlation structure clustered by session — about ten lines.

7. Report every headline number under three weightings. Pooled-event, session-equal, and reweighted to your true agent mix. If a number moves across a threshold under reweighting, say so; do not pick the flattering one.

Libraries and models: pandas or polars for the event table, statsmodels for the GEE, scipy or a hand-written bootstrap, a small fast model (Haiku 4.5) for Tier-2 path labelling. No training, no GPUs, no fine-tuning. This is an ETL project with statistics on the end.

Where the effort actually goes: roughly 60% of it is step 2. Transcript formats are undocumented, change between versions, and hide the information you need in string fields. Budget accordingly.

How to Improve It

Five directions, each testable, ordered by how much they would change what anyone does.

1. Run the intervention study the paper explicitly declines to run. This is observational — it can show correlation and absence, never effect. The obvious next move: take 50 repos, randomise half to receive an executable specification (doctests, schema contracts, a named make check) and half to receive equivalent prose, then measure whether Validate events go from zero to non-zero, and whether merge rates move. The paper’s own §6.1 calls this a “hypothesis for intervention studies.” Whoever runs it owns the finding, and it is a two-week project with a modern agent harness.

2. Fix the instrument’s biggest blind spot: in-source documentation. Docstrings and inline comments are invisible because documentation is identified by path. That is a real gap — a Python-heavy repo does most of its documenting inside .py files. Improvement: parse edited/read regions (line ranges from the diff or the read call) and classify whether the touched region is docstring/comment or code. This is tractable with tree-sitter and would let you answer a question nobody has: do agents maintain docstrings, or only prose files? My prior is that the 60.5% agent-facing share drops meaningfully once in-source docs are counted.

3. Extend the horizon and the model to catch mediated influence. The near-zero adjacent transition (0.002) is a first-order statement, and the three-event lift is a thin patch. Reads are followed by reasoning 24.5% of the time — so influence almost certainly flows through reasoning text the tool-call instrument cannot see. Improvement: run a sequence model (or simply a longer-horizon lift with horizons 3/5/10/20) over the full symbol alphabet, and separately, if you have the reasoning text, embed it and test whether post-read reasoning is more semantically similar to the read document than baseline reasoning is. That would convert “unresolved” into an answer.

4. Validate the coding scheme, then publish the κ. A quarter of the headline number is unvalidated LLM labelling. Dual human coding of 200–300 events with Cohen’s κ or Krippendorff’s α is a day of work for two people and would move the agent_working_note finding from provisional to solid. Low glamour, high value — and it is the authors’ own stated next step, so it is available.

5. Separate the two mechanisms the paper cannot distinguish. Is documentation working memory (context-window pressure) or a bypassed reference (the test suite is a cheaper oracle)? Testable: vary context window across otherwise-identical runs. If working-notes production drops as the context window grows, mechanism one. If it does not budge, mechanism two. This matters commercially — if agent notes are a context-window symptom, they will shrink as windows grow and you should not build tooling around them; if they are a genuine externalisation strategy, they are permanent and worth investing in.

Bonus, and the one I would actually build: the paper measures behaviour but never measures quality. Nobody knows whether a well-written CLAUDE.md produces better outcomes than a random one — and one cited study found random rules help as much as curated ones. A benchmark that holds the task fixed and varies only the instruction file, scored on task success rather than on agent self-report, would be the highest-value missing artefact in this whole area. It is also the sort of thing worth publishing in its own right if you run the benchmark.

Glossary

• Adjacent transition probability — the chance that event B immediately follows event A. P(edit code | read doc) = 0.002 means that of 1,328 documentation reads, exactly 3 were followed by a code edit as the very next action.
• Agent instruction file — a file whose purpose is to configure agent behaviour: AGENTS.md, CLAUDE.md, SKILL.md, Cursor/Copilot rule files. 35.4% of observed documentation interaction.
• Agent working note — a durable prose artefact the agent wrote for its own use: plans, thoughts/ directories, brainstorms, verification logs. 25.1% of interaction; a category that did not exist in the authors’ initial scheme.
• AIDev — a public dataset of pull requests opened by coding agents on public GitHub repos, with commits, file-level diffs, reviews, and timelines. Used here for artefact-level evidence.
• Anchor — in the lift analysis, a documentation consultation event whose following three events are examined. 1,615 of them.
• Cluster bootstrap — a confidence-interval method that resamples whole groups (sessions, repositories) rather than individual observations, because observations inside a group are correlated. Widened intervals here by up to 14.4×.
• Co-change — two artefact types (here code and documentation) modified within the same pull request. Observed in 32.0% of agentic PRs.
• Cohen’s κ (kappa) — a statistic for agreement between two human coders that corrects for agreement by chance. The authors report none; they name it as the missing validation step.
• GEE (Generalised Estimating Equation) — a regression that handles correlated observations within clusters. Used here to adjust the lift analysis for development stage, session position, session length, and agent family.
• Lift — the probability of an action near a documentation consultation, divided by its background probability elsewhere in the same sessions. 1.0 = indistinguishable from background.
• machine_readable flag — marks OpenAPI, JSON Schema, and Protobuf artefacts, which are simultaneously API documentation and executable specifications.
• Odds ratio (adjusted) — the association between consultation and a following action after statistically removing the influence of the confounders above. Reported alongside the raw lift, not in place of it.
• SWE-bench — the benchmark that scores an agent by whether its patch makes the repo’s tests pass. Relevant here because it rewards no documentation behaviour, so it generates no documentation evidence.
• SWE-chat — a public dataset of real agentic coding sessions contributed by developers using command-line agent tools, with complete transcripts. Opt-in telemetry; 87% from one agent family.
• Sticky stage heuristic — the rule that once a failure signal appears, a session is labelled debugging until a test or build passes. Inflates the debugging share, which is why the authors only advance the negative claim about stage.
• Tier 1 / Tier 2 classification — Tier 1 is deterministic path rules; Tier 2 is LLM labelling of the paths Tier 1 could not resolve. Tier 2 is where agent_working_note was discovered and is the unvalidated part.
• Two-lobed cycle — the paper’s replacement for the assumed linear documentation journey: a self-recurrent consultation lobe and a large production lobe, loosely coupled to each other and only weakly coupled to code modification.
• Vendored path — a third-party file (e.g. under node_modules/) that an agent may read but the repository does not own. Flagged rather than dropped, because reading it is still a genuine documentation interaction.
• Wilson interval — a confidence interval for a proportion that assumes independent trials. Retained in this paper as a reference only, because the independence assumption is false here.