Outcome Primitives: A Framework for Measuring AGI Value in the World

TL;DR

Benchmarks are at war with each other — the same week in March 2026, Gemini 3.1 Pro beat human PhDs on GPQA Diamond by thirty points while every frontier model scored under 1% on ARC-AGI-3. None of them tell you whether anyone’s life got better. This paper introduces VCF (Value Classification Framework): three orthogonal axes that tag any “value episode” (one human trying to get one outcome with AI help) by what it was (Outcome Primitive, 10 types), how big it was (Outcome Magnitude, measured in human-hours saved), and how sure we are it happened (Evidence Tier, 4 levels from “we shipped output” to “controlled experiment”). They ran it on 25,000 live agents (2,678 opted in), auto-classified 7,718 daily files over 21 days, and estimated ~182,000 hours of human-effort-equivalent (~88 person-years, ~$32M of artifact value) produced in three weeks. The payoff move: AGI gets a falsifiable, outcome-grounded definition — consistently produce program-scale (OM4) outcomes across all ten categories with externally-verifiable (E2+) evidence, and you’ve “done the work.”

Problem & Motivation

The pain in one sentence: we measure what AI can do on tests, and we have no honest, comparable way to measure what AI actually did for real people.

Concretely, the field’s evaluation stack is broken in three ways:

1. Benchmarks disagree on what the models even are. GPQA says superhuman; ARC-AGI-3 says sub-1%. Both are “capability in vitro” — performance on a frozen task set. Neither measures transfer to the messy, open-ended work people actually bring to an agent.
2. The proxies everyone falls back on measure the wrong thing. Usage metrics count engagement (did they keep clicking?). Satisfaction surveys count vibes. Token economics counts cost. Safety evals count harm avoided. None of them count outcomes — the thing that was accomplished in the world.
3. Everyone secretly reports at the highest confidence while measuring at the lowest. A demo shows “the model built an app.” Did the app get deployed? Used? Did it work? Most evaluation “delivers output and presents assumptions as evidence.” There’s no discipline that forces you to say “we don’t actually know if this mattered.”

For someone running an AI services company this is the daily reality: a client asks “is this agent worth it?” and you have token bills and a satisfaction score, neither of which answers “how much real work did it produce and how do we know.” VCF is an attempt to make that answerable.

What’s New (Core Contribution)

The paper’s novelty is a measurement schema plus its first large-scale field deployment, not a model or an algorithm. Four genuine contributions:

• Three orthogonal axes instead of one number. Before: a benchmark collapses everything into a single score (88% on MMLU). Now: every unit of value is a point in a 3D space — OP (what kind of work), OM (how much human effort it replaced), E (how confident we are it happened). Orthogonality is the key claim: a tiny zero-hour intervention (flagging an unsafe supplement) and a 300-hour infrastructure build are different on every axis and shouldn’t be averaged together.
• The “value episode” as the atomic unit. Before: evaluation units are the message, the session, or the user. Now: the unit is one coherent human attempt at one outcome — and a single 10-minute conversation may contain three episodes (emotional support, business planning, a quick lookup), each scored separately. This is the right granularity and also the hardest part (boundary detection is unsolved — see Limitations).
• An honesty axis (Evidence Tiers). Before: confidence is implicit and usually overstated. Now: E0–E3 forces you to label whether you have zero verification (E0), behavioral signal (E1, user came back and built on it), an external trace (E2, deployed URL / processed payment / published book), or a controlled experiment (E3). The paper reports E3 = 0% as a feature, not a footnote.
• An outcome-grounded definition of AGI. Before: AGI = passes some hard test. Now: AGI = a system that consistently produces OM4 outcomes across all ten OP categories at E2+ evidence. It reframes AGI from “can it pass?” to “can it do the work — every kind of work humans do — and leave a trace?”

What’s repackaged rather than new: the dissatisfaction with benchmarks is Chollet’s (2019); the “measure outcomes not activity” instinct is standard in professional-services management. The novelty is the specific 3-axis taxonomy and the in-vivo measurement instrument.

How It Works (Technically)

This is a measurement framework, so “how it works” = the three axes, the unit they classify, and the instrument that does the classifying. There’s almost no math — the one quantitative move is the aggregate-value derivation, which I demystify below.

The three axes.

• OP — Outcome Primitive (10 categories). A coarse, deliberately stable taxonomy of what the human was trying to achieve, defined by intent, not by tool. Examples: OP.AP Asset Production (make a deliverable — code, docs, media), OP.IS Intelligence Synthesis (research/analysis), OP.DS Decision Support, OP.OA Operational Automation, OP.IN Interpersonal Navigation (emotional/relational), OP.TF Transaction Facilitation (deals/trading), OP.HO Health Optimization, OP.SA Skill Acquisition, OP.CA Compliance Assurance, OP.SP Security Probing. Ten is a design choice: coarse enough to be stable across systems, fine enough to be meaningful.

• OM — Outcome Magnitude (OM0–OM4), measured in Human Effort Equivalent (HEE). HEE = “hours a qualified professional would need to produce this without AI.” This is the clever decoupling: magnitude is divorced from compute cost. OM0 = atomic (≤1h), OM1 = bounded session (1–8h), OM2 = scoped deliverable (8–80h), OM3 = initiative (80–320h), OM4 = program (>320h, team-month+). Note the framework distinguishes om_step (event-local, used for routing in real time) from om_goal (episode-level, used for capability analysis) — a single OM3 initiative is hundreds of OM0–OM1 steps stitched together.

• E — Evidence Tier (E0–E3). The confidence axis. E0 = output delivered, no verification. E1 = behavioral signal (user returned, iterated, referenced it later). E2 = external trace (deployed URL, processed payment, published work, executed trade, accepted legal filing). E3 = controlled experiment with a counterfactual. The whole point: most evaluation “operates implicitly at E0 and presents findings as E2.”

The atomic unit. A value episode — one human, one intended outcome, one AI-assisted attempt. Each episode gets one (OP, OM, E) triple. The OP×OM grid forms a capability landscape: a 2D heatmap of “what AI actually does in the world,” with E as a per-cell confidence overlay.

The measurement instrument — the agent itself. This is the cleverest design decision. They never read raw conversations. Instead they treat the agent the way a hospital treats a physician’s chart: the agent, as a byproduct of doing its job, emits three structured artifacts they classify on:

1. Memory files — what the agent learned about the user (goals, projects, progress, emotional patterns).
2. Workspace artifacts — code, documents, deployed apps, research directories.
3. Behavioral metadata — tool-call patterns, session timing, cron jobs, sub-agent spawns.

The classifier is a batch keyword/heuristic scanner, not an LLM judge. For each user’s daily memory file it: (1) keyword-matches OP category patterns → assigns the primary OP; (2) estimates OM from daily file size + tool-invocation density + workspace artifact complexity + calendar span of sustained activity; (3) assigns E from keyword patterns (deploy/publish/payment/hire → E2; complete/return/progress → E1; else E0); (4) detects episode boundaries from topic-shift markers and distinct OP categories in one file. This is genuinely simple — and the simplicity is why it scales to 7,718 files, and also why it systematically undercounts internal outcomes (a first-time emotional disclosure leaves almost no structural signal).

Tracing one input through. Take the real “complete language course” episode. A teacher spends multiple days conversing with the agent; the agent produces 110 lesson files (audio, exercises, tests) in the workspace and logs progress in memory. The scanner sees: workspace artifact complexity = high, calendar span = multi-day → OM3 (~120h HEE). Keyword match on lesson/curriculum/teaching intent → OP.AP (Asset Production). The memory notes the course “was used with students” → an external-use trace → E2. Output triple: (OP.AP, OM3, E2). One point on the landscape.

Demystifying the one piece of math — the $32M. People will fixate on this number, so here’s exactly how it’s built and how soft it is:

• HEE per OM band uses the geometric midpoint of each range, not the arithmetic mean. Why geometric? The bands are log-spaced (1–8, 8–80, 80–320), and for right-skewed data within a band the geometric mean sits below the arithmetic mean → deliberately conservative. E.g. OM2 (8–80h) → √(8·80) = √640 ≈ 25h, not 44h. OM4 (open-ended) is capped at 400h even though they admit some episodes exceed 1,000h.
• Multiply each band’s midpoint by its file count → 182,425 HEE hours ≈ 88 person-years. Sensitivity: band-minimums give 84k hours, band-maximums give 443k; the headline sits at the 27th percentile of that range.
• Two independent dollar models. Model A prices by complexity tier (OM) at market replacement rates ($50–200/h, what a client pays) → $31.6M. Model B prices by professional domain (OP) at BLS median wages ($30–55/h, what the worker earns) → $9.0M. The 3.5× ratio between them is the well-known wage-vs-billing markup (firms bill 2–4× salary). Two different axes, two different rate sources, “converge on the same order of magnitude” — that convergence is the actual evidentiary argument, not either point estimate.

Architecture & data flow

flowchart TB
  U[User pursues a goal] -->|conversation| AG[Persistent AI agent<br/>Claude Opus 4.6 + Gemini fallback]
  AG -->|byproduct, not surveillance| ART[Structured outputs]
  ART --> MEM[Memory files<br/>goals, progress, patterns]
  ART --> WS[Workspace artifacts<br/>code, docs, deployed apps]
  ART --> META[Behavioral metadata<br/>tool calls, cron, timing]
  MEM --> SC[Batch keyword scanner]
  WS --> SC
  META --> SC
  SC -->|keyword match| OP[OP: which of 10 types]
  SC -->|file size + tool density + span| OM[OM0-OM4: HEE hours]
  SC -->|deploy/pay/publish keywords| E[E0-E3: confidence]
  OP --> EP[Value episode = OP x OM x E]
  OM --> EP
  E --> EP
  EP --> LAND[Capability Landscape<br/>OP x OM heatmap + E overlay]
  LAND --> AGI{Consistent OM4 across<br/>all 10 OP at E2+?}
  AGI -->|yes| DEF[Definition of AGI satisfied]
  AGI -->|no, today| GAP[Map still filling in]

Interactive capability landscape: the OP×OM grid the paper builds. Hover a cell to see the episode density (log-scaled, schematic from the paper's reported distribution). Notice Asset Production dominating every magnitude and the sparse, high-value OM3–OM4 frontier — the "have we done the work" region.

The algorithm, simplified

# The VCF classifier: turn one agent's daily artifacts into value episodes.
# No LLM judge here — it's a keyword/heuristic batch scanner. That's the point:
# cheap enough to run on every user every day; biased toward structural signal.

OP_KEYWORDS = {                      # intent patterns -> Outcome Primitive
    "AP": ["wrote", "built", "generated", "deployed file", "lesson", "design"],
    "IS": ["research", "analysis", "due diligence", "summarize", "dataset"],
    "IN": ["feeling", "relationship", "support", "struggling", "coaching"],
    "TF": ["trade", "negotiate", "invoice", "payment", "procurement"],
    # ... 6 more categories ...
}
E2_KEYWORDS = ["deployed", "published", "processed payment", "executed trade", "filing accepted"]
E1_KEYWORDS = ["returned", "iterated", "progress", "completed", "referenced earlier"]

def classify_daily_file(memory_text, workspace, metadata):
    episodes = []
    # 1) Which OP categories fired? >=2 keyword hits = a real episode, not noise.
    fired = {op: count_hits(memory_text, kws) for op, kws in OP_KEYWORDS.items()}
    active_ops = [op for op, n in fired.items() if n >= 2]   # multi-OP day -> multi-episode

    for op in active_ops:
        # 2) OM from STRUCTURE, not self-report: bigger/longer/denser -> larger magnitude.
        hee = estimate_hee(
            file_bytes      = len(memory_text),
            artifact_score  = workspace.complexity(),          # # files, code vs. text, sizes
            tool_density    = metadata.tool_calls_per_session,
            calendar_span_d = metadata.days_of_sustained_work,  # an OM3 spans days
        )
        om = bucket_om(hee)        # <=1h->OM0, 1-8->OM1, 8-80->OM2, 80-320->OM3, >320->OM4

        # 3) Evidence tier: highest external trace wins; default to "we don't know."
        if any(k in memory_text or workspace.has_trace(k) for k in E2_KEYWORDS):
            e = "E2"               # left a trace in the WORLD (deploy/pay/publish)
        elif any(k in memory_text for k in E1_KEYWORDS):
            e = "E1"               # behavioral signal: came back, built on it
        else:
            e = "E0"               # output delivered, mattered? unknown. Stated honestly.

        episodes.append((op, om, e))
    return episodes                # each (OP, OM, E) is one point on the landscape

Built on Prior Work

Results & Evidence

The setup. Platform launched Feb 14 2026, grew to ~25,000 agents; 2,678 users (~10%) opted in. Each got an isolated agent with persistent memory, a sandboxed workspace, and tools (code exec, browsing, file/media gen), running Claude Opus 4.6 (Gemini 3.1 Pro fallback). Participants: 30+ countries, ages 21–70+, software engineers to restaurateurs. The 21-day classification window (Mar 10–30) processed 7,718 daily files from 1,305 users → ~17,921 estimated value episodes.

Headline numbers:

• All ten OP categories are populated. Asset Production dominates (~40.7% of episodes), with 36 episodes reaching OM4 (program-scale).
• Distribution center of mass is small work: OM1 = 52%, OM1+OM2 = 86%. The OM3–OM4 frontier is 7.4% of files (571 episodes) — the multi-day deployed platforms, full curricula, trading infra.
• The evidence distribution (the paper’s self-declared most important result): E1 = 43.3% (largest — users build on outputs), E2 = 19.7% (1,524 episodes with external traces — ~1 in 5 left a verifiable mark in the world), E0 = 36.9% (delivered, unverified), E3 = 0% (no controlled experiments).
• Stability: OM2+ held at 40.2% (σ=5.8pp) and E2 at 19.0% (σ=3.6pp) across all 21 days; a Mar 25 influx of 367 users diluted both, recovered in 3 days. Stability is their argument that the distribution is structural, not transient.
• Aggregate: ~182,000 HEE hours ≈ 88 person-years ≈ $31.6M (Model A) / $9.0M (Model B floor).

What the evidence does NOT establish — read this part skeptically:

• It’s a single platform, single product, single population, 21 days. The OP taxonomy may not generalize; OM calibration rests on subjective HEE estimates.
• OM is auto-estimated from file size / tool density / span — proxies, not ground truth. A verbose memory file inflates OM; a quiet life-changing conversation registers OM0. The authors openly say structural measurement systematically undercounts internal outcomes (OP.IN, OP.HO).
• E2 detection is keyword-based. “deployed” in a memory file is not proof of a working deployment. E2 here means “a trace plausibly exists,” not “verified by the research team” (only 153 curated episodes got human verification).
• The dollar figures are estimates of estimates. HEE is an assumption; the rate cards are assumptions; the “convergence” of Model A and B is partly engineered (both use the same HEE hours, just different $/h). Treat $32M as an order-of-magnitude marketing-grade number, not an audited one.
• E3 = 0% means nothing here is causal. No counterfactual: we don’t know what these users would have produced without the agent. The headline “88 person-years created” assumes the AI did work that otherwise wouldn’t have happened — unproven.

To their credit, the paper states all of this in the open. The honesty is the contribution; the numbers are the illustration.

How You’d Use It

For an AI services company, VCF is less a research artifact and more a reporting and pricing layer you can wrap around any agent product you ship. Concrete slots:

• Client value reporting / QBRs. Replace “the agent handled 4,200 sessions this month” with “the agent produced 312 value episodes: 41% Asset Production, 18% Decision Support; 22% left externally-verifiable traces; estimated ~1,900 hours of human-effort-equivalent.” That is a renewal conversation that justifies the bill. The E-tier honesty actually builds trust — you’re telling them what you can and can’t prove.
• Value-grounded pricing / unit economics. The paper’s sharpest commercial point: “a $50 OM3 outcome and a $50 OM0 outcome have identical cost profiles and entirely different value profiles.” Tag episodes with (OM, token-cost) and you can compute artifact-value-per-dollar-of-inference per client, per workflow — and price on outcome, not tokens.
• Model/integration A-B selection. Plot two configurations (model + memory + tools + workflow) on the same OP×OM landscape. The paper notes a weaker model with better integration can outproduce a stronger bare-chat model — exactly the kind of claim you want to make to a client choosing between a cheap-integrated and an expensive-raw option.
• Routing in a multi-agent system. Their om_step vs om_goal split maps directly onto MAS orchestration: use the cheap event-local OM estimate to route (escalate OM3-looking goals to a planner/sub-agent swarm; keep OM0 lookups on a small model), and the episode-level OM to report.
• A productized “AI Value Audit” offering. Run the scanner over a prospect’s existing agent logs/workspaces and hand them a capability landscape + evidence distribution. It’s a foot-in-the-door deliverable that requires no model access — only their artifacts.

Build Your Own (Minimal Recipe)

You can stand up an 80%-value version in a few days because the hard intellectual work is the taxonomy, and they’ve published it.

Components, in build order:

1. Artifact capture. Make sure each agent persists three things per day: a memory/summary file, a workspace dir, and a metadata log (tool calls, timestamps, cron). If you run any agent framework today you mostly have these already.
2. OP classifier. Start with their 10 categories and a keyword/regex map. Upgrade path: swap to an LLM classifier with the OP definitions in the prompt — better recall on intent, at a per-file cost. (Honestly, given today’s prices, an LLM judge is the right call over keywords for a real product.)
3. OM estimator. A simple scoring function over file_bytes, artifact_complexity, tool_density, calendar_span → bucket into OM0–OM4 via HEE thresholds. Calibrate the thresholds against ~20 hand-labeled episodes.
4. E classifier. Keyword tiers for E2 (deploy/pay/publish), E1 (return/iterate/progress), else E0. Upgrade: actually verify E2 — ping the deployed URL, check the payment webhook — to earn real E2 instead of keyword-E2.
5. Episode boundary splitter. Cheapest version: one episode per distinct OP with ≥2 hits in a daily file. This is the acknowledged weak point.
6. Reporting layer. The OP×OM heatmap + E distribution + the aggregate HEE/$ rollup.

The 1–2 genuinely hard parts:

• Episode boundary detection. Users interleave episodes within minutes; “where does one attempt end and another begin” is unsolved in the paper. Anything beyond the naive per-OP split is real research.
• OM ground truth / calibration. HEE is subjective. Without a labeled set, your magnitudes drift. Budget time to hand-label a calibration set and re-tune.

Reach for: any agent framework with persistent memory + sandboxed workspace (LangGraph, your own); an LLM for the classifier (the OP/OM/E prompts are short); pandas for the rollups; a heatmap lib for the landscape. No training required — this is all inference + bookkeeping.

How to Improve It

Limitations are the leverage. Five concrete, testable upgrades:

1. Close the E3 gap with a randomized holdout. The fatal weakness is zero causal evidence. Randomly assign a cohort to a degraded agent (no persistent memory, no tools) and compare OM/E distributions. Even a small RCT moves the headline claim from “88 person-years existed” to “88 person-years were caused.” This is the single highest-value extension and the paper begs for it.

• Verified-E2 instead of keyword-E2. Replace E2 keyword matching with active verification: hit the deployed URL and check it returns 200, confirm the payment via the processor API, fetch the published artifact. Turns a soft proxy into a hard trace and would likely lower the E2 rate — a more honest, more defensible number.

2. An internal-outcome channel for OP.IN / OP.HO. Structural measurement undercounts emotional and health value by design. Add an optional, consented user-reported signal (a one-tap “this mattered” or periodic check-in) as a separate evidence sub-tier, so the framework stops being blind to its most human category.
3. Temporal aggregation across episodes. Per-episode classification misses compounding: 30 OM0 health reminders over a month may be an OM2 health outcome. Add a longitudinal roll-up that links episodes by user+goal and re-scores the trajectory, not just the snapshot.
4. LLM-judge classifier with calibration + inter-rater agreement. Swap the keyword scanner for an LLM classifier, then measure agreement against the 153 human-curated episodes. Report a confusion matrix per axis so consumers know the classifier’s own error bars — addressing the “the instrument has systematic error” limitation head-on.
5. Multi-platform replication to test taxonomy generality. Run the identical scanner on a second product/population. If the OP ranking (AP first, DS/IN second tier) holds, the taxonomy is structural; if it scrambles, the 10 categories are platform artifacts. This is the cheapest way to test the biggest external-validity threat.

Glossary

• VCF (Value Classification Framework) — the paper’s 3-axis system for classifying any AI-assisted value episode.
• Value episode — the atomic unit: one human’s coherent attempt to achieve one specific outcome with AI help.
• OP (Outcome Primitive) — what the AI was used for; 10 intent-defined categories (Asset Production, Intelligence Synthesis, etc.).
• OM (Outcome Magnitude) — how big the outcome was, on a 5-level scale OM0–OM4.
• HEE (Human Effort Equivalent) — hours a qualified professional would need to produce the outcome without AI; the unit OM is measured in. Decouples value from compute cost.
• E (Evidence Tier) — how sure we are the outcome happened: E0 (none), E1 (behavioral), E2 (external trace), E3 (controlled experiment).
• Capability landscape — the OP×OM grid; a heatmap of what AI actually does in the world.
• om_step vs om_goal — event-local magnitude (for real-time routing) vs episode-level magnitude (for capability analysis); an OM3 goal = hundreds of OM0–OM1 steps.
• In vivo / in vitro — measuring AI “in the wild” with real users vs. on a frozen benchmark.
• Geometric midpoint — √(low·high) of a band; used for HEE because the bands are log-spaced, yielding deliberately conservative (sub-arithmetic) estimates.
• GPQA / SWE-bench / MMLU / ARC-AGI-3 — capability benchmarks the paper contrasts itself against; they score the model on tasks, not outcomes in the world.
• Outcome-empirical AGI definition — AGI = consistently producing OM4 outcomes across all 10 OP categories at E2+ evidence.