Measuring AI Ability to Complete Long Software Tasks

TL;DR

Benchmark percentages don’t tell you what an AI can actually do for you, and they saturate fast, so you can’t compare GPT-2 to Claude 3.7 on the same scale. METR’s fix: time every task by how long it takes a skilled human to do it, then find the task length at which a given AI model succeeds 50% of the time — its “50% time horizon.” Across 170 software/ML-research tasks (from 1-second sanity checks to 8-hour research projects) and 12 frontier models from 2019–2025, that horizon has grown from about 2 seconds (GPT-2) to about 110 minutes (o3), doubling roughly every 207 days. The growth survives a battery of robustness and external-validity checks (a different benchmark, “messiness”-controlled tasks, real internal engineering tickets), though absolute numbers should be read as fuzzy and domain-specific. Naively extrapolated, the trend implies AI that can autonomously handle a full month (167 working hours) of software tasks arrives somewhere between mid-2028 and mid-2030 — sooner if the apparent 2024–2025 acceleration holds.

Problem & Motivation

Two AI labs release models six months apart. Both score well on their respective flagship benchmarks. What can you actually say about the real difference? Not much — because:

• Benchmark scores are relative, not absolute. “62% on SWE-bench” means nothing about real work until you know how hard SWE-bench tasks are relative to a human’s day.
• Benchmarks saturate. Once models hit ~90%+ on a benchmark, that benchmark stops being informative, and the field has to build a new one — which resets the scale and breaks any before/after comparison.
• Benchmarks are often adversarially selected. Datasets like HellaSwag and Humanity’s Last Exam were built by filtering for questions that already-good models get wrong, which biases them against showing genuine capability growth and makes them a poor tool for tracking a long-run trend.
• There’s no shared unit. You cannot meaningfully compare “AI performance” across GPT-2 (2019) and Claude 3.7 (2025) because they were never measured against the same yardstick.

The result: everyone doing AI safety planning, product roadmapping, or public capability claims has been eyeballing “vibes” of qualitative benchmark difficulty instead of a number they can put a trend line through. METR’s proposed yardstick is time — a unit every human intuitively understands and that transfers across any task, any benchmark, any model generation.

What’s New (Core Contribution)

• The time-horizon metric itself. Before: capability = % correct on benchmark X. Now: capability = the length of task (in human-minutes/hours) a model can complete with a chosen success probability (50%, 80%, etc.). This single number is comparable across models, benchmarks, and years, because it’s anchored to something external and stable (how long the task takes a competent human) rather than to the difficulty of whatever test happens to exist today.
• Software Atomic Actions (SWAA) — a new 66-task, 1–30 second benchmark. Before: the shortest tasks on HCAST/RE-Bench take ~1 minute, which is already too hard for GPT-2/GPT-3-era models to attempt meaningfully — you’d measure a flat zero and learn nothing about early-model capability. SWAA extends the measurable range down to single-digit seconds, which is what let the trend be anchored all the way back to 2019 instead of starting in 2023.
• Difficulty grounded in large-scale, real human timing, not guesswork. Before: task “difficulty” in most agent benchmarks is assigned by the benchmark author’s judgment or an LLM’s estimate. Here, task difficulty is the geometric mean time of successful human baseline attempts — over 800 timed human runs, 2,529 human-hours total, across professionals in software engineering, ML, and security.
• A dated, quantified, stress-tested trend — plus a specific forecast. Before: no one had fit an actual doubling-time curve to “how much longer are the tasks AI can now do,” let alone checked it against an independent benchmark and against “messiness”-adjusted and real (non-contaminated) tasks. This paper does both, and turns the fitted curve into a concrete, falsifiable forecast: 1-month-task-capable AI in the mid-2028–mid-2030 window.

How It Works (Technically)

The core idea in one sentence: measure how long a task takes a human, measure whether the AI succeeds at that task, then find — per model — the task length at which the AI’s success probability crosses 50%; that length is the model’s time horizon, and plotting it against release date gives you a trend.

Step 1 — Build the task suite. 170 tasks from three sources: HCAST (97 tasks, 1 minute–30 hours — general software engineering), RE-Bench (7 tasks, all fixed at 8 hours — hard ML research engineering), and the paper’s own SWAA (66 tasks, 1–30 seconds — atomic software actions like “which file is a shell script?”). Every task has an automatic scorer (continuous score or binary pass/fail).

Step 2 — Time the humans. Skilled professionals (“baseliners,” ~5 years’ average experience) attempt tasks and their completion time is recorded (successful attempts only). Each task’s difficulty rating t_task is the geometric mean of these successful human times. 148 of 169 tasks got real human timing; the rest use researcher estimates.

Step 3 — Run the AI agents. 12 frontier models (2019–2025) plus 4 near-frontier, each run 8 times per task through a consistent agent scaffold (mostly a basic ReAct-style tool loop called modular-public; a different scaffold, triframe/duet, for models like o1 that struggled with the simple one). Every run is scored and reduced to a binary success/fail against the same bar the human aimed for.

Step 4 — Fit a logistic curve per agent. This is the one equation that matters:

p_success(agent, task) = σ((log h_agent − log t_task) · β_agent)

Plain English: for a given agent, plot task length (log-scaled) on the x-axis and success probability on the y-axis. That curve is an S-shape (a sigmoid, σ). h_agent is the task length where the curve crosses 50% — the time horizon. β_agent is how steep the drop-off is around that point — a high β means the model is either reliably good or reliably bad depending on which side of h_agent the task falls on (sharp cliff); a low β means performance degrades gradually across a wide range of task lengths (the model is inconsistent even on tasks well below its horizon). Fitting is maximum-likelihood logistic regression, done separately per agent — no other model’s performance affects a given model’s horizon estimate.

This is deliberately a simplified cousin of Item Response Theory (used in human psychometric testing): standard 2-parameter-logistic IRT would jointly estimate both a task-difficulty parameter and a person-ability parameter from the response data itself, and would give you an abstract “ability score” you’d then have to convert into something meaningful. METR skips both steps — task difficulty is fixed in advance from real human clock time (not learned), and the fitted parameter is the time horizon already, in real minutes/hours. The trade-off: you give up IRT’s per-task diagnostic tools (e.g., Fisher information), but you gain an interpretable, directly comparable number with far fewer free parameters (one slope per agent instead of one per task).

Step 5 — Trend and uncertainty. Time horizon is plotted against each model’s release date; log(time horizon) is linearly regressed against release date to get a doubling time. Confidence intervals come from a three-level hierarchical bootstrap — resampling task families, then tasks within families, then individual runs within tasks, 10,000 times — because tasks within the same “family” (e.g., issues from the same code repo) are correlated, and naive resampling would understate uncertainty.

Architecture & data flow

flowchart LR
  subgraph Suite["Task suite (170 tasks)"]
    HCAST["HCAST\n97 tasks, 1min-30hr"]
    REB["RE-Bench\n7 tasks, 8hr"]
    SWAA["SWAA (new)\n66 tasks, 1-30sec"]
  end
  Suite --> HB["Human baseliners\n800+ timed runs"]
  Suite --> AG["AI agent + scaffold\n12 models x 8 runs/task"]
  HB --> TT["Task time rating\n(geometric mean success time)"]
  AG --> SR["Binary success per run"]
  TT --> FIT["Per-agent logistic fit\n(Eq. 1)"]
  SR --> FIT
  FIT --> TH["Time horizon h_agent\n+ slope beta_agent"]
  TH --> TREND["Plot vs release date\n-> doubling-time trend"]

flowchart TD
  T["One task, e.g. 'munge data'\n(56 min human time)"] --> H["Humans attempt it\n-> geometric-mean time = t_task"]
  T --> M["Agent attempts it 8x\n-> success rate at t_task"]
  H --> C["Repeat across all 170 tasks\nof varying t_task"]
  M --> C
  C --> S["Fit sigmoid: P(success) vs log(t_task)\nfor this one agent"]
  S --> X["Read off where curve = 50%\n-> this agent's time horizon"]

The equation made visual: two agents with different time horizons produce the same S-shaped curve, just shifted along the (log) task-length axis. A model with a longer horizon doesn't just "do better everywhere" — its 50%-crossing point moves right. Watch where each curve crosses the dashed 50% line.

Time horizon (log scale: seconds to months) vs. release date, anchored to the paper's own reported numbers for GPT-2 (~2 sec, 2019), o1 (~39 min, mid-2024) and o3 (~110 min, 2025), with the fitted ~207-day-doubling trend extended forward. The shaded band marks the paper's 80% CI for when a 1-month (167-hour) time horizon arrives; the dashed line shows how much sooner that lands if the faster 2024–2025 rate continues.

The algorithm, simplified

import numpy as np
from scipy.optimize import minimize

def fit_time_horizon(results):
    """
    results: list of (t_task_minutes, success) pairs for ONE agent, across
    every task it attempted. Mirrors the paper's Eq. 1:
        P(success) = sigmoid((log(h) - log(t_task)) * beta)
    Returns (h, beta): the 50% time horizon in minutes, and the reliability slope.
    """
    def neg_log_likelihood(params):
        log_h, beta = params
        nll = 0.0
        for t_task, success in results:
            z = (log_h - np.log(t_task)) * beta
            p = 1 / (1 + np.exp(-z))            # sigmoid: model's predicted P(success)
            p = np.clip(p, 1e-6, 1 - 1e-6)      # avoid log(0) at the extremes
            nll -= success * np.log(p) + (1 - success) * np.log(1 - p)
        return nll

    log_h0, beta0 = np.log(10.0), 1.0           # start the search near a 10-minute horizon
    fit = minimize(neg_log_likelihood, x0=[log_h0, beta0])
    log_h, beta = fit.x
    return np.exp(log_h), beta                  # h in same units as t_task; beta = curve steepness

# In the paper, each (t_task, success) pair is also down-weighted by
# 1/sqrt(family_size) so that one over-represented task family (e.g. many
# similar GitHub issues from one repo) can't dominate the fit.

Built on Prior Work

Results & Evidence

Headline numbers:

• 50% time horizon doubles roughly every 207 days (95% CI: 166–240 days) from 2019 to early 2025.
• GPT-2 (2019): ~2-second horizon. o1 (mid-2024): ~39 minutes. o3 (2025): ~110 minutes, with some individual successes on tasks that take humans over 4 hours.
• Model success rate is strongly, negatively correlated with human task length — success falls off roughly exponentially as log(task time) increases (R² ≈ 0.80–0.83 across the two ways they report this fit).
• The 80% time horizon (a stricter, more deployment-relevant reliability bar) follows a similar doubling time (~204 days) but is 4–6x shorter in absolute terms than the 50% horizon — the reliability gap between “sometimes succeeds” and “usually succeeds” is large and doesn’t seem to be closing faster than the headline number suggests.
• o3 sits statistically above the long-run trend line (p = 0.006), one of the pieces of evidence the paper cites for the 2024–2025 rate possibly being faster than the 2019–2025 average.

External-validity checks (does this generalize beyond METR’s own tasks?):

• SWE-bench Verified replication: the same method applied to an independent, industry-standard benchmark shows the same exponential shape — but a shorter doubling time (~70 days vs. ~143 days for HCAST+RE-Bench+SWAA over the same 2024-only window). The likely cause isn’t a genuinely faster trend: SWE-bench’s official time annotations underestimate how long real contract baseliners take on the easiest tasks, which mechanically inflates the apparent growth rate.
• “Messiness” scoring: tasks were rated on 16 factors that make real work harder than a clean benchmark (resource limits, irreversible mistakes, dynamic environments, weak feedback loops, etc.). Higher-messiness tasks get lower success rates at a given length, as expected — but the growth rate over time looks similar for low- and high-messiness subsets, i.e. no sign (yet) of the trend stalling specifically on messier work.
• Internal, uncontaminated PR tasks: on METR’s own real (non-benchmark, unseen-by-models) pull requests, contractors took 5–18x longer than the original repo maintainers to fix the same issues. AI performance tracked the contractor-based time estimates, not the maintainer-based ones — a reminder that “time horizon” is relative to whose clock and what context you measure against, not an absolute property of the task.

What the evidence does not establish:

• The task suite is narrow: overwhelmingly software engineering / ML research, no GUI/mouse interaction, no multi-agent coordination or negotiation with other people, and few high-reliability-required tasks — all common in real work.
• Elicitation effort was wildly uneven: o1 and the original Claude 3.5 Sonnet each got roughly 2–3 engineer-weeks of scaffold/prompt tuning; most other models got the same generic scaffold with minimal adjustment. Results should be read as a lower bound on capability, and cross-model comparisons partly reflect METR’s tuning effort, not pure model capability.
• Only 7 frontier models fall in the 2024–2025 window, so the “trend may have accelerated” claim carries wide uncertainty — the data can’t yet distinguish a genuinely faster 2024–2025 slope from noise.
• Human baseline times come from small per-task samples, count only successful attempts, and may be systematically biased by baseliners giving up on tasks that were actually within reach.
• Reliable measurement above ~90% success rate isn’t possible with this dataset size — exactly the reliability range that matters most for real autonomous deployment.

How You’d Use It

• Your business. Set honest expectations before you commit to automating something. Time the human version of the workflow, then check it against a model’s known (or self-measured) time horizon at 80–95% reliability, not the flashier 50% number — 50% is “sometimes,” not “production-ready.” The paper’s own cost math helps here too: at $143.61/hour loaded human cost, over 80% of the AI agent’s successful runs already cost under 10% of the human-equivalent labor cost — a concrete number to weigh when deciding what to automate now versus wait on.
• Your workflows and methodologies. Build your own house benchmark. The method transfers directly: apply the same logistic fit (see “Build Your Own” below) to a couple dozen of your own recurring task types. That gives you a defensible, domain-specific capability number per model release instead of trusting vendor marketing or a generic public leaderboard — and a real trend line (“time horizon doubles roughly every 7 months”) to anchor your own planning instead of generic “AI is moving fast” hand-waving.
• Your automations and business processes. Use the messiness checklist as due diligence before you automate anything. Score the target workflow against the paper’s 16 messiness factors (resource-limited, irreversible mistakes, dynamic environment, weak feedback, etc.). High-messiness workflows deserve a much bigger reliability discount than a raw benchmark score would suggest.
• Your harness. Run the same benchmark method against any model before you wire it into your own agent stack, and re-run it whenever a new model drops. Treat it as an ongoing gate on your harness, not a one-off evaluation — the doubling trend means today’s “good enough” horizon is a moving target.

Build Your Own (Minimal Recipe)

Components:

1. A task set (15–30 is enough to start) spanning a wide range of durations in your target domain, each with an automatic or rubric-based pass/fail check.
2. A handful of timed human completions per task (2–3 people, self-timed is fine to start) to get a geometric-mean “task time” — this becomes each task’s x-position.
3. One consistent agent scaffold (a basic tool-calling/ReAct loop is enough) run identically across every model you want to compare, several trials per task to average out run-to-run noise.
4. Binarize every run’s outcome against the same bar you asked the humans to hit.
5. Fit: run fit_time_horizon (above) separately per model to get h_agent (its time horizon) and beta_agent (its reliability slope).
6. Plot h_agent against model release date over time — that’s your own trend line.

The genuinely hard parts:

• Getting believable human baseline times cheaply. METR spent 2,529 human-hours on this — you won’t match that, so be upfront that a small-sample house benchmark has wide error bars, and weight it toward relative model comparisons rather than absolute claims.
• Writing pass/fail graders that don’t rubber-stamp partial credit. A grader that’s too lenient will inflate every model’s horizon and erase the differences you’re trying to measure — this is the part worth spending your engineering time on, not the scaffold.

Reach for: scipy.optimize or statsmodels.Logit for the fit itself; any simple agent harness you already have from MAS work for running the tasks — the scaffold is genuinely the easy 20% here.

How to Improve It

• Multi-domain suites. Extend past software/ML-research into other domains you work in (support, legal review, data entry) to see whether the doubling time differs by domain — directly useful for cross-vertical capability claims.
• Reliability-weighted horizon. A single 50%-horizon number hides the fact that two models with the same 50% horizon can have 4–6x different 80% horizons (shown in this paper’s own data). A “horizon at your required reliability” calculator, using both h_agent and beta_agent, would be more decision-useful than either number alone.
• Cost-adjusted horizon. Combine the time-horizon x-axis with the paper’s own $/successful-run economics to produce a “capability per dollar” curve — a natural extension for anyone pricing agent work against human labor.
• Disclosed, matched elicitation effort. The paper openly admits uneven tuning effort across models. Reporting engineer-hours spent per model alongside the horizon number (or deliberately capping/matching effort) would make cross-model comparisons fairer and more reproducible.
• Inference-time compute scaling as a lever. Given the paper’s own finding of large cost headroom below human-labor cost, a natural follow-on is testing whether spending more inference compute (best-of-k, extended reasoning budgets) shifts a model’s horizon rightward at a fixed dollar budget — especially relevant now that reasoning-style models are the default frontier.

Glossary

• Time horizon — the length of task (in human-equivalent minutes/hours) a model can complete at a chosen success rate; the paper’s core metric.
• 50%-time-completion time horizon — the specific version used as the headline: the task length at which the model succeeds half the time.
• Sigmoid / logistic function (σ) — an S-shaped curve mapping any real number to a probability between 0 and 1; here it converts “how far a task’s length is from the model’s horizon” into a success probability.
• Item Response Theory (IRT) — a family of statistics techniques (from human testing) for estimating a test-taker’s ability and a question’s difficulty from right/wrong answers; this paper’s method is a simplified, one-directional version of it.
• HCAST — Human-Calibrated Autonomy Software Tasks; the 97-task general software-engineering benchmark this paper’s main suite is built on.
• RE-Bench — a 7-task, 8-hour-each benchmark of hard ML research engineering problems, graded against human experts.
• SWAA (Software Atomic Actions) — the 66 new 1–30-second tasks this paper contributes, built to make pre-2023 models measurable at all.
• SWE-bench Verified — an industry-standard benchmark of real GitHub issues, used here as an independent check on the trend.
• Scaffold — the surrounding code that turns a raw LLM into an agent: the tool-calling loop, prompt structure, and context management (here, mostly a ReAct-style loop called modular-public).
• ReAct — an agent pattern that interleaves reasoning (“thinking out loud”) with taking actions/tool calls, rather than jumping straight to an action.
• Baseliner — a skilled human professional paid to attempt a task under timed, recorded conditions, used to set that task’s difficulty rating.
• Doubling time — the number of days it takes a growing quantity (here, time horizon) to double, assuming exponential growth; ~207 days in this paper’s headline result.
• Geometric mean — the “multiplicative average” (nth root of the product of n numbers); used here instead of the arithmetic mean because task completion times are heavily right-skewed (a few humans taking much longer would otherwise distort the average).
• Hierarchical bootstrap — a resampling technique for estimating uncertainty that respects nested structure in the data (here: task family → task → individual run), used to avoid understating error bars from correlated tasks.
• Messiness factor — one of 16 rated properties (resource limits, irreversible mistakes, dynamic environments, etc.) that make a task more like messy real-world work and less like a clean benchmark.
• Elicitation — the effort spent (prompting, scaffolding, tuning) to get a model to perform near its actual capability ceiling, rather than being held back by a poorly matched harness.