Tendem: A Hybrid AI+Human Platform

TL;DR

Autonomous AI agents (like ChatGPT Agent) are fast and cheap but fumble ambiguous briefs, long documents, and judgment calls — they hallucinate, drop requirements, and can’t always tell when they’re wrong. Human freelance platforms like Upwork are more reliable but slow and inconsistent in both price and quality. Tendem is a production system that runs an AI agent for the routine, tool-heavy parts of a task, but inserts a vetted Human Expert at defined checkpoints (“step gates”) whenever the AI’s plan is risky, its output is uncertain, or an automated quality check fails — then runs the result through a second layer of automated and human QA before it reaches the customer. On a new 94-task benchmark built from real freelance-platform categories, Tendem delivered 74.5% “delivery-ready” (Good) results versus 53.2% for Upwork freelancers and 40.4% for ChatGPT Agent, while cutting median delivery time by 53% and median price by 36% compared to Upwork. The paper’s real contribution isn’t a new model or algorithm — it’s an operational proof that a well-instrumented hybrid pipeline can beat both pure-AI and pure-human execution on quality, speed, and cost simultaneously, which is the trade-off anyone routing real work between AI and paid humans is implicitly trying to solve.

Problem & Motivation

Two existing ways to get knowledge work done both have a specific, well-known failure mode:

• AI-only agents (ChatGPT Agent, Claude computer-use style agents, etc.) optimize for speed and cost. They’re great at bounded, well-specified tool use, but they struggle with under-specified briefs, synthesizing across multiple documents, and “non-local” quality criteria — problems where you have to hold the whole task in your head to notice something’s wrong, not just check one step in isolation. In this paper’s own evaluation, ChatGPT Agent scored “Bad” on 36.2% of tasks.
• Human-only marketplaces (Upwork, Fiverr, Freelancer) give you accountability, reputation history, and real judgment — but they’re slow (35 hours median turnaround in this study) and inconsistent: quality varies a lot by which freelancer picks up the job, and price isn’t fixed.

Neither failure mode is news. What’s missing is a clean way to measure the trade-off on real work, because public agent benchmarks (BrowseComp, GAIA, HLE — see Results) test narrow, well-defined skills like web search or exam questions, not the messy reality of someone handing over a spreadsheet, three PDFs, and a vague goal with an implicit acceptance bar. Tendem’s authors built both the hybrid system and a new benchmark designed to close that measurement gap, because you can’t argue “hybrid is better” without a way to test all three approaches on the same, realistic tasks.

What’s New (Core Contribution)

1. Step-gated hybrid execution. Instead of “AI does everything, human reviews the final output” or “human does everything,” Tendem defines explicit checkpoints inside the AI’s planning loop (plan audit, plan-step check, draft refinement, etc.) where a Human Expert is pulled in only if that specific step is flagged as high-risk or uncertain. The human’s time is spent on the 10-30% of the task that actually needs judgment, not re-doing the routine 70%.
2. Layered QA, not a single gate. Quality checks run during execution (online QA — catches errors before they compound into the next step) and again as a full pass after execution (offline QA — multi-step verification against the original requirements, with escalation to a human QA specialist on uncertain cases). Before this, most agent systems check quality once, at the end, if at all.
3. A new real-world benchmark (94 tasks). Public agent benchmarks are saturated or synthetic. The authors commissioned genuinely new tasks across four business areas (Sales, Operations, Marketing/Analysis) sourced from real Upwork/Fiverr/Freelancer category distributions, with attached files and explicit acceptance criteria, and had trained human raters score every system’s output blind (system identity hidden) on four axes: Accuracy, Completeness, Style & Formatting, and Overall.
4. The three-way comparison itself. Most “AI vs. human” papers pick one axis (usually quality or cost). This paper measures quality, time, and price for the same 94 tasks across AI-only, human-only, and hybrid — and shows the hybrid wins on all three at once, which is the paper’s central empirical claim.

How It Works (Technically)

There’s no new model or training procedure here — Tendem is a systems and orchestration paper. The “mechanism” is the workflow: who does what, and exactly when control passes from AI to human and back.

Three roles:

• Customer — states the goal in plain language, attaches files/references, expects a result within a time window, at a predictable cost.
• AI Agent + Tools — runs a plan → act → observe → verify loop (the ReAct pattern; Yao et al., 2023) with tool access: web browsing, file I/O for office formats, a sandboxed Python runtime for analysis/charting, and an isolated shell. Every action runs with minimal permissions and is logged for traceability. Continuous automated QA tools run alongside it: spec conformance checks, unit/total reconciliation (do the numbers in the output actually add up to what the source data says?), citation matching (does the cited source actually say what’s claimed?), and lightweight self-consistency checks (asking the model to double-check its own output, in the spirit of SelfCheckGPT — Manakul et al., 2023).
• Human Expert — a vetted professional (admitted via tests/exams, then continuously QA’d) who steps in at step gates and performs the final offline QA pass. Tracked via QA pass rate and rework rate.

The 8-step pipeline (this is the actual mechanism — trace a task through it):

1. Customer request — plain-language ask + attached files.
2. Clarify and formalize — the AI inspects the files and requirements and asks targeted clarifying questions (this is where under-specification gets caught early, instead of silently guessed at).
3. Plan with gated steps — the AI decomposes the task into steps and tags which ones are high-risk (these get a mandatory Human Expert gate before execution continues).
4. Routing and matching — the system finds a suitable expert by skill set and estimates time.
5. Hybrid execution — the AI does the work; the Human Expert steps in whenever output needs adjustment, live, not after the fact.
6. Online QA — lightweight automated checks run on both AI and human edits during execution; if something fails, the plan updates and the loop iterates rather than pushing a known-bad intermediate result forward.
7. Offline QA — after execution, a full automated multi-step verification against the customer’s original requirements and attached materials; uncertain cases escalate to a human QA expert.
8. Finalization — deliver to the customer, or bounce back for rework if material issues remain.

The core idea worth remembering: escalation is conditional and local, not global. A task doesn’t get “assigned to AI” or “assigned to a human” as a whole — individual steps inside the plan get flagged, and only those steps pull in a human. That’s the structural difference from “AI drafts, human reviews everything” (expensive, slow) and from “fully autonomous, no human” (fast, but brittle on the steps that need judgment).

On the statistics used to report results: the paper uses a one-sided z-test to compare the share of “Good” outcomes between two systems — this just answers “how likely is a gap this large (Tendem’s 74.5% vs. Upwork’s 53.2%) to have happened by chance if the two systems were actually equally good?” (answer: p = 0.0012, i.e. very unlikely). For median times and prices, they use bootstrap confidence intervals — repeatedly resampling the 94 tasks with replacement to see how much the median would wobble if you’d happened to sample a slightly different set of 94 tasks. Both are standard tools for saying “this difference is real, not noise,” given a fairly small sample (n=94).

Architecture & data flow

flowchart LR
  C[Client request] --> CL[Clarify & formalize]
  CL --> PL[Plan with gated steps]
  PL --> RT[Route to Human Expert]
  subgraph Hybrid Execution
    AI[AI Agent: plan-act-observe-verify + tools]
    HE[Human Expert: judgment, refinement]
    AI <-->|step gate escalation| HE
  end
  RT --> AI
  AI --> OQ[Online QA: spec check, reconciliation, citations, self-consistency]
  OQ -->|fail: iterate plan| PL
  OQ -->|pass| OFQ[Offline QA: full verification vs requirements]
  OFQ -->|uncertain: escalate| HE
  OFQ -->|pass| F[Finalize & deliver]
  OFQ -->|material issues| REWORK[Rework]
  REWORK --> AI

The same 8-step pipeline, animated: a task moves left to right through Tendem's stages. Watch for the two step-gate points (plan check, offline QA) where the flow can branch down to a Human Expert instead of continuing straight through — that branching is the entire mechanism of the paper.

The algorithm, simplified

The paper doesn’t give code, but the control logic implied by the step-gate design is straightforward to sketch. This is the shape you’d actually implement:

# The core idea: escalate individual PLAN STEPS, not the whole task.
def run_hybrid_task(task, client_files):
    spec = clarify(task, client_files)              # AI asks targeted questions, fills gaps
    plan = decompose(spec)                            # list of steps, each with a risk_score

    results = []
    for step in plan:
        if step.risk_score > RISK_GATE_THRESHOLD:      # e.g. conflicting sources, high-impact step
            output = human_expert.do(step, context=results)
        else:
            output = ai_agent.act(step, tools=[web, files, python_sandbox, shell])

        check = online_qa(output, step.acceptance_criteria)  # spec conformance, reconciliation,
                                                                # citation match, self-consistency
        if not check.passed:
            plan = replan(plan, step, check.reason)     # iterate rather than push a bad result forward
            continue

        results.append(output)

    draft = assemble(results)
    verdict = offline_qa(draft, spec)                  # full pass against original requirements
    if verdict.uncertain:
        draft = human_expert.review(draft, verdict.flags)   # final human QA pass
    return draft if verdict.ok_to_ship else rework(draft, verdict)

The one thing to notice: risk_score and RISK_GATE_THRESHOLD are the entire “secret sauce” of the system, and the paper doesn’t publish how they’re computed or tuned. That’s the part you’d have to design yourself if you built this (see Build Your Own and How to Improve It).

Built on Prior Work

Results & Evidence

In-house benchmark (94 tasks, three systems: Tendem, Upwork freelancers, ChatGPT Agent):

The quality gap is statistically significant (one-sided z-test, Tendem vs. Upwork, p = 0.0012). The biggest single-criterion gap is Completeness (+22.3 percentage points over Upwork) — the paper attributes this to the step-gate plan structure catching omissions that a single freelancer working alone would miss. ChatGPT Agent is fast and effectively free per task but has the worst quality by a wide margin — 36.2% of its outputs were rated outright Bad.

External agentic benchmarks (Tendem’s AI agent alone, no human involvement): on BrowseComp (deep web browsing), Tendem’s agent scores 71.0%, close to the best reported ChatGPT Agent number (68.9%) and ahead of most other systems listed. On GAIA (tool-use/assistant tasks) it scores 78.2%, near the top of the comparison table. On HLE (Humanity’s Last Exam — hard, broad knowledge) it scores 39.0%, behind GPT-5 pro with tools (42.0%) but ahead of several others. This establishes that the automated half of the hybrid isn’t weak — it’s a genuinely competitive agent on its own, which matters because the hybrid system’s economics depend on the AI carrying most of the volume.

What the evidence does not establish, and the paper is explicit about several of these:

• Single-rater labeling. Each task got one human QA rater’s Good/Mediocre/Bad/Decline judgment, with only “a large fraction” spot-checked — no published inter-rater agreement number, so some of the reported gaps could partly reflect one rater’s taste rather than a robust consensus.
• No LLM-as-judge validation. The paper explicitly says they did not validate an LLM-judge against their human raters, so there’s no independent cross-check on the human labels themselves.
• Small, narrow sample. 94 tasks across four business areas, deliberately excluding regulated/high-stakes domains (medical, legal, senior-level programming). Confidence intervals on individual quality numbers run ±4-5 percentage points — real, but not enormous.
• Vendor-reported external baselines. The comparison table for BrowseComp/HLE/GAIA mixes Tendem’s own measured numbers with other vendors’ self-reported numbers from system cards and blog posts, under different (sometimes browsing-enabled, sometimes not) setups. The paper flags this itself as imperfectly comparable.
• Who built and graded the benchmark. The 94-task benchmark, the rating rubric, and the raters were all assembled by the company that makes Tendem, to evaluate Tendem. The GitHub release of inputs/outputs is a genuine transparency move, but it doesn’t remove the structural conflict of interest in a first-party benchmark showcasing a first-party commercial product.
• No reported escalation rate. The paper never states what fraction of tasks or plan steps actually triggered a human step gate. Without that number you can’t tell how “hybrid” Tendem really is in practice versus how close it runs to human-only on the hard tasks — which materially affects how its cost and speed would scale with volume.

How You’d Use It

This paper is less “here’s a technique to copy” and more “here’s the operating model for any AI-plus-human system you run.” The direct takeaway: the deliverable isn’t “AI or human,” it’s a system that decides, at the level of individual plan steps, which one handles this step — and that decision needs to be instrumented, logged, and QA’d twice (online and offline), not left to vibes.

Concretely, this maps onto:

• Your business’s delivery pipeline. If you run any operation where AI agents produce work someone else consumes — internal reports, automated deliverables, customer-facing output — this is a template for the escalation layer you need to build in from day one, not bolt on later: define step-level risk tags in your planner, wire a human-review queue behind the ones that fail automated checks, and track pass/rework rates per reviewer the way Tendem tracks QA and rework rates.
• Your harness’s QA layer. The paper’s four-axis rubric (Accuracy, Completeness, Style & Formatting, Overall, each Good/Mediocre/Bad/Decline) is a reusable template for grading your own agent outputs against a human-only or fully-autonomous baseline before you trust a new automation with real output.
• Your cost model. The finding that hybrid can be cheaper at the median than human-only, despite involving humans, is a useful data point if you’ve assumed “human in the loop” automatically means “more expensive.” The average cost is still higher (a few escalated, high-touch tasks pull it up) — worth building that into your own cost model up front.

Build Your Own (Minimal Recipe)

You can get a meaningful fraction of the value with a fairly small build:

1. A plan-act-observe-verify loop over your target task type — a ReAct-style agent with tool access (web, files, code execution) is table stakes; frameworks like LangGraph make the loop-with-branches structure straightforward to implement as a graph.
2. A risk/uncertainty tag on each plan step. This is the one genuinely hard, genuinely novel part. You need some signal — model self-reported confidence, a rule (e.g., “step touches financial totals” or “step required conflicting sources”), or a lightweight classifier — that decides whether a step needs a human. Start crude (a short list of trigger conditions) and tune from there; don’t try to build a learned risk model on day one.
3. A human-review queue and UI. Whoever handles escalated steps needs the task context, the AI’s partial output, and a fast way to edit/approve/reject. This is as much an ops problem (who is on call, what’s the SLA) as an engineering one.
4. Two QA passes, not one. An online check that runs after each step (does this step’s output satisfy its own acceptance criteria — reconciliation, citation match, spec conformance) and an offline check that runs once at the end against the original requirements. Build the offline check as an explicit checklist derived from the stated deliverables, not a vague “does this look right” prompt.
5. A benchmark before you trust it. Steal the paper’s rubric (Accuracy/Completeness/Style/Overall, Good/Mediocre/Bad/Decline) and run it on ~20-50 real tasks from your own workload, blind-labeled, before you claim hybrid beats either baseline for your use case.

The two genuinely hard parts are #2 (calibrating when to escalate — too eager and you lose the cost/speed advantage over human-only, too lax and you lose the quality advantage over AI-only) and #3 (building or contracting a reliable, vetted expert pool with fast turnaround — a staffing/ops problem, not a code problem).

How to Improve It

1. Publish the escalation rate. What fraction of steps actually hit a human gate? This single number would let a reader model how Tendem’s cost and latency scale with task difficulty and volume — right now it’s a black box.
2. Validate an LLM-as-judge against the human raters. The paper flags this as a known gap. Running both in parallel on the same 94 tasks would either let them scale the benchmark cheaply or reveal where the human raters and an LLM judge disagree, which is useful information either way.
3. Report inter-rater agreement. Double-rate a subsample and publish a Cohen’s kappa or similar — single-rater-plus-spot-checks doesn’t tell you how noisy the Good/Mediocre/Bad boundary is.
4. Test escalation-rate stability at scale. Does the hybrid advantage hold when expert supply becomes the bottleneck (100x the task volume)? The 94-task study says nothing about queueing effects once you can’t staff every escalation instantly.
5. Break results down by task category. Table 1 gives category counts (e.g., “Build Multi-step Automation Workflows,” “Run Exploratory Data Analysis”) but the quality tables are aggregated. A category-level breakdown would show whether the hybrid advantage is uniform or concentrated in a few categories where AI-only is especially weak — which is exactly the information you’d need to decide where hybrid is worth the overhead for your own use case.

Glossary

• Step gate — a defined checkpoint inside the AI agent’s plan where the workflow can hand control to a Human Expert if the step is flagged high-risk or uncertain.
• Plan-act-observe-verify loop — the ReAct-style pattern where an agent reasons about what to do (plan), takes an action (act, often a tool call), reads the result (observe), and checks it (verify) before continuing.
• Online QA — automated quality checks that run during task execution, after each step, so errors don’t compound before anyone notices.
• Offline QA — a full automated (plus human, if uncertain) verification pass that runs after the task is otherwise complete, checking the whole result against the customer’s original requirements.
• Escalation — routing a step or task to a Human Expert because an automated check failed or a risk tag was triggered.
• ReAct — “Reasoning and Acting,” the technique of interleaving an LLM’s chain-of-thought reasoning with tool calls in one loop (Yao et al., 2023).
• Self-consistency / SelfCheckGPT — checking an LLM’s output for hallucination by asking it the same thing multiple ways and seeing if the answers agree, without needing external ground truth.
• Percentage point (pp) — the raw difference between two percentages (e.g., 74.5% − 53.2% = 21.3 pp), as opposed to a relative “percent change.”
• z-test (one-sided) — a statistical test used here to check whether the gap in “Good” rate between two systems is larger than you’d expect from random sampling noise alone.
• Bootstrap (confidence interval) — a way of estimating how much a statistic (like a median) would vary by repeatedly resampling the observed data with replacement, used here because medians don’t have a simple formula for their uncertainty.
• BrowseComp / GAIA / HLE — three public agent benchmarks used to test the AI agent alone: BrowseComp (deep web browsing for verifiable answers), GAIA (general assistant tasks needing tool use), HLE (Humanity’s Last Exam — hard, broad expert knowledge).
• Decline — a rating label meaning the system refused the task outright (safety filter or lack of domain coverage), distinct from a low-quality attempt.