OpenJarvis: Personal AI, On Personal Devices

TL;DR

Personal AI stacks like OpenClaw and Hermes Agent route almost everything, including sensitive personal data, to a cloud frontier model. Just dropping in a local open-weight model instead (same prompts, same tools, same everything else) tanks accuracy by 25–39 percentage points, because the prompts, tool descriptions, memory setup, and runtime settings were all co-designed for the cloud model. OpenJarvis fixes this by describing the whole personal AI system as a typed “spec” with five independently editable pieces (model, runtime, agent logic, tools/memory, and the optimizer), then runs LLM-guided spec search: a frontier cloud model reads failure traces and proposes coordinated edits across the spec, but an edit is only kept if it passes a strict “don’t make anything else worse” test. The cloud model is only used during this offline search; the resulting spec runs 100% on-device. The payoff: on-device specs match or beat cloud accuracy on 4 of 8 benchmarks, land within 3.2 percentage points of cloud on average, and do it at roughly 800× lower marginal API cost and 4× lower latency, while spec search itself closes 13–32 points of the remaining gap for 7–11× less optimization cost than tuning prompts or weights alone.

Problem & Motivation

Personal AI agents (write my email, research this, manage my calendar) increasingly run through frameworks like OpenClaw and Hermes Agent. Almost all of them assume a cloud frontier model sits at the center: that costs real money every month, sends personal data off-device, needs network connectivity, and burns far more energy per token than local inference would.

The obvious fix — run an open-weight model locally instead — doesn’t work if you just substitute the model and leave everything else alone. The authors tested this directly: take OpenClaw or Hermes Agent exactly as shipped, and replace their intended cloud model (Claude Opus 4.6) with a solid open-weight local model (Qwen3.5-9B), changing nothing else. Accuracy drops 25–39 percentage points on PinchBench and GAIA (two personal-AI-style benchmarks).

Why does swapping the model break so much? Because these frameworks are monolithic: the agent prompts, the tool descriptions, the memory configuration, and the runtime settings are all written and tuned assuming a specific, very capable cloud model sits behind them. Pull that model out and put in a weaker local one, and every one of those components is now miscalibrated at once — the prompt over-assumes reasoning ability, the tool descriptions assume the model will infer intent it can’t, the memory retrieval settings assume more context capacity than the runtime affords.

You might think: fine, just tune what you can. But in these frameworks, the only thing you can tune without touching the source code is the prompt. The authors ran state-of-the-art prompt optimizers (GEPA and DSPy) against the local-model swap and only closed 5 percentage points of the 25–39 point gap. Optimizing one lever (the prompt) can’t fix a problem that’s spread across five different levers at once.

What’s New (Core Contribution)

• The spec abstraction. Before: frameworks hardcode or fuse model choice, runtime, agent prompts, tools, and memory into one bundle tied to a specific cloud model — you can edit at most the prompt. Now: OpenJarvis represents the whole system as a typed configuration object (a “spec”) with five independent, swappable fields — Intelligence (model + weights), Engine (runtime), Agents (reasoning loop/prompts/tool policy), Tools & Memory (integrations + persistent state), and Learning (the optimizer that edits the other four). Every piece becomes a first-class, editable degree of freedom instead of baked-in framework code.
• Joint accuracy–efficiency evaluation. Before: benchmarks report accuracy alone (GAIA, SWE-bench) or measure one efficiency axis in isolation (energy-only tools like Zeus, cost-only tools). Now: every query run against a spec is instrumented for accuracy, energy, latency, power, and dollar cost simultaneously, so you can see the actual accuracy-for-cost tradeoff of a complete configuration, not just a model.
• LLM-guided spec search. Before: every existing optimizer touches one primitive — LoRA/SFT/GRPO update model weights only; DSPy/GEPA update prompts only. Now: a frontier cloud model reads real usage traces, groups failures into clusters, and proposes coordinated edits across up to four primitives at once (model, runtime, agent logic, tools/memory) — and each candidate edit is only accepted if it passes a held-out “no regressions elsewhere” gate. The cloud model is used only at search time; the winning spec runs entirely on local hardware at inference time.
• Evidence that this actually closes the gap, and why. The paper isolates how much each idea contributes: the spec alone (no search) recovers 56–77% of the local-substitution accuracy drop; search on top of that closes another 13–32 percentage points, and ablations show both “which model proposes edits” and “how many primitives it’s allowed to touch” matter independently.

How It Works (Technically)

1. The spec. A spec is a typed object with five fields:

Spec S := {
  Intelligence: (model, params, quant),
  Engine      : (backend, batch, kv_cache),
  Agent       : (loop, prompts, tool_strategy),
  Tools       : (set, descriptions, memory),
  Learning    : (optimizer, reward, gate)
}

Each field is independently swappable — you can change the model without touching the runtime, or rewrite a tool description without retraining anything. Learning is different from the other four: it isn’t part of the runtime system, it’s the optimizer that’s allowed to edit the other four fields. Different optimizers are just different restrictions on which fields Learning may touch: LoRA only edits Intelligence; DSPy and GEPA only edit Agent; LLM-guided spec search can edit Intelligence, Engine, Agent, and Tools jointly.

2. Joint evaluation. For every query run against a spec, a wrapper records accuracy (exact match or an LLM-judge win rate), energy (measured on local hardware, estimated for cloud via prior work), latency (wall-clock), power (energy ÷ latency), and dollar cost (API pricing for cloud; $0 marginal cost for local, with hardware/electricity reported separately). This gives a spec a full cost-quality profile instead of a single accuracy number.

3. LLM-guided spec search — the core loop. This is a local–cloud division of labor: the cloud model is good at reading many traces and reasoning about coordinated changes, so it does that (offline, at search time); local hardware is good at running the result cheaply and fast, so it does that (at inference time). Concretely, each search session:

1. Diagnose. The frontier “teacher” model reads a batch of traces from the current spec and groups the failures into failure clusters — sets of traces that fail for the same underlying reason, with a plain-language description (e.g. “student fails multi-hop scheduling questions because it never invokes the calendar tool”).

2. Propose. The teacher proposes an edit targeting one cluster. An edit can span any combination of the four editable primitives at once — e.g. rewrite a tool’s description and add a matching one-shot example to the agent prompt and bump the context window in the Engine settings, all in one proposal.

3. Apply & evaluate. The edit is applied to produce a candidate spec S', which is scored on a held-out set covering the targeted cluster and every other tracked cluster.

4. Gate. The edit is accepted only if it strictly helps the cluster it targeted, and doesn’t meaningfully hurt anything else:

   Gc(S') > Gc(S) and Gc'(S') ≥ Gc'(S) − ε for every other cluster c'

   In plain terms: Gc(S) is “how well does spec S do on failure cluster c, measured on held-out data.” The rule says the targeted cluster must get strictly better, and no other cluster is allowed to drop by more than ε (default 1%). This is a “do no harm” gate — like a canary check for a config change instead of a code change: ship the change only if it fixes what it was meant to fix and doesn’t quietly break something else.

5. Repeat. Accepted edits become the new spec for the next session; rejected edits are discarded (rolled back). The loop stops when the gate score stops improving for k sessions (default 5) or the search budget runs out.

Optional: training-time reward for Intelligence edits. When a proposed edit trains model weights (via GRPO, an RL method), each candidate response y to a query q isn’t scored on correctness alone — it’s scored by a weighted composite:

R(q, y) = α·R_acc(q, y) − β·Ê(q, y) − γ·L̂(q, y) − δ·Ĉ(q, y)

with default weights (α, β, γ, δ) = (0.5, 0.1, 0.1, 0.3). In plain English: reward correctness, but subtract penalties for energy, latency, and dollar cost, so the RL step optimizes toward answers that are correct and cheap/fast, not just correct. The overall spec is still judged by the same held-out gate — this reward only shapes what “good” means inside one Intelligence-edit training run.

Architecture & data flow

flowchart LR
  subgraph Spec["Spec (5 typed primitives)"]
    I[Intelligence<br/>model + params]
    E[Engine<br/>runtime + quant]
    A[Agents<br/>reasoning loop + prompts]
    T["Tools & Memory<br/>integrations + state"]
    L[Learning<br/>optimizer + gate]
  end
  L -->|edits| I
  L -->|edits| E
  L -->|edits| A
  L -->|edits| T
  Spec --> Wrap[Instrumented wrapper]
  Wrap --> M["accuracy · energy · latency · power · cost"]

flowchart TD
  Start[Current spec S] --> Diag["Teacher.diagnose(traces)"]
  Diag --> Clusters[Failure clusters]
  Clusters --> Prop["Teacher.propose(edit)"]
  Prop --> Apply["apply edit -> candidate spec S'"]
  Apply --> Gate{"GateOK?<br/>target cluster improves AND<br/>others regress ≤ ε"}
  Gate -->|accept| Commit[S <- S'] --> Diag
  Gate -->|reject| Rollback[keep S] --> Diag
  Commit -.stagnant k sessions or budget exhausted.-> Done[Return S]

The acceptance gate, live. Toggle a proposed edit and watch which failure clusters move — the edit only gets accepted if the targeted cluster (bold) improves and every other cluster stays within the tolerance band.

The algorithm, simplified

# LLM-guided spec search — the search-time loop (paper's Algorithm 1)
# S: current spec. teacher: frontier cloud model, used only here, not at inference.
# gate: held-out evaluator. epsilon: max allowed regression on a non-target cluster.
def llm_guided_spec_search(S0, teacher, gate, epsilon=0.01, k=5, budget=None):
    S = S0
    stagnant = 0
    cost = 0
    while stagnant < k and cost < budget:
        clusters = teacher.diagnose(traces(S))          # group failures by root cause
        target, edit = teacher.propose(S, clusters)      # may touch Intelligence/Engine/Agent/Tools jointly
        S_candidate = apply(S, edit)

        scores_before = gate.score_all_clusters(S)
        scores_after = gate.score_all_clusters(S_candidate)
        cost += training_cost(edit)                      # only real if edit trains weights

        target_improved = scores_after[target] > scores_before[target]
        no_collateral_damage = all(
            scores_after[c] >= scores_before[c] - epsilon
            for c in scores_before if c != target
        )

        if target_improved and no_collateral_damage:
            S = S_candidate                               # accept: greedy hill-climb
            stagnant = 0
        else:
            stagnant += 1                                 # reject: roll back, try again next session

    return S   # this spec runs 100% on-device at inference time

Built on Prior Work

Results & Evidence

The suite: 8 personal-AI benchmarks (PinchBench, GAIA, LiveCodeBench, τ-Bench V2, τ²-Bench Telecom, ToolCall-15, DeepResearchBench, LiveResearchBench), 11 local models across 4 model families, 3 cloud baselines, 7 hardware platforms, 508 tasks total, every number averaged over 5 independent runs, scored by GPT-5-mini as an LLM judge except where a benchmark grades deterministically.

Accuracy vs. marginal cost per query, schematic from the paper's reported numbers. Drag the accuracy-gap slider to see how few points separate the best local spec from the best cloud model, at roughly three orders of magnitude difference in cost.

• The spec alone (no search) recovers most of the substitution damage. Swapping Claude Opus 4.6 for Qwen3.5-9B inside OpenClaw/Hermes Agent drops accuracy 24.8–38.8 pp. Re-wrapping the same local model in an OpenJarvis spec (retargeting only Engine/Agent/Tools, no search, no model change) shrinks the residual gap to 5.6–16.5 pp — recovering 77% of the PinchBench drop and 56–57% of the GAIA drop. GAIA recovers less because its deep multi-hop reasoning demands more raw capability than a 9B model has, regardless of how the surrounding stack is configured.
• The best local spec lands within 3.2 pp of the best cloud model on average, and matches or beats cloud on 4 of 8 benchmarks (ToolCall-15, PinchBench, LiveCodeBench, τ-Bench V2). Qwen3.5-122B reaches 80.3% average accuracy versus Claude Opus 4.6’s 83.5%. Remaining gaps concentrate on GAIA, τ²-Bench Telecom, and DeepResearchBench.
• At ~800× lower marginal API cost and ~4× lower latency. Qwen3.5-122B costs roughly a thousandth of a cent per query versus $0.009 for Claude Opus 4.6. Local specs also finish full agentic workloads about 4× faster in this protocol, though the authors note single-shot prompts can still favor cloud serving due to time-to-first-token optimizations cloud providers have invested in.
• LLM-guided spec search closes 13–32 more percentage points. The strongest search-optimized Qwen3.5-9B student reaches 100.0% on PinchBench, 83.0% on LiveCodeBench, and 91.0% on LiveResearchBench (up from the unoptimized spec). Across all 8 benchmarks, average gains per student model range 13.1–31.5 pp.
• Search beats single-primitive tuning at lower cost, not just higher accuracy. LLM-guided spec search beats LoRA (the strongest single-primitive baseline) by 1.1–8.8 pp and prompt-only GEPA by 5.0–18.8 pp, while costing 7.1–10.9× less to run — because rejected edits never trigger a full training run, only accepted ones do.
• Both “who proposes” and “how much they can touch” matter, independently. At the same four-primitive move space, the LLM teacher beats an evolutionary-search proposer by 10.0 pp on average and beats random edit sampling by 14.0 pp on average — so the edit catalog and gate alone don’t explain the gain, the teacher’s reasoning does. Separately, with the LLM teacher fixed, expanding the editable set from 1 primitive to all 4 adds 5.5–16.5 pp accuracy and 2.65–3.45× latency speedup; forcing primitives to move together (merging pairs) reverses this, losing 2.8–9.7 pp accuracy.
• Accepted edits spread across all four primitives, not concentrated on model weights: weight edits are only 16–44% of accepted edits. Which primitive dominates tracks the failure type — Intelligence edits dominate code tasks, Agent edits dominate customer-service/agentic tasks, Tool edits dominate tool-calling and research tasks.

Caveats the paper is upfront about (and one it should be pushed on harder): results come from 5 runs per configuration, use an LLM judge (GPT-5-mini) that may carry its own bias, and were evaluated on a single machine setup. Also worth flagging as a reader: PinchBench and LiveResearchBench read as benchmarks closely associated with this same research group rather than fully independent third-party suites, and the “spec search” step itself isn’t free — the paper’s own math shows the amortized teacher cost only drops below $0.001/query after six months at 100 queries/day, which matters if you’re evaluating this for a low-volume or one-off use case rather than a standing assistant.

How You’d Use It

• Your harness. The transferable idea isn’t “run a local model” (that’s commodity now) — it’s the methodology: decompose your own agent stack into independently swappable primitives (model, runtime, agent logic, tools/memory, optimizer), generate real failure traces from it, use a strong frontier model as a one-time optimization teacher, and gate every accepted change against a held-out set. If your harness currently hardcodes prompts, tool descriptions, and runtime settings around one specific cloud model, this is a concrete refactor target: pull those apart so swapping the model underneath doesn’t silently break the other four.
• Your applications. If you’re shipping a personal-AI-style feature (an assistant, an agent that touches sensitive user data) and want it on-device for privacy, latency, or cost reasons, this is a working recipe for closing the accuracy gap between a frontier cloud model and a local one — cluster failures, propose coordinated edits, gate them — rather than assuming a bigger local model is the only lever. It’s the strongest fit where “the data never leaves the device” and “no recurring per-token bill” matter more than squeezing out the last few points of benchmark accuracy.
• Your workflows and methodologies. The gate rule itself — accept a change only if the thing it targeted improved and nothing else got meaningfully worse — is usable immediately, even without building the full spec system, as a discipline for tuning any agent’s prompts, tool descriptions, or RAG config by hand. It also reframes a pattern familiar from any local-model-swap work: most “the local model is worse” failures are stack-misalignment failures, not model-capability failures, and the fix is coordinated, gated edits across prompt/tool/runtime — not just a bigger model.
• What it costs: the eval harness and held-out gate discipline are the real investment, not the search loop itself. The paper’s own math shows a teacher-optimization pass only pays for itself after roughly six months at 100 queries/day — worth running the same amortization check against your own expected query volume before treating this as free.

Build Your Own (Minimal Recipe)

You don’t need the paper’s full framework to get most of the value. A toy version needs:

1. A typed spec — even a plain dataclass or TOML file with four editable fields: model (name + generation params), runtime (backend, quantization, context length), agent (system prompt + tool-selection policy), tools (tool descriptions + memory/retrieval settings).
2. An eval harness — run a fixed task suite against a spec, log every trace: input, output, pass/fail, cost, latency. Split into an “optimization” set and a held-out “gate” set up front, and don’t leak between them.
3. A teacher step — batch the failed traces, send them to a frontier model (Claude/GPT) with a prompt asking it to (a) cluster failures by likely cause and (b) propose one concrete edit to one field targeting one cluster.
4. Apply + gate — apply the edit, re-score on the held-out set, and keep it only if the targeted cluster’s score improved and nothing else dropped by more than your chosen tolerance (start at 1%, same as the paper).
5. Loop with a stop rule — stop after N sessions with no accepted edit, or when you hit a time/dollar budget.

Build order: start with just the agent and tools fields editable — they’re pure text edits, cheapest to test, and the paper’s own ablation shows expanding the move space (even without touching weights) is where a big chunk of the gain comes from. Add runtime settings next. Save weight edits (LoRA/GRPO) for last — they’re the slowest and most expensive part of the loop, and the paper shows they’re often not even the dominant fix (16–44% of accepted edits).

The genuinely hard parts: (1) getting the teacher’s failure clustering to be reliable and non-trivial — with too few or too homogeneous traces, everything looks like “one cluster” and the targeted-improvement signal is noisy; (2) held-out set discipline — the paper doesn’t specify exactly how large the held-out set needs to be per session, and reusing a small held-out set across many candidate edits risks quietly overfitting to the gate itself, the same way a shrinking validation set overfits in classic ML.

Reach for: whatever local serving stack you already run (Ollama, vLLM, llama.cpp), any frontier API as the teacher, and a lightweight harness — a Python script plus a folder of JSON traces is enough to prototype this in an afternoon.

How to Improve It

1. Multi-teacher ensembling. The paper tests three proposer models but always uses one at a time per search. Proposing with multiple teachers and only accepting edits where they agree (or where the intersection of their non-regressing edits overlaps) could cut down on edits that pass the gate by luck rather than genuine improvement.
2. Active held-out set growth. The gate’s reliability depends entirely on held-out set quality and size, which isn’t detailed. An active-learning step that expands the held-out set specifically around each newly accepted cluster would guard against overfitting to the gate across many search sessions.
3. Cross-deployment spec transfer. Since specs are typed and versioned, test whether a spec optimized for one deployment’s traffic pattern transfers — with a light re-optimization pass — to a similar deployment. If it does, the “one optimization run pays for many deployments” economics get even better for anyone running more than one instance.
4. Cost-aware stopping rule. Right now the loop stops on gate-score stagnation or a flat budget. A rule that factors in expected future query volume (tie the stopping point to ”$ this search will save per future query vs. $ already spent”) would make the amortization math in the paper (six months to break even at 100 q/day) something you could tune per deployment instead of accepting as fixed.
5. A narrow inference-time escape hatch for the hard cases. The authors admit GAIA-style deep reasoning gaps aren’t closeable by spec search alone — they’re capability-bound, not configuration-bound. Worth testing whether a cheap, rare, per-failure-cluster fallback to cloud at inference time (not the full “100% local” claim, just for the specific cluster search couldn’t fix) recovers more of that residual gap while keeping the vast majority of queries fully on-device.

Glossary

• Spec — the typed configuration object describing an entire personal AI system as five fields.
• Primitive — one of the spec’s five independently editable parts.
• Intelligence — the primitive covering model choice, weights, and generation parameters.
• Engine — the primitive covering the inference runtime: serving backend, batching, quantization, KV-cache.
• Agents — the primitive covering the reasoning loop, system prompts, and tool-use policy.
• Tools & Memory — the primitive covering external tool interfaces, retrieval, and persistent user state.
• Learning — the primitive that specifies the optimizer used to edit the other four from traces.
• LLM-guided spec search — this paper’s search algorithm: a cloud teacher proposes edits across the spec, a gate accepts only non-regressing ones.
• Gate — the held-out accept/reject rule applied to each candidate spec edit.
• Failure cluster — a group of traces the teacher groups together because they fail for the same underlying reason.
• Teacher / proposer — the frontier cloud model used only at search time to diagnose failures and propose edits; never called at inference time in the headline results.
• Student — the local model/spec being optimized.
• Trace — a logged record of one query run against a spec (input, output, pass/fail, cost, latency).
• GRPO — Group Relative Policy Optimization, a reinforcement-learning method used here for optional Intelligence weight edits.
• LoRA — Low-Rank Adaptation, a lightweight way to fine-tune a small subset of a model’s weights instead of all of them.
• DSPy / GEPA — prior automatic prompt/agent optimization frameworks that only edit the Agent prompt text, used here as single-primitive baselines.
• Pareto frontier — the set of configurations where you can’t improve one metric (e.g. accuracy) without making another (e.g. cost) worse.
• pp — percentage points, the unit used throughout for accuracy differences (not percent change).