Harness-Bench

Harness-Bench

Quick answer: Harness-Bench is a May 2026 benchmark from Peking University and Qiyuan Tech that holds the task set and model pool fixed while swapping the agent harness (scaffolding) around them — 106 sandboxed, oracle-checkable tasks run across 6 harnesses and 8 model backends, producing 5,088 trajectories. It isolates how much of an agent's score is attributable to the harness rather than the model, finding a 23.8-point gap between the best harness (NanoBot, 76.2) and the worst (OpenClaw, 52.4) on the exact same models and tasks.

At a Glance

What it tests: Whether the agent harness — not just the underlying model — determines task completion, security, and process quality, by running identical tasks and models through 6 different harness implementations.

Why it matters: Most agent benchmarks conflate model capability with harness quality, since they only ever test one harness per model. Harness-Bench is one of the first benchmarks to explicitly design for the opposite: same model, same task, different scaffolding — making the harness's contribution directly measurable.

Known limitations: Only 6 harnesses and 8 model backends are covered, and the underlying task suite (106 tasks) is smaller than large-scale coding/agent benchmarks like SWE-bench Pro. Full per-harness, per-model score breakdowns beyond the headline best/worst gap are only available in the source paper.

What Harness-Bench Measures

Harness-Bench is built around a simple but underused experimental design: instead of comparing models under a single fixed harness (the norm for most agent leaderboards), it fixes 106 sandboxed, oracle-checkable tasks and runs each one through 6 configurable harnesses — OpenClaw, ZeroClaw, Hermes, Moltis, NullClaw, and NanoBot — combined with 8 different model backends, plus Codex as a non-harness reference point. That produces 5,088 total trajectories, enough to statistically separate "the model got it right" from "the harness let the model get it right."

Each trajectory is scored across three dimensions: task completion (did the agent achieve the oracle-verified outcome), security (did the agent avoid unsafe or destructive actions), and process quality (robustness to unexpected tool output, correct tool-use sequencing, and consistency across repeated runs). These three dimensions are combined into the benchmark's composite 0–100 score.

The paper's headline finding is that harness choice moves scores by up to 23.8 points on identical models and tasks — comparable to or larger than many model generation upgrades. It also finds that stronger models are more harness-tolerant (smaller score swings across harnesses), while weaker models are far more sensitive to which harness they're run through. Separately, the paper attributes 36% of all observed failures to output-contract or schema violations rather than reasoning errors — a harness-level bug class, not a model-level one.

Benchmark Specifications

How Harness-Bench Is Scored

Each of the 106 tasks is run to completion inside a sandbox and checked against an oracle (a verifiable ground-truth outcome), removing the need for subjective LLM-as-judge scoring on the completion dimension. Security scoring penalizes unsafe or destructive actions taken en route to the goal. Process scoring evaluates robustness (recovery from unexpected tool output), correct tool-use sequencing, and run-to-run consistency. The three sub-scores are combined into a single 0–100 composite per harness–model pair, then averaged across the 8 model backends to produce a per-harness ranking.

A composite score in the mid-70s (the current best, NanoBot at 76.2) represents the top of the observed range; scores in the low-50s (OpenClaw at 52.4) represent harnesses that reliably lose ground purely from scaffolding weaknesses — schema violations, poor tool-call sequencing, or inconsistent recovery from errors — despite running the same underlying models as the top-ranked harness.

State-of-the-Art Results

Scores below are harness-level averages across all 8 model backends, not single-model results — this is a benchmark of harnesses, not models.

Codex is evaluated as a non-harness reference baseline. Per-harness scores for ZeroClaw, Hermes, Moltis, and NullClaw, along with the full per-model breakdown, are reported in the source paper but not reproduced here — see the paper for the complete leaderboard.

Scores sourced from the Harness-Bench paper. Results depend on harness version, model backend, and sandbox configuration — see the source paper for full methodology.

Harness-Bench on Benchgen

No Benchgen results yet — be the first to run Harness-Bench.

Harness-Bench vs Other Benchmarks

EnterpriseClawBench and Harness-Bench both report harness–model pairs rather than model-only scores, but Harness-Bench is purpose-built to isolate the harness's contribution via a controlled task/model matrix, while EnterpriseClawBench focuses on realistic enterprise artifact delivery. Use Harness-Bench when you need to attribute a score change to scaffolding rather than model capability.

Run Harness-Bench on Your Model

Benchgen lets teams run the same task against multiple harness configurations while holding the model constant, making it possible to see exactly how much of your agent's score is coming from your scaffolding versus your model choice — and to catch harness regressions before they reach production.

Related Models & Benchmarks

• Related benchmarks: EnterpriseClawBench · SWE-bench Pro · Terminal-Bench · MCP Atlas

Frequently Asked Questions