# Harness-Bench ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Composite%20Score-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-106-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-May%202026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202605.27922-b31b1b)](https://arxiv.org/abs/2605.27922) > **Quick answer:** Harness-Bench is a May 2026 benchmark from Peking University and Qiyuan Tech that holds the task set and model pool fixed while swapping the agent harness (scaffolding) around them — 106 sandboxed, oracle-checkable tasks run across 6 harnesses and 8 model backends, producing 5,088 trajectories. It isolates how much of an agent's score is attributable to the harness rather than the model, finding a 23.8-point gap between the best harness (NanoBot, 76.2) and the worst (OpenClaw, 52.4) on the exact same models and tasks. ## At a Glance **What it tests:** Whether the agent harness — not just the underlying model — determines task completion, security, and process quality, by running identical tasks and models through 6 different harness implementations. **Why it matters:** Most agent benchmarks conflate model capability with harness quality, since they only ever test one harness per model. Harness-Bench is one of the first benchmarks to explicitly design for the opposite: same model, same task, different scaffolding — making the harness's contribution directly measurable. **Known limitations:** Only 6 harnesses and 8 model backends are covered, and the underlying task suite (106 tasks) is smaller than large-scale coding/agent benchmarks like SWE-bench Pro. Full per-harness, per-model score breakdowns beyond the headline best/worst gap are only available in the source paper. ## What Harness-Bench Measures Harness-Bench is built around a simple but underused experimental design: instead of comparing models under a single fixed harness (the norm for most agent leaderboards), it fixes 106 sandboxed, oracle-checkable tasks and runs each one through 6 configurable harnesses — OpenClaw, ZeroClaw, Hermes, Moltis, NullClaw, and NanoBot — combined with 8 different model backends, plus Codex as a non-harness reference point. That produces 5,088 total trajectories, enough to statistically separate "the model got it right" from "the harness let the model get it right." Each trajectory is scored across three dimensions: task completion (did the agent achieve the oracle-verified outcome), security (did the agent avoid unsafe or destructive actions), and process quality (robustness to unexpected tool output, correct tool-use sequencing, and consistency across repeated runs). These three dimensions are combined into the benchmark's composite 0–100 score. The paper's headline finding is that harness choice moves scores by up to 23.8 points on identical models and tasks — comparable to or larger than many model generation upgrades. It also finds that stronger models are more harness-tolerant (smaller score swings across harnesses), while weaker models are far more sensitive to which harness they're run through. Separately, the paper attributes 36% of all observed failures to output-contract or schema violations rather than reasoning errors — a harness-level bug class, not a model-level one. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Agent (harness comparison) | | Metric | Composite score (0–100): completion + security + process | | Number of tasks | 106 (sandboxed, oracle-checkable) | | Harnesses evaluated | OpenClaw, ZeroClaw, Hermes, Moltis, NullClaw, NanoBot (+ Codex reference) | | Model backends | 8 | | Total trajectories | 5,088 | | Saturation | Low (best composite: 76.2) | | Created by | Peking University & Qiyuan Tech | | Source paper | [Harness-Bench (arXiv 2605.27922)](https://arxiv.org/abs/2605.27922) | | GitHub | Not yet public | | Dataset | Not yet public | ## How Harness-Bench Is Scored Each of the 106 tasks is run to completion inside a sandbox and checked against an oracle (a verifiable ground-truth outcome), removing the need for subjective LLM-as-judge scoring on the completion dimension. Security scoring penalizes unsafe or destructive actions taken en route to the goal. Process scoring evaluates robustness (recovery from unexpected tool output), correct tool-use sequencing, and run-to-run consistency. The three sub-scores are combined into a single 0–100 composite per harness–model pair, then averaged across the 8 model backends to produce a per-harness ranking. A composite score in the mid-70s (the current best, NanoBot at 76.2) represents the top of the observed range; scores in the low-50s (OpenClaw at 52.4) represent harnesses that reliably lose ground purely from scaffolding weaknesses — schema violations, poor tool-call sequencing, or inconsistent recovery from errors — despite running the same underlying models as the top-ranked harness. ## State-of-the-Art Results Scores below are harness-level averages across all 8 model backends, not single-model results — this is a benchmark of harnesses, not models. | Rank | Harness | Avg. composite score (across 8 models) | Source | Date | |---|---|---|---|---| | 1 | NanoBot | 76.2 | [Harness-Bench paper](https://arxiv.org/abs/2605.27922) | 2026-05 | | 6 (last) | OpenClaw | 52.4 | [Harness-Bench paper](https://arxiv.org/abs/2605.27922) | 2026-05 | *Codex is evaluated as a non-harness reference baseline. Per-harness scores for ZeroClaw, Hermes, Moltis, and NullClaw, along with the full per-model breakdown, are reported in the source paper but not reproduced here — see the paper for the complete leaderboard.* *Scores sourced from the Harness-Bench paper. Results depend on harness version, model backend, and sandbox configuration — see the source paper for full methodology.* ## Harness-Bench on Benchgen > No Benchgen results yet — [be the first to run Harness-Bench](/benchmarks/harness-bench). ## Harness-Bench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **Harness-Bench** | Harness quality, holding model & task fixed | 106 | Low | | [EnterpriseClawBench](/benchmarks/enterprise-claw-bench) | Enterprise agent workflows (harness–model pairs) | 852 | Low | | [SWE-bench Pro](/benchmarks/swe-bench-pro) | Harder software engineering tasks | — | Low | | [Terminal-Bench](/benchmarks/terminalbench) | Terminal/CLI agent task completion | — | Low | EnterpriseClawBench and Harness-Bench both report harness–model pairs rather than model-only scores, but Harness-Bench is purpose-built to isolate the harness's contribution via a controlled task/model matrix, while EnterpriseClawBench focuses on realistic enterprise artifact delivery. Use Harness-Bench when you need to attribute a score change to scaffolding rather than model capability. ## Run Harness-Bench on Your Model Benchgen lets teams run the same task against multiple harness configurations while holding the model constant, making it possible to see exactly how much of your agent's score is coming from your scaffolding versus your model choice — and to catch harness regressions before they reach production. ## Related Models & Benchmarks - Related benchmarks: [EnterpriseClawBench](/benchmarks/enterprise-claw-bench) · [SWE-bench Pro](/benchmarks/swe-bench-pro) · [Terminal-Bench](/benchmarks/terminalbench) · [MCP Atlas](/benchmarks/mcp-atlas) ## Frequently Asked Questions

What is Harness-Bench?

Harness-Bench is a 2026 benchmark from Peking University and Qiyuan Tech that evaluates 6 agent harnesses across 8 model backends on 106 oracle-checkable tasks, isolating how much of an agent's score comes from the harness rather than the model.

What does a good Harness-Bench score look like?

Scores are composite (0–100), combining completion, security, and process quality. The best-performing harness (NanoBot) averages 76.2 across all 8 models; the worst (OpenClaw) averages 52.4 — a 23.8-point gap on identical models and tasks.

Who created Harness-Bench?

Harness-Bench was created by researchers at Peking University and Qiyuan Tech, published on arXiv in May 2026 (arXiv:2605.27922).

Why does Harness-Bench report harness scores instead of model scores?

The benchmark's explicit goal is to separate harness quality from model quality. By running the same 106 tasks and 8 models through 6 different harnesses, it produces a harness-level ranking that a model-only benchmark can't reveal.

Is Harness-Bench saturated?

No. The best observed composite score is 76.2 out of 100, leaving significant headroom, and the 23.8-point best-to-worst harness gap shows the benchmark still meaningfully differentiates scaffolding quality.

Harness-Bench — Evaluations