WildClawBench

WildClawBench

Quick answer: WildClawBench is a native-runtime agent benchmark of 60 human-authored, bilingual, multimodal tasks that run inside a real CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to genuine tools rather than mocked services. The best-performing model, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, showing the benchmark remains far from saturated.

At a Glance

What it tests: Long-horizon, real-tool agent task completion — each task averages ~8 minutes of wall-clock time and 20+ tool calls, spanning six thematic categories and running inside a reproducible Docker container hosting an actual CLI harness.

Why it matters: Most agent benchmarks rely on synthetic sandboxes, short-horizon tasks, and mock-service APIs; WildClawBench tests whether agents can complete realistic work in the exact production-grade runtimes where they're deployed.

Known limitations: Only 60 tasks (a small evaluation set), and switching harness alone can shift a single model's score by up to 18 points — meaning scores aren't directly comparable across different agent harnesses.

What WildClawBench Measures

WildClawBench drops agents into a live, containerized runtime hosting a genuine CLI agent harness — OpenClaw, Claude Code, Codex, or Hermes Agent — with access to real tools instead of mocked services. Its 60 tasks are human-authored, bilingual, and multimodal, spanning six thematic categories such as clipping highlights from a video, negotiating meeting times across multiple rounds of messages, and other everyday "in-the-wild" agent workflows. Each task averages roughly 8 minutes of wall-clock time and more than 20 tool calls, making it a genuine long-horizon test rather than a short single-turn interaction.

Grading is hybrid: deterministic rule-based checks, environment-state auditing of side effects (did the agent actually produce the right file/state change), and an LLM/VLM judge for semantic verification of open-ended outputs. Because the benchmark runs inside the actual harness rather than a simplified simulation, results are sensitive to which CLI harness is used — the paper reports up to an 18-point swing for the same model across harnesses.

Benchmark Specifications

How WildClawBench Is Scored

Models are scored on % overall task success within a specific CLI harness (OpenClaw, Claude Code, Codex, or Hermes Agent). Because grading combines deterministic checks with an LLM/VLM judge, scores reflect both whether the agent produced the correct end-state and whether open-ended outputs meet semantic criteria. Across 19 frontier models tested in the original paper, the best score (Claude Opus 4.7 under OpenClaw) reached only 62.2%, with every other model below 60% — indicating substantial headroom remains.

WildClawBench on Benchgen

No Benchgen results yet — be the first to run WildClawBench.

WildClawBench vs Other Benchmarks

WildClawBench is the right choice for evaluating an agent's real CLI-harness behavior on long-horizon, multi-tool tasks, complementing GUI-focused benchmarks like OSWorld-Verified and simulated multi-agent scenarios like GAIA2.

Run WildClawBench on Your Model

Benchgen lets teams run WildClawBench against their own model and harness combinations, tracking scores across different CLI harnesses over time — important given how sensitive results are to harness choice — rather than relying on a single vendor-reported number.

Frequently Asked Questions

Benchmark definition based on Ding et al., "WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation" (arXiv:2605.10912). Last updated 2026-08-10.