# WildClawBench ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Success-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-60-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-May%202026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2605.10912) [![GitHub](https://img.shields.io/badge/GitHub-InternLM%2FWildClawBench-181717?logo=github&logoColor=white)](https://github.com/InternLM/WildClawBench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/internlm/WildClawBench) > **Quick answer:** WildClawBench is a native-runtime agent benchmark of 60 human-authored, bilingual, multimodal tasks that run inside a real CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to genuine tools rather than mocked services. The best-performing model, Claude Opus 4.7, reaches only 62.2% overall under OpenClaw, showing the benchmark remains far from saturated. ## At a Glance **What it tests:** Long-horizon, real-tool agent task completion — each task averages ~8 minutes of wall-clock time and 20+ tool calls, spanning six thematic categories and running inside a reproducible Docker container hosting an actual CLI harness. **Why it matters:** Most agent benchmarks rely on synthetic sandboxes, short-horizon tasks, and mock-service APIs; WildClawBench tests whether agents can complete realistic work in the exact production-grade runtimes where they're deployed. **Known limitations:** Only 60 tasks (a small evaluation set), and switching harness alone can shift a single model's score by up to 18 points — meaning scores aren't directly comparable across different agent harnesses. ## What WildClawBench Measures WildClawBench drops agents into a live, containerized runtime hosting a genuine CLI agent harness — OpenClaw, Claude Code, Codex, or Hermes Agent — with access to real tools instead of mocked services. Its 60 tasks are human-authored, bilingual, and multimodal, spanning six thematic categories such as clipping highlights from a video, negotiating meeting times across multiple rounds of messages, and other everyday "in-the-wild" agent workflows. Each task averages roughly 8 minutes of wall-clock time and more than 20 tool calls, making it a genuine long-horizon test rather than a short single-turn interaction. Grading is hybrid: deterministic rule-based checks, environment-state auditing of side effects (did the agent actually produce the right file/state change), and an LLM/VLM judge for semantic verification of open-ended outputs. Because the benchmark runs inside the actual harness rather than a simplified simulation, results are sensitive to which CLI harness is used — the paper reports up to an 18-point swing for the same model across harnesses. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Agent | | Metric | % overall task success (hybrid rule-based + environment-state + LLM/VLM judge) | | Number of tasks | 60 (bilingual, multimodal, ~8 min / 20+ tool calls average) | | Saturation | Low | | Created by | Ding et al. (Shanghai AI Lab / InternLM) | | Source paper | [Ding et al. 2026](https://arxiv.org/abs/2605.10912) | | GitHub | [InternLM/WildClawBench](https://github.com/InternLM/WildClawBench) | | Dataset | [internlm/WildClawBench](https://huggingface.co/datasets/internlm/WildClawBench) | ## How WildClawBench Is Scored Models are scored on % overall task success within a specific CLI harness (OpenClaw, Claude Code, Codex, or Hermes Agent). Because grading combines deterministic checks with an LLM/VLM judge, scores reflect both whether the agent produced the correct end-state and whether open-ended outputs meet semantic criteria. Across 19 frontier models tested in the original paper, the best score (Claude Opus 4.7 under OpenClaw) reached only 62.2%, with every other model below 60% — indicating substantial headroom remains. ## WildClawBench on Benchgen > No Benchgen results yet — [be the first to run WildClawBench](/benchmarks/wildclawbench). ## WildClawBench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **WildClawBench** | **Native-runtime, long-horizon CLI agent tasks with real tools** | **60** | **Low** | | [GAIA2](/benchmarks/gaia2) | Dynamic, time-aware, multi-agent scenarios | 800 | Low | | [OSWorld-Verified](/benchmarks/osworld-verified) | Real desktop/web GUI computer-use tasks | 369 | Low | WildClawBench is the right choice for evaluating an agent's real CLI-harness behavior on long-horizon, multi-tool tasks, complementing GUI-focused benchmarks like OSWorld-Verified and simulated multi-agent scenarios like GAIA2. ## Run WildClawBench on Your Model Benchgen lets teams run WildClawBench against their own model and harness combinations, tracking scores across different CLI harnesses over time — important given how sensitive results are to harness choice — rather than relying on a single vendor-reported number. ## Frequently Asked Questions

What is WildClawBench?

WildClawBench is a native-runtime benchmark of 60 human-authored, bilingual, multimodal agent tasks that run inside a real CLI agent harness (OpenClaw, Claude Code, Codex, or Hermes Agent) with access to genuine tools.

What does a good WildClawBench score look like?

As of the original paper, the best model (Claude Opus 4.7) reached only 62.2% overall under the OpenClaw harness, with every other tested model below 60% — so scores in the 40-60% range represent strong frontier performance.

Who created WildClawBench?

WildClawBench was created by Shuangrui Ding and collaborators at Shanghai AI Lab / InternLM, released in May 2026.

Why do scores vary by harness on WildClawBench?

Because tasks run inside the actual CLI harness rather than a simplified simulation, differences in tool-calling conventions and scaffolding between harnesses (OpenClaw, Claude Code, Codex, Hermes Agent) can shift a single model's score by up to 18 points.

Is WildClawBench saturated?

No — the best-performing model in the original evaluation reached only 62.2% overall, leaving substantial headroom for improvement.

--- *Benchmark definition based on Ding et al., "WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation" (arXiv:2605.10912). Last updated 2026-08-10.*

Rank	Model	Score
1	dots3-note-preview	61.7
2	k2-horizon-375b-a23b	50.9
3	qwen3-8-27b	48.02
4	muse-glimmer	47.6

WildClawBench — Results

WildClawBench — Evaluations