EnterpriseClawBench

EnterpriseClawBench

Quick answer: EnterpriseClawBench is an enterprise agent benchmark built from real workplace sessions, introduced by Zhong et al. (FrontisAI, June 2026). It contains 852 reproducible tasks derived from proprietary agent sessions, each paired with recovered file fixtures, rewritten prompts, role/skill annotations, hard rules, and semantic rubrics. It uniquely evaluates complete harness–model systems rather than base models alone. The current best result is 66.3%, achieved by Codex with GPT-5.5, while the benchmark average across all tested configurations is 48.6%.

At a Glance

What it tests: Whether a complete coding-agent system (harness + model) can operate inside a persistent workspace, read heterogeneous files, invoke tools, and deliver usable business artifacts — assessed against hard rules and semantic rubrics derived from real enterprise sessions.

Why it matters: Most coding benchmarks test isolated code-generation tasks. EnterpriseClawBench evaluates the full agent system on realistic enterprise workflows including spreadsheet generation, document creation, web page production, and multi-file analysis — the tasks enterprise teams actually deploy AI agents to perform. Critically, it reports harness–model combinations rather than base model scores, acknowledging that the scaffolding matters as much as the model.

Known limitations: The benchmark data is not publicly released (it derives from proprietary enterprise sessions); only the construction pipeline, evaluation protocol, and aggregate leaderboard are open. Results are therefore not independently reproducible against the private task set. Scores are also sensitive to prompt format and harness configuration — comparisons across configurations should account for these differences.

What EnterpriseClawBench Measures

EnterpriseClawBench constructs each task from a real workplace agent session through a privacy-preserving pipeline: raw sessions are filtered, fixtures are recovered, tasks are rewritten to be self-contained, and each task is annotated with a role class, skill subclass, hard validation rules, and a semantic rubric. The agent receives a prompt and any necessary file fixtures, executes in a sandbox, and produces natural-language responses plus file artifacts. Evaluation applies both rule-based checks (hard rules) and LLM-as-judge rubric scoring.

The benchmark covers a broad range of enterprise artifact types: spreadsheets, documents, web pages, data exports, and structured reports. Because the tasks derive from real sessions rather than synthetic construction, the task distribution reflects actual enterprise workloads — including the edge cases, ambiguous specifications, and multi-step dependencies that characterise production agent deployments.

A key design decision is evaluating harness–model combinations. The harness (the agent scaffolding: tool routing, retry logic, memory, multi-step planning) can account for as much variance in score as the underlying model. EnterpriseClawBench makes this explicit by treating each harness–model pair as a distinct entry on the leaderboard, with cost and runtime reported alongside quality scores.

Benchmark Specifications

How EnterpriseClawBench Is Scored

Each of the 852 tasks is scored on a 0–1 scale. The score combines hard rule validation (binary pass/fail on structural requirements like file format, presence of required fields, and computational correctness) with semantic rubric scoring (LLM-as-judge evaluation of response quality, artifact usability, and instruction adherence).

The final score reported for a harness–model configuration is the average across all tasks. A score of 0.663 (66.3%) represents the current best result. The benchmark average across all tested configurations is 48.6%, meaning more than half of all tested systems score below this level. Scores below 30% generally indicate configurations where the harness and model combination struggles to reliably complete multi-step enterprise tasks.

Cost (in ¥ CNY for a full benchmark run) and runtime (in minutes) are reported alongside quality scores, since enterprise deployment decisions require balancing quality against operational cost.

State-of-the-Art Results

Scores are reported per harness–model combination. The average score across all 32 tested configurations is 48.6%.

Full leaderboard: 32 harness–model combinations. Source: Zhong et al. 2026 / frontisai.github.io/EnterpriseClawBench

Scores sourced from the EnterpriseClawBench paper and public leaderboard. Results depend on harness, prompt format, and sandbox configuration — see the source paper for full methodology.

EnterpriseClawBench on Benchgen

No Benchgen results yet — be the first to run EnterpriseClawBench.

EnterpriseClawBench vs Other Benchmarks

SWE-bench Verified and SWE-bench Pro are the closest analogues for coding-agent evaluation, but they focus on repository-level code patching rather than enterprise artifact delivery. EnterpriseClawBench is the only benchmark that explicitly evaluates harness–model systems on realistic workplace workflows including multi-format artifact generation, making it more representative of how enterprise AI agents are actually deployed.

Run EnterpriseClawBench on Your Model

Benchgen lets teams run standardised benchmarks against their own model and harness configurations, compare results across runs, and detect regressions before they reach production. Rather than relying on vendor-reported one-time scores from the official leaderboard, running EnterpriseClawBench through Benchgen gives your team version-controlled benchmark history tied to your specific deployment configuration.

Related Models & Benchmarks

• Top performers: GPT-5.5 · Claude Sonnet 4.6 · Claude Opus 4.6
• Related benchmarks: SWE-bench Verified · SWE-bench Pro · τ³ Banking

Frequently Asked Questions