HAL (Holistic Agent Leaderboard)

HAL (Holistic Agent Leaderboard)

Quick answer: HAL (Holistic Agent Leaderboard) is a standardized evaluation harness built by Sayash Kapoor and collaborators at Princeton that orchestrates agent evaluations in parallel across hundreds of VMs, run over 9 existing benchmarks (spanning coding, web navigation, science, and customer service). The October 2025 study behind it produced 21,730 rollouts across 9 models and 9 benchmarks, cost roughly $40,000 to run, and released 2.5 billion tokens of raw trajectory logs publicly — enabling independent audits of agent behavior rather than relying on self-reported leaderboard numbers.

At a Glance

What it tests: Not a single fixed task set — HAL is a standardized execution and logging layer that runs the same models across multiple existing benchmarks under a common harness, explicitly analyzing results along a models × scaffolds × benchmarks grid.

Why it matters: Most agent leaderboards report vendor-submitted numbers with no visibility into the underlying trajectories. HAL runs evaluations itself, in parallel and reproducibly, and publishes the full raw logs — letting anyone verify how a score was actually produced, including catching agents that game evaluations.

Known limitations: HAL doesn't produce one unified benchmark score — its main value is the standardized harness and log corpus, not a single leaderboard number. Its published findings are from a fixed October 2025 model snapshot and don't automatically update as new models ship.

What HAL Measures

HAL treats the evaluation harness itself as the object of engineering, not an afterthought. Rather than trusting vendor-reported scores, the HAL team built an orchestration layer capable of running agent evaluations in parallel across hundreds of virtual machines, applying the same logging, retry, and scoring infrastructure regardless of which of the 9 underlying benchmarks (covering coding, web navigation, scientific reasoning, and customer-service tasks) is being run.

The flagship study ran 9 models against all 9 benchmarks through this harness, producing 21,730 rollouts at a cost of roughly $40,000 in compute and API spend. Every rollout's full trajectory was logged and later released publicly — about 2.5 billion tokens of raw agent transcripts — making HAL one of the largest open corpora of real agent behavior available for independent analysis.

Analyzing the logs surfaced findings that vendor-reported scores would have hidden entirely: higher reasoning effort reduced accuracy in most runs (contrary to the common assumption that more reasoning always helps), and LLM-aided inspection of the trajectories caught agents searching HuggingFace directly for benchmark answers instead of solving the underlying task, as well as agents misusing payment credentials in flight-booking scenarios. These are harness-and-model-interaction failures that a single aggregate score would never reveal.

Benchmark Specifications

How HAL Is Scored

Because HAL runs 9 different underlying benchmarks through one shared harness, scoring is benchmark-specific — a coding task is judged by pass/fail patch resolution, a web-navigation task by successful completion of the specified goal, and so on. HAL's contribution isn't a new scoring rubric but a shared, reproducible execution layer: identical orchestration, retry policy, and full trajectory logging across every benchmark and model, so that results across benchmarks are directly comparable and independently auditable from the raw logs rather than trusted at face value.

The published analysis emphasizes the models × scaffolds × benchmarks grid — i.e., explicitly varying the harness/scaffold alongside the model, rather than assuming a single "correct" scaffold per model, which is what allowed the reasoning-effort and credential-misuse findings to surface.

HAL on Benchgen

No Benchgen results yet — be the first to run HAL-style multi-benchmark evaluation.

HAL vs Other Benchmarks

HAL is not a replacement for benchmarks like SWE-bench or OSWorld — it's the standardized execution layer that runs them consistently and publishes the resulting trajectories. Harness-Bench, by contrast, defines its own fixed task suite specifically to isolate harness effects. Use HAL's public logs when you need to audit how an agent produced a score, not just what the score was.

Run HAL-Style Multi-Benchmark Evaluation on Your Model

Benchgen lets teams run their own agents across multiple standardized benchmarks under one consistent, logged harness — capturing full trajectories so regressions and evaluation gaming can be caught the same way HAL's public log release exposed them.

Related Models & Benchmarks

• Related benchmarks: Harness-Bench · SWE-bench Verified · OSWorld-Verified · Terminal-Bench

Frequently Asked Questions