AgentLens
1 phaseActive
Process-level re-scoring benchmark for agent trajectories, separating verified solutions from 'lucky' blind-retry passes.
Overview
AgentLens
Quick answer: AgentLens is a May 2026 process-level re-scoring benchmark that re-examines 2,614 existing OpenHands SWE-agent trajectories, distinguishing genuinely verified solutions from "lucky passes" — cases where an agent's blind retries or missing verification happened to produce a passing result without a solid underlying fix. It finds that up to 23.2% of passes in the original trajectories were lucky, and that model rankings shift by up to 5 positions once process quality (not just binary pass/fail) is factored into scoring.
At a Glance
What it tests: Whether a coding agent's "pass" on a task reflects a genuinely verified, solid solution or a lucky outcome produced by blind retries, guesswork, or missing verification steps — applied retrospectively to existing SWE-agent trajectory data.
Why it matters: Binary pass/fail scoring, the norm across most coding-agent benchmarks, treats a lucky pass identically to a rigorously verified one. AgentLens shows this materially distorts model rankings — up to 5 position shifts — meaning published leaderboards may overstate the reliability of some models relative to others.
Known limitations: AgentLens re-scores existing trajectories (from OpenHands SWE-agent runs) rather than generating new tasks, so its findings are bounded by whatever trajectory data was originally collected; it's an evaluation-methodology contribution more than a new task benchmark.
What AgentLens Measures
AgentLens takes 2,614 existing SWE-agent trajectories, originally produced by OpenHands and scored with standard binary pass/fail criteria, and re-scores them at the process level — examining how each trajectory reached its outcome rather than just whether it reached a passing state. This process-level lens surfaces "lucky passes": cases where the agent's final submitted patch happened to satisfy the test suite despite the agent never verifying its own work, relying on blind repeated attempts, or otherwise arriving at a passing state without a defensible underlying fix.
The paper's central finding is that a substantial share of nominal "passes" — up to 23.2% — fall into this lucky category. Because different models and harnesses accumulate lucky passes at different rates, re-ranking systems by process-adjusted score (rather than raw pass/fail) shifts model rankings by as much as 5 positions relative to the original leaderboard. This is a direct empirical argument for the "Trace" evaluation surface highlighted across the broader 2026 harness-observability literature: outcome-only scoring is insufficient to assess real agent reliability.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent (process-level re-scoring / evaluation methodology) |
| Metric | Lucky-pass rate (%) and process-adjusted ranking |
| Trajectories re-scored | 2,614 (OpenHands SWE-agent) |
| Saturation | Low |
| Created by | AgentLens authors |
| Source paper | AgentLens (arXiv 2605.12925) |
| GitHub | Not linked here — see paper |
| Dataset | Derived from existing OpenHands SWE-agent trajectories |
How AgentLens Is Scored
Each of the 2,614 trajectories is examined process-by-process rather than judged solely on its final pass/fail outcome. AgentLens flags a pass as "lucky" when the trajectory shows no genuine verification step, relies on blind retries rather than diagnosed fixes, or otherwise reaches a passing test result without evidence the agent understood why its change worked. The lucky-pass rate (up to 23.2% across the re-scored corpus) is reported alongside a process-adjusted ranking that re-orders models by process quality instead of raw pass rate — this re-ranking is what produces the reported 5-position rank shifts.
AgentLens on Benchgen
No Benchgen results yet — be the first to run AgentLens-style process scoring.
AgentLens vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| AgentLens | Process-level re-scoring of existing agent trajectories | 2,614 | Low |
| SWE-bench Verified | Code patch resolution on real GitHub issues (pass/fail) | 500 | Low |
| HAL | Standardized multi-benchmark harness with public trajectory logs | — | Low |
| Harness-Bench | Harness quality on a fixed academic task suite | 106 | Low |
AgentLens and HAL are complementary: HAL provides the standardized harness and public trajectory logs, while AgentLens demonstrates a concrete method for extracting process-quality signal from exactly that kind of trajectory data — showing why raw pass/fail leaderboards like SWE-bench Verified can understate the true reliability gap between models.
Run AgentLens-Style Process Scoring on Your Model
Benchgen captures full agent trajectories, not just pass/fail outcomes, letting teams apply process-level checks — like AgentLens's lucky-pass detection — to their own coding-agent runs and catch models or harnesses that are quietly passing tasks without a defensible fix.
Related Models & Benchmarks
- Related benchmarks: SWE-bench Verified · HAL · Harness-Bench