# AgentLens ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Lucky--Pass%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-2614-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-May%202026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202605.12925-b31b1b)](https://arxiv.org/abs/2605.12925) > **Quick answer:** AgentLens is a May 2026 process-level re-scoring benchmark that re-examines 2,614 existing OpenHands SWE-agent trajectories, distinguishing genuinely verified solutions from "lucky passes" — cases where an agent's blind retries or missing verification happened to produce a passing result without a solid underlying fix. It finds that up to 23.2% of passes in the original trajectories were lucky, and that model rankings shift by up to 5 positions once process quality (not just binary pass/fail) is factored into scoring. ## At a Glance **What it tests:** Whether a coding agent's "pass" on a task reflects a genuinely verified, solid solution or a lucky outcome produced by blind retries, guesswork, or missing verification steps — applied retrospectively to existing SWE-agent trajectory data. **Why it matters:** Binary pass/fail scoring, the norm across most coding-agent benchmarks, treats a lucky pass identically to a rigorously verified one. AgentLens shows this materially distorts model rankings — up to 5 position shifts — meaning published leaderboards may overstate the reliability of some models relative to others. **Known limitations:** AgentLens re-scores existing trajectories (from OpenHands SWE-agent runs) rather than generating new tasks, so its findings are bounded by whatever trajectory data was originally collected; it's an evaluation-methodology contribution more than a new task benchmark. ## What AgentLens Measures AgentLens takes 2,614 existing SWE-agent trajectories, originally produced by OpenHands and scored with standard binary pass/fail criteria, and re-scores them at the process level — examining *how* each trajectory reached its outcome rather than just whether it reached a passing state. This process-level lens surfaces "lucky passes": cases where the agent's final submitted patch happened to satisfy the test suite despite the agent never verifying its own work, relying on blind repeated attempts, or otherwise arriving at a passing state without a defensible underlying fix. The paper's central finding is that a substantial share of nominal "passes" — up to 23.2% — fall into this lucky category. Because different models and harnesses accumulate lucky passes at different rates, re-ranking systems by process-adjusted score (rather than raw pass/fail) shifts model rankings by as much as 5 positions relative to the original leaderboard. This is a direct empirical argument for the "Trace" evaluation surface highlighted across the broader 2026 harness-observability literature: outcome-only scoring is insufficient to assess real agent reliability. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Agent (process-level re-scoring / evaluation methodology) | | Metric | Lucky-pass rate (%) and process-adjusted ranking | | Trajectories re-scored | 2,614 (OpenHands SWE-agent) | | Saturation | Low | | Created by | AgentLens authors | | Source paper | [AgentLens (arXiv 2605.12925)](https://arxiv.org/abs/2605.12925) | | GitHub | Not linked here — see paper | | Dataset | Derived from existing OpenHands SWE-agent trajectories | ## How AgentLens Is Scored Each of the 2,614 trajectories is examined process-by-process rather than judged solely on its final pass/fail outcome. AgentLens flags a pass as "lucky" when the trajectory shows no genuine verification step, relies on blind retries rather than diagnosed fixes, or otherwise reaches a passing test result without evidence the agent understood why its change worked. The lucky-pass rate (up to 23.2% across the re-scored corpus) is reported alongside a process-adjusted ranking that re-orders models by process quality instead of raw pass rate — this re-ranking is what produces the reported 5-position rank shifts. ## AgentLens on Benchgen > No Benchgen results yet — [be the first to run AgentLens-style process scoring](/benchmarks/agentlens). ## AgentLens vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **AgentLens** | Process-level re-scoring of existing agent trajectories | 2,614 | Low | | [SWE-bench Verified](/benchmarks/swe-bench-verified) | Code patch resolution on real GitHub issues (pass/fail) | 500 | Low | | [HAL](/benchmarks/hal-agent-leaderboard) | Standardized multi-benchmark harness with public trajectory logs | — | Low | | [Harness-Bench](/benchmarks/harness-bench) | Harness quality on a fixed academic task suite | 106 | Low | AgentLens and HAL are complementary: HAL provides the standardized harness and public trajectory logs, while AgentLens demonstrates a concrete method for extracting process-quality signal from exactly that kind of trajectory data — showing why raw pass/fail leaderboards like SWE-bench Verified can understate the true reliability gap between models. ## Run AgentLens-Style Process Scoring on Your Model Benchgen captures full agent trajectories, not just pass/fail outcomes, letting teams apply process-level checks — like AgentLens's lucky-pass detection — to their own coding-agent runs and catch models or harnesses that are quietly passing tasks without a defensible fix. ## Related Models & Benchmarks - Related benchmarks: [SWE-bench Verified](/benchmarks/swe-bench-verified) · [HAL](/benchmarks/hal-agent-leaderboard) · [Harness-Bench](/benchmarks/harness-bench) ## Frequently Asked Questions

What is AgentLens?

AgentLens is a benchmark that re-scores 2,614 existing OpenHands SWE-agent trajectories at the process level, distinguishing genuinely verified passes from "lucky passes" produced by blind retries or missing verification.

What is a "lucky pass" in AgentLens?

A lucky pass is a trajectory that satisfies the test suite (a nominal "pass") without evidence the agent verified its own work or understood why the change worked — often the product of blind repeated attempts rather than a diagnosed fix.

Who created AgentLens?

AgentLens was published on arXiv in May 2026 (arXiv:2605.12925) as a process-level re-scoring study of existing SWE-agent trajectory data.

How much do model rankings change under AgentLens's process scoring?

Up to 23.2% of nominal passes in the re-scored corpus are found to be lucky, and re-ranking models by process-adjusted score (instead of raw pass/fail) shifts rankings by as much as 5 positions relative to the original leaderboard.

Is AgentLens a new task benchmark or an evaluation method?

AgentLens is primarily an evaluation methodology — it re-scores existing trajectories from OpenHands SWE-agent runs rather than introducing a new set of tasks, making it a contribution to *how* agent trajectories should be scored rather than *what* they're scored on.

AgentLens — Evaluations