Long-Horizon Terminal-Bench (LHTB)

Long-Horizon Terminal-Bench (LHTB)

Quick answer: Long-Horizon Terminal-Bench (LHTB) is an agent benchmark from Lehigh University and Tencent that drops a model into a Docker container with a goal requiring hundreds of dependent terminal actions — then grades the outcome with a hidden, fake-proof verifier using continuous partial credit. Across 46 tasks and 18 frontier models, the best result (Grok 4.5) is only 0.505 mean reward, and 29 of 46 tasks have never been fully solved by any model.

At a Glance

What it tests: Whether an AI agent can sustain coherent multi-step work inside a live terminal environment over a 90-minute budget — carrying state across hundreds of actions, recovering from dead ends, and knowing when a complex goal is actually complete.

Why it matters: Most agent benchmarks end within minutes; LHTB targets the regime where long-horizon state management and sustained reasoning are the limiting factor, not baseline task knowledge. It reveals a failure mode — losing the thread — that short benchmarks structurally cannot expose.

Known limitations: All baseline runs use the identical Terminus-2 harness, so scores reflect a specific scaffold rather than the model's raw capability. Evaluation cost is high ($2.50–$73 per task), limiting the number of providers who can submit fresh runs.

What Long-Horizon Terminal-Bench Measures

LHTB is built around three design axes that run through every task. First, each task demands a long horizon: solutions require hundreds of dependent actions and sustained state — a single task averages 69–93 minutes of wall-clock time and roughly 120–320 agent steps. Second, the benchmark actively resists shortcutting: a hidden verifier replays the agent's evidence using deterministic seeds and held-out answer keys so that claimed progress never counts — only replayed work does. Third, scoring uses a continuous graded reward rather than binary pass/fail. On tasks where nearly every frontier model scores zero under a strict threshold, partial credit is what keeps the ranking informative.

The 46 tasks span nine categories: software & reverse engineering, scientific computing, earth/climate & energy, multimodal & imaging, research reproduction, systems & security, professional (APEX) workflows, interactive games, and logic & constraint puzzles. Every task follows the same contract: a Docker container, an instruction file, a live environment, and a hidden test suite that grades the outcome. A task counts as solved at reward ≥ 0.95.

All baseline evaluations run under the Terminus-2 harness with a fixed 90-minute budget per task. The harness uses a JSON action parser, proactive context summarisation, and full terminal-session recording. No model-specific tools or bespoke agents are allowed; each model differs only in the API endpoint behind a single identical harness. A model that times out retains whatever partial credit the verifier can replay at the deadline.

Benchmark Specifications

How LHTB Is Scored

Each task yields a reward in the continuous range [0, 1]. Banded or proportional partial credit is awarded per task by a hidden verifier that replays the agent's evidence using seeded environments and held-out answer keys. A reward of exactly 0 means no verifiable progress at all; a reward ≥ 0.95 counts as a full solve. The headline metric is mean reward averaged over all 46 tasks. A secondary metric — solve count — reports how many tasks crossed the 0.95 threshold; this is a sparser, harder signal.

Partial credit matters because on tasks of this difficulty, binary scoring compresses nearly every model to zero. In the published runs, 7% of all 782 model-task pairs crossed the solve threshold; under strict pass/fail, the other 93% would be indistinguishable. The continuous reward spreads that mass: more than half of all runs land in the low-partial band (0–0.25), where models differ by how far they get before losing the thread.

State-of-the-Art Results

Scores sourced from the LHTB paper and official leaderboard (Li et al. 2026). All runs use the Terminus-2 harness with a 90-minute budget.

Price Is Not Performance

A striking result from the LHTB paper is the weak correlation between cost and quality. Per-task spend ranges nearly 30×, from approximately $2.50 (Hy3) to $73 (Claude Fable 5), while mean reward spans only 2.5× from the best to worst result. Grok 4.5 posts the top mean reward at roughly $11 per task — combining the best score with low cost. MiniMax M3 and Qwen3.6 Plus deliver 60–80% of the top score at $4–6 per task, making them the value-efficient picks for production workloads that require repeated long-horizon runs.

LHTB on Benchgen

No Benchgen results yet — be the first to run Long-Horizon Terminal-Bench.

LHTB vs Other Benchmarks

Run Long-Horizon Terminal-Bench on Your Model

Comparing long-horizon terminal performance across model versions is precisely the kind of regression-sensitive evaluation where Benchgen adds the most value. Because LHTB tasks run for 60–90 minutes each and use hidden verifiers, a single ad-hoc run cannot tell you whether an improvement in one update was offset by a regression in another. Running LHTB through Benchgen gives you version-controlled, reproducible results with trajectory-level diagnostics — so you can see not just whether the score improved, but where and why the agent lost the thread.

Related Models & Benchmarks

Top-performing models: Claude Sonnet 5, Claude Opus 4.8, Claude Fable 5, GPT-5.6 Sol, GPT-5.5

Related benchmarks: TerminalBench 2.1, SWE-bench Verified, SWE-bench Pro, CyberGym, TAU3 Banking

Frequently Asked Questions